EmoSynth for Sensorium AI
EmoSynth: A combination of ‘Emotion’ and ‘Synthesizer’, highlighting the device’s ability to synthesize emotional expressions.
Designers
- Lang Qin
- Junru Chen
- Emily Lei
- Kassia Zheng
- Muqing Wang
Final Presentation
Project Description
“We are Sensorium AI Group from the Multisensory class. We designed a wearable device that works with Sensorium Ex, Luke, Michael, and our professor, Lauren. In order to fulfill the design objectives of facilitating real-time artistic expression and modulation for performers with cerebral palsy and/or limited speech, our device allows the user to freely select the emotion and level of emotion they want to express. Additionally, there is a selection of high and low audio to represent diverse genders, along with a reset button to prevent accidental touches during movement on stage. Each selection on the device is clearly denoted by an embedded LED hint to better inform users.
Timeline
- 09/14/2023: Primary research ( Reference cases and Inspiration)
- 09/21/2023: Ideas brainstorm and sketch
- 09/28/2023: Finalized our project idea
- 10/05/2023:Developed first quick prototype
- 10/12/2023: User testing with the quick prototype
- 10/19/2023: Implementation of user testing result
- 10/26/2023: Finalize the sensors and buy materials
- 11/02/2023: Prototyping with the sensors (input signal)
- 11/09/2023: Start to work on the visual and audio output
- 11/16/2023: Finalize our soft material and the functions of each button
- 11/30/2023: Connect flex sensors, lights, and buttons with the Arduino
- 12/07/2023: Finalized the fabrication part
- 12/14/2023: Documentation and final testing/troubleshooting.
Weekly Updates
Click here for our Google Slides Weekly Presentation link (66 pages).
Click here for our Final Google Slides Presentation link (15 pages).
09/14/23
Reference
Inspiration 1: http://xhslink.com/6NqCyu
Inspiration 2: https://www.media.mit.edu/projects/alterego/overview/
Inspiration 3: https://www.deafwest.org/
Other Possibilities:
Emotion AI: https://www.affectiva.com/about-affectiva/
EMOTIC Dataset: Context Analysis: https://s3.sunai.uoc.edu/emotic/index.html
Brainwave: https://prezi.com/p/fvxrqad30mjq/music-from-brainwaves/
09/21/2023
Background:
Takeaways from meeting with Luke
Based on our target user, Thomas, and the general audience, we have proposed two ideas: sign language translation and alternative emotional expression.
“ Translation” of sign language
Sketch:
Alternative Emotional Expression
Notes from Crit:
- Invisible and visible design
- Thomas’s frequent language/issues
- Pre-set device? Technical issues.
- The generated voice is more vivid than the machinery voice.
- How to train AI mode?
- Product or part of art/opera?
- Connect to Thomas.
09/28/2023
10/05/2023
10/12/2023
Hypothesis
An educated guess on what you think the results of the study will be.
If we can create a wearable device that can recognize sign languages and analyze the emotions of the user based on an AI model, then it may help them to effectively communicate with people who do not understand sign languages. In the case of the Sensorium AI, with a pre-set movement, content, and voice output, it may allow Thomas to express the context with the sign language as triggers and translate it into voices while he is performing on the stage.
Research Goals
- Help individuals with speech impairments communicate with others and express their emotions more effectively.
- Specifically, focus on performers in opera with speech impairments to help them express emotions effectively and timely on stage.
Methodology
Number of participants: 5
Location: NYC
Age range: not limited
Gender: not limited
Disability Identity: speech impairments
Experience: prefer people with performance experience
Procedure
Data collection: brainstorming, feasibility testing Find more methods here
Length: approximately 30 minutes
Location: Unknown
Testing Script
Informed Consent
Hi, my name is <XX>. Today, we would like to invite you to participate in a research study designed to explore the potential benefits of wearable devices in aiding non-verbal performers in conveying content and emotions on stage. Before you decide whether or not to participate, it is important for you to understand the purpose of the study, your role, and your rights as a participant
Please let me know if you’d like a break, have any access requirements, or need any accommodations. There are no right or wrong answers, and we can stop at any point. Do I have your permission to video and audio record? Quotes will be anonymized, and all data will only be shared internally with my team, stored on a secure, password-protected cloud server, and deleted after the study is completed.
General questions:
- What’s your first impression of this wearable device?
-
- Do you feel any barriers from this device?
- Will anything from this device affect the costumes and stage design?
- What is the safest way to have this device attached to you by considering your interactions with other performers on the stage?
User Task Feedback:
Mary:
It will be cool to wear it on stage.
There might be some kind of conflict between recognizing movements and the freedom of performers on the stage.
To ensure a wearable device, I might need to move in a very specific way.
Maybe the speed of movement should be considered.
Cindy:
This looks very futuristic.
There might be some unintended activation.
Nicole:
The appearance of this device kind of aligns with the story’s background.
The device may make the performer concerned that their movements will damage the device so that they cannot perform as well as they used to.
I am concerned about the learning curve of this device.
Are there any tips I need to learn or remember by wearing this device?
Josh:
Will sweating and makeup influence the circuit?
Potential risk to the body’s security since it’s an electronic device.
Fanny:
It’s the solo device for each individual? The construction of relationships is inevitable on the stage/in the opera, multi-player might be the concept that you can dive in.
Edward：
I like the design because it expands the expression of dancing and movement on the stage. As a dancer who performs in large theaters, we are concerned about how to affect audiences far away from the stage. I think the design would help us express and visualize our emotions more dramatically. I am looking forward to your next step! The one suggestion I would like to make is that be sure to pay attention to the comfort as well as the beauty of design. It’s quite hard to imagine wearing a winter glove all the time while dancing. Find some breathable material and try it on yourself for several hours before you decide to use the material.
10/19/2023
Comments on the form:
It will be cool to wear it on stage.
This looks very futuristic.
The appearance of this device kind of aligns with the story’s background.
Potential Constraints in the Performance
There might be some kind of conflict between recognizing movements and the freedom of performers on the stage.
To ensure a wearable device, I might need to move in a very specific way.
The device may make the performer concerned that their movements will damage the device so that they cannot perform as well as they used to.
Damage Concerns
There might be some unintended activation.
Will sweating and makeup influence the circuit?
There’s a potential risk to body security since it’s an electronic device.
Learning Process
I am concerned about the learning curve of this device.
Are there any tips I need to learn or remember while wearing this device?
Our question for the weekly meeting with Luke, Jerome, and Michael
- What’s going to happen in this opera? Such as movement and positioning…
- How many people are going to use this product?
- What’s the maximum of the movement going to be?
- Can this prototype be used in this opera with some movement preset into the performance?
Takeaway from meeting with Luke, Jerome and Michael
Regulation
Number of users: 10–15 performers
Suggestion:
Maybe the speed of movement should be considered.
Take away from in-person hack with Luke
We learned about how to use MaxMSP to create output, which helped us get a better understanding of what would the final prototype be.
10/26/2023
our Prototype
The main component in the circuit
lock/unlock
Reset
Reference link for prototype: Physical computing project from another website
Reference link for slider: YouTube video
In our weekly group meeting, we also bought materials for the final prototype including different sensors and parts:
11/01/2023
11/07/2023
11/14/2023
11/19/2023
11/30/2023
12/07/2023
12/14/2023
This week, we filmed our documentation with the whole team.
5 W’s Chart
| Components | Questions and considerations |
| Background (Why) | Upon receiving the mission, we got to know an incredible opera team called Sensorium Ex. To fulfill the design objectives of facilitating real-time artistic expression and modulation for performers with cerebral palsy and/or limited speech, we resolved to craft a user-controlled device aimed at alleviating the inconveniences experienced on the stage. |
| Impressions (What) | Anticipating the availability of pre-existing AAC devices, we fashioned a wearable voice-emotion synthesizer with a soft interface. In our design process, we considered various factors that may influence human voices, encompassing aspects such as gender, age, emotions, accents, and more. |
| Events (When) | This device is designed to be both accessible and user-friendly, requiring no prolonged learning curve. Furthermore, we considered sizing and explored diverse placement methods, culminating in the development of an adjustable extension fabric. Consequently, it can be comfortably worn on the arms, legs, or wheelchair handles, accommodating a spectrum of usage scenarios. |
| Sensory Elements (How) | In the final stages, we prototyped pivotal components, including pitch selections representing diverse genders, visualizations of emotional expressions as placeholders for the yet-to-be-incorporated model, and a reset button to forestall inadvertent activation during on-stage movement or dancing. Our design showcases a highly tactile and instinctive control system, empowering users to effortlessly navigate the device without the need for visual attention. Each selection on the device is clearly denoted by an embedded LED hint. |
| Designer/User (Who/Whom) | The designers are Kassia Zheng, Junru Chen, Lang Qin, Emily Lei, and Muqing Wang. Lang did the coding associated with Arduino and electronic connections. Junru and Kassia handled the fabrication part, connecting sensors and sewing all the materials into one piece. Emily was responsible for the visual output using TouchDesigner, while Muqing learned MaxMSP and actively communicated with Luke and Michael. As a team, we worked together on the final output and documentation. |
Workflow
Final Sketch
