KISS orchestra

Designers

• Jane Liu
• Ri Chang
• Beatrice Mai
• Flora Zhang

Project Description

Kiss Orchestra is an installation that aims to explore the synesthetic relationship between our sense of hearing and touch playfully through an instrument that features curious textures and is played using a lesser-used body part. The result is the “kiss board”, a synthesizer pad that you can play by kissing. There are two boards in the installation, one with comfortable textures which are paired with strange and unpleasant sounds, and another with uncomfortable textures that are paired with pleasant and beautiful sounds.

What does it look like?

What does it do technically?

Each kiss board consists of 4 keys. Every key consists of a square force sensory resistor that serves as a sensor. On top of the sensor is a foam piece carefully mounted so that there is a gap between the foam and the sensor in the resting position. On the user-facing side of the foam are velcros, on which the various textures for kissing will be mounted. The use of velcros allows fresh textures to be applied after every use.

As the user kisses the texture, the foam will be depressed, making contact with the FSR sensors, which would in turn send a signal to Max, a visual coding software, to play the associated sound or note. The board is also programmed in such a way that the harder the user kisses the board, the louder the resulting sound will be.

What do users do in the project?

In its usual setup, the installation will require two participants. They will stand behind the kiss board and kiss it to make sounds. They will later be encouraged to create a “duet” together using the kiss boards, hence forming a two-person orchestra.

Timeline

• 09/12/2024: team set up & inspiration collection
• 09/19/2024: project ideas generation
• 09/26/2024: iteration of project ideas
• 10/03/2024: Milestone 1– finalized project idea
• 10/10/2024: preparation for the user test
• 10/17/2024: Milestone 2 – user test
• 10/24/2024: Milestone 3 – iteration of the project
• 10/31/2024: collection of material lists
• 11/07/2024: Milestone 4 – project diagram
• 11/14/2024: Milestone 5 – project construction
• 11/21/2024: Milestone 6 – user test & iteration of the project
• 12/03/2024: Milestone 7 – documentation
• 12/12/2024: Final Presentation

Weekly Updates

Questions to Consider:

1. Could a blind person use and understand my project?
2. Could a deaf person use and understand my project?
3. Could a person with mobility restriction use and understand my project?

09/12/24

Inspiration Slides

09/19/24

project Ideas Slides

09/26/24

Feedback from Crit

Idea #1: Mystery Box

the good

• meets the goal of having users explore with a sense of touch
• there’s a lot of potential for interaction
• components seem sound
• featuring textured surfaces in the box instead of an object subverts the expectations people have for a mystery box and maybe invites more curiosity

food for thought

• how do users know how and when to start the experience?
  • sign of a good interface is you don’t have to explain it
• why does the box feature textured surfaces and not objects?
• might including a reveal of the textures inside (a transparent window) be more satisfying for the user

---

Idea #2: Stimming wearables (shoes)

the good

• trying to use other body parts to experience touch while making stimming accessible and convenient in daily life is a great idea
• many people are always moving their feet or shaking their feet
• there are many stimming products but none for feet!

food for thought

• why bread? what other smells might be interesting?
  • dog feet smell like Dorito and yeast
• could the interface offer a personalized smell?

---

Idea #3 Kiss Soundmaker (piano) 👑

the good

• seems fun
• seems similar to the WeScream: gustosonic system work

food for thought

• hygiene concerns
  • use barriers?
• how might barriers affect how the textures feel or experience?
• interesting textures
  • slime
  • silicone
  • something slippery
• potentially problematic textures
  • furry- may cause ambiguous sensor feedback (flat texture)
• collecting lipstick marks on the piece might be interesting
• adapting this to become an interactive piece featuring dead musicians (e.g. Beethoven)
  • kissing the head of Beethoven and having it conduct music to the user via touch (bone conduction)
• how could this interaction accommodate multiple users and facilitate interaction between them?

Workshopping Final Idea

Kiss Orchestra Possible Iteration Ideation

Iteration 1: Music-Stand Iteration

The Music Stand Iteration mimics a traditional setup of orchestras. Sensors are built into individual music stands, each stand is assigned a sound. Each participant is given a texture that they can stick on the music stand, that they then kiss to activate the sound.

Iteration 2: Personal Keyboards

The Personal Keyboard Iteration mimics the setup of a piano or keyboard. Each participant is given a set of “piano keys”. Each set of keys has 3 sensors on them, each assigned to a sound.

On top of every key, different textures are attached on top. For hygiene purposes, every participant’s keyboard is unique and not meant to be shared.

rapid prototyping User Testing Goals

We rapid prototyped and user-tested in class for the first time!

1) How the interaction feels

Using cardboard and different textures to make a keyboard mockup, we tested out the following 2 possible iterations to see which sparked the most curiosity and interest. 

1. Orchestra Mode
2. Two sides of an acrylic panel facing the other user

2) Various hygiene solutions

• Stick-on personal textures (personalized effects to the original sound?)
• Peel-away layers of the same texture on a shared keyboard
• Barriers & thinning texture

3) Trying out various textures

• Comfortable Textures vs. Uncomfortable Textures
• Edible Textures vs. Inedible Textures
• Visual Expectation vs. Physical Sensation vs. Sound

3) Trying out various Sounds

• Weird Sounds vs. Musical Notes
• Unexpected Sounds vs. Expected Sounds
• Visual & Sound Contrast (Does a texture sound like the way it looks?)

10/03/24

Finalized idea: A Kissboard Duet

After user testing our rapid prototyping in class, we have decided to go with the Individual Kissboard Iteration.

Two music stands facing each other

The “musicians” stand behind each stand and kiss the board to play music together

Tech concept

• 2 kissboards with sensors embedded and velcros on the surface
  • each kissboard plays 3 notes
• Every user has an envelope of x-number textures
  • each texture will have velcros on the back

Acrylic stand = kiss board, cardboard textures = textures

Kissboards are hooked up to a microcontroller that is then connected to the computer, which will play the assigned sound

keys/textures

1. Food theme only
2. Food is included but other textures can also apply

10/10/24

User Testing Script

Project: Kiss Piano/Board/Duet

Hypothesis:
If given a keyboard consisting of keys of various textures that make sounds only when kissed, users will prefer to kiss food textures and have the keyboard generate specific musical notes.

Research Goals

1. Determine whether users prefer food-related or non-food textures.
2. Understand how different textures influence the likelihood of interaction (kissing the key).
3. Identify whether users prefer structured musical notes or random sounds when interacting with the textures.
4. Establish what physical or emotional criteria users feel must be met for the interaction to be considered “complete” or meaningful.
5. Determine the optimal duration of the interaction for maintaining user engagement and emotional connection.

Methodology

• Number of participants: 3 pairs (2 users at a time)
• Location: United States only
• Age range: 18-45
• Gender: All
• Disability Identity:
• Experience: all level musical proficiency, any participant with a general interest in music

Procedure

Data collection:

• A/B Testing: Compare user reactions between food and non-food textures, as well as between structured notes and random sounds.
• Think-Aloud Protocol: Participants will describe their thoughts and feelings during each task.
• Observation: Record user behavior, facial expressions, body language, and any hesitation or excitement.

Length: 30-45 minutes / each pair, participant can be strangers or non

Location: NYU ITP Floor, phone booth

Test Setup: Two zip lock bags, each including six different textures, will be given to participants. Two boards each made of six texture-based keys, each with corresponding sounds, will be given to the participants. Each key is associated with different sound sets that will rotate throughout the testing process.

• Part 1 will focus on the tactile experience of kissing different textures without sound.
• Part 2 will add sound, testing three different sound sets.

Testing Script

Hi, thank you for participating in today’s study.

We are exploring how physical textures and sounds can create meaningful interactions when engaged in a non-traditional way—specifically through touching and kissing different materials.

We will ask you to interact with different textures, first by touching them and then by kissing them. Throughout the test, we encourage you to speak your thoughts aloud as you complete each task, as it will help us understand your experience.

During any point of the study, please let me know if you’d like a break, have any access requirements, or need any accommodations. There are no right or wrong answers and we can stop at any point.

We will also be recording video and audio throughout the study. Quotes will be anonymized and all data will only be shared internally with my team, and will be deleted after the study is completed.

Do I have your consent to record video and audio?

For hygienic purposed, please use the hand sanitizer provided to desensitize your hands.

Part 1: Texture Test

[ Task 1: Touching Textures (5 min) ]

For this first task, you will be each be given a clear zip lock bag. Inside, you will find six different textures.

Present the 2 bags of texture to both participants.

Please touch and feel each texture with your hands. As you do, think out loud and describe what you feel. Take your time to explore each texture. When you are done, please leave the textures on the table. Then, we’ll ask you a few questions.

Follow-up Questions (after touching):

• • • How would you describe the physical feeling of each texture? Which textures felt the most pleasant to touch? Why?
    • On a scale of 1-4, how comfortable were you touching each texture? (We will go through each texture one by one.)
    • Did you have a preference for food-related textures (e.g., marshmallows) or non-food textures (e.g., yarn, tile)? Why?
    • How did touching the textures make you feel emotionally? Did any textures trigger specific emotions or memories?

Collect bags of texture.
Present 2 bags of texture boards.

[ Task 2: Kissing Textures (5 min) ] 

Now we will move on to Task 2.

Present zip lock bag with board of textures in side.

Inside the zip lock bag, you will find the same set of textures, but attached on a board. Each board is designed for single-person-use, and not meant to be shared with others.

You will interact with the textures again, but this time, using your lips.

Please bring each texture to your lips, and if comfortable, please kiss the texture. Feel free to take your time.

Once you feel like you are done exploring the textures, just place the board back on the table. We will then ask you a few questions.

Follow-up Questions (after kissing):

1. 1. 1. Which texture felt the most pleasant or meaningful to kiss? Why?
      2. On a scale of 1-4, how comfortable were you kissing each texture? (We will go through textures one by one.
      3. Did your preference change between touching and kissing the textures? Why or why not?
      4. How did the act of kissing the texture change your emotional response, if at all?

Part 2: Music Interaction

Task Instruction Script: In the next part, you will be kissing the same textures again, but now each texture is linked to a sound. You may kiss the textures in any order or at any speed.

There are three sets of sounds, and you will have 2 minutes to experience each set.

After each set, we will ask you a few questions.

Set 1: Musical Notes Only

• • • Start 2 minute timer.
    • Present the sound set consisting of musical notes.
    • Play sound according to each key being kissed.Follow-Up Questions
      1. How did the sound element affect your experience? Did it feel more complete, engaging, or emotional with the sound?
      2. Did the presence of musical notes make you feel more or less comfortable while interacting with the textures?
      3. How did the musical notes make you feel during the interaction? Did they evoke any specific emotions or memories?
      4. Did the musical notes feel appropriate or did you find yourself wanting a different kind of sound?

Set 2: Non-Musical Sounds

• • • Start 2 minute timer.
    • Present the sound set of random, non-musical sounds (cat purring, water running, etc.).
    • Play sound according to each key being kissed.

Follow-up Questions:

1. 1. 1. How did the non-musical sounds (e.g., cat purr, water running) affect your emotional experience? Were any of them surprising or unexpected?
      2. Did the random, non-musical sounds make you feel more or less comfortable? Did any sounds feel jarring or pleasant?
      3. Did these everyday, ambient sounds make you more or less interested in interacting with the textures? Why?
      4. Did these sounds change your perception of the textures? Were any sounds distracting or pleasant?
      5. Did the non-musical sounds fit the experience, or did you prefer the structure of musical notes from the previous set?

Set 3: Mixed Sounds

• • • Start 2 minute timer.
    • Present the mixed sound set.
    • Play sound according to each key being kissed.

Follow-up Questions:

1. 1. 1. How did the combination of musical and non-musical sounds affect your emotional response? Did the mix of sounds feel more engaging or confusing?
      2. Did this mixture of sounds make you feel more or less comfortable than the previous sets? Why?
      3. Did the combination of sounds make you more curious and engaged with the textures, or was it distracting?
      4. Do you prefer the variety of mixed sounds, or would you rather have one type of sound (musical or non-musical) for this experience?

Final Round-up Questions:

1. 1. Overall, how did touching and kissing differ for you in terms of physical and emotional experience?
   2. Which combination of textures and sounds felt the most meaningful or engaging?
   3. Did the sounds enhance or detract from your interaction with the textures?
   4. Do you have any urges or desires (missed opportunities) when interacting with this object that you were not able to fulfill due to constraints of this test?
   5. How much time felt right for this experience— is 2 minutes too long, too short, or just right? Did your lips or face get tired?
   6. What emotion or memory did this experience evoke, if any?
   7. Do you think kissing the textures created a more unique interactive experience than just touching them? Why or why not?
   8. From your perspective, what went well in the interaction? What didn’t go well?
   9. Please describe any areas of the interface you had questions about.

Closing Remarks:

Thank you so much for helping me out today. This will really help me improve the interface. Is there anything else you’d like to ask questions about?

10/17/24

Key Findings from User Testing

describe the 3-5 key findings from the user feedback session

Key Findings	Action Item
Social Comfort: personal comfort level of kissing keys in the presence of others largely depends on the social intimacy level between participants	– Consider using low light setting; Incentivize goal of interaction (eg. playing a game, create a tune); Target participants with specific social intimacy levels (eg. friends, loved-ones, acquaintances).
User Preference: Textures/smells/sounds arrangement are affected by all modalities in combination and subjective association.	– Consider the combinations among all modalities, also including the unexpected effects. – Give options to users to decide what textures to use.
User Preference: Popular textures were mostly pleasant/comforting food textures that are either nice to smell or touch (e.g. marshmallow, orange peel)	– Consider incorporating more food textures that are pleasant to smell or touch or kiss
Interface Comfort: Holding board was more comfortable than kissing wall. Users seem to kiss the board with less force as time goes on as they get tired of the kissing motion.	– Fine-tune haptic feedback and the sensing threshold of the sensor to respond to force of kiss – Make the interface a handheld board rather than a stationary stand
User enjoyment: Any element of surprise increased user’s enjoyment and delight when using the kissboard	– Curate sounds and textures and combine them in ways that will surprise users

10/24/24

Prototype Update

We iterated on our project featuring the following key findings:

1. social comfort
2. texture preference
3. user enjoyment

Here is our final instructions & user experience.

Setup

• 2 participants
• 2 handled kiss board (1 board has extremely comfortable textures but weird sounds, and the other board has extremely uncomfortable textures but beautiful sounds)
• 1 laptop for sound output and running program
• 1 speaker
• 2 pairs of wired headphones
• 2 tables/stands for kiss-board placement

Introduction

Welcome to the kiss board experience!

This experience requires two people, you’re welcome to do this with a friend or partner. But if you’d like to experience this with an stranger, that is okay too.

Instruction

1. Please stand at the designated spot.
2. Please stand up and pick up/hold this kiss board in front of you.
3. Please put on the headphones.
4. Each section of the board will play a corresponding sound when it is kissed. Try kissing different parts of the board to find out what sound it makes! Listen carefully. (Try it for ~1 min)
5. For the next 2 minutes, try to create am improv duet with the other participant by kissing the board. You may verbally communicate with each other if you wish.
6. When time is up, you will receive a recording of your collaborative duet, which will be played out-loud publicly.

Materials:

Questions for crit

• Do you want to have the audio emailed to you or only can be played back once at the installation?
• should we offer them the option of playing their collaborative piece out loud before step 4?

10/31/24

Material list for the project

texture material (pleasant & unpleasant)

food texture

• soft materials (e.g. loaf, marshmallow)
• fruit peels (e.g. Citrus Fruits， yuzu) with fresh smells
• hard materials (e.g. rolls) with artificial smells

non-food texture

• smooth materials (e.g. tiles, furry things)
• rough materials (e.g. nets, sponges)
• petals? with a fresh smell

hardware material

face mask 1

https://a.co/d/iTGroXr

face mask 2

https://a.co/d/b7xciVC

Electronics material

sensor https://tinkersphere.com/sensors/3744-square-fsr-force-sensor-arduino-compatible.html

Conductive Paint (Jane already have)

Musical Stands

Pressure-Sensitive Conductive Sheet (Velostat/Linqstat)

Media Commons – Training & Scheduling

Headphones

Final Iteration Documentation – Script

11/7/24

Structure DESIGN DIAgram

 

11/14/24

First Prototype

Code

Draft version 2, updated 11/13

Prototype & Test video

Sources

あ / Ah – Team Panai

HIBIKI Interactive Bar

Invisible Sculptures (Seires) – Yeseul Song

4DSound – Roy Nachum

A Blind Legend – Dowino

2023 用手去看見世界 (Seeing through Hands)

Provenance – Alexandra Palovc

Flavor – MoFad

Final Presentation

click to see the video