Listening to Music Together. Source: Tunes over Zooms? How to listen to music with friends and family. Dec 2020. 

Project Intro:

Our mid-term project in Interaction Lab is called Music Taste Matchy Matchy. With my project partner @Jess Jiapeng Tang, we worked together on this project which utilizes audio information as the core element of interaction and aims at testing participants’ music taste compatibility as social features. In this documentation blog, I will demonstrate the inspirations, designs, process of making, and reflections throughout the project duration.

Context & Significance:

In my last group project, our team presented an interaction between robotics and human participants with physical interactions, audio communications, and visual representations. And this time, we shifted gears and decided to only focus on how audio information can be the core element of interaction projects. Jess and I researched the list of potential topics and agreed that music can always be a great way to bring people together with unique aesthetic features, artistic expressions, or simply, just pure fun. Therefore, we decide to base our tentative project on the concept of music.

With this fundamental understanding in mind, we went back to consult two interaction projects we researched before: the Lightwaves and 21 Swings. Interestingly, both of the projects allow participants to freely and collaboratively create music that provides real-time feedback. This allows a large number of participants to engage in projects based in a macro urban setting, but it is not so feasible for our “small” lab environments. Since we might have to limit the number of participants due to technology, ability and space constraints, we want to add “reward and punishment” features to enrich the experience: music will be played when two people have matching tastes, otherwise, it would be stopped. We expected these features to make our interactive game more challenging rather than plain creation. In this sense, our definition of “interaction” extends beyond signaling and receiving feedback to include successful interactions and unsuccessful attempts.

21 Musical Swings in Montreal, Canada. Source: https://www.dailytouslesjours.com/en/work/musical-swings

Uniquely, we tried to position our project at the intersection of music and social networking. This is primarily targeted at Generation Z who consider pop music an important part of their social lives and tend to hang out with people with similar music preferences. This project helps them to mingle with others through music as a social language and identify “people of the same kind” (liking the same artists, genres, songs, etc.). But we later realized that music is a universal and transgenerational symbol and should not just be limited to young people, so incorporating popular genres from different eras is imperative and helpful to make our project more inclusive and appealing to a wider range of users.

Conception & Design:

Our original idea was that two users would individually select one music genre they like the most from several options, and if their choices match, the corresponding music clip would be played. Therefore, we decided to create a setup where two people are visually blocked from each other’s choice-making process so their true compatibility (and gut feelings) could be accurately tested. We designed two sets of buttons on a horizontal cardboard and then uses another vertical cardboard as a blocking, as well as a speaker and two LEDs in the middle which can be seen by both users. The lighting of the green LED signals a successful match while the red indicates a failed attempt. The photo below is a sketch made by Jess to demonstrate our idea:

However, after the user test recitation, we shifted gears and changed our design. Professor Eric Parren recommended we use only audio messages and avoid using too many text-based descriptions. We accordingly optimized the project with music playing at the beginning and then allowing users to decide whether they like the music or not. As long as any of the users select “dislike”, the red LED would light up and the music would get cut off.

We decided to use buttons because we thought they were the devices that users can intuitively interact with (although they might be “boring”). However, when we looked back at it, potentiometers can also be used to replace buttons (especially slide pots) because they can provide a rating of like/dislike instead of just an arbitrary YES/NO answer.

Fabrication & Production:

「Prepare」

We started our project with getting all equipment ready. Except for our Arduino set and toolkit, we attempted to get these items we needed:

• An OLED display (which we did not use ultimately),
• An MP3 player module,
• An amplifier connecting to the MP3 player module,
• A micro SD card (to store the music clips),
• An SD card reader.

After these hardware items are in place, we also downloaded the library for the MP3 player <DYPlayerArduino.h> and explored how to use it.

「Connect & Construct」

Then, in our circuit, we placed our two sets of buttons on two separate breadboards connected to the same Arduino microcontroller, which also extends wires to another smaller breadboard with the LEDs and the MP3 module on it. See the picture below.

After we completed our circuit, we built a cardboard gadget to include the circuit for protection and stabilization purposes. To achieve this, we also had to extend the wires to fit in. The cardboard gadget looked like this:

And this is a basic structure of our project, which we worked on improving how it looks later.

「Coding」

Moving on from the hardware to the software part, coding is the most challenging part of this project for me since we both have no coding experience. But it was also a rewarding process for us to see our code work with the circuit. First, we wrote the code for the button part and the player part separately. To increase efficiency, Jess worked more on the button part with the help of our IMA fellow Sylvia and I worked more on the player part with the help of Kevin. They have been super helpful and we managed to finish our code cooperatively and put it together with adjustments.

To make our project work with a sequence of music clips, we decided to use the “times” and “state” functions: the pressing of a button is counted as one time and one music clip is one state. We had to sort out the logic chain and make the corresponding code coherent with the continuation, pause, stop, and switch of songs, as well as the lighting and dimming of LEDs. It was difficult because we had to carefully plan what every press of a button leads to and make sure our use of conditionals does not conflict with each other, with every possibility in mind. Since we used a lot of “if”s with “if”s in them, as well as “else if”s, we had to rule out every logical fallacy when debugging.

Before the user test, we had a rough version of code that enables users to simply try the buttons’ functionality and our interaction ideas:

// mp3
#include 
#include 
#include "DYPlayerArduino.h"
SoftwareSerial SoftSerial(10, 11);  //RX and TX from Arduino
DY::Player player(&SoftSerial);     //should connect them to io0 and io1
//button
const int button1Pin = 2;  //a
const int button2Pin = 3;  //b
const int button3Pin = 4;  //a
const int button4Pin = 5;  //b
const int greenLedPin = 12;
const int redLedPin = 13;
int buttonState1 = LOW;
int buttonState2 = LOW;
int buttonState3 = LOW;
int buttonState4 = LOW;
int pb1 = 0;
int pb2 = 0;
int pb3 = 0;
int pb4 = 0;
bool pressedA = false;  // save which button is pressed
bool pressedB = false;
bool pressedC = false;
bool pressedD = false;
// save the pressed to an array, compare with the previous pressed > only print when two are different
//round
float state = 1;
int times = 0;
void setup() {
  Serial.begin(9600);
  // mp3 part
  Serial.println("Starting the Player...");
  player.begin();
  player.setPlayingDevice(DY::Device::Sd);  //SD card, USB storage volume is
  player.setVolume(30);                     // 30 is 100% of Volume; with 15 you get 50% Volume
                                            // player.setCycleMode(DY::PlayMode::Repeat); // Play all and repeat.
  //player.play();
  //buttons part
  pinMode(button1Pin, INPUT);
  pinMode(button2Pin, INPUT);
  pinMode(button3Pin, INPUT);
  pinMode(button4Pin, INPUT);
  pinMode(greenLedPin, OUTPUT);
  pinMode(redLedPin, OUTPUT);
}
void loop() {
  buttonState1 = digitalRead(button1Pin);
  buttonState2 = digitalRead(button2Pin);
  buttonState3 = digitalRead(button3Pin);
  buttonState4 = digitalRead(button4Pin);
  // checking the button input and save them into booleans (pressedA,B,....)
  // user 1
  if (buttonState1 == HIGH) {
    pressedB = false;
    if (buttonState1 != pb1) {
      times++;
    }
    if (pressedB == false) {
      pressedA = true;
    }
  }
  if (buttonState2 == HIGH) {
    pressedA = false;
    if (buttonState2 != pb2) {
      times++;
    }
    if (pressedA == false) {
      pressedB = true;
    }
  }
  Serial.print("User 1: ");
  Serial.print(pressedA);
  Serial.print(",");
  Serial.print(pressedB);
  // user 2
  if (buttonState3 == HIGH) {
    pressedD = false;

    if (buttonState3 != pb3) {
      times++;
    }
    if (pressedD == false) {
      pressedC = true;
    }
  }
  if (buttonState4 == HIGH) {
    pressedC = false;
    if (buttonState4 != pb4) {
      times++;
    }
    if (pressedC == false) {
      pressedD = true;
    }
  }
  Serial.print(" User 2: ");
  Serial.print(pressedC);
  Serial.print(",");
  Serial.print(pressedD);
  Serial.println();
  // Serial.print(" times: ");
  // Serial.println(times);
  if (state == 1) {
    if (times == 2) {
      if (pressedA == true && pressedC == true) {
        digitalWrite(greenLedPin, HIGH);
        digitalWrite(redLedPin, LOW);
        //player.play();
      } else if (pressedA != pressedC) {
        digitalWrite(redLedPin, HIGH);
        digitalWrite(greenLedPin, LOW);
        //player.stop();
      }
      if (pressedB == true && pressedD == true) {
        digitalWrite(greenLedPin, HIGH);
        digitalWrite(redLedPin, LOW);
        //player.play();
      } else if (pressedB != pressedD) {
        digitalWrite(redLedPin, HIGH);
        digitalWrite(greenLedPin, LOW);
        // player.stop();
      }
    }
    else if (times == 3) {
      pressedA = false;  // reset all the buttons
      pressedB = false;
      pressedC = false;
      pressedD = false;
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
      state = 2;
      //Serial.println("state 2");
    }
    else {
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
    }
  }
  if (state == 2) {
    if (times == 3) {
      pressedA = false;  // reset all the buttons
      pressedB = false;
      pressedC = false;
      pressedD = false;
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
    } else if (times == 5) {
      if (pressedA == true && pressedC == true) {
        digitalWrite(greenLedPin, HIGH);
        digitalWrite(redLedPin, LOW);
        // state++;
      } else if (pressedA != pressedC) {
        digitalWrite(redLedPin, HIGH);
        digitalWrite(greenLedPin, LOW);
        // state++;
      }
      if (pressedB == true && pressedD == true) {
        digitalWrite(greenLedPin, HIGH);
        digitalWrite(redLedPin, LOW);
        // state++;
      } else if (pressedB != pressedD) {
        digitalWrite(redLedPin, HIGH);
        digitalWrite(greenLedPin, LOW);
        // state++;
      }
    } else {
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
    }
  }
  pb1 = buttonState1;
  pb2 = buttonState2;
  pb3 = buttonState3;
  pb4 = buttonState4;
}

We also encountered some challenges in the MP3 player part. With the sample code from the downloaded library, we successfully got it to play some music clips but we failed to play the specific clip we want. With the help of Kevin, I went on google and found the corresponding open-source references. After trying a few possible solutions, we finally found a workable way to rename the files and play specific ones.  We fixed the code after the user test as shown below (which is our final version):

// rule insturctions:
// (1) The music will automatically play music clip one
// (2) The two sides should press one of their buttons only ONCE to choose whether they like it or not
// (3) After the first two presses, the third one will stop the music clip one
// (4) The 4th press will play the second song
// (5) The 5th and 6th presses will be choosing likes/dislikes again
// (6) The 7th press will dim the LEDs
// (7) The 7th press will end the music clip
// set up the button
const int button1Pin = 2;  //a
const int button2Pin = 3;  //b
const int button3Pin = 4;  //a
const int button4Pin = 5;  //b
const int greenLedPin = 12;
const int redLedPin = 13;
int buttonState1 = 0;
int buttonState2 = 0;
int buttonState3 = 0;
int buttonState4 = 0;
int pb1 = 0;
int pb2 = 0;
int pb3 = 0;
int pb4 = 0;
bool pressedA = false;  // save which button is pressed
bool pressedB = false;
bool pressedC = false;
bool pressedD = false;  // save the pressed to an array, compare with the previous pressed > only print when two are different
// set up the MP3 player module
#include 
#include 
#include 
SoftwareSerial SoftSerial(10, 11);  //RX and TX from Arduino
DY::Player player(&SoftSerial);     //should connect them to io0 and io1
char s1[] = "/00001.MP3";
char s2[] = "/00002.MP3";
char s3[] = "/00003.MP3";
// set the default round
float state = 1;
int times = 0;  // Set round one as the default start.
void setup() {
  Serial.begin(9600);
  // buttons part
  pinMode(button1Pin, INPUT);
  pinMode(button2Pin, INPUT);
  pinMode(button3Pin, INPUT);
  pinMode(button4Pin, INPUT);
  pinMode(greenLedPin, OUTPUT);
  pinMode(redLedPin, OUTPUT);
  // mp3 part
  Serial.begin(9600);
  Serial.println("Starting the Player...");
  player.begin();
  player.setPlayingDevice(DY::Device::Sd);  //SD card, USB storage volume is
  player.setVolume(30);                     // 30 is 100% of Volume; with 15 you get 50% Volume
  // player.setCycleMode(DY::PlayMode::Repeat); // Play all and repeat.
  player.playSpecifiedDevicePath(DY::Device::Sd, s1);  // choose s1 or s2 or s3 to play different songs.
}
void loop() {
  buttonState1 = digitalRead(button1Pin);
  buttonState2 = digitalRead(button2Pin);
  buttonState3 = digitalRead(button3Pin);
  buttonState4 = digitalRead(button4Pin);
  // checking the button input and save them into booleans (pressedA,B,....)
  // user 1
  if (buttonState1 == HIGH) {
    pressedB = false;
    if (buttonState1 != pb1) {
      times++;
    }
    if (pressedB == false) {
      pressedA = true;
    }
  }
  if (buttonState2 == HIGH) {
    pressedA = false;
    if (buttonState2 != pb2) {
      times++;
    }
    if (pressedA == false) {
      pressedB = true;
    }
  }
  Serial.print("User 1: ");
  Serial.print(pressedA);
  Serial.print(",");
  Serial.print(pressedB);
  // user 2
  if (buttonState3 == HIGH) {
    pressedD = false;
    if (buttonState3 != pb3) {
      times++;
    }
    if (pressedD == false) {
      pressedC = true;
    }
  }
  if (buttonState4 == HIGH) {
    pressedC = false;
    if (buttonState4 != pb4) {
      times++;
    }
    if (pressedC == false) {
      pressedD = true;
    }
  }
  Serial.print(" User 2: ");
  Serial.print(pressedC);
  Serial.print(",");
  Serial.print(pressedD);
  Serial.println();
  // Serial.print(" times: ");
  // Serial.println(times);
  if (state == 1) {
    if (times == 2) {
      if (pressedA == true && pressedC == true) {
        digitalWrite(greenLedPin, HIGH);
        digitalWrite(redLedPin, LOW);
        //player.play();
      } else if (pressedA != pressedC) {
        digitalWrite(greenLedPin, LOW);
        digitalWrite(redLedPin, HIGH);
        delay(200);
        digitalWrite(redLedPin, LOW);
        delay(200);
        player.stop();
      }
      if (pressedB == true && pressedD == true) {
        digitalWrite(greenLedPin, LOW);
        digitalWrite(redLedPin, HIGH);
        delay(200);
        digitalWrite(redLedPin, LOW);
        delay(200);
        player.stop();
      } else if (pressedB != pressedD) {
        digitalWrite(greenLedPin, LOW);
        digitalWrite(redLedPin, HIGH);
        delay(200);
        digitalWrite(redLedPin, LOW);
        delay(200);
        player.stop();
        // player.stop();
      }
    } else if (times == 3) {
      pressedA = false;  // reset all the buttons
      pressedB = false;
      pressedC = false;
      pressedD = false;
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
      player.stop();
      state = 2;
      //Serial.println("state 2");
    } else {
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
    }
  }
  if (state == 2) {
    if (times == 3) {
      pressedA = false;  // reset all the buttons
      pressedB = false;
      pressedC = false;
      pressedD = false;
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
      if (times > 2) {
        player.playSpecifiedDevicePath(DY::Device::Sd, s2);
      }
    } else if (times == 6) {
      if (pressedA == true && pressedC == true) {
        digitalWrite(greenLedPin, HIGH);
        digitalWrite(redLedPin, LOW);
        // state++;

      } else if (pressedA != pressedC) {
        digitalWrite(greenLedPin, LOW);
        digitalWrite(redLedPin, HIGH);
        delay(200);
        digitalWrite(redLedPin, LOW);
        delay(200);
        player.stop();
        // state++;
      }
      if (pressedB == true && pressedD == true) {
        digitalWrite(greenLedPin, LOW);
        digitalWrite(redLedPin, HIGH);
        delay(200);
        digitalWrite(redLedPin, LOW);
        delay(200);
        player.stop();
        // state++;
      } else if (pressedB != pressedD) {
        digitalWrite(greenLedPin, LOW);
        digitalWrite(redLedPin, HIGH);
        delay(200);
        digitalWrite(redLedPin, LOW);
        delay(200);
        player.stop();
        // state++;
      }
    } else {
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
    }
    if (times == 8) {
      player.stop();
      digitalWrite(greenLedPin, LOW);
      digitalWrite(redLedPin, LOW);
    }
  }
  pb1 = buttonState1;
  pb2 = buttonState2;
  pb3 = buttonState3;
  pb4 = buttonState4;
}

「User Test」

In the user test session, we displayed our half-finished project (did not get the player to play specific songs) to professors, fellows, learning assistants and other students. Most users expressed their interest in the idea we presented to interact through music, and they also gave us these constructive suggestions:

• The wires and cables are kind of messy. They should be organized or “hidden”.
• The buttons can be modified.
• We could use bigger speakers to increase the volume and audio quality.
• Change the logic: play the music first then make people choose if they like it.

These suggestions are of high value and they are well-received and appreciated. We were surprised by people’s positive feedback on the two-player compatibility test feature. During the session, it was also helpful for us to learn from other people’s projects.

「Optimization」

After the user test session, we refined our project according before the final presentation in terms of both software and hardware. We figured out the code (as demonstrated above in the “code” section) to play two specific songs, hid the wires, and updated our physical appearance with wider coverage. We tried to get louder speakers but failed to get them, so we used our original amplifiers due to time constraints.

Conclusions & Reflections:

The final project and its working process are shown in this video:

Our project aims to facilitate user interaction by matching their shared music preferences, allowing them to connect with each other through an intuitive interface that enables them to express their likes and dislikes of a given music clip. However, despite our best efforts, the project is still in its infancy and has room for improvement. While the current design offers rewards and punishments to incentivize user engagement, the limited options available for expressing preferences calls for a more proactive exchange of signals and messages. Although our project is designed to be intuitive, some users may need additional guidance throughout the process to fully understand how to interact with the platform. As a solution, we suggest adding audio instructions between music clips to inform users how to navigate the system.

If given more time, there are some improvements we could work on: (1) get a louder speaker that improves the volume and audio quality, which could greatly elevate the user experience; (2) add more songs to the storage (the order can be randomized) so people have a more immersive and rich experience; (3) instead of using buttons to give YES/NO answers, use pots to create a rating system on a standardized scale, giving users larger freedom to express their degree of like/dislike; (4) insert audio instructions in between music clips to guide the users throughout the process to better navigate the system; (5) instead of using LEDs, use a more visually stimulating display that can also aid the audio interactions; (6) further improve the physical appearance and manage the circuit.

With the challenges and setbacks, I have come to realize the value of a solid idea and framework as the foundation of a successful project. This can help avoid unnecessary changes and potential flaws in the future. Additionally, I have learned that debugging involves not only identifying and fixing bugs, but also optimizing the code to make it more efficient and streamlined, ultimately leading to clearer logic and improved running efficiency. It is also important to be proficient in using open-source content available on the web as a crucial source of assistance.

While this project had its challenges, we are happy to have received positive feedback which supports our idea of a two-player, collaborative, interactive game using music as its core social feature. Our achievements also highlighted the significance of user testing, critical suggestions, and self-evaluation. We recognize that our project has many flaws and much room for improvement, but it was a valuable learning experience to know more about hardware, software, coding, online resources and the importance of collaboration in Interaction Lab.

Annex:

References:

1. MANA – Global New Media Art Platform – Art / Design / Technology Cross-Industry Innovation. https://www.manamana.net/video/detail?id=70417#!zh. Accessed 21 Mar. 2023.
2. Musical Swings | Daily Tous Les Jours. https://www.dailytouslesjours.com/en/work/musical-swings. Accessed 21 Mar. 2023.