MUSIC DRUM GAME –JESSIE NIE– RUDI

CONCEPTION AND DESIGN:

For the final project, Sid and I teamed up. We wanted to make a game from the very beginning. We wanted to make a game that was interactive, engaging, and fun. The idea came about in part because we wanted to make a game that would help people relax. Because the main group of people who would see and use our project was NYUSH students, who were under pressure from their studies. Additionally, we wanted to make a game that was not disposable, unlike games such as the charades, and that would provide long-term engagement. After the whole semester of study, we thought we already had the ability to combine and interact with Arduino and Processing. So we wanted to combine the videos shown on Processing with some sensors that Arduino can connect flexibly to drive some physical activities, so we planned to make a video game with strong physical interaction.

In my previous research, I saw a project called UW-Makeathon: Laser Drums. For this project, the designers made some drums with laser cut, and when these different drums are hit, they make different sounds. I was also inspired by the element of drums, and I thought drumming would be a good physical interaction. When I was having a brainstorming discussion with Sid, I mentioned drumming as a way to interact physically. He said he was also reminded of a Japanese rhythm game called Taiko No Tatsujin. This video game is a rhythm game where the user hits the drum on cue when the screen shows that the drum needs to be hit. So we decided to make a drumming video game like this. The idea of the game itself may be simple, but what makes it special is that we made the drums in the video into real drums.

At the same time, we thought it would be too basic if we made it a drumming rhythm game for a single player. Existing rhythm games are basically single-player ones, so we considered making it a two-player game, because if you work with someone else, your experience is always different because that person who works with you is different. To put our own ideas into action, we wanted to make three drums and have those drums associated with the three primary colors. But with only three colors, it would be a bit too simple for two players. So we added mixed colors to the game, and the user had to hit the drums of the two primary colors in order to eliminate these mixed color blocks on the screen. This required communication and collaboration between the two players, so that there would be more interaction between the player and the game, and between the two players. So, in the end, we decided to have three drums and each player had one drumstick, and they had to collaborate and play the drums according to the color blocks on the screen. The screen would show six color blocks in total, which are the three primary colors, red, yellow and blue; and mixed colors like green, orange and purple. Players must have proper timing because, for the mixed colors, they must each play the drums at the same time.

And to make this game more interesting and engaging, we intend to set up three different modes: easy, medium, and hard. Each mode corresponds to a song, from soothing music to fast-paced music. During the User Testing session, we had only one mode for now, and the color blocks in the video were set to fall randomly according to one speed. But the whole process worked very smoothly and exceeded our expectations.

This is a recorded video from user testing：

According to the testers’ feedback, they thought our game was fun and the sensor was very accurate. However, there were six tracks of color blocks falling on the screen at the same time, making it difficult for them to focus and thus not knowing which drum to hit. Especially when the mixed colors fell together, it was more confusing. Because one mixed color would make them react for a while, let alone three mixed colors. They also generally reported that our color blocks fell too fast for people to react. Based on user feedback, we decided to slow down the entire color drop and change the six tracks to four. The first three tracks are the three primary colors, while the last track is the mixed colors. And there will be no mixed colors falling on the screen in simple mode, only three primary colors. In addition, we have added an instruction video to familiarize users with the composition of mixed colors before the game starts.

I think the user testing was quite useful for our project, because according to Sid’s feedback, people liked our project overall during our final presentation. Our game was interactive enough to get people involved, which means that the changes we made to the difficulty were effective. And we found that even two people who didn’t know each other well would unconsciously communicate when playing our game. They would unconsciously say things like “green is coming, hit yellow and blue”. So we feel that our whole project went through in a way that was very much in line with what we envisioned at the beginning.

FABRICATION AND PRODUCTION:

As for the production of the whole game, I think Sid and I worked well both together and divided the work. We found the game sample we wanted to reference together, and then he was in charge of the programming part and I was in charge of the fabrication part.

Let me start by describing the part Sid and I made together. Before we officially started working on the final, we had been looking for some drum game samples in open processing, and we found a pretty good one at first. But it uses p5.js. language but not java, which we learned in class. We tried to adapt it, but we gave up and started looking for a new sample game code when we thought that all the changes afterwards would require this new language. And then we found another sample game code, which we actually used for our final.

And this is the game of which we took the code as a reference. Even though it looks very different from the game interface we had afterwards, it really helped us a lot.

We also brainstormed what sensors could be used on the drums to make the drums sense the signals of hitting more accurately. At first we thought of the pressure sensor, which we wanted to attach to the back of the drum, but found out that this sensor could not be used in this way. It needs to sense the force of pressure on it rather than a sudden hit. Then we came up with the volume sensor, which I thought would be placed inside the drum, because it would be very loud when the top of the drum was struck. But we hadn’t tested the volume sensor yet when a better idea came up, my friend Evelyn suggested we use a vibration sensor, which I didn’t know that we had in our equipment room. If the vibration sensor is glued to the back of the drum, it would catch a strong vibration when it is hit. So we borrowed some vibration sensors and tested them. 

Our vibration sensor looks like this, but the connecting part is a bit different. We didn’t take a picture of it. It’s a picture of the sensor online.

But I have to say that the vibration sensors in the equipment room were really insensitive. During the night before the user testing, when my fabrication part and Sid’s programming part were basically done, we found out to our dismay that many of the sensors we borrowed were very insensitive. As a consequence, when the drum was hit, the Arduino would not detect that it was vibrating, and thus the color blocks on the video would not disappear. Sid and I were very frustrated at the time because the sensitivity of the sensors determined to some extent the meaning of our game. If the sensor does not work, then the interaction between people and the game can not be connected, and everything will be meaningless. So that night we thought of a lot of ways to make the sensors more sensitive, we tried to change the position of its paste on the drum, paste the degree of tightness, we even re-solder some sensors on the wire. The good thing is that in the end, they worked way more sensitively in the user testing session, and after several rounds of testing, it worked so well and accurately during the final presentation.

Below I will talk about the parts I was mainly responsible for. I was in charge of the creation of the three drums and the handles. As well as the production of the introductory and instructional videos at the beginning of the game. I think the whole process of fabrication was very interesting, I mainly used the technology of laser cut. How to use the laser cut machine, how to design the pattern, and how to choose the right material are all very important, and I am very happy to learn them in this final production process.

For the drums, I chose cardboard for the drum body because the cardboard can be folded into a cylinder. As for the drum surface, because our drums need to represent a color, I directly chose the acrylic board with its own color instead of painting other materials. The whole process was pretty smooth, but a little time-consuming. I folded the laser-cut cardboard into a cylinder, glued the acrylic board and sensor together, and then glued it to the drum body. But the initial drum was a bit too unattractive, so I went to the fabrication lab to find some materials. I then wrapped the drum body with black cloth, and decorated it with a layer of white yarn. Finally, I decorated the acrylic drum surface and the drum body with a circle of cotton to cover the traces of hot melt glue.

As for the handles, because we found that the vibration sensor responded more accurately to a larger area hit, we decided to replace the drumsticks with round handles in the shape of tapes. I also used laser cut to make the handles, I cut ten concentric circles on a 5mm wood board and glued five together to make one handle. But the handle was not very comfortable to hold. So I wrapped the handles with a towel material.

The intro and color guide video at the beginning of the game I made in Canva, I wanted it to look nice and interesting. And I think I did it.

Then I will introduce the part that Sid was responsible for. He was mainly working on the Code part of the game, and he changed our sample code to what it is now. He put in the video game the option to choose a mode, the scoring function, and the design of the four tracks, different color blocks will fall down from different tracks. He also inserted my intro video into the game, which took him a lot of time because there was only one void draw in Processing, and this video had a lot of trouble inserting it into the game at first. He really did a lot of work. We tested sensors together, and he did resoldering and a series of tweaks to the insensitive sensors.

Here’s the video of people playing the game we made, which is also the final documentation video:

CONCLUSIONS:

Our goal from the beginning of this project was to make a two-player musical drumming game, which could provide the players with long-term engagement. We wanted the game to be fun, fully engaging, and to unconsciously promote interaction between the two players. From the results of the players’ reactions, the game was relatively successful and met our expectations at the beginning.

I think it’s interesting to see how our players interact with this game, and how two players interact with each other. When the players are playing our game, their attention is very focused on the screen, and they will be very focused on hitting the drum of the corresponding color according to the color blocks that fall on the screen. And once they eliminate the color blocks, the game will give points to players, and at the end, it will also show the players’ scores. So I think the interaction between our game and each player is very adequate. We also found that the interactions between different two players were very interesting in different combinations, many of them would communicate directly by talking. They would assign drums in advance and remind each other to hit the drum of a certain color. But some other players might just say a word or two before the game starts and simply assign tasks, and then play the game in silence and concentration. But I found that even the “silent players” were actually doing teamwork, just in a quieter way. They would look at each other when the mixed-color blocks arrived or just hit two different drums at the same time in perfect silence. This made me realize that our game did make the two players cooperate and interact, and they were both very engaged in the game and focused.

To be honest, our project didn’t really change my general definition of the concept of “interaction”. I think interaction still refers to the process of mutual “input” and “output” based on each other’s responses. But I have gained some new insights into multiple people interacting with the same thing. I found that even if we think of two or more people as interacting with one another thing, there are bound to be various forms of interaction between these people, and they are connected in any case. I think that’s the extension characteristic of “interaction”. Each individual involved in the interaction is in fact unintentionally interacting with each other in various ways. And that’s also what I think is the most meaningful part of our project, when two people play this game together, they are engaged in our game, but they are also engaged in the cooperation with each other, no matter how this cooperation is in the form of. During this game, for one minute or two, you must consider that there is another person doing the same thing with you and that you need to collaborate in order to achieve a high score. We’ve actually met people who wanted to play this game alone, and we let them try. But we found that people didn’t laugh or show much joy when they played the game alone. But when two people play, there is more communication between players in general, and people show more anticipation and smile more, regardless of the outcome of their game.

I actually think we’ve accomplished what we were aiming for at the beginning, but if I had more time, I think I’d work with Sid to refine our introduction of mixed colors, and we could have a few mixed color blocks fall out on the screen before the game starts, and then let the player try to eliminate them to get them before the game officially starts.

As for the gains, technically I learned to be proficient in laser cut, and I found that this technic can really assist in many things, which is very useful. I’m satisfied with the overall design of our drums and handles, so I think I’m doing well with my hands-on design skills. Also, I have a clearer understanding of the logic of designing a project. We need to have the goal of “what I want to do”, but we also need to make the idea into something that the user can really use and understand. For example, at the beginning Sid and I had discussed the mixed color idea many times, so we kind of defaulted that everyone knows about the mixed colors, and the color compositions. But the fact is not as we thought. During the user testing session, people were confused when they saw the green and orange color blocks. Then I realized that people are not familiar with these colors, and we needed to make instructions. I think this point is also what I learned during the whole project production process.

To sum up, our project is a successful two-player musical drumming game. It is very interactive and engaging, and is a very complete game that does not confuse the player with the steps. The physical interaction form of drumming is also good in my view. Most importantly, it requires two players to participate together, and while the players are fully engaged in the game, there is a lot of interaction and connection between the players themselves. This is the most interesting point of our project to me, and a window to think about many new thoughts about “interaction.”

ANNEX

Processing Code (based on this Jeppe Brown Richardson’s code):

import processing.serial.*;
import osteele.processing.SerialRecord.*;
import processing.sound.*;
import processing.video.*;

// declare move object
Movie movie1;
Movie movie2;

// declare a SoundFile object
SoundFile sound1;
SoundFile sound2;
SoundFile sound3;

Serial serialPort;
SerialRecord serialRecord;

ArrayList tile = new ArrayList();
static int gameWidth = 1200, gameHeight = 600;
int score=0;
int misses=0;
String questionMessage1;
String questionMessage2;
boolean first_time = true;
boolean display_question = false;
int blue;
int yellow;
int red;
int difficulty_num;
float difficulty_speed = 1.5;
float duration;
int gamestatus = 0;

void setup() {
  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);
  serialRecord = new SerialRecord(this, serialPort, 3);

  fullScreen();
  noStroke();
  questionMessage1 = "Do You Want To Play Again?";
  questionMessage2 = "Choose Your Difficulty";
  askingQuestion = true;
  textSize(18);

  sound1 = new SoundFile(this, "christmas.mp3"); // It's Beginning To Look A Lot Like Christmas - Perry Como
  sound2 = new SoundFile(this, "tree.mp3"); // Christmas Tree Farm - Taylor Swift
  sound3 = new SoundFile(this, "santa.mp3"); // Santa Tell Me - Ariana Grande

  movie1 = new Movie(this, "opening.mp4");
  movie1.play();

  //image(movie, 0, 0, width, height);

  //beginning video or instruction video
}

void draw() {

  serialRecord.read();
  blue = serialRecord.values[1]; //blue
  yellow = serialRecord.values[2]; //yellow
  red = serialRecord.values[0]; //red

  background(50, 150, 200, 200);

  if (gamestatus == 0) {
    if (movie1.available()) {
      movie1.read();
      print(questionMessage2);
    } else if (movie1.available() == false) {
      print(questionMessage1);
      gamestatus = 1;
    }
    image(movie1, 0, 0, width, height);
  } else if (gamestatus == 1) {




    //image(movie, 0, 0, width, height);


    tiles til = new tiles(int(random(difficulty_num)), int(random(3)));

    fill(#aa0000, 50);
    rect(0, 600, width, 100);

    if (frameCount%50==6) {            //amount of notes generates per second, frameCount%10=6 notes/second
      tile.add(til);
    }

    fill(#aaaaaa);
    rect(200, 0, 20, height);
    rect(450, 0, 20, height);
    rect(700, 0, 20, height);
    rect(950, 0, 20, height);

    for (int i=0; i<tile.size(); i++) {

      tiles ta = (tiles) tile.get(i);
      ta.run();
      ta.display();
      ta.move(difficulty_speed);
      if ( red > 10 && ta.location.y > 550 && ta.location.y < 750&& ta.location.x == 160.0) { //red
        ta.gone=true;
        serialRecord.read();

        red = serialRecord.values[0];
      }
      if (blue > 10 && ta.location.y > 550 && ta.location.y < 750 && ta.location.x==410.0) { //blue
        ta.gone=true;
        serialRecord.read();
        blue = serialRecord.values[1];
      }
      if (yellow > 10 && ta.location.y > 550 && ta.location.y < 750 && ta.location.x==660.0) { //yellow
        ta.gone=true;
        serialRecord.read();
        yellow = serialRecord.values[2];
      }
      if (yellow > 30 && blue > 30 && ta.location.y > 550 && ta.location.y < 700 &&ta.location.x==910.0) { //green
        ta.gone=true;
        serialRecord.read();
        
        yellow = serialRecord.values[2];
        blue = serialRecord.values[1];
      }
      if (red > 30 && blue > 30 && ta.location.y > 550 && ta.location.y < 700&&ta.location.x==910.0) { //purple
        ta.gone=true;
        serialRecord.read();
        
        red = serialRecord.values[0];
        blue = serialRecord.values[1];
      }
      if (red > 30 && yellow > 30 && ta.location.y > 550 && ta.location.y < 700 &&ta.location.x==910.0) { //orange
        ta.gone=true;
        serialRecord.read();
        
        red = serialRecord.values[1];
        yellow = serialRecord.values[2];
      }


      if (ta.location.y>1000) {
        tile.remove(i);
        misses++;
      }
      if (ta.gone==true) {
        score+=ta.location.y>650?30:ta.location.y>600?20:10;    //scoring system(you get more points if you do better)
        tile.remove(i);
      }
    }

    fill(#0000aa); // score display
    textAlign(CENTER);
    textSize(50);
    text(score, 1300, 130);

    if (sound1.isPlaying() == false && sound2.isPlaying() == false && sound3.isPlaying() == false) {
      if (first_time == true) {
        drawDifficulty();
        first_time = false;
      } else if (display_question == false) {
        display_question = true;
        askingQuestion = true;
        for (int i=0; i<tile.size(); i++) { // removes all tiles after song stops
          tile.remove(i);
        }

        drawScore();
      } else if (display_question == true) {
        askingQuestion = true;
        display_question = false;
        drawDifficulty();
        score = 0;
        misses = 0;
      }
    }
  } else if (gamestatus == 2) {
    movie2 = new Movie(this, "ending.mp4");
    movie2.play();
    if (movie2.available()) {
      movie2.read();
      println("Game Over! Thanks for playing, please refresh the page when you are ready to PLAY AGAIN!");
    } else if (movie2.available() == false) {
      println("bug!");
      gamestatus = 3;
    }
    image(movie2, 0, 0, width, height);
  } else if (gamestatus == 3) {
    endGame();
  }
}

void mousePressed() {
  int check1 = checkAnswer1();
  if (check1 == YES) {
    loop();
  }
  if (check1 == NO) {

    // add end video
    gamestatus = 2;
  }

  int check2 = checkAnswer2();
  if (check2 == EASY) {
    print(check2);
    difficulty_num = 3;
    difficulty_speed = 1.5;
    sound1.play();
    duration = sound1.duration();
    loop();
  } else if (check2 == MEDIUM) {
    difficulty_num = 4;
    difficulty_speed = 1.5;
    print(check2);
    sound2.play();
    duration = sound2.duration();
    loop();
  } else if (check2 == HARD) {
    difficulty_num = 4;
    difficulty_speed = 2.8;
    sound3.play();
    duration = sound3.duration();
    loop();
    print(check2);
  }
}


class tiles {
  PVector location;
  Boolean gone=false;
  color tile_color;

  tiles(int i, int j) {
    color[] colors = {#19ad05, #9504de, #fc8403};
    location = new PVector((i*250) + 160, 0);
    tile_color = colors[j];
  }


  void run() {
    display();
    move(difficulty_speed);
  }

  void display() { //#19ad05, #9504de, #edd602
    fill(location.x>=0 && location.x<200 ?#de0404: location.x>200 && location.x<=410 ?#0416de:
      location.x>410 && location.x<=660 ?#edd602: location.x>660 && location.x<=910?tile_color: location.x>900 && location.x<1200?#fc7600:#fafafa);
    rect(location.x, location.y, 100, 50, 40);
  }

  void move(float speed) {
    location.y+=speed;
    //note speed, changing this will up the difficulity, putting it too high will make
  }                                  //it literally impossible
}


static boolean askingQuestion = false, answer = false;
static int questionX=gameWidth/2-150, questionY=gameHeight/2-40, questionWidth=370, questionHeight=100;
final static int NO_ANSWER = 0;
final static int YES = 1;
final static int NO = 2;
final static int EASY = 3;
final static int MEDIUM = 4;
final static int HARD = 5;

int checkAnswer2() { //Check to see if user clicked which difficulty
  if (mouseX >= questionX && mouseX <= questionWidth + 650 && mouseY >= questionY + 50 && mouseY <= questionY+questionHeight && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return EASY;
  }
  if (mouseX >= questionX && mouseX <= questionWidth + 650 && mouseY >= questionY + 100 && mouseY <= questionY + 140 && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return MEDIUM;
  }
  if (mouseX >= questionX && mouseX <= questionWidth + 650 && mouseY >= questionY + 150 && mouseY <= questionY + 190 && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return HARD;
  }

  return NO_ANSWER;
}

int checkAnswer1() { //Check to see if user clicked yes or no
  if (mouseX >= questionX + 600 && mouseX <= questionX + 783 && mouseY >= questionY + 150 && mouseY <= questionY + 190 && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return YES;
  }
  if (mouseX >= questionX + 783 && mouseX <= questionX + 950 && mouseY >= questionY + 150 && mouseY <= questionY + 190 && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return NO;
  }

  return NO_ANSWER;
}


void drawScore() {
  stroke(225);
  fill(50);
  //rect(questionX - 2, questionY - 2, questionWidth + 10, questionHeight + 4);
  rect(questionX + 600, questionY, questionWidth, questionHeight - 50);
  rect(questionX + 600, questionY + 50, questionWidth, questionHeight - 50);
  rect(questionX + 600, questionY + 100, questionWidth, questionHeight - 50);
  rect(questionX + 600, questionY + 150, questionWidth, questionHeight - 50);
  rect(questionX + 783, questionY + 150, questionWidth - 183, questionHeight - 50);
  stroke(225);
  fill(225);
  textSize(30);
  text(questionMessage1, questionX + 790, questionY + questionHeight - 66);

  text("Score: " + score, questionX + 685, questionY + questionHeight - 18);
  text("Misses: " + misses, questionX + 682, questionY + questionHeight + 34);
  text("Yes", questionX + 684, questionY + questionHeight + 83);
  text("No", questionX + 872, questionY + questionHeight + 83);
  noLoop();
}

void drawDifficulty() {
  stroke(225);
  fill(50);
  //rect(questionX - 2, questionY - 2, questionWidth + 10, questionHeight + 4);
  rect(questionX, questionY, questionWidth + 160, questionHeight - 50);
  rect(questionX, questionY + 50, questionWidth + 160, questionHeight - 50);
  rect(questionX, questionY + 100, questionWidth + 160, questionHeight - 50);
  rect(questionX, questionY + 150, questionWidth + 160, questionHeight - 50);
  stroke(225);
  fill(225);
  textSize(30);
  text(questionMessage2, questionX + 250, questionY + questionHeight - 66);
  textSize(20);
  text("Easy: It's Beginning To Look A Lot Like Christmas - Perry Como", questionX + 267, questionY + questionHeight - 18);
  text("Medium: Christmas Tree Farm - Taylor Swift", questionX + 190, questionY + questionHeight + 34);
  text("Hard: Santa Tell Me - Ariana Grande", questionX + 158, questionY + questionHeight + 81);
  noLoop();
}

void endGame() {
  noLoop();
  exit();
}

 Arduino Code:

#include "SerialRecord.h"

SerialRecord writer(3);

void setup() {
  Serial.begin(9600);
}

void loop() {
  
  int value1 = analogRead(0);
  int value2 = analogRead(2);
  int value3 = analogRead(4);
  writer.send();
  //if (value1 > 0) {
  writer[0] = value1;
  writer[1] = value2;
  writer[2] = value3;
    //writer.send(0);
      //
  //}
  //if (value2 > 0) {
  
  
      //
  //}
  
  // This delay slows down the loop, so that it runs less frequently. This can
  // make it easier to debug the sketch, because new values are printed at a
  // slower rate.
  delay(20);
  
}