MUSIC DRUM GAME –JESSIE NIE– RUDI

CONCEPTION AND DESIGN:

For the final project, Sid and I teamed up. We wanted to make a game from the very beginning. We wanted to make a game that was interactive, engaging, and fun. The idea came about in part because we wanted to make a game that would help people relax. Because the main group of people who would see and use our project was NYUSH students, who were under pressure from their studies. Additionally, we wanted to make a game that was not disposable, unlike games such as the charades, and that would provide long-term engagement. After the whole semester of study, we thought we already had the ability to combine and interact with Arduino and Processing. So we wanted to combine the videos shown on Processing with some sensors that Arduino can connect flexibly to drive some physical activities, so we planned to make a video game with strong physical interaction.

In my previous research, I saw a project called UW-Makeathon: Laser Drums. For this project, the designers made some drums with laser cut, and when these different drums are hit, they make different sounds. I was also inspired by the element of drums, and I thought drumming would be a good physical interaction. When I was having a brainstorming discussion with Sid, I mentioned drumming as a way to interact physically. He said he was also reminded of a Japanese rhythm game called Taiko No Tatsujin. This video game is a rhythm game where the user hits the drum on cue when the screen shows that the drum needs to be hit. So we decided to make a drumming video game like this. The idea of the game itself may be simple, but what makes it special is that we made the drums in the video into real drums.

At the same time, we thought it would be too basic if we made it a drumming rhythm game for a single player. Existing rhythm games are basically single-player ones, so we considered making it a two-player game, because if you work with someone else, your experience is always different because that person who works with you is different. To put our own ideas into action, we wanted to make three drums and have those drums associated with the three primary colors. But with only three colors, it would be a bit too simple for two players. So we added mixed colors to the game, and the user had to hit the drums of the two primary colors in order to eliminate these mixed color blocks on the screen. This required communication and collaboration between the two players, so that there would be more interaction between the player and the game, and between the two players. So, in the end, we decided to have three drums and each player had one drumstick, and they had to collaborate and play the drums according to the color blocks on the screen. The screen would show six color blocks in total, which are the three primary colors, red, yellow and blue; and mixed colors like green, orange and purple. Players must have proper timing because, for the mixed colors, they must each play the drums at the same time.

And to make this game more interesting and engaging, we intend to set up three different modes: easy, medium, and hard. Each mode corresponds to a song, from soothing music to fast-paced music. During the User Testing session, we had only one mode for now, and the color blocks in the video were set to fall randomly according to one speed. But the whole process worked very smoothly and exceeded our expectations.

This is a recorded video from user testing：

According to the testers’ feedback, they thought our game was fun and the sensor was very accurate. However, there were six tracks of color blocks falling on the screen at the same time, making it difficult for them to focus and thus not knowing which drum to hit. Especially when the mixed colors fell together, it was more confusing. Because one mixed color would make them react for a while, let alone three mixed colors. They also generally reported that our color blocks fell too fast for people to react. Based on user feedback, we decided to slow down the entire color drop and change the six tracks to four. The first three tracks are the three primary colors, while the last track is the mixed colors. And there will be no mixed colors falling on the screen in simple mode, only three primary colors. In addition, we have added an instruction video to familiarize users with the composition of mixed colors before the game starts.

I think the user testing was quite useful for our project, because according to Sid’s feedback, people liked our project overall during our final presentation. Our game was interactive enough to get people involved, which means that the changes we made to the difficulty were effective. And we found that even two people who didn’t know each other well would unconsciously communicate when playing our game. They would unconsciously say things like “green is coming, hit yellow and blue”. So we feel that our whole project went through in a way that was very much in line with what we envisioned at the beginning.

FABRICATION AND PRODUCTION:

As for the production of the whole game, I think Sid and I worked well both together and divided the work. We found the game sample we wanted to reference together, and then he was in charge of the programming part and I was in charge of the fabrication part.

Let me start by describing the part Sid and I made together. Before we officially started working on the final, we had been looking for some drum game samples in open processing, and we found a pretty good one at first. But it uses p5.js. language but not java, which we learned in class. We tried to adapt it, but we gave up and started looking for a new sample game code when we thought that all the changes afterwards would require this new language. And then we found another sample game code, which we actually used for our final.

And this is the game of which we took the code as a reference. Even though it looks very different from the game interface we had afterwards, it really helped us a lot.

We also brainstormed what sensors could be used on the drums to make the drums sense the signals of hitting more accurately. At first we thought of the pressure sensor, which we wanted to attach to the back of the drum, but found out that this sensor could not be used in this way. It needs to sense the force of pressure on it rather than a sudden hit. Then we came up with the volume sensor, which I thought would be placed inside the drum, because it would be very loud when the top of the drum was struck. But we hadn’t tested the volume sensor yet when a better idea came up, my friend Evelyn suggested we use a vibration sensor, which I didn’t know that we had in our equipment room. If the vibration sensor is glued to the back of the drum, it would catch a strong vibration when it is hit. So we borrowed some vibration sensors and tested them. 

Our vibration sensor looks like this, but the connecting part is a bit different. We didn’t take a picture of it. It’s a picture of the sensor online.

But I have to say that the vibration sensors in the equipment room were really insensitive. During the night before the user testing, when my fabrication part and Sid’s programming part were basically done, we found out to our dismay that many of the sensors we borrowed were very insensitive. As a consequence, when the drum was hit, the Arduino would not detect that it was vibrating, and thus the color blocks on the video would not disappear. Sid and I were very frustrated at the time because the sensitivity of the sensors determined to some extent the meaning of our game. If the sensor does not work, then the interaction between people and the game can not be connected, and everything will be meaningless. So that night we thought of a lot of ways to make the sensors more sensitive, we tried to change the position of its paste on the drum, paste the degree of tightness, we even re-solder some sensors on the wire. The good thing is that in the end, they worked way more sensitively in the user testing session, and after several rounds of testing, it worked so well and accurately during the final presentation.

Below I will talk about the parts I was mainly responsible for. I was in charge of the creation of the three drums and the handles. As well as the production of the introductory and instructional videos at the beginning of the game. I think the whole process of fabrication was very interesting, I mainly used the technology of laser cut. How to use the laser cut machine, how to design the pattern, and how to choose the right material are all very important, and I am very happy to learn them in this final production process.

For the drums, I chose cardboard for the drum body because the cardboard can be folded into a cylinder. As for the drum surface, because our drums need to represent a color, I directly chose the acrylic board with its own color instead of painting other materials. The whole process was pretty smooth, but a little time-consuming. I folded the laser-cut cardboard into a cylinder, glued the acrylic board and sensor together, and then glued it to the drum body. But the initial drum was a bit too unattractive, so I went to the fabrication lab to find some materials. I then wrapped the drum body with black cloth, and decorated it with a layer of white yarn. Finally, I decorated the acrylic drum surface and the drum body with a circle of cotton to cover the traces of hot melt glue.

As for the handles, because we found that the vibration sensor responded more accurately to a larger area hit, we decided to replace the drumsticks with round handles in the shape of tapes. I also used laser cut to make the handles, I cut ten concentric circles on a 5mm wood board and glued five together to make one handle. But the handle was not very comfortable to hold. So I wrapped the handles with a towel material.

The intro and color guide video at the beginning of the game I made in Canva, I wanted it to look nice and interesting. And I think I did it.

Then I will introduce the part that Sid was responsible for. He was mainly working on the Code part of the game, and he changed our sample code to what it is now. He put in the video game the option to choose a mode, the scoring function, and the design of the four tracks, different color blocks will fall down from different tracks. He also inserted my intro video into the game, which took him a lot of time because there was only one void draw in Processing, and this video had a lot of trouble inserting it into the game at first. He really did a lot of work. We tested sensors together, and he did resoldering and a series of tweaks to the insensitive sensors.

Here’s the video of people playing the game we made, which is also the final documentation video:

CONCLUSIONS:

Our goal from the beginning of this project was to make a two-player musical drumming game, which could provide the players with long-term engagement. We wanted the game to be fun, fully engaging, and to unconsciously promote interaction between the two players. From the results of the players’ reactions, the game was relatively successful and met our expectations at the beginning.

I think it’s interesting to see how our players interact with this game, and how two players interact with each other. When the players are playing our game, their attention is very focused on the screen, and they will be very focused on hitting the drum of the corresponding color according to the color blocks that fall on the screen. And once they eliminate the color blocks, the game will give points to players, and at the end, it will also show the players’ scores. So I think the interaction between our game and each player is very adequate. We also found that the interactions between different two players were very interesting in different combinations, many of them would communicate directly by talking. They would assign drums in advance and remind each other to hit the drum of a certain color. But some other players might just say a word or two before the game starts and simply assign tasks, and then play the game in silence and concentration. But I found that even the “silent players” were actually doing teamwork, just in a quieter way. They would look at each other when the mixed-color blocks arrived or just hit two different drums at the same time in perfect silence. This made me realize that our game did make the two players cooperate and interact, and they were both very engaged in the game and focused.

To be honest, our project didn’t really change my general definition of the concept of “interaction”. I think interaction still refers to the process of mutual “input” and “output” based on each other’s responses. But I have gained some new insights into multiple people interacting with the same thing. I found that even if we think of two or more people as interacting with one another thing, there are bound to be various forms of interaction between these people, and they are connected in any case. I think that’s the extension characteristic of “interaction”. Each individual involved in the interaction is in fact unintentionally interacting with each other in various ways. And that’s also what I think is the most meaningful part of our project, when two people play this game together, they are engaged in our game, but they are also engaged in the cooperation with each other, no matter how this cooperation is in the form of. During this game, for one minute or two, you must consider that there is another person doing the same thing with you and that you need to collaborate in order to achieve a high score. We’ve actually met people who wanted to play this game alone, and we let them try. But we found that people didn’t laugh or show much joy when they played the game alone. But when two people play, there is more communication between players in general, and people show more anticipation and smile more, regardless of the outcome of their game.

I actually think we’ve accomplished what we were aiming for at the beginning, but if I had more time, I think I’d work with Sid to refine our introduction of mixed colors, and we could have a few mixed color blocks fall out on the screen before the game starts, and then let the player try to eliminate them to get them before the game officially starts.

As for the gains, technically I learned to be proficient in laser cut, and I found that this technic can really assist in many things, which is very useful. I’m satisfied with the overall design of our drums and handles, so I think I’m doing well with my hands-on design skills. Also, I have a clearer understanding of the logic of designing a project. We need to have the goal of “what I want to do”, but we also need to make the idea into something that the user can really use and understand. For example, at the beginning Sid and I had discussed the mixed color idea many times, so we kind of defaulted that everyone knows about the mixed colors, and the color compositions. But the fact is not as we thought. During the user testing session, people were confused when they saw the green and orange color blocks. Then I realized that people are not familiar with these colors, and we needed to make instructions. I think this point is also what I learned during the whole project production process.

To sum up, our project is a successful two-player musical drumming game. It is very interactive and engaging, and is a very complete game that does not confuse the player with the steps. The physical interaction form of drumming is also good in my view. Most importantly, it requires two players to participate together, and while the players are fully engaged in the game, there is a lot of interaction and connection between the players themselves. This is the most interesting point of our project to me, and a window to think about many new thoughts about “interaction.”

ANNEX

Processing Code (based on this Jeppe Brown Richardson’s code):

import processing.serial.*;
import osteele.processing.SerialRecord.*;
import processing.sound.*;
import processing.video.*;

// declare move object
Movie movie1;
Movie movie2;

// declare a SoundFile object
SoundFile sound1;
SoundFile sound2;
SoundFile sound3;

Serial serialPort;
SerialRecord serialRecord;

ArrayList tile = new ArrayList();
static int gameWidth = 1200, gameHeight = 600;
int score=0;
int misses=0;
String questionMessage1;
String questionMessage2;
boolean first_time = true;
boolean display_question = false;
int blue;
int yellow;
int red;
int difficulty_num;
float difficulty_speed = 1.5;
float duration;
int gamestatus = 0;

void setup() {
  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);
  serialRecord = new SerialRecord(this, serialPort, 3);

  fullScreen();
  noStroke();
  questionMessage1 = "Do You Want To Play Again?";
  questionMessage2 = "Choose Your Difficulty";
  askingQuestion = true;
  textSize(18);

  sound1 = new SoundFile(this, "christmas.mp3"); // It's Beginning To Look A Lot Like Christmas - Perry Como
  sound2 = new SoundFile(this, "tree.mp3"); // Christmas Tree Farm - Taylor Swift
  sound3 = new SoundFile(this, "santa.mp3"); // Santa Tell Me - Ariana Grande

  movie1 = new Movie(this, "opening.mp4");
  movie1.play();

  //image(movie, 0, 0, width, height);

  //beginning video or instruction video
}

void draw() {

  serialRecord.read();
  blue = serialRecord.values[1]; //blue
  yellow = serialRecord.values[2]; //yellow
  red = serialRecord.values[0]; //red

  background(50, 150, 200, 200);

  if (gamestatus == 0) {
    if (movie1.available()) {
      movie1.read();
      print(questionMessage2);
    } else if (movie1.available() == false) {
      print(questionMessage1);
      gamestatus = 1;
    }
    image(movie1, 0, 0, width, height);
  } else if (gamestatus == 1) {




    //image(movie, 0, 0, width, height);


    tiles til = new tiles(int(random(difficulty_num)), int(random(3)));

    fill(#aa0000, 50);
    rect(0, 600, width, 100);

    if (frameCount%50==6) {            //amount of notes generates per second, frameCount%10=6 notes/second
      tile.add(til);
    }

    fill(#aaaaaa);
    rect(200, 0, 20, height);
    rect(450, 0, 20, height);
    rect(700, 0, 20, height);
    rect(950, 0, 20, height);

    for (int i=0; i<tile.size(); i++) {

      tiles ta = (tiles) tile.get(i);
      ta.run();
      ta.display();
      ta.move(difficulty_speed);
      if ( red > 10 && ta.location.y > 550 && ta.location.y < 750&& ta.location.x == 160.0) { //red
        ta.gone=true;
        serialRecord.read();

        red = serialRecord.values[0];
      }
      if (blue > 10 && ta.location.y > 550 && ta.location.y < 750 && ta.location.x==410.0) { //blue
        ta.gone=true;
        serialRecord.read();
        blue = serialRecord.values[1];
      }
      if (yellow > 10 && ta.location.y > 550 && ta.location.y < 750 && ta.location.x==660.0) { //yellow
        ta.gone=true;
        serialRecord.read();
        yellow = serialRecord.values[2];
      }
      if (yellow > 30 && blue > 30 && ta.location.y > 550 && ta.location.y < 700 &&ta.location.x==910.0) { //green
        ta.gone=true;
        serialRecord.read();
        
        yellow = serialRecord.values[2];
        blue = serialRecord.values[1];
      }
      if (red > 30 && blue > 30 && ta.location.y > 550 && ta.location.y < 700&&ta.location.x==910.0) { //purple
        ta.gone=true;
        serialRecord.read();
        
        red = serialRecord.values[0];
        blue = serialRecord.values[1];
      }
      if (red > 30 && yellow > 30 && ta.location.y > 550 && ta.location.y < 700 &&ta.location.x==910.0) { //orange
        ta.gone=true;
        serialRecord.read();
        
        red = serialRecord.values[1];
        yellow = serialRecord.values[2];
      }


      if (ta.location.y>1000) {
        tile.remove(i);
        misses++;
      }
      if (ta.gone==true) {
        score+=ta.location.y>650?30:ta.location.y>600?20:10;    //scoring system(you get more points if you do better)
        tile.remove(i);
      }
    }

    fill(#0000aa); // score display
    textAlign(CENTER);
    textSize(50);
    text(score, 1300, 130);

    if (sound1.isPlaying() == false && sound2.isPlaying() == false && sound3.isPlaying() == false) {
      if (first_time == true) {
        drawDifficulty();
        first_time = false;
      } else if (display_question == false) {
        display_question = true;
        askingQuestion = true;
        for (int i=0; i<tile.size(); i++) { // removes all tiles after song stops
          tile.remove(i);
        }

        drawScore();
      } else if (display_question == true) {
        askingQuestion = true;
        display_question = false;
        drawDifficulty();
        score = 0;
        misses = 0;
      }
    }
  } else if (gamestatus == 2) {
    movie2 = new Movie(this, "ending.mp4");
    movie2.play();
    if (movie2.available()) {
      movie2.read();
      println("Game Over! Thanks for playing, please refresh the page when you are ready to PLAY AGAIN!");
    } else if (movie2.available() == false) {
      println("bug!");
      gamestatus = 3;
    }
    image(movie2, 0, 0, width, height);
  } else if (gamestatus == 3) {
    endGame();
  }
}

void mousePressed() {
  int check1 = checkAnswer1();
  if (check1 == YES) {
    loop();
  }
  if (check1 == NO) {

    // add end video
    gamestatus = 2;
  }

  int check2 = checkAnswer2();
  if (check2 == EASY) {
    print(check2);
    difficulty_num = 3;
    difficulty_speed = 1.5;
    sound1.play();
    duration = sound1.duration();
    loop();
  } else if (check2 == MEDIUM) {
    difficulty_num = 4;
    difficulty_speed = 1.5;
    print(check2);
    sound2.play();
    duration = sound2.duration();
    loop();
  } else if (check2 == HARD) {
    difficulty_num = 4;
    difficulty_speed = 2.8;
    sound3.play();
    duration = sound3.duration();
    loop();
    print(check2);
  }
}


class tiles {
  PVector location;
  Boolean gone=false;
  color tile_color;

  tiles(int i, int j) {
    color[] colors = {#19ad05, #9504de, #fc8403};
    location = new PVector((i*250) + 160, 0);
    tile_color = colors[j];
  }


  void run() {
    display();
    move(difficulty_speed);
  }

  void display() { //#19ad05, #9504de, #edd602
    fill(location.x>=0 && location.x<200 ?#de0404: location.x>200 && location.x<=410 ?#0416de:
      location.x>410 && location.x<=660 ?#edd602: location.x>660 && location.x<=910?tile_color: location.x>900 && location.x<1200?#fc7600:#fafafa);
    rect(location.x, location.y, 100, 50, 40);
  }

  void move(float speed) {
    location.y+=speed;
    //note speed, changing this will up the difficulity, putting it too high will make
  }                                  //it literally impossible
}


static boolean askingQuestion = false, answer = false;
static int questionX=gameWidth/2-150, questionY=gameHeight/2-40, questionWidth=370, questionHeight=100;
final static int NO_ANSWER = 0;
final static int YES = 1;
final static int NO = 2;
final static int EASY = 3;
final static int MEDIUM = 4;
final static int HARD = 5;

int checkAnswer2() { //Check to see if user clicked which difficulty
  if (mouseX >= questionX && mouseX <= questionWidth + 650 && mouseY >= questionY + 50 && mouseY <= questionY+questionHeight && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return EASY;
  }
  if (mouseX >= questionX && mouseX <= questionWidth + 650 && mouseY >= questionY + 100 && mouseY <= questionY + 140 && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return MEDIUM;
  }
  if (mouseX >= questionX && mouseX <= questionWidth + 650 && mouseY >= questionY + 150 && mouseY <= questionY + 190 && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return HARD;
  }

  return NO_ANSWER;
}

int checkAnswer1() { //Check to see if user clicked yes or no
  if (mouseX >= questionX + 600 && mouseX <= questionX + 783 && mouseY >= questionY + 150 && mouseY <= questionY + 190 && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return YES;
  }
  if (mouseX >= questionX + 783 && mouseX <= questionX + 950 && mouseY >= questionY + 150 && mouseY <= questionY + 190 && askingQuestion == true) {
    loop();
    askingQuestion = false;
    answer = true;
    return NO;
  }

  return NO_ANSWER;
}


void drawScore() {
  stroke(225);
  fill(50);
  //rect(questionX - 2, questionY - 2, questionWidth + 10, questionHeight + 4);
  rect(questionX + 600, questionY, questionWidth, questionHeight - 50);
  rect(questionX + 600, questionY + 50, questionWidth, questionHeight - 50);
  rect(questionX + 600, questionY + 100, questionWidth, questionHeight - 50);
  rect(questionX + 600, questionY + 150, questionWidth, questionHeight - 50);
  rect(questionX + 783, questionY + 150, questionWidth - 183, questionHeight - 50);
  stroke(225);
  fill(225);
  textSize(30);
  text(questionMessage1, questionX + 790, questionY + questionHeight - 66);

  text("Score: " + score, questionX + 685, questionY + questionHeight - 18);
  text("Misses: " + misses, questionX + 682, questionY + questionHeight + 34);
  text("Yes", questionX + 684, questionY + questionHeight + 83);
  text("No", questionX + 872, questionY + questionHeight + 83);
  noLoop();
}

void drawDifficulty() {
  stroke(225);
  fill(50);
  //rect(questionX - 2, questionY - 2, questionWidth + 10, questionHeight + 4);
  rect(questionX, questionY, questionWidth + 160, questionHeight - 50);
  rect(questionX, questionY + 50, questionWidth + 160, questionHeight - 50);
  rect(questionX, questionY + 100, questionWidth + 160, questionHeight - 50);
  rect(questionX, questionY + 150, questionWidth + 160, questionHeight - 50);
  stroke(225);
  fill(225);
  textSize(30);
  text(questionMessage2, questionX + 250, questionY + questionHeight - 66);
  textSize(20);
  text("Easy: It's Beginning To Look A Lot Like Christmas - Perry Como", questionX + 267, questionY + questionHeight - 18);
  text("Medium: Christmas Tree Farm - Taylor Swift", questionX + 190, questionY + questionHeight + 34);
  text("Hard: Santa Tell Me - Ariana Grande", questionX + 158, questionY + questionHeight + 81);
  noLoop();
}

void endGame() {
  noLoop();
  exit();
}

 Arduino Code:

#include "SerialRecord.h"

SerialRecord writer(3);

void setup() {
  Serial.begin(9600);
}

void loop() {
  
  int value1 = analogRead(0);
  int value2 = analogRead(2);
  int value3 = analogRead(4);
  writer.send();
  //if (value1 > 0) {
  writer[0] = value1;
  writer[1] = value2;
  writer[2] = value3;
    //writer.send(0);
      //
  //}
  //if (value2 > 0) {
  
  
      //
  //}
  
  // This delay slows down the loop, so that it runs less frequently. This can
  // make it easier to debug the sketch, because new values are printed at a
  // slower rate.
  delay(20);
  
}

Arduino

As for the Arduino physical controller, I chose this variable resistor (it may not be called that, I’m not sure about the name). I finished the wiring part quickly since it was pretty easy. All I needed to do is connect the resistor, one top to the pin, and the other two to GRD and 5V. Here’s a picture of the wiring connection.

Then I was going to write the codes of Arduino. I opened the SerialRecord example library and opened the SendSingleValue example. And also I opened another example in Arduino. Then I combine the two example code together, and it was like the below:

But there was no need to have the LED part, which was included in the example code, so I just deleted them. And at first, the Arduino part was done.

Processing

In the Processing part, I chose to use the image, so I downloaded a landscape of Santorini image on google. Then I referred to slide 22 and learned how to play the image on the sketch of Processing. At first, I had a problem that my image was not in this Processing document, but I found out that I did not add the “jpg” suffix to the image name when I entered it. Once I added it, the image was displayed without any problems. 

But I found something wrong with the size of this image, so I edited the size in the code and the image was displayed in full.

Then I opened the SerialRecord example of Processing, which was ReceiveSingleValue. And I combined this and the displaying image one. When I ran the code, though the image could zoom in and zoom out according to the variable resistor on the breadboard, there were several lines on the screen.

So I added a black background to the sketch and at the end the code of Processing  was like the below:

So finally, when Arduino and Processing’s code run together, the size of the image in Processing’s sketch will change according to the movement of the variable resistor connected to Arduino. The final result video looks like this.

Reference

Picture: https://www.klook.com/en-HK/city/405-Santorini-things-to-do/

In this recitation class, we needed to combine laser cut and a motor to make a rotatable object. Lesley and I were partners for this class! 

Step 1: Design

As for the design, we used cuttle.xyz, which is a website introduced in class before. While doing the stand design, I was in charge of the right side one. It was not hard, but we should carefully follow the instruction to pay attention to the units, data, location. 

As for the pattern design, I watched the instructional video and then designed a figure inside the circle of the specified size and then duplicated it several times by building groups, and this figure I designed for the outside part was finished.

Then, I exported 2 of my files and went to 823 fabrication lab to laser cut.

Step 2: Laser cut

We were the first ones to go to laser cut and we found that the professors had already cut the stands in advance on wooden boards. All we needed to cut was the pattern of our own design for the spin.

The rectangle in my design was inside my pattern and didn’t intersect with my pattern, so printing down my pattern was going to break. So Andy made some adjustments to it. Finally it was successfully cut out.

Step 3: Assemble 

There was nothing wrong with the process of assembling our stand, and there was nothing wrong with the wiring on the Arduino as well. However, we found that the servo and the acrylic board we cut were a bit unassembled, so we taped them to make them fit together, although not very firmly.

Both Lesley’s and my computer had a problem with the Processing version, and we couldn’t find the example in the library, so we borrowed Jason’s computer to run the code, and we found that ours ran smoothly.

Reflection

I found this class very interesting and I really liked the use of cuttle.xyz which is new to us, this graphic design is very attractive. We will also use laser cut in our final, so I have been watching the steps carefully. One of the shortcomings of our design is that Lesley and I didn’t pay much attention to which cut would be in front and which would be behind, so his design was smaller than mine but behind my board, which was not well thought out.

Project title: Two-player drumming rhythm game

Project Statement of Purpose:

When we were thinking about this project, we considered that college students are generally under high stress, so we wanted to design a game that could relieve people’s stress in a fun way. So we came up with the idea of a rhythm game, but we felt that a one-player game was too basic, and we wanted to have two players collaborate on this game, so we wanted to design a rhythm game that two players would work together to play. We think that with this kind of teamwork players can feel more fun and engaged.

Project Proposal Plan:

Generally, what we want to do is a rhythm game for two players. The main format we use is to display different color blocks on the screen according to a certain clip of music, and players will have different color drums in front of them, which color appears on the screen players need to hit the corresponding color drum. We also chose the drums because we felt that the drums would allow more physical interaction with the players, and the players moved their upper limbs instead of just their fingers. To make this more cooperative for two players, we decide to add some mixed colors, for example, if green appears on the screen, the players need to hit the yellow and red drums at the same time. We will also have three modes for players to choose from, easy medium and hard. These are the ways Sid and I came up with to make the game more engaging and more enjoyable for players, and also how to let them collaborate more.

We will also ask our friends and other classmates during the whole design process how we think we can improve the game to make it more interesting and more interactive for the two players.

Nov. 24th – Nov. 31st Building the programming part: finding similar code in open processing and making changes to it； Selecting a sensor: what sensor to connect to the drum to better deliver information accurately and without delay to the Arduino; we need to experiment and select.

December 1st – December 4th  Complete the programming design part, drum-making part (using laser cut), and wiring construction part. Basically, complete the project.

December 5th Check the whole project for problems and make minor changes

Context and Significance:

In my previous research, I came across a project called “UW-Makeathon: Laser Drums”, a project that uses laser cut technology to make drums, but its main theme is audio-visual integration, so it composes drums and fast LEDs. I got the inspiration from this project that I could use laser cutting to make drums and design a game with rhythm and music.

I think my vision of this project is quite in line with the definition of interaction, in my view it’s a fully interactive game. The color blocks on the screen mobilize the user to beat the corresponding drum, and the player’s response translates into a score on the screen that determines whether they can move to the next level of the game. The user is highly motivated and engaged, and it is not a one-sided input and output from either the project or the player.

Our game is unique in that it is not a single-player ordinary rhythm game, which already existed. It is a rhythm game where two players need to cooperate fully; two players control three drums and they need to decide for themselves how to distribute the tasks so that their motivation will be very actively mobilized.

I think the value of our project is that it is very simple; it is very easy to get started, and it brings remarkable results, that is, two players will definitely have a lot of communication when playing this game, will definitely cooperate, and will definitely be very natural and relaxed work in a team. This allows people to increase their communication with others in the game and also achieve the purpose of decompression and relaxation.

Task #1:  Make a Processing Etch-A-Sketch

For the first individual task, we needed to draw an Etch-A-Sketch, by using  SerialRecord to connect Processing and Arduino as well. In this task, Processing needs to send the value, and Processing is the receiver of the value.

I first built the circuit to send two analog values from Arduino to Processing via serial communication using potentiometers. This step isn’t difficult or unusual, so I moved on quickly. Then I opened the “SendMultiple value” example and created a sketch to read the two serial values. I uploaded the code to the Arduino board and verified that the code and potentiometers were functional. The results demonstrated that both are operational; as I twisted each potentiometer, the serial values displayed on the screen changed.

Then I started to modify the “RecieveMultipleValue” example on Processing with the help of Kaylee, the learning assistant, and the whole process went smoothly too. 

This task’s interaction occurred when I twisted the potentiometers to change the position of the circle on the screen. After I twisted the potentiometer, the serial values on Arduino were sent to Processing, which then processed the values and changed the position of the circle via the x and y axes.

Here’s the documentation video:

The Arduino Code:

import processing.serial.*;
import osteele.processing.SerialRecord.*;

Serial serialPort;
SerialRecord serialRecord;

void setup() {
  fullScreen();
   

  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);

  // If the Arduino sketch sends a different number of values, modify the number
  // `2` on the next line to match the number of values that it sends.
  serialRecord = new SerialRecord(this, serialPort, 2);
}

void draw() {
  background(0);

  serialRecord.read();
  int value1 = serialRecord.values[0];
  int value2 = serialRecord.values[1];
 

  float x = map(value1, 0, 1024, 0, width);
  float y = map(value2, 0, 1024, 0, height);
  fill(255);
  circle(x, y, 60);
}

And how to modify the code to turn it into lines, at first I thought it was very simple. I drew a line on Processing and changed its position to (x, y, x, y), but I soon realized how can the beginning of the line and the end of the line be the same？ So with Kaylee’s help, I added prevaluex and prevaluey. 

The Processing Code：

import processing.serial.*;
import osteele.processing.SerialRecord.*;

Serial serialPort;
SerialRecord serialRecord;
float preValuex;
float preValuey;

void setup() {
  fullScreen();
  preValuex = 0;
  preValuey = 0;
  background(0);

  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);

  // If the Arduino sketch sends a different number of values, modify the number
  // `2` on the next line to match the number of values that it sends.
  serialRecord = new SerialRecord(this, serialPort, 2);
}

void draw() {
  

  serialRecord.read();
  int value1 = serialRecord.values[0];
  int value2 = serialRecord.values[1];
 
  float x = map(value1, 0, 1024, 0, width);
  float y = map(value2, 0, 1024, 0, height);
  stroke(255);
  strokeWeight(20);
 line(preValuex, preValuey, x, y);
 
 preValuex= x;
 preValuey= y;

}

Task #2:

Ragnor was my partner, and we started by figuring out the procedures for task 2. Initially, we planned to create the Arduino and Processing code separately and then combine them at the end. Shake wrote the Arduino code, and I wrote the Processing code. My goal for the Processing section was to create a circle that moved continuously from the left to the right side of the screen and then from right to left. And the circle had to send a signal to the Arduino board every time it reached the side.

Ragnor wrote the Arduino part, but when Processing came with our Arduino, it just didn’t work, I didn’t know what was wrong. Ragnor and I worked on the 826 for a long time on Friday afternoon, but didn’t find the problem, Ragnor told me it was the Arduino’s Code that was wrong and Steve came to help us. Finally we found that the frequency of send value was too fast to cause the servo on the receive side to not move, we adjusted this and the whole program finally ran smoothly.

But we didn’t add anything to the servo, so it looks less complete overall.

The Processing Code：

import processing.serial.*;
import osteele.processing.SerialRecord.*;

float x = 0;
float y;
int speed = 10;

Serial serialPort;
SerialRecord serialRecord;

void setup() {
fullScreen();
y = height/2;
serialPort = new Serial(this, "COM8", 9600);
serialRecord = new SerialRecord(this, serialPort, 1);
}

void draw() {
background(0);
circle(x, y, 20);
x = x + speed;
if(x > width){
speed = -10;
serialRecord.values[0] = 1;
serialRecord.send();
}
if (x < 0){
speed = 10;
serialRecord.values[0] = 0;
serialRecord.send();
}
delay(50);
}

The Arduino Code：

#include "SerialRecord.h"
#include <Servo.h>

Servo myservo1;
Servo myservo2;  

SerialRecord reader(1);

void setup() {
 myservo1.attach(8);
 myservo2.attach(9);
 Serial.begin(9600);
}

void loop() {
  reader.read();
  Serial.println(reader[0]);
 if(reader[0] == 0){
  myservo1.write(90);
  delay(500);
  myservo1.write(0);
 }
 if(reader[0] == 1){
   myservo2.write(90);
   delay(300);
   myservo1.write(0);
 }
}

Reflection

I think the 2nd task is a bit difficult for us, we needed to complete the process of processing send value and accepting value from Arduino independently, and also connect it with the servo. This independent process also made me understand more about coding, but if we had more time, I would like to add something to the servo to make this little project more complete and interactive.

First: Dancing with the rhythm

First I’m trying to come up with something to help University students release their pressure. Because I feel that the pressure on today’s college students is too heavy and something should be done to help them release their academic pressure. And the first thing that came to my mind was some simple and passionate way of exercise, so what exercise can be made into a project by us, I continued to think. Then I thought of the dancing machine, where people would step back and forth according to the instructions on the screen, and then score points based on how correctly they danced. So my first idea was a project somewhat similar to the dance machine, we wanted to wear sensors on the user’s hands and feet, and then follow the rhythm of the music we played, some images would appear on the screen, and people’s hands and feet would need to move according to the images and dance to the music. Depending on the performance of different users, users will also receive different rating feedback. I think dancing to the rhythm of the music is a very suitable way for college students to relieve stress, and it’s also a very engaging game.

Second: Punch to release pressure

My second project idea was also aimed at reducing student stress, so I naturally came up with the idea of hitting something, like a boxing sandbag, etc. So I envisioned putting the pressure sensor in some soft material that can be punched, people can use our project to vent their emotions, and whenever they punch our project, some corresponding text and images will appear on the screen to further reduce the user’s pressure. These words can be encouraging words like “take your time, things will always get better”; they can also be some witty and humorous words. It doesn’t really matter what form they take, the main thing is that something will appear on the screen after people hit our project to make them feel better and less stressed.

Third: “Color-Drum-Rhythm” Game

My next project is still focused on the student community, I want to make a game between students, it can be a game of cooperation or competition between two people. Because I want people to fully engage, I plan to make a two-player cooperative game. And when designing the form of this game, I thought of the research I did earlier, the project about laser cutting drums. So I think this game can also be combined with music, it will be made into a rhythm game, the screen shows some color blocks, and the player needs to hit the corresponding drum according to the different colors. There may be several drums of different colors. I haven’t quite figured out exactly how to implement it, because it’s pretty difficult to make the drums and install the sensors, but the direction and format I think is a good idea: a two-player cooperative music rhythm game.

New thoughts about “interaction”:

What contributes to a successful interactive experience is a very insightful question. And at first, when we were doing the group project, we just thought of interaction as people and the machine having mutual input and output and that’s enough. So we made a robot that can provide different help according to the petrified state of the person, it was relatively intelligent, but the input provided by the people was actually not enough. And we had a kind of similar problem when we were doing our midterm. So the users just wore the heartbeat sensor on their ears and pressed the button, and that was basically all for what the users could do. Then we would show the visualized image of their heartbeat frequency and let them know how fast their heartbeat was. Though the feedbacks from the users were generally good, we still think that the interaction between users and our project was kind of lacking. The above is my further understanding of interactivity from my group project and midterm.

So now I think “interaction” should be more mutual and users need to do more input, and in turn, they will get a deeper experience. People will have a better experience in this project after the interaction is more in-depth, In the book, Ernest Edmonds says that “an interaction, be it with another human, an artwork, or a game, can influence internal states.” So I now think that interaction should also have a deeper impact on the user, rather than just staying in a momentary sensory experience.

The first interactive project I chose is “Gesture Controlled Interactive Table Light.” This project mainly consists of a sensor and FastLED, and the user’s gestures will be transmitted to this sensor, and then FastLED will change the color of the light. I think its interactive part is successful, it fully changes the light color according to the user’s input, which is their gesture. And the color of the lights will have a mental impact on the user while the user is using this table light. So the combination of user input and the output of this project is well worth learning from.

The second interactive project I chose is “UW-Makeathon: Laser Drums.“The main body of the project is a number of drums made of laser-cut wooden boards that emit different types of sounds when struck, and are visualized on the neopixel. I think this combination of music and visuals and drums is very good. And I might work with music and instruments as well, and also apply some visual stuff to make the whole thing like a game.

In this recitation class, we needed to use light strip in combination with Processing and Arduino, We were asked to connect the sound to the light strip and the light strip to the computer.

Task one：Test the NeoPixel

First, we were asked to connect the computer to the light strip and make sure the program was running properly. I built the circuit by using three jumper cables to connect the NeoPixel strips to the Arduino.

I downloaded the FastLed library and tested several examples, for example, I tested blink. The whole thing was running very smoothly. I also modified the code to make it flash a different color every other light

Task two: Use your computer to light up NeoPixels

I downloaded the Serial Record library before, so I started the second step directly. I tested my code with Serial Monitor and the whole process went well and smoothly. I had no problem with my Arduino, I programmed my Arduino with the code we had in class, which I simply copied and pasted from the link provided on the Recitation page. Then, I entered the serial values (CSV) of 1,255,0,100 into my Serial Monitor. The first value represents the pixel number, the second the content of red, then green, and finally blue. The outcome indicated that my Serial Monitor was operational.

But when I got to the third step of Processing part, my computer couldn’t run it for some reason. In the end, with the help of our professor, the students who were using Macbook air downloaded another version of the file and solved the problem.

Then we managed to implement the interaction between Precessing and Arduino at Fast Led. Although it’s not something I programmed myself, I understood the logic in general, and I thought it was pretty cool.

Here’s a documentation video:

Task three: Add Music!

This part was quite interesting that we downloaded a music clip from the internet and tried to visualize it. So I chose the music called “The Glimpse of Us” as the one going to be played and dragged the downloaded file into the Processing file. I replaced beat.aiff with joji.mp3. Then I also connected it to the Arduino’s FastLed, which means that when the song was playing, in addition to the screen displaying the video based on volume and rhythm, the FastLed’s light strip would also be displayed accordingly. I also modified the color of the screen visualization video according to the rhythm and style of the whole song.

But I also found some problems and did not find useful solutions, which was that the LED response to the music was relatively lagging. And when the amplitude of the music changes, for example, when the amplitude changes from high level to low level, the LED does not change. But overall I was still satisfied with the whole work. I feel that this class gave me a rough idea of the logic between Arduino and Processing.

But my video doesn’t seem to have recorded the part with the LED strip, but you can see from my code that I did include it.

Final Code：

import processing.serial.*;
import osteele.processing.SerialRecord.*;
import processing.sound.*;

Serial serialPort;
SerialRecord serialRecord;

int W;         //width of the tiles
int NUM = 60;  //amount of pixels
int[] r = new int[NUM]; //red of each tile
int[] g = new int[NUM]; //red of each tile
int[] b = new int[NUM]; //red of each tile


SoundFile sample;
Amplitude analysis;

void setup() {
  size(600, 200);
  W = width/NUM;


  // You can use this syntax and change COM3 for your serial port
  // printArray(Serial.list());
  // serialPort = new Serial(this, "COM3", 9600);
  // in MacOS it looks like "/dev/cu.usbmodem1101"
  //or you can try to use this instead:

  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);
  serialRecord = new SerialRecord(this, serialPort, 4);
  serialRecord.logToCanvas(false);
  rectMode(CENTER);

  // load and play a sound file in a loop
  sample = new SoundFile(this, "joji.mp3");
  sample.loop();

  // create the Amplitude analysis object
  analysis = new Amplitude(this);
  // analyze the playing sound file
  analysis.input(sample);
}

void draw() {
  println(analysis.analyze());
  background(75, 120, 237);
  noStroke();
  fill(125, 212, 77);

  // analyze the audio for its volume level
  float volume = analysis.analyze();

  // volume is a number between 0.0 and 1.0
  // map the volume value to a useful scale
  float diameter = map(volume, 0, 1, 0, width);
  // draw a circle based on the microphone amplitude (volume)
  circle(width/2, height/2, diameter);

int n = floor(map(volume, 0, 1, 0, 60));
  r[n] = floor(map(volume, 0, 1, 0, 255));
  g[n] = floor(random(map(volume, 0, 1, 0, 255)));
  b[n] = floor(random(map(volume, 0, 1, 0, 255)));

  serialRecord.values[0] = n;     // which pixel we change (0-59)
  serialRecord.values[1] = r[n];  // how much red (0-255)
  serialRecord.values[2] = g[n];  // how much green (0-255)
  serialRecord.values[3] = b[n];  // how much blue (0-255)
  serialRecord.send();            // send it!
} 

At the beginning of this poster, I made three dots, two moving from the left to the right of the screen and one moving from the right to the left. There was also a square moving on the screen, which would flash on the screen. Finally, I wrote the time and place of the IMA show on a white background. What bothers me about this poster is that if the dots move at normal speed, the square will be too fast to be controlled. So now the dot is moving very slowly in order to make the square move normally.

int x=0;
int y=808;
float z=0;
float o=0;
float p=40;
float q=60;

void setup(){
  size(1024,768);
}

void draw(){
  background(144,212,255);
   C1(x);
     frameRate(200);
   x++;
   if (x==width+40){
     x=-40;
   }
   
   C2(y);
     frameRate(200);
   y=y-1;
   if (y==width-808){
     y=808;
   }
   
    C3(x);
      frameRate(200);
   //x++;
   if (x==width+40){
     x=-40;
   }
   
   R1(z,o,p,q);
   frameRate(20);
   z=z+10;
   o=o+10;
   p=p+20;
   q=q+20;
}

void C1(int a){
  //鹅黄色
  fill(254,237,146);
  noStroke();
  ellipse (a,200,80,80);
}

void C2(int b){
   fill(254,237,146);
   noStroke();
   ellipse (b,400,80,80);
}

void C3(int a){
  //鹅黄色
  fill(254,237,146);
  noStroke();
  ellipse (a,600,80,80);
}

void R1(float c,float d,float e,float f){
  int r=int(random(255));
  int g=int(random(255));
  int b=int(random(255));
  fill(r,g,b);
  noStroke();
  rect(c,d,e,f);
  
    noStroke();
  fill(255);
  rect(206, 335, 800, 300);
  fill(0);
  textSize(50);
  text("IMA Fall 22 Show!", 240, 420);
  fill(0);
  textSize(25);
  text("Friday, December 16 | 6pm to 8pm", 240, 480);
  text("Location: 8th Floor", 240, 540);
  fill(190);
   circle(225, 380, 20);
   circle(225, 580, 20);
  
  
}

Homework

I really didn’t think the poster I made in class looked good, so I studied the homework outside of class. I drew a figure consisting of three circles, generated 100 of them, and followed the generated positions. I learned the use of “push” and “pop” in the previous lesson, and I used them. I also used keyPressed function to control them according to the instruction. When the “o” is pressed, the whole pattern will be laid out in a different way.

At first, the color choice was a bit weird, but then I edited it a bit. I defined three RGB variables and used the map function to map random numbers ranging from 0 to 255 into a specific range, resulting in visually harmonious colors.

First version:

Final version:

float r, g, b;
void setup() {
  size(1024, 768);
  background(255);
  for(int x= 0; x<10; x++){
  for(int y=0; y<10; y++) {
      r = map(random(255), 0, 255, 200 ,255);
      g = map(random(255), 0, 255, 180 ,240);
      b = map(random(255), 0, 255, 175 ,235);
 
   push();
   translate( random(0, width),random(0, height));
   //translate(y * 100,0);
   noStroke();
  fill(r,g,b);
  ellipse(60, 40, 60, 60);
  fill(253,255,156);
  ellipse(50,54, 52, 52);
  fill(r,g,b);
  ellipse(50, 55, 20, 20);
  fill(r,g,b);
  pop();
    }  
  }
}

void changePos(){
  background(255);
  for(int x= 0; x<10; x++){
    for(int y=0; y<10; y++) {
       r = map(random(255), 0, 255, 200 ,255);
      g = map(random(255), 0, 255, 180 ,240);
      b = map(random(255), 0, 255, 175 ,235);
 
 
   push();
   translate( random(0, width),random(0, height));
   //translate(y * 100,0);
   noStroke();
  fill(r,g,b);
  ellipse(60, 40, 60, 60);
  fill(253,255,156);
  ellipse(50,54, 52, 52);
  fill(r,g,b);
  ellipse(50, 55, 20, 20);
  fill(r,g,b);
  pop();
    }  
  }

}

void changeColor(){
  

}
void keyPressed() {
  if (key == 'o') {
    changePos();
  }
 
}

void draw(){

}

This is my Wechat profile, and I think it’s quite cute, so I chose to use Processing draw it. Since I’m not that good at drawing, here my sketch is not that pretty.

I drew the dog in a kind of symmetrical way because I thought it would be easier to draw it on Processing. And due to my absence from the previous class, when Rudi introduced the basics of Processing, it was even harder for me to apply Processing to draw pictures and this kind of work.

But I browsed Processing’s website and learned how to draw some basic shapes, such as ellipses, rectangles, and lines. Kaylee, my learning assistant, and Sid, my friend, also taught me how to fill in the colors and how to find the parameters of the desired color.

So I started the first try of coding with Processing. I started the drawing with a big pink ellipse. But I just needed a small part of it to be the pink hat of the puppy. And then I draw a relatively smaller ellipse inside the big one in lighter pink, to be the face. 

Then I’m going to start drawing the eyes. The dog’s eyes are tilted in the sketch, so I want to draw a tilted eye too. And this means I needed to rotate the ellipses as the eyes on the puppy’s face to some degree. But I really had no idea how to do it, so I turned to my professor Rudi for help.

Rudi told me to use “rotate” to do this, and applied “push” and “pop”. The code was like this:

After knowing how to rotate any angle, I easily drew the eyes, with ellipses of different sizes. So the puppy was almost done, just a nose and a tongue were needed. And I quickly coded a nose. Then I saw the prompt, it said if you want, you can make some moving animation.

And I thought if the color of the puppy’s hat could change, that would be very cool. So I asked my friend, the learning assistant in our class–Kaylee, about how to make the animation move.

She asked me whether I wanted to make the color change automatically or when I click the mouse, it changes once. I chose the mouse clicking changing mode. 

So she introduced the “mousePressed” function to me, and told me to make the color value of the hat some variables. Then after the mousePressed instruction, I could easily change the color of the hat. I got her point and coded another blue hat. After clicking the mouse, the hat would change into blue.

It’s very cool.

And after the recitation class, I added a tongue and two ears to the puppy. It then looked very cute, at least to me!

Here’s the code: