Category: Interaction Lab

A. Project Title: Rhythm Master
Group members: Siwei Fang (Wendy), Siwei Chen
Instructor: Gottfried Haider

B. CONCEPTION AND DESIGN:
In our previous research, we were inspired by the music games “Osu!” and “Dance revolution”, in which players perform actions with the rhythm of the music and earn points according to the accuracy and correctness of their play. This is a kind of game with a strong sense of participation. So we decide to design a rhythm game (it’s really interactive! ) In the game, the player presses the button according to the rhythm and the prompt on the screen to interact with the artifact. In order to increase the fun and engagement, we also designed an additional session where the use of a whistle at a specified rhythm point can change the effect of the lights (unfortunately, because of the epidemic this session could not be realized by the game).

(Our first proposal:)

In the user testing, the game was mostly recognized by the players, they thought the experience was great, the lighting was cool, and the difficulty was challenging. The feedbacks and suggestions we received include: adding a tutorial section, changing the lighting effect (such as covering a white paper on top of Neopixel), adding different songs, adjusting the speed of falling stars, etc. Based on feedback, we made adjustments accordingly. I made the project into three parts, the tutorial part, the normal part, and the harder part. We also pasted white paper on all the LEDs to make the overall look more beautiful. In the final presentation, the player experience and feedbacks were better.

C. FABRICATION AND PRODUCTION:
The most significant part of the project is the rhythm, judgment area in processing, how and when to send and receive value with Arduino, and the coding of the Neopixel part on Arduino.

The most important part of the music game is the rhythm, I originally intended to calculate the BPM of each song and then write a for loop to achieve automatic rhythm recording, so that the beat points can be very accurate. But the beat points are exactly the same, which may be too simple and dull. So we directly input the rhythm manually (use “println” to record the beat), which may lead to some errors in the rhythm, but can make beat points variable.

The screen part is the same as our previous idea, there is a red heart in the middle of the screen protected by a white circle, and four stars will appear from the four corners, when the stars touch the white line a moment, the player should press the button to protect the red heart.

Then came the judgment part, because when playing music in Processing, it will have a delay, and there is no convenient way to measure the delay. So I used the most direct method – listening to the music and looking at the beat point (when the star hit the white line) to manually adjust the delay. In order to improve the game experience, I wrote the judgment interval as one second, that is, if the player presses the button within one second before or after the beat point, it will be judged as a success, if pressed outside of one second, it will be judged as a failure. I first completed the Processing part, using the keyboard instead of buttons, and this part can already be played as a tiny rhythm game.

Then came the harder part: how to connect Processing with Arduino? Because the star has four directions, so correspondingly, we used four buttons. First, before each star hits the white line, we have to send a signal to the Arduino to make the corresponding button glow as a reminder to the player; second, if a button in the same direction as the star is pressed, the Arduino has to send a message to Processing, and this message will be put into the judgment interval to determine whether the player scores. I defined a lot of arrays to complete this part and finally succeed.

Then it occurred to us that if we let players who are not familiar with the operation and have never played a game before, it is difficult to play the game by looking at the computer screen and not at the buttons. So we wanted to make the LED board in the middle of the button synchronized with the display of the stars on the computer screen (only the direction of movement is opposite because the LED moves from the center to the button). This step is the most painful because I have not touched the Neopixel 8*8 board before at all. We first measured the LED numbering order, and then add some basic changes, such as blink, and fade. But I’m not clear on how to make the LED and the star make a synchronous movement. With the help of the professor and my friend, we listed 20 cases to divide the situation and used “case” and “break” to control the movement of the LEDs (this step took a long time). After many attempts, we basically finished the Arduino part of the programming. But we still have some problems. The first problem is in the Processing part, we use a red cross as a reminder to the player that he does not press accurately in the judgment interval, the red cross is completely controlled by buttons but it sometimes appears when we even haven’t connected the button. We have no idea but to delete the cross. Another problem is the Neopixel board, our code is built on the basis that the next LED will glow only after the previous one finishes glowing and goes off, sometimes the LED will somehow get stuck in one place and not move, causing the whole LED board to go wrong. 

After testing countless times (very painful), I came up with the final solution: reduce the beat points (not too many beat points in the Processing, otherwise the Arduino part will have problems); limit the song length. We also wanted to let the board make other instructions in the whistle part (such as changing the LED color), but we couldn’t do it due to coding problems. However, the final work is nice (to me).

The circuit:

The codes: (Arduino)

#include <FastLED.h>
  #include "SerialRecord.h"

  #define NUM_LEDS 64 // How many leds in your strip?
  #define DATA_PIN 10
  int state[83];
CRGB leds[NUM_LEDS]; // Define the array of leds
int prems[83];
int direction[83];
int num = 0;
int start;
int pre_state1 = LOW;
int pre_state2 = LOW;
int pre_state3 = LOW;
int pre_state4 = LOW;
int buttonState1 = 0;
int buttonState2 = 0;
int buttonState3 = 0;
int buttonState4 = 0;

SerialRecord reader(1);
SerialRecord writer(1);

//button a-d:led  button1-4:switch
int buttona = 2;
int buttonb = 3;
int buttonc = 4;
int buttond = 5;
int button1 = 6;
int button2 = 7;
int button3 = 8;
int button4 = 9;


void setup() {
  Serial.begin(115200);
  pinMode(button1, INPUT_PULLUP); //switch
  pinMode(button2, INPUT_PULLUP);
  pinMode(button3, INPUT_PULLUP);
  pinMode(button4, INPUT_PULLUP);
  pinMode(buttona, OUTPUT); //led
  pinMode(buttonb, OUTPUT);
  pinMode(buttonc, OUTPUT);
  pinMode(buttond, OUTPUT);
  FastLED.addLeds<NEOPIXEL, DATA_PIN>(leds, NUM_LEDS);  // GRB ordering is assumed

  for (int i = 0; i < 63; i++) {
    leds[i] = CRGB::Black;
  }
  FastLED.show();
  for (int i = 0; i < 83; i++) {
    prems[i] = -10000;
  }
}

void loop() {
  buttonState1 = digitalRead(button1);
  buttonState2 = digitalRead(button2);
  buttonState3 = digitalRead(button3);
  buttonState4 = digitalRead(button4);
  if (!buttonState1 == HIGH && buttonState1 != pre_state1) { //if button pressed, send value to processing
    writer[0] = 1;
    writer.send();
  }
  if (!buttonState2 == HIGH && buttonState2 != pre_state2) {
    writer[0] = 2;
    writer.send();
  }
  if (!buttonState3 == HIGH && buttonState3 != pre_state3) {
    writer[0] = 3;
    writer.send();
  }
  if (!buttonState4 == HIGH && buttonState4 != pre_state4) {
    writer[0] = 4;
    writer.send();
  }
  pre_state1 = buttonState1;
  pre_state2 = buttonState2;
  pre_state3 = buttonState3;
  pre_state4 = buttonState4;
  if (reader.read()) { // if receive value, LED light up accordingly
    if (reader[0] == 1) {
      digitalWrite(2, HIGH);
    }
    if (reader[0] == 2) {
      digitalWrite(3, HIGH);
    }
    if (reader[0] == 3) {
      digitalWrite(4, HIGH);
    }
    if (reader[0] == 4) {
      digitalWrite(5, HIGH);
    }
    if (reader[0] == 5) { // when the ball reach center, all LED put out
      digitalWrite(2, LOW);
      digitalWrite(3, LOW);
      digitalWrite(4, LOW);
      digitalWrite(5, LOW);
    }

    if (reader[0] != 5) {
      prems[num] = millis();
      Serial.println(num);
      //void PinReaderMap[] = {0, 0,1,3,2}
      if (reader[0] == 1) {
        direction[num] = 0;
      }
      if (reader[0] == 2) {
        direction[num] = 1;
      }
      if (reader[0] == 3) {
        direction[num] = 3;
      }
      if (reader[0] == 4) {
        direction[num] = 2;
      }

      num++;
    }
  }

  int now = millis();

  for (int i = 0; i < 83; i++) { // to make neopixel 8*8 work with the screen

    if (now - prems[i] >= 0 && now - prems[i] < 150 && state[i] == 0) {
      // Serial.println("?");
      switch (direction[i]) {
      case 0:
        {
          leds[35] = CRGB::Red;
          FastLED.show();
          break;
        }
      case 1:
        {
          leds[36] = CRGB::Red;
          FastLED.show();
          break;
        }
      case 2:
        {
          leds[27] = CRGB::Blue;
          FastLED.show();
          break;
        }
      case 3:
        {
          leds[28] = CRGB::Blue;
          FastLED.show();
          break;
        }
      }
      state[i]++;
    } else if (now - prems[i] >= 150 && now - prems[i] < 300 && state[i] == 1) {
      switch (direction[i]) {
      case 0:
        {
          leds[35] = CRGB::Black;
          leds[45] = CRGB::Red;
          FastLED.show();
          break;
        }
      case 1:
        {
          leds[36] = CRGB::Black;
          leds[42] = CRGB::Red;
          FastLED.show();
          break;
        }
      case 2:
        {
          leds[27] = CRGB::Black;
          leds[21] = CRGB::Blue;
          FastLED.show();
          break;
        }
      case 3:
        {
          leds[28] = CRGB::Black;
          leds[18] = CRGB::Blue;
          FastLED.show();
          break;
        }
      }
      state[i]++;
    } else if (now - prems[i] >= 300 && now - prems[i] < 450 && state[i] == 2) {
      switch (direction[i]) {
      case 0:
        {
          leds[45] = CRGB::Black;
          leds[49] = CRGB::Red;
          FastLED.show();
          break;
        }
      case 1:
        {
          leds[42] = CRGB::Black;
          leds[54] = CRGB::Red;
          FastLED.show();
          break;
        }
      case 2:
        {
          leds[21] = CRGB::Black;
          leds[9] = CRGB::Blue;
          FastLED.show();
          break;
        }
      case 3:
        {
          leds[18] = CRGB::Black;
          leds[14] = CRGB::Blue;
          FastLED.show();
          break;
        }
      }
      state[i]++;
    } else if (now - prems[i] >= 450 && now - prems[i] < 600 && state[i] == 3) {
      switch (direction[i]) {
      case 0:
        {
          leds[49] = CRGB::Black;
          leds[63] = CRGB::Red;
          FastLED.show();
          break;
        }
      case 1:
        {
          leds[54] = CRGB::Black;
          leds[56] = CRGB::Red;
          FastLED.show();
          break;
        }
      case 2:
        {
          leds[9] = CRGB::Black;
          leds[7] = CRGB::Blue;
          FastLED.show();
          break;
        }
      case 3:
        {
          leds[14] = CRGB::Black;
          leds[0] = CRGB::Blue;
          FastLED.show();
          break;
        }
      }
      state[i]++;
    } else if (now - prems[i] >= 600 && state[i] == 4) {
      switch (direction[i]) {
      case 0:
        {
          leds[63] = CRGB::Black;
          FastLED.show();
          break;
        }
      case 1:
        {
          leds[56] = CRGB::Black;
          FastLED.show();
          break;
        }
      case 2:
        {
          leds[7] = CRGB::Black;
          FastLED.show();
          break;
        }
      case 3:
        {
          leds[0] = CRGB::Black;
          FastLED.show();
          break;
        }
      }
      state[i]++;
    }
  }
}

(Processing)
Instruction:

import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;

Serial serialPort;
SerialRecord serialRecord;
SoundFile sound;

boolean[] flag=new boolean[18];
boolean[] flaga=new boolean[18];
boolean[] flagb=new boolean[18];
boolean[] flagc=new boolean[18];
boolean[] sendstate=new boolean[18];
int playStart;
boolean success=false;
int premil=0;
int value=0;
int num=0;
int[]beats= {
  752,
  863,
  979,
  1090,
  1486,
  1543,
  1598,
  1653,
  2337,
  2506,
  2677,
  3004,
  3059,
  3116,
  3172,
  3229,
  3284,
  3339,
};

int[] xpos=new int[18];
int[] ypos=new int[18];
int x;
int y;
int[] ball=new int[18];
int a=0;

void star(float x1, float y1, float radius1, float radius2, int npoints) { //ball=start
  float angle = TWO_PI / npoints;
  float halfAngle = angle/2.0;
  beginShape();
  for (float a = 0; a < TWO_PI; a += angle) {
    float sx = x1 + cos(a) * radius2;
    float sy = y1 + sin(a) * radius2;
    vertex(sx, sy);
    sx = x1 + cos(a+halfAngle) * radius1;
    sy = y1 + sin(a+halfAngle) * radius1;
    vertex(sx, sy);
  }
  endShape(CLOSE);
}

void keyPressed() {
 println(frameCount+",");
  //println(millis());
}

void drawBigHeart() { // if player press the button correctly, the heart becomes bigheart
  fill(255, 0, 0);
  rectMode(CENTER);
  rect(height/2, width/2, 10, 10);
  rect(height/2-10, width/2, 10, 10);
  rect(height/2+10, width/2, 10, 10);
  rect(height/2-20, width/2, 10, 10);
  rect(height/2+20, width/2, 10, 10);
  rect(height/2-30, width/2, 10, 10);
  rect(height/2+30, width/2, 10, 10);
  rect(height/2+10, width/2-10, 10, 10);
  rect(height/2-10, width/2-10, 10, 10);
  rect(height/2+20, width/2-10, 10, 10);
  rect(height/2-20, width/2-10, 10, 10);
  rect(height/2-15, width/2-20, 10, 10);
  rect(height/2+15, width/2-20, 10, 10);
  rect(height/2, width/2+10, 10, 10);
  rect(height/2-10, width/2+10, 10, 10);
  rect(height/2+10, width/2+10, 10, 10);
  rect(height/2-20, width/2+10, 10, 10);
  rect(height/2+20, width/2+10, 10, 10);
  rect(height/2, width/2+20, 10, 10);
  rect(height/2-10, width/2+20, 10, 10);
  rect(height/2+10, width/2+20, 10, 10);
  rect(height/2, width/2+30, 10, 10);
}

void drawHeart() { // normal heart
  fill(255, 0, 0);
  rect(height/2, width/2, 10, 10);
  rect(height/2-10, width/2, 10, 10);
  rect(height/2+10, width/2, 10, 10);
  rect(height/2-20, width/2, 10, 10);
  rect(height/2+20, width/2, 10, 10);
  rect(height/2+10, width/2-10, 10, 10);
  rect(height/2-10, width/2-10, 10, 10);
  rect(height/2, width/2+10, 10, 10);
  rect(height/2-10, width/2+10, 10, 10);
  rect(height/2+10, width/2+10, 10, 10);
  rect(height/2, width/2+20, 10, 10);
}

void setup() {
  size(800, 800);
  frameRate(60);
  delay(1000);
  sound = new SoundFile(this, "g.mp3");
  playStart = millis();
  sound.play();

  noStroke(); // the text
  textSize(30);
  textAlign(CENTER, CENTER);

  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 115200);
  serialRecord = new SerialRecord(this, serialPort, 1);

  for (int w=0; w<18; w++) {
    sendstate[w]=true;
  }
}

void key(int k) {   //key is controlled by buttons
  //println((frameCount - playStart)+",");
  for (int i=0; i<18; i++) { //judgement
    if (abs(frameCount-beats[i])<10&&ball[i]==k&&!flaga[i]) {
      success=true;  //if press button correctly, the heart become bigger
      //println("success=true");
      premil=millis();
      a=a+1;
      flaga[i]=true;
    }
  }
}

void draw() {
  background(0);
  fill(255, 100);
  ellipse(400, 400, 180, 180);
  fill(0);
  ellipse(400, 400, 160, 160);
  rectMode(CENTER);
  drawHeart();  //heart

  
  if (serialRecord.read()) {
    value = serialRecord.get();
    key(value);
    //println(frameCount);
  }

  if (success==true) {
    if (millis()-premil<300) {
      fill(255);
      ellipse(400, 400, 180, 180);
      fill(0);
      ellipse(400, 400, 160, 160);
      drawBigHeart();
    } else {
      success=false;
    }
  } else drawHeart();

int now = millis() - playStart;
  push();
  if (now > 2000 && now < 5000) {
    clear();
    textSize(96);
    fill(255, 255, 255);
    text("instruction", height/2, width/2);
  }
  pop();
  
  if (now > 5000 && now < 8783) {
    clear();
    fill(255, 255, 255);
    text("Pressed Button when the star hits White Line", height/2, width/2);
  }
  if (now > 20066 && now < 23850) {
    clear();
    fill(255, 255, 255);
    text("You can also look at the pixel LED in the middle of the box", height/2, width/2-32);
    text("LEDs and the stars are moving in sync", height/2, width/2);
    text("Press button when LED Reach the Edge", height/2, width/2+32);
  }
  if (now > 29400 && now < 34500) {
    clear();
    fill(255, 255, 255);
    text("If you press correctly,", height/2, width/2-32);
    text("the heart becomes bigger after each pressing", height/2, width/2);
    text("Your score is displayed in the top left corner", height/2, width/2+32);
  }
  if (now > 32500) {
    fill(255);
    text("Your score is:", 125, 65);
    text(a, 100, 100);
  }
  if (now >34500 && now < 37800) {
    clear();
    fill(255, 255, 255);
    text("If the star is", height/2, width/2-32);
    fill(#FFF534);
    text("YELLOW", height/2, width/2);
    fill(255);
    text("Use Whistle and Press Button at the same time! ", height/2, width/2+32);
  }

  push();
  if (now>46533&&now<48350) {
    clear();
    fill(255);
    textSize(96);
    text(" ENJOY PLAYING !", height/2, width/2);
  }
  pop();
  if(frameCount>3500){
    clear();
    fill(255);
    text("This is the end of the tutorial section",height/2, width/2);
  }
  
  //ball
  int cfc=frameCount;
  for (int i=max(0, num-2); i<18; i++) {
    if (cfc-beats[i]>-31&&!flag[i]) {
      fill(255);
      if (abs(cfc-(beats[i]-31))<2) {
        xpos[i]=800*floor(random(0, 2));
        ypos[i]=800*floor(random(0, 2));
        if (xpos[i]==0&&ypos[i]==0) {
          ball[i]=1;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 1;
            serialRecord.send();
            //println("1");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==800&&ypos[i]==0) {
          ball[i]=2;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 2;
            serialRecord.send();
            //println("2");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==0&&ypos[i]==800) {
          ball[i]=3;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 3;
            serialRecord.send();
            //println("3");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==800&&ypos[i]==800) {
          ball[i]=4;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 4;
            serialRecord.send();
            //println("4");
            sendstate[i]=false;
            num++;
          }
        }
      }

      if (xpos[i]<400 && ypos[i]<400) {
        xpos[i] = xpos[i] + 10;
        ypos[i]= ypos[i] + 10;
      }
      if (xpos[i]>400&&ypos[i]<400) {
        xpos[i] = xpos[i]-10;
        ypos[i]=ypos[i]+10;
      }
      if (xpos[i]>400&&ypos[i]>400) {
        xpos[i] = xpos[i]-10;
        ypos[i]=ypos[i]-10;
      }
      if (xpos[i]<400&&ypos[i]>400) {
        xpos[i] = xpos[i]+10;
        ypos[i]=ypos[i]-10;
      }
      noStroke();
      if (ypos[i]<=400) {
        fill(255, 0, 0);
      }
      if (ypos[i]>=400) {
        fill(#30CEFC);
      }
      if (2300<cfc&&cfc<2700) {
        fill(#FFF534);
        //serialRecord.values[0]=6;
       // serialRecord.send();
        //flagb[i]=true;
      }
      
      star(xpos[i], ypos[i], 40, 20, 5);
      if (xpos[i]==400&&ypos[i]==400) {
        flag[i]=true;
        serialRecord.values[0] = 5;
        serialRecord.send();
      }
    }
  }
  }

Normal:

import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;

Serial serialPort;
SerialRecord serialRecord;
SoundFile sound;

boolean[] flag=new boolean[83];
boolean[] flaga=new boolean[83];
boolean[] sendstate=new boolean[83];
int playStart;
boolean success=false;
int premil=0;
int value=0;
int num=0;
int[]beats= {
  199,
  265,
  330,
  397,
  463,
  530,
  599,
  667,
  731,
  801,
  869,
  935,
  1004,
  1071,
  1106,
  1141,
  1175,
  1210,
  1244,
  1277,
  1310,
  1344,
  1379,
  1414,
  1445,
  1479,
  1513,
  1548,
  1580,
  1615,
  1647,
  1680,
  1714,
  1750,
  1783,
  1816,
  1851,
  1884,
  1918,
  1951,
  2018,
  2052,
  2085,
  2119,
  2152,
  2187,
  2221,
  2254,
  2286,
  2319,
  2353,
  2388,
  2422,
  2456,
  2489,
  2523,
  2557,
  2591,
  2623,
  2657,
  2691,
  2724,
  2758,
  2791,
  2825,
  2859,
  2894,
  2927,
  2959,
  2977,
  2995,
  3013,
  3029,
  3046,
  3063,
  3080,
  3097,
  3113,
  3129,
  3146,
  3163,
  3181,
  3196,
};

int[] xpos=new int[83];
int[] ypos=new int[83];
int x;
int y;
int[] ball=new int[83];
int a=0;

void star(float x1, float y1, float radius1, float radius2, int npoints) { //ball=start
  float angle = TWO_PI / npoints;
  float halfAngle = angle/2.0;
  beginShape();
  for (float a = 0; a < TWO_PI; a += angle) {
    float sx = x1 + cos(a) * radius2;
    float sy = y1 + sin(a) * radius2;
    vertex(sx, sy);
    sx = x1 + cos(a+halfAngle) * radius1;
    sy = y1 + sin(a+halfAngle) * radius1;
    vertex(sx, sy);
  }
  endShape(CLOSE);
}

void keyPressed() {
  //println(frameCount+",");
  //println(millis());
}

void drawBigHeart() { // if player press the button correctly, the heart becomes bigheart
  fill(255, 0, 0);
  rectMode(CENTER);
  rect(height/2, width/2, 10, 10);
  rect(height/2-10, width/2, 10, 10);
  rect(height/2+10, width/2, 10, 10);
  rect(height/2-20, width/2, 10, 10);
  rect(height/2+20, width/2, 10, 10);
  rect(height/2-30, width/2, 10, 10);
  rect(height/2+30, width/2, 10, 10);
  rect(height/2+10, width/2-10, 10, 10);
  rect(height/2-10, width/2-10, 10, 10);
  rect(height/2+20, width/2-10, 10, 10);
  rect(height/2-20, width/2-10, 10, 10);
  rect(height/2-15, width/2-20, 10, 10);
  rect(height/2+15, width/2-20, 10, 10);
  rect(height/2, width/2+10, 10, 10);
  rect(height/2-10, width/2+10, 10, 10);
  rect(height/2+10, width/2+10, 10, 10);
  rect(height/2-20, width/2+10, 10, 10);
  rect(height/2+20, width/2+10, 10, 10);
  rect(height/2, width/2+20, 10, 10);
  rect(height/2-10, width/2+20, 10, 10);
  rect(height/2+10, width/2+20, 10, 10);
  rect(height/2, width/2+30, 10, 10);
}

void drawHeart() { // normal heart
  fill(255, 0, 0);
  rect(height/2, width/2, 10, 10);
  rect(height/2-10, width/2, 10, 10);
  rect(height/2+10, width/2, 10, 10);
  rect(height/2-20, width/2, 10, 10);
  rect(height/2+20, width/2, 10, 10);
  rect(height/2+10, width/2-10, 10, 10);
  rect(height/2-10, width/2-10, 10, 10);
  rect(height/2, width/2+10, 10, 10);
  rect(height/2-10, width/2+10, 10, 10);
  rect(height/2+10, width/2+10, 10, 10);
  rect(height/2, width/2+20, 10, 10);
}

void setup() {
  size(800, 800);
  frameRate(60);
  delay(1000);
  sound = new SoundFile(this, "t.mp3");
  playStart = millis();
  sound.play();

  noStroke(); // the text
  textSize(90);
  textAlign(CENTER, CENTER);

  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 115200);
  serialRecord = new SerialRecord(this, serialPort, 1);

  for (int w=0; w<83; w++) {
    sendstate[w]=true;
  }
}


void key(int k) {   //key is controlled by buttons
  //println((frameCount - playStart)+",");
  for (int i=0; i<83; i++) { //judgement
    if (abs(frameCount-beats[i])<10&&ball[i]==k&&!flaga[i]) {
      success=true;  //if press button correctly, the heart become bigger
      //println("success=true");
      premil=millis();
      a=a+1;
      flaga[i]=true;
    }
    /*if (success==false) {  //draw cross
     fill(255, 0, 0);
     push();
     translate(400, 400);
     rotate(PI/4);
     rect(0, 0, 400, 100);
     pop();
     push();
     translate(400, 400);
     rotate(-PI/4);
     rect(0, 0, 400, 100);
     pop();
     }*/
  }
}

void draw() {
  background(0);
  fill(255, 100);
  ellipse(400, 400, 180, 180);
  fill(0);
  ellipse(400, 400, 160, 160);
  rectMode(CENTER);
  drawHeart();  //heart

  // we add a text
  int now = millis() - playStart;
  if (now > 1000 && now < 1500) {
    fill(255);
    text("3", height/2, width/4);
  }
  if (now > 1500 && now < 2000) {
    fill(255);
    text("2", height/2, width/4);
  }
  if (now > 2000 && now < 2500) {
    fill(255);
    text("1", height/2, width/4);
  }
  if (now > 2500 && now < 3000) {
    fill(255);
    text("Start!", height/2, width/4);
  }

  if (now > 17000 && now < 18500) {
    fill(255, 255, 255);
    text("It will be faster", height/2, width/4);
  }
  if (now > 48500 && now < 49500) {
    fill(255, 255, 255);
    text("Even more faster", height/2, width/4);
  }
  push();
  textSize(32);
  fill(255);
  text("Your score is:", 125, 65);
  text(a, 100, 100);
  text("/83", 135, 100);
  pop();
  if (serialRecord.read()) {
    value = serialRecord.get();
    key(value);
    //println(frameCount);
  }

  if (success==true) {

    if (millis()-premil<300) {
      fill(255);
      ellipse(400, 400, 180, 180);
      fill(0);
      ellipse(400, 400, 160, 160);
      drawBigHeart();
    } else {
      success=false;
    }
  } else drawHeart();

  //ball
  int cfc=frameCount;
  for (int i=max(0, num-2); i<83; i++) {
    if (cfc-beats[i]>-31&&!flag[i]) {
      fill(255);
      if (abs(cfc-(beats[i]-31))<2) {
        xpos[i]=800*floor(random(0, 2));
        ypos[i]=800*floor(random(0, 2));
        if (xpos[i]==0&&ypos[i]==0) {
          ball[i]=1;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 1;
            serialRecord.send();
            //println("1");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==800&&ypos[i]==0) {
          ball[i]=2;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 2;
            serialRecord.send();
            //println("2");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==0&&ypos[i]==800) {
          ball[i]=3;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 3;
            serialRecord.send();
            //println("3");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==800&&ypos[i]==800) {
          ball[i]=4;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 4;
            serialRecord.send();
            //println("4");
            sendstate[i]=false;
            num++;
          }
        }
      }

      if (xpos[i]<400 && ypos[i]<400) {
        xpos[i] = xpos[i] + 10;
        ypos[i]= ypos[i] + 10;
      }
      if (xpos[i]>400&&ypos[i]<400) {
        xpos[i] = xpos[i]-10;
        ypos[i]=ypos[i]+10;
      }
      if (xpos[i]>400&&ypos[i]>400) {
        xpos[i] = xpos[i]-10;
        ypos[i]=ypos[i]-10;
      }
      if (xpos[i]<400&&ypos[i]>400) {
        xpos[i] = xpos[i]+10;
        ypos[i]=ypos[i]-10;
      }
      noStroke();
      if (ypos[i]<=400) {
        fill(255, 0, 0);
      }
      if (ypos[i]>=400) {
        fill(#30CEFC);
      }
      if (1024<=cfc&&cfc<=1072||1526<=cfc&&cfc<=1578||2100<=cfc&&cfc<=2148||2600<=cfc&&cfc<=2648) {
        fill(#FFF534);
      }
      star(xpos[i], ypos[i], 40, 20, 5);
      if (xpos[i]==400&&ypos[i]==400) {
        flag[i]=true;
        serialRecord.values[0] = 5;
        serialRecord.send();
      }
    }
  }
  if(millis()>58500){
    clear();
    fill(255);
  text("Your score is:", height/2,width/2-50);
  text(a, height/2-50,width/2+50);
  text("/83", height/2+50,width/2+50);
    
  }   
}
Harder:

import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;

Serial serialPort;
SerialRecord serialRecord;
SoundFile sound;

boolean[] flag=new boolean[74];
boolean[] flaga=new boolean[74];
boolean[] sendstate=new boolean[74];
int playStart;
boolean success=false;
int premil=0;
int value=0;
int num=0;
int[]beats= {
186,
210,
232,
255,
279,
302,
324,
348,
373,
388,
419,
435,
464,
487,
509,
532,
557,
579,
602,
626,
650,
672,
695,
720,
744,
761,
789,
804,
833,
857,
880,
904,
926,
950,
973,
997,
1022,
1045,
1068,
1090,
1113,
1136,
1159,
1185,
1208,
1231,
1255,
1274,
1300,
1323,
1346,
1368,
1392,
1417,
1439,
1463,
1484,
1502,
1531,
1548,
1576,
1601,
1627,
1650,
1672,
1690,
1719,
1735,
1762,
1786,
1811,
1835,
1858,
1951,
};
int[] xpos=new int[74];
int[] ypos=new int[74];
int x;
int y;
int[] ball=new int[74];
int a=0;

void star(float x1, float y1, float radius1, float radius2, int npoints) { //ball=start
  float angle = TWO_PI / npoints;
  float halfAngle = angle/2.0;
  beginShape();
  for (float a = 0; a < TWO_PI; a += angle) {
    float sx = x1 + cos(a) * radius2;
    float sy = y1 + sin(a) * radius2;
    vertex(sx, sy);
    sx = x1 + cos(a+halfAngle) * radius1;
    sy = y1 + sin(a+halfAngle) * radius1;
    vertex(sx, sy);
  }
  endShape(CLOSE);
}

void keyPressed() {
  println(frameCount+",");
  //println(millis());
}

void drawBigHeart() { // if player press the button correctly, the heart becomes bigheart
  fill(255, 0, 0);
  rectMode(CENTER);
  rect(height/2, width/2, 10, 10);
  rect(height/2-10, width/2, 10, 10);
  rect(height/2+10, width/2, 10, 10);
  rect(height/2-20, width/2, 10, 10);
  rect(height/2+20, width/2, 10, 10);
  rect(height/2-30, width/2, 10, 10);
  rect(height/2+30, width/2, 10, 10);
  rect(height/2+10, width/2-10, 10, 10);
  rect(height/2-10, width/2-10, 10, 10);
  rect(height/2+20, width/2-10, 10, 10);
  rect(height/2-20, width/2-10, 10, 10);
  rect(height/2-15, width/2-20, 10, 10);
  rect(height/2+15, width/2-20, 10, 10);
  rect(height/2, width/2+10, 10, 10);
  rect(height/2-10, width/2+10, 10, 10);
  rect(height/2+10, width/2+10, 10, 10);
  rect(height/2-20, width/2+10, 10, 10);
  rect(height/2+20, width/2+10, 10, 10);
  rect(height/2, width/2+20, 10, 10);
  rect(height/2-10, width/2+20, 10, 10);
  rect(height/2+10, width/2+20, 10, 10);
  rect(height/2, width/2+30, 10, 10);
}

void drawHeart() { // normal heart
  fill(255, 0, 0);
  rect(height/2, width/2, 10, 10);
  rect(height/2-10, width/2, 10, 10);
  rect(height/2+10, width/2, 10, 10);
  rect(height/2-20, width/2, 10, 10);
  rect(height/2+20, width/2, 10, 10);
  rect(height/2+10, width/2-10, 10, 10);
  rect(height/2-10, width/2-10, 10, 10);
  rect(height/2, width/2+10, 10, 10);
  rect(height/2-10, width/2+10, 10, 10);
  rect(height/2+10, width/2+10, 10, 10);
  rect(height/2, width/2+20, 10, 10);
}

void setup() {
  size(800, 800);
  frameRate(60);
  delay(1000);
  sound = new SoundFile(this, "t.wav");
  playStart = millis();
  sound.play();

  noStroke(); // the text
  textSize(90);
  textAlign(CENTER, CENTER);

  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 115200);
  serialRecord = new SerialRecord(this, serialPort, 1);

  for (int w=0; w<74; w++) {
    sendstate[w]=true;
  }
}


void key(int k) {   //key is controlled by buttons
  //println((frameCount - playStart)+",");
  for (int i=0; i<74; i++) { //judgement
    if (abs(frameCount-beats[i])<10&&ball[i]==k&&!flaga[i]) {
      success=true;  //if press button correctly, the heart become bigger
      //println("success=true");
      premil=millis();
      a=a+1;
      flaga[i]=true;
    }
  }
}

void draw() {
  background(0);
  fill(255, 100);
  ellipse(400, 400, 180, 180);
  fill(0);
  ellipse(400, 400, 160, 160);
  rectMode(CENTER);
  drawHeart();  //heart

  // we add a text
  int now = millis() - playStart;
  if (now > 1000 && now < 1500) {
    fill(255);
    text("3", height/2, width/4);
  }
  if (now > 1500 && now < 2000) {
    fill(255);
    text("2", height/2, width/4);
  }
  if (now > 2000 && now < 2500) {
    fill(255);
    text("1", height/2, width/4);
  }
  if (now > 2500 && now < 3000) {
    fill(255);
    text("Start!", height/2, width/4);
  }

  push();
  textSize(32);
  fill(255);
  text("Your score is:", 125, 65);
  text(a, 100, 100);
  text("/74", 135, 100);
  pop();
  if (serialRecord.read()) {
    value = serialRecord.get();
    key(value);
    //println(frameCount);
  }

  if (success==true) {

    if (millis()-premil<300) {
      fill(255);
      ellipse(400, 400, 180, 180);
      fill(0);
      ellipse(400, 400, 160, 160);
      drawBigHeart();
    } else {
      success=false;
    }
  } else drawHeart();

  //ball
  int cfc=frameCount;
  for (int i=max(0, num-2); i<74; i++) {
    if (cfc-beats[i]>-31&&!flag[i]) {
      fill(255);
      if (abs(cfc-(beats[i]-31))<2) {
        xpos[i]=800*floor(random(0, 2));
        ypos[i]=800*floor(random(0, 2));
        if (xpos[i]==0&&ypos[i]==0) {
          ball[i]=1;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 1;
            serialRecord.send();
            //println("1");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==800&&ypos[i]==0) {
          ball[i]=2;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 2;
            serialRecord.send();
            //println("2");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==0&&ypos[i]==800) {
          ball[i]=3;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 3;
            serialRecord.send();
            //println("3");
            sendstate[i]=false;
            num++;
          }
        }
        if (xpos[i]==800&&ypos[i]==800) {
          ball[i]=4;
          if (sendstate[i]==true) {
            serialRecord.values[0] = 4;
            serialRecord.send();
            //println("4");
            sendstate[i]=false;
            num++;
          }
        }
      }

      if (xpos[i]<400 && ypos[i]<400) {
        xpos[i] = xpos[i] + 10;
        ypos[i]= ypos[i] + 10;
      }
      if (xpos[i]>400&&ypos[i]<400) {
        xpos[i] = xpos[i]-10;
        ypos[i]=ypos[i]+10;
      }
      if (xpos[i]>400&&ypos[i]>400) {
        xpos[i] = xpos[i]-10;
        ypos[i]=ypos[i]-10;
      }
      if (xpos[i]<400&&ypos[i]>400) {
        xpos[i] = xpos[i]+10;
        ypos[i]=ypos[i]-10;
      }
      noStroke();
      if (ypos[i]<=400) {
        fill(255, 0, 0);
      }
      if (ypos[i]>=400) {
        fill(#30CEFC);
      }
      star(xpos[i], ypos[i], 40, 20, 5);
      if (xpos[i]==400&&ypos[i]==400) {
        flag[i]=true;
        serialRecord.values[0] = 5;
        serialRecord.send();
      }
    }
  }
  if(frameCount>2040){
    clear();
    fill(255);
  text("Your score is:", height/2,width/2-50);
  text(a, height/2-50,width/2+50);
  text("/74", height/2+50,width/2+50);
  }   
}

Wendy did the Neopixel strip part. In this part, the Neopixel strip changes its color according to the loudness of the music. We also add a microphone function code in the Processing to sense and analyze the frequency of the whistle, so that every time we blow the whistle, the LEDs will change their color obviously, which looks very cool.

The circuit:

The code: (Arduino)

#include "SerialRecord.h"
#include <FastLED.h>

#define NUM_LEDS 60  // How many leds in your strip?
#define DATA_PINA 3  
#define DATA_PINB 4 
#define DATA_PINC 5
 // Which pin are you connecting Arduino to Data In?
CRGB leds[NUM_LEDS];
// Change this number to the number of values you want to receive
SerialRecord reader(4);

void setup ()  
{   
  Serial.begin(9600);
 FastLED.addLeds<NEOPIXEL, DATA_PINA>(leds, NUM_LEDS);  // Initialize 
 FastLED.addLeds<NEOPIXEL, DATA_PINB>(leds, NUM_LEDS);
 FastLED.addLeds<NEOPIXEL, DATA_PINC>(leds, NUM_LEDS);
 FastLED.setBrightness(10); // BEWARE: external power for full (255)
}  
 
void loop() {
 if (reader.read()) {
   int n = reader[0];
   int r = reader[1];
   int g = reader[2];
   int b = reader[3];

   leds[reader[0]] = CRGB(reader[1], reader[2], reader[3]);  //  Prepare the color information using CRGB( Red, Green, Blue
   FastLED.show();  //  Pass the information of color to the LED
 } 
}

 (The processing: For different songs, we just change the music name)

import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;
import processing.sound.*;

// declare an AudioIn object
AudioIn microphone;
// declare an Frequency analysis object to detect the frequencies in a sound
FFT freqAnalysis;
// declare a variable for the amount of frequencies to analyze
// should be a multiple of 64 for best results
int frequencies = 1024;
// Define the frequencies wanted for our visualization.  Above that treshold frequencies are rarely atteigned and stay flat.
int freqWanted = 128;
// declare an array to store the frequency anlysis results in
float[] spectrum = new float[freqWanted];
// Declare a drawing variable for calculating the width of the
float circleWidth;
float a;
Serial serialPort;
SerialRecord serialRecord;
SoundFile sample;
Amplitude analysis;
int NUM = 60;  //amount of pixels
int[] r = new int[NUM]; //red of each tile
int[] g = new int[NUM]; //red of each tile
int[] b = new int[NUM]; //red of each tile
int startTime;

void setup() {
  fullScreen();
  // load and play a sound file in a loop
  sample = new SoundFile(this, "t.wav");
  sample.loop();

  // create the Amplitude analysis object
  analysis = new Amplitude(this);
  // analyze the playing sound file
  analysis.input(sample);
   String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);
  serialRecord = new SerialRecord(this, serialPort, 4);
  serialRecord.logToCanvas(false);
  rectMode(CENTER);
  startTime = millis();
  circleWidth = width/float(freqWanted);
  // create the AudioIn object and select the mic as the input stream
  microphone = new AudioIn(this, 0);
  // start the mic input without routing it to the speakers
  microphone.start();
  // create the Frequency analysis object and tell it how many frequencies to analyze
  freqAnalysis = new FFT(this, frequencies);
  // use the microphone as the input for the analysis
  freqAnalysis.input(microphone);
}

void draw() {
  println(analysis.analyze());

  long t = millis() - startTime;
 if (t < 1500) {
   background(125, 255, 125);
 } else if ((t>=1500) && t<3000) {
   background(255);
 } else {
   background(55);
 }
 
  noStroke();
  fill(255, 0, 150);

  // analyze the audio for its volume level
  float volume = analysis.analyze();

  // volume is a number between 0.0 and 1.0
  // map the volume value to a useful scale
  float diameter = map(volume, 0, 1, 0, width);
  float light = map(volume, 0, 1, 0, 255);
  //int n = int(random(60));
  a = random(0, 255);
  int n = frameCount % 60;
    r[n] = floor(light/2);
    g[n] = floor(255-light);
    b[n] = floor(a);
    serialRecord.values[0] = n;     // which pixel we change (0-59)
    serialRecord.values[1] = r[n];  // how much red (0-255)
    serialRecord.values[2] = g[n];  // how much green (0-255)
    serialRecord.values[3] = b[n];  // how much blue (0-255)
    serialRecord.send();    
  // draw a circle based on the microphone amplitude (volume)
  circle(width/2, height/2, diameter);
  // analyze the frequency spectrum and store it in the array
  freqAnalysis.analyze(spectrum);
  //printArray(spectrum);
  background(0);
  fill(200, 0, 100, 150);
  noStroke();
  float loudest = 0;
  int loudestIdx = 0;
  for (int i=0; i<freqWanted; i++) {
    if (spectrum[i] > loudest) {
      loudest = spectrum[i];
      loudestIdx = i;
    }
  }
  if ((loudestIdx == 19 || loudestIdx == 20) && loudest > 0.02) {
  background(255,255,255);
  int n1 = frameCount % 60;
  r[n1] = floor(255);   
  g[n1] = floor(255);  
  b[n1] = floor(255);
    serialRecord.values[0] = n1;  // which pixel we change (0-59)    
    serialRecord.values[1] = r[n1];  // how much red (0-255)     
    serialRecord.values[2] = g[n1];  // how much green (0-255)     
    serialRecord.values[3] = b[n1];  // how much blue (0-255)// how much blue (0-255)
    serialRecord.send();            // send it!

  }
  if (loudest > 0.01) {
    println(loudestIdx + " @ " + loudest);
  }

  // draw circles based on the values stored in the spectrum array
  // adjust the scale variable as needed
  float scale = 600;
  for (int i=0; i<freqWanted; i++) {
    circle(i*circleWidth, height/2, spectrum[i]*scale);
  }
}
The music we use: 

(The music of the harder part is “.wav” and cannot be attached)

 The video:

D. CONCLUSIONS:
The goal of our project is to make an interactive rhythm game. I think our project has achieved this goal from the player’s playing process and feedback. The player presses the button according to the rhythm of the music and can adjust their own speed according to the output (whether they get a score/hit correctly), which constitutes a kind of interaction. And my previous definition of interaction is between the user and artifact, the user gives an input to the artifact and receives an output from the artifact, and this output can change the behavior of the user. So this project aligns with my definition of interaction.

If I have more time, I will find a better way to program the Neopixel board, which can run more LED movements at the same time, and also make more changes, such as LED color changes, making specified patterns, etc.

The programming part of this project was difficult, but I learned more programming languages, such as boolean, using defining a flag to make a true/false judgment for start/stop a loop, and I also explored and learned some Neopixel usage by myself. In fact, in the beginning, I couldn’t even imagine that I could make a rhythm game by myself. From this project, although it took a lot of time and effort, I gained a lot, and I feel a sense of accomplishment when I look at the final work! The code we finished has some versatility, if you want to add another song to play, you can just change the beat point and play it. I also understood and learned some new functions of Processing more deeply, and learned how to make the Processing and Arduino deliver value and cooperate with each other. And through this project, I also have a deeper understanding of the rhythm game, I understand some of its logic and the way it is written, and also broaden my understanding of interaction.

E. Some other interesting projects…

In this recitation class, I used a potentiometer to build a circuit and sent its value to processing to control the speed of the video.

I connected one side of the potentiometer to pin2 and connected the other side to 5V and ground. And I use the example of “Send single value” to send the value of the potentiometer to processing. In the processing part, I use “map” and “myMovie.speed” to control the speed of the video. When the value of the potentiometer is smaller, the speed of the video will be slower; when the value is bigger, the speed will become faster.

The code: (Arduino)

#include "SerialRecord.h"

SerialRecord writer(2);

void setup() {
  Serial.begin(9600);
}

void loop() {
  int sensorValue = analogRead(A0);

  writer[0] = millis() % 1024;
  writer[1] = sensorValue;
  writer.send();

  delay(20);
}

(Processing):

import processing.video.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;

Serial serialPort;
SerialRecord serialRecord;
Movie myMovie;

void setup() {
  background(0);
  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);
  serialRecord = new SerialRecord(this, serialPort, 2);

  size(512, 288);
  myMovie = new Movie(this, "Jing'An.mp4");
  myMovie.loop();
}


void movieEvent(Movie movie) {
  myMovie.read();  
}

void draw() { 
  serialRecord.read();
  int value1 = serialRecord.values[0];
  
  image(myMovie, 0, 0);   
  float newSpeed = map(value1, 0, width, 0.1, 5);
  myMovie.speed(newSpeed);
} 

Step 1.1: Stand design

Cuttle is almost a new thing for me. I followed the instructions given, doing the work of “student A” and used various functions like “Round corners” and “Boolean Difference” to make a sketch of the cutting of the wooden board.

Step 1.2: Pattern design

What’s unlucky is that I forgot how to draw the curve. I tried by myself many times and watch the video many times. And finally, I found out that I should drag the line instead of just “clicking”. Using rotational repeat and outline stroke, I drew my own pattern.

Step 2: Laser cut

We went to fabrication lab 823 to do laser cutting. Laser cutting is really amazing!

Step 3: Assemble 

 

A. PROJECT TITLE: Rhythm Master

B. PROJECT STATEMENT OF PURPOSE

Genre: Music.
We searched for some projects about rhythm games (Reference: https://en.wikipedia.org/wiki/Rhythm_game). For example, arcade rhythm games: Dance Revolution, and mobile rhythm games: osu! Combining what have researched, we decided to make a rhythm game that not only requires the player to catch the pace of the music but also the player needs quick react and make movements accordingly (press the button and blow the whistle with the prop). The challenge for us is to figure out how to write and calculate the range of judgment (the input from the player can only be recognized in the range of judgment time) because we should leave some time for players to react. And we also should consider the player’s game experience, like how to make the instruction more obvious, and we need to figure out how much time should the indicator light flash in advance.

C. PROJECT PROPOSAL PLAN

During the first week, we are going to finish the external device connected to the Arduino. We need to set four small buttons on a large cardboard, install an LED board in the middle, and put 4 Neopixel Strips around it. We will select a piece of music with lyrics, and select and record part of the rhythm because it is very difficult to get the computer to analyze the music directly. We would then artificially program these rhythms into the computer, and four different Neopixel Strips would light up with the rhythm. When a specific Neopixel lights up, the player needs to press the corresponding button. We will pick a specific lyric in the song, and when the word is sung, the middle LED will light up, at this time, the player needs to blow the whistle with the prop.

In the second week, we will finish the code for Processing. When a specific light band is lit up, the computer screen will have a small ball in the corresponding direction smashing into the stickman in the middle. When the player presses the button, he will make the ball bounce, and if the player does not react in time, then a big red cross will appear on the whole screen, and then all the light strips will turn red together.

At last, to make the game more interactive, we hope to add a sound sensor again, when a specific word appears, there will be small balls in all four directions to smash people, so players need to whistle, at this time the sound sensor will receive the signal, and the stickman will be safe.

D. CONTEXT AND SIGNIFICANCE

The preparatory research of rhythm games shows a great combination of the music and the movements of the players and inspired us. So we decided to put this feature of rhythm game into our project, but change its form of expression and add some interactive elements like blowing the whisper into the game. The different instructions (press buttons and blow whistle) require players to think and react quickly, and the different inputs from players will create various outputs. If pressing the button correctly, the stickman will be safe, if miss the opportunity to press the button or whistle, the stickman will be hit by the ball. The Player will adjust his pace according to the output (the flash of LEDs). So the game aligns with my definition of interactive. In the processing part, the game was presented in the form of a mini-game to the audience, to make the player’s action visualized to the audience, and make the game more interesting.

 

Task #1:  Make a Processing Etch-A-Sketch

Using the code from the example: Send Multiple Values and Receive Multiple Values, and building the circuit of the two potentiometers, I could draw an Etch-A-Sketch by twisting the potentiometers. After the code for drawing circles worked, I modified circles into the line and add “previous x” and “previous y”, using “previous x=x” and “previous y=y” to keep the previous tracks of lines.

In this Etch-A-Sketch, one potentiometer controls the circle’s x-axis movement, and the other controls the circle’s y-axis movement. It’s convenient to draw straight lines, but difficult to draw curves because it is hard to control. So I just wrote a “Hello” with the mouse.

The videos:

The code: (Arduino part)

#include "SerialRecord.h"

// Change this number to send a different number of values
SerialRecord writer(2);

void setup() {
  Serial.begin(9600);
}

void loop() {
  int Value1 = analogRead(A0);
  int Value2 = analogRead(A1);

  writer[0] = Value1;
  writer[1] = Value2;
  writer.send();

  // This delay slows down the loop, so that it runs less frequently. This
  // prevents it from sending data faster than a Processing sketch that runs at
  // 60 frames per second will process it. It also makes it easier to debug the
  // sketch, because values are received at a slower rate.
  delay(20);
}

(Processing part:)

import processing.serial.*;
import osteele.processing.SerialRecord.*;

 float prex;
 float prey;
  
Serial serialPort;
SerialRecord serialRecord;

void setup() {
  size(500, 500);

  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);

  // If the Arduino sketch sends a different number of values, modify the number
  // `2` on the next line to match the number of values that it sends.
  serialRecord = new SerialRecord(this, serialPort, 2);

size(640, 360);
  background(102);
}

void draw() {

  serialRecord.read();
  int value1 = serialRecord.values[0];
  int value2 = serialRecord.values[1];

  float x = map(value1, 0, 1024, 0, width);
  float y = map(value2, 0, 1024, 0, height);
  circle(x, y, 20);
  stroke(255);

    //line(x, y, prex, prey);
    prex=x;
    prey=y;
  }

Task #2:

My partner Jinnuo (Jim) Liu took charge of the processing part, which should make the ball “bounce” and send values when the ball reaches the edge of the screen. I wrote Arduino code and built the circuit with two servos. My thought is when the ball hits the edge and sends a value to Arduino, one servo will work (turn to 120 degrees and then turn back), and when the ball reaches another edge, another value will be sent to make another servo work. When we first combined our code and circuit, only one servo worked, then we changed some parts of the code. We put “else” between the servo turning to 120 degrees and turning to 60 degrees. After a few changes, the servo did work with the ball, but it seems like having a delay between them.

The code: (Arduino)

#include <Servo.h>
#include "SerialRecord.h"
Servo servo1;
Servo servo2;
SerialRecord reader(2);

void setup() {
  Serial.begin(9600);
  servo1.attach(9);
  servo2.attach(10);
}
void loop() {
  reader.read()；
    if (reader[0] == 1) {
      servo1.write(120);
      delay(100);
    }else servo1.write(60);
    if (reader[1] == 1) {
      servo2.write(60);
      delay(100);
    }else servo2.write(120);
}

(Processing: )

import processing.serial.*;
import osteele.processing.SerialRecord.*;

Serial serialPort;
SerialRecord serialRecord;

int x=75;
int  direction=20;
int radius=75;

void setup() {
  fullScreen();
  background(0);
  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);
  serialRecord = new SerialRecord(this, serialPort, 2);
  ellipseMode(CENTER);
}
void draw() {
  fill(0);
  rect(0, 0, width, height);
  noStroke();
  // display instructions
  fill(255);
  ellipse(x, height/2, 2*radius, 2*radius);
  if (x<radius||x>width-radius) {
    direction*=-1;
  }
  x=x+direction;

  if (x<radius) {
    serialRecord.values[0] = 1;
  } else serialRecord.values[0] = 0;
  if (x>width-radius)
  {
    serialRecord.values[1] = 1;
  } else  serialRecord.values[1] = 0;

  serialRecord.send();
}

We struggled for a long time to find ways to reduce the delay between the ball hitting the edge and the servo turns. With the help of professors, we changed the code to decrease the frequency that Processing sent signals to Arduino. And used “if(reader. read())” instead of “reader. read()”. Also, we deleted the “else” in the Arduino code to directly let the servo turn back. Finally, the servos worked as expected.

The code:(Arduino)

#include <Servo.h>
#include "SerialRecord.h"
Servo servo1;
Servo servo2;
SerialRecord reader(2);

void setup() {
  Serial.begin(9600);
  servo1.attach(9);
  servo2.attach(10);
}
void loop() {
  if (reader.read()) {

    if (reader[0] == 1) {
      servo1.write(120);
      delay(100);
      servo1.write(60);
    }
    if (reader[1] == 1) {
      servo2.write(60);
      delay(100);
      servo2.write(120);
    }
  }
}

(Processing:)

import processing.serial.*;
import osteele.processing.SerialRecord.*;

int pre1, pre2;

Serial serialPort;
SerialRecord serialRecord;

int x=75;
int  direction=20;
int radius=75;

void setup() {
  fullScreen();
  background(0);
  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, serialPortName, 9600);
  serialRecord = new SerialRecord(this, serialPort, 2);
  ellipseMode(CENTER);
}

void draw() {
  fill(0);
  rect(0, 0, width, height);
  noStroke();
  // display instructions
  fill(255);
  ellipse(x, height/2, 2*radius, 2*radius);
  if (x<radius||x>width-radius) {
    direction*=-1;
  }
  x=x+direction;

  if (x<radius) {
    serialRecord.values[0] = 1;
  } else serialRecord.values[0] = 0;
  if (x>width-radius)
  {
    serialRecord.values[1] = 1;
  } else  serialRecord.values[1] = 0;

  if (serialRecord.values[0]!=pre1||serialRecord.values[1]!=pre2) {
    serialRecord.send();
  }
  pre1=serialRecord.values[0];
  pre2=serialRecord.values[1];
}

The final video:

1. The Snake
Genre: Game
Snake is a video game genre where the player maneuvers a line that grows bigger or longer after eating something, typically apples, making the snake a primary obstacle to itself. The player controls a dot, square, or object on a bordered plane. As it moves forward, it leaves a trail behind, resembling a moving snake. In some games, the end of the trail is in a fixed position, so the snake continually gets longer as it moves. (Wikipedia, https://en.wikipedia.org/wiki/Snake_(video_game_genre))
I want to simplify the map and add some music functions to this game. Using 4 NeoPixel strips to build a square frame where the snake (made of LED lights) can automatically travel through this frame. The snake moves with the pace of the music. There will be apples randomly appearing in the frame, represented as a blinking LED. Different from the original game, players should press the button when the snake comes across the apple as “eat the apple”, else, the snake will bump into the apple and lose its length. It’s a game that exercises the player’s reaction speed because the apple will appear randomly and unexpectedly, and players also have to press the button according to the rhythm of the music. These functions will make the game more interactive. The application of led adds a sense of technology to the whole game. I also want to use processing to make a thumbnail image which let the player see the location of the snake more clearly, and the screen can also display some data, such as the length of the snake, the number of apples, etc.

2. Rhythm Game
Genre: Game, Music
Rhythm game or challenges a player’s sense of rhythm. It focuses on dance or the simulated performance of musical instruments and requires players to press buttons in a sequence dictated on the screen. (Wikipedia, https://en.wikipedia.org/wiki/Rhythm_game)
I want to use some sensors to do a rhythm game that requires hand-arm movement, and the player makes the specified action according to the rhythm of the music and the prompt on the screen. For example, I can use a photoresistor to detect the player’s hand position, and arm position and use the tilt switch to input the movement data. When the game begins, there may be some instruction words appearing on the screen, players should move and make gestures with instructions and follow the rhythm. Or, I can just use some buttons, the player should timely and accurately press the button according to the instructions (for example, falling squares to catch, circle to point), and pressing with the pace of the music increase the engagement of players. Imagining a bit more, the game could be designed with two players competing with their speed of reaction and accuracy of the time of pressing buttons, which is more interactive.

3. The painting
After learning how to use a photoresistor to control the patterns presented in processing, I think maybe we can use temperature sensors and photoresistors as well as some other sensors to draw a unique image through the control of light, changing of temperature, and some other data.
For example, if my hand covers the photoresistor, then the image will be darker, and if I move my hand, the image will be lighter. Also, if I put my hand closer to the temperature sensor, the image will add some red and orange color as a “warm” feeling, otherwise, it will show in a cold tone. With the distance between the sensor and users changing, the image will become either vague or clear. If there are sensors that can detect other things like the wind and the humidity, and forms a simulation through processing to control the movement of the image (for example, the image moves and shakes because of the wind, some blue points appear in the image as raindrops when the humidity is high), the work will be more interactive.

In the midterm project, my partner and I designed some new playing methods based on the game “Truth or Dare”. Two players are involved in the game, having a competition on their speed to shake the clapping toy and on their luck. In my opinion, interactive means communication between the artifact and the users, or between users and users. From the article, I know more about interactive and different levels of art: Dynamic-Passive: the simple reaction of the artifact, which its output is completely predictable; Dynamic-Interactive: the interactive artifact that can deliver its output to the users, making the user think and change their input actions; Dynamic-Interactive (Varying): the highest level is that the work is unpredictable and interactive. It will learn experience and adjust its output, which adds more random and interesting functions to the performance.
(Ernest Edmonds, Art, Interaction and Engagement).

I am inspired by the game I played named “Osu!”. It is a great rhythm game, combining the music with clicking, tapping keyboards, and moving the mouse. I like the “osu! mania” mode in this game. In the game, the key notes appear in the form of squares, and the moment the squares hit the line, the player should immediately press the corresponding button to get a “combo” (“perfect”). If the error between the key pressing and the rhythm of the original song is larger, the “perfect” will become “good”, “bad” or “miss” displayed (the error will reduce the accuracy rate and the song score).”The map of the music is then played with accompanying music, simulating a sense of rhythm as the player interacts with the objects to the beat of the music.” (Wikipedia. https://en.wikipedia.org/wiki/Osu!)
When playing osu, I am completely immersed in the music. If I got a “miss” or “bad” mistake, I will immediately adjust my pace, it is interactive. Although this game looks simple compared to other games, this simple style and interactive playing method inspired me. In the final project, I may also try adding some combinations of music and physical control to make the game more interesting.

(The video I recorded myself playing the game.)

The other project is the artifact designed by Venezuelan artist Jesús Rafael Soto (1923 – 2005), made of colorful linear materials, appears to be a monolith, but the interior is free to travel, allowing visitors to enter this magical space, move freely, play, and enjoy as if they could touch the golden sunlight. The combination of linearity forms different shapes visually. The visitors also can travel through it, which is very attractive. If the linear combination can change with people’s movement, and position, it will be more interactive as Dynamic-Interactive. If the artifact can use some algorithmic to learn experiences from the users, and depending on the history of interactions with users, performing and modifying its specification, it will be more interactive and interesting.

The pictures:

Task 1 & Task 2

In task 1 and task 2, I checked the download of the library and used the example to make sure the NeoPixel Led strip worked correctly. In the code of task 2, I can input the number to the monitor and make the LED light according to the order and color I entered.

Task 3 & Task 4

Combining the given code in task 1 and task 2, the NeoPixel Led strip can change color randomly with the mouse’s drag and click. At the same time, the music I chose was played. Then I struggled with how to make the color of the NeoPixel Led strip change with the music’s volume. I tried to modify the code, letting the number of led light-emitting can be controlled by the volume of music, and adding random in colors to make Led strip change more dynamic and ornamental.

With the help of the professor and partner, I found a way to control the light and fade of LEDs better by using “if” and “for”. If the volume is higher than before, the lit LEDs will keep their previous colors and the newly added LEDs will show random colors. If the volume is lower than before, the LEDs which show the volume higher at the moment will fade. 

The only regret is that I didn’t have the time and could not understand the “FFT analysis” code in Task 4 to integrate it into my code😢

The code:

/* This is a code example for Processing, to be used on Recitation 7
 You need to have installed the SerialRecord library.
 
 Interaction Lab
 IMA NYU Shanghai
 2022 Fall
 */
import processing.sound.*;
int pre_n=0;
import processing.serial.*;
import osteele.processing.SerialRecord.*;

SoundFile sample;
Amplitude analysis;

Serial serialPort;
SerialRecord serialRecord;

int W;         //width of the tiles
int NUM = 60;  //amount of pixels
int[] r = new int[NUM]; //red of each tile
int[] g = new int[NUM]; //red of each tile
int[] b = new int[NUM]; //red of each tile

void setup() {
  size(640, 480);

  // load and play a sound file in a loop
  sample = new SoundFile(this, "Feryquitous F9 - Monochrome Re.Surgence.mp3");
  sample.loop();

  // create the Amplitude analysis object
  analysis = new Amplitude(this);
  // analyze the playing sound file
  analysis.input(sample);

  printArray(Serial.list());
  String serialPortName = SerialUtils.findArduinoPort();
  serialPort = new Serial(this, "/dev/cu.usbmodem1101", 9600);
  serialRecord = new SerialRecord(this, serialPort, 4);
  serialRecord.logToCanvas(false);
  rectMode(CENTER);

  /*for (int i=0; i < NUM; i++) {
   serialRecord.values[0] = i;     // which pixel we change (0-59)
   serialRecord.values[1] = 0;  // how much red (0-255)
   serialRecord.values[2] = 0;  // how much green (0-255)
   serialRecord.values[3] = 0;  // how much blue (0-255)
   serialRecord.send();            // send it!
   }*/
}

void draw() {
  println(analysis.analyze());
  noStroke();
  for (int i=0; i<height; i++) {
    float b = map(i, 0, -height, 100, 255);
    stroke(0, 0, b);
    line(0, i, width, i);
  }
  fill(255, 100);

  // analyze the audio for its volume level
  float volume = analysis.analyze();

  // volume is a number between 0.0 and 1.0
  // map the volume value to a useful scale
  float vol = map(volume, 0, 1, 0, 60);
  // draw a circle based on the microphone amplitude (volume)
  circle(width/2, height/2, vol);

  int n=floor(vol);
  println(n);

 if (n>pre_n) {
    for (int i=pre_n; i<n; i ++) {

      r[n] = floor(random(0, 255));
      g[n] = floor(random(100, 255));
      b[n] = floor(random(160, 255));

      serialRecord.values[0] = i;     // which pixel we change (0-59)
      serialRecord.values[1] = r[i];  // how much red (0-255)
      serialRecord.values[2] = g[i];  // how much green (0-255)
      serialRecord.values[3] = b[i];  // how much blue (0-255)
      serialRecord.send();            // send it!
    }
  }
  if (n<pre_n) {
    for (int j=pre_n; j>n; j--) {
      serialRecord.values[0] = j;     // which pixel we change (0-59)
      serialRecord.values[1] = 0;  // how much red (0-255)
      serialRecord.values[2] = 0;  // how much green (0-255)
      serialRecord.values[3] = 0;  // how much blue (0-255)
      serialRecord.send();            // send it!
    }
  }
pre_n = n;
  
}

The video:

I learned some new functions from examples such as making the ellipse move and leave track (which will disappear later) by constantly overlaying a big rectangle filled with background color, and letting the ellipse “bounce” to change its direction. Also, I tried to use some new code like “angle” and “translate” rotate” to make an eye that is “looking” at the mouse. It’s surprising that I can add some mathematical symbols into the code like “arctan” sin” cos”.

The poster:

 

The code:

float beginX = 0;
float endX = 1024;
float distX;
float x = 0;
float y = 0;
float step = 0.015;
float c = 0.0;
int i = 0;
float angle=0.0;

int rad = 60;
float xpos, ypos;

float xspeed = 10.0;
float yspeed = 10.0;

int xdirection = 1;
int ydirection = 1;
PFont myFont;

void setup() {
  size(1024, 768);
  background(0);
  distX = endX - beginX;
  xpos = random(0, width);
  ypos = 700;

  String[] fontList = PFont.list();
  printArray(fontList);
  myFont = createFont("BMJUAOTF", 32);
  textFont(myFont);
}

void draw() {

  noStroke();
  fill(0, 15);
  rect(0, 0, width, height);

  c += step;
  x = beginX + (c * distX);
  y = 1024 - (c * distX);

  xpos = xpos + ( xspeed * xdirection );
  ypos = ypos + ( yspeed * ydirection );

  if (xpos > width-rad || xpos < rad) {
    xdirection *= -1;
  }
  if (ypos > height-rad || ypos < rad) {
    ydirection *= -1;
  }

  ellipse(xpos, ypos, rad, rad);

  noStroke();
  fill(#FF4BD2); //red
  ellipse(x, height/2-150, 100, 100);
  fill(#FEFF34); //yellow
  ellipse(y, height/2, 100, 100);
  fill(#29FFFD); //blue
  ellipse(x, height/2+150, 100, 100);
  fill(#7E83FF, 100);
  ellipse(random(0, 1024), random(0, 768), 100, 100);
  fill(#69FF62);//green
  ellipse(xpos, ypos, 100, 100);


  if (x > 1024) {
    c = 0.0;
    beginX = 0;
    x=0;
    y=0;
  }

  push();
  noFill(); //rect
  stroke(random(0, 255));
  strokeWeight(6);
  rectMode(CENTER);
  rect(width/2-10, height/2+30, 350, 240);
  pop();

  fill(random(0, 255)); //text
  textSize(196);
  textAlign(CENTER, CENTER);
  text("IMA", width/2, height/2);

  noFill();
  stroke(#E5E286);
  strokeWeight(8);
  beginShape();
  curveVertex(width/2-100, 125);
  curveVertex(width/2-100, 125);
  curveVertex(width/2, 125-50);
  curveVertex(width/2+100, 125);
  curveVertex(width/2+100, 125);
  endShape();
  beginShape();
  curveVertex(width/2-100, 125);
  curveVertex(width/2-100, 125);
  curveVertex(width/2, 125+50);
  curveVertex(width/2+100, 125);
  curveVertex(width/2+100, 125);
  endShape();


  strokeWeight(12);
  noFill();
  ellipse(width/2, 125, 95, 95);
  angle=atan2(mouseY-150, mouseX-width/2);
  translate(width/2, 125);
  rotate(angle);
  fill(255);
  noStroke();
  ellipse(100/4, 0, 100/2, 100/2);

  //println(mouseX, mouseY);
} 

Homework:

 

The code:

float angle=0;
void setup() {
  size(1024, 768);
  background(#F0EDA7);
}

void draw() {

  for (int i=50; i<=width; i+=250) {
    for (int j=150; j<=height; j+=250) {
      drawEye(i, j);
    }
  }
}
void drawEye(int x, int y) {
  noFill();
  stroke(0);
  strokeWeight(8);
  beginShape();
  curveVertex(x, y);
  curveVertex(x, y);
  curveVertex(x+100, y-50);
  curveVertex(x+200, y);
  curveVertex(x+200, y);
  endShape();
  beginShape();
  curveVertex(x, y);
  curveVertex(x, y);
  curveVertex(x+100, y+50);
  curveVertex(x+200, y);
  curveVertex(x+200, y);
  endShape();


  strokeWeight(8);
  fill(#A7CCF0);
  ellipse(x+100, y, 95, 95);

  angle=atan2(mouseY-y, mouseX-x);
  push();
  translate(x+100, y);
  fill(0);
  ellipse(95/4*cos(angle), 95/4*sin(angle), 95/2, 95/2);
  pop();
}
 

Step 1: Choose your motif

This photo was taken in Gansu province and the great sand dune is called MingSha mountain. In the image, the color of the dunes tones in the color of blue sky, which not only has an aesthetic quality but also has geometric features. 

Step 2: Draw your image on paper

I first drew and simplified the image on graph paper. I used curve and bezier to draw the dunes and its shadow side. But I found it difficult to sketch the shape of cloud, so I added the moon and the sun instead of the cloud to decorate the sky.

Step 3: Draw your image with code & Step 4: Going beyond

I didn’t find the approach to make the color of sky change gradually, so I used several rectangles and filled in blue colors from dark to light. And I animate the positions of sun and moon to let them alternate.

float a=0;
float c=280;
float d=280;
float b=-800;
int mode=1;
void setup() {
size(800, 600);
}
void draw() {
// Your drawing code goes here
background(146,159,191);
fill(131,163,204);//sky
rect(0,0,800,150);
fill(161,178,208);
rect(0,150,800,300);
fill(183,197,223);
rect(0,300,800,450);
fill(174,188,215);
rect(0,450,800,600);
noStroke();
fill(90,75,68,200);//mountain1
beginShape();
vertex(-100,600);
bezierVertex(78,510,167,490,310,600);
endShape();
fill(158,103,73,150);//3
beginShape();
vertex(347,600);
bezierVertex(676,500,760,520,1000,600);
endShape();
fill(158,103,73);//mountain2
beginShape();
vertex(88,600);
bezierVertex(168,588,368,450,664,600);
endShape();
fill(90,75,68);//mountain2'
beginShape();
vertex(347,600);
bezierVertex(388,516,368,510,664,600);
endShape();

fill(#FFFA6A,170);
ellipse(a,c,100,100);
a=a+4;
if(a>=0&&a<=400){
c=c-1;
}
if(a>=400){
c=c+1;
}
if(a==800) {a=-800;c=280;}
b=b+4;
if(b<=400&&b>=0){
d=d-1;
}
if(b>=400){
d=d+1;
}
if(b==800) {b=-800;d=280;}
fill(#FF6040,170);
ellipse(b,d,100,100);
println(mouseX,mouseY);
}

I think drawing in Processing is interesting because it can realize the combination of images and movements. And maybe I can depict a short story with Processing, it is a good means of realizing design.