A. Project Title: Rhythm Master
Group members: Siwei Fang (Wendy), Siwei Chen
Instructor: Gottfried Haider
B. CONCEPTION AND DESIGN:
In our previous research, we were inspired by the music games “Osu!” and “Dance revolution”, in which players perform actions with the rhythm of the music and earn points according to the accuracy and correctness of their play. This is a kind of game with a strong sense of participation. So we decide to design a rhythm game (it’s really interactive! ) In the game, the player presses the button according to the rhythm and the prompt on the screen to interact with the artifact. In order to increase the fun and engagement, we also designed an additional session where the use of a whistle at a specified rhythm point can change the effect of the lights (unfortunately, because of the epidemic this session could not be realized by the game).
(Our first proposal:)
In the user testing, the game was mostly recognized by the players, they thought the experience was great, the lighting was cool, and the difficulty was challenging. The feedbacks and suggestions we received include: adding a tutorial section, changing the lighting effect (such as covering a white paper on top of Neopixel), adding different songs, adjusting the speed of falling stars, etc. Based on feedback, we made adjustments accordingly. I made the project into three parts, the tutorial part, the normal part, and the harder part. We also pasted white paper on all the LEDs to make the overall look more beautiful. In the final presentation, the player experience and feedbacks were better.
C. FABRICATION AND PRODUCTION:
The most significant part of the project is the rhythm, judgment area in processing, how and when to send and receive value with Arduino, and the coding of the Neopixel part on Arduino.
The most important part of the music game is the rhythm, I originally intended to calculate the BPM of each song and then write a for loop to achieve automatic rhythm recording, so that the beat points can be very accurate. But the beat points are exactly the same, which may be too simple and dull. So we directly input the rhythm manually (use “println” to record the beat), which may lead to some errors in the rhythm, but can make beat points variable.
The screen part is the same as our previous idea, there is a red heart in the middle of the screen protected by a white circle, and four stars will appear from the four corners, when the stars touch the white line a moment, the player should press the button to protect the red heart.
Then came the judgment part, because when playing music in Processing, it will have a delay, and there is no convenient way to measure the delay. So I used the most direct method – listening to the music and looking at the beat point (when the star hit the white line) to manually adjust the delay. In order to improve the game experience, I wrote the judgment interval as one second, that is, if the player presses the button within one second before or after the beat point, it will be judged as a success, if pressed outside of one second, it will be judged as a failure. I first completed the Processing part, using the keyboard instead of buttons, and this part can already be played as a tiny rhythm game.
Then came the harder part: how to connect Processing with Arduino? Because the star has four directions, so correspondingly, we used four buttons. First, before each star hits the white line, we have to send a signal to the Arduino to make the corresponding button glow as a reminder to the player; second, if a button in the same direction as the star is pressed, the Arduino has to send a message to Processing, and this message will be put into the judgment interval to determine whether the player scores. I defined a lot of arrays to complete this part and finally succeed.
Then it occurred to us that if we let players who are not familiar with the operation and have never played a game before, it is difficult to play the game by looking at the computer screen and not at the buttons. So we wanted to make the LED board in the middle of the button synchronized with the display of the stars on the computer screen (only the direction of movement is opposite because the LED moves from the center to the button). This step is the most painful because I have not touched the Neopixel 8*8 board before at all. We first measured the LED numbering order, and then add some basic changes, such as blink, and fade. But I’m not clear on how to make the LED and the star make a synchronous movement. With the help of the professor and my friend, we listed 20 cases to divide the situation and used “case” and “break” to control the movement of the LEDs (this step took a long time). After many attempts, we basically finished the Arduino part of the programming. But we still have some problems. The first problem is in the Processing part, we use a red cross as a reminder to the player that he does not press accurately in the judgment interval, the red cross is completely controlled by buttons but it sometimes appears when we even haven’t connected the button. We have no idea but to delete the cross. Another problem is the Neopixel board, our code is built on the basis that the next LED will glow only after the previous one finishes glowing and goes off, sometimes the LED will somehow get stuck in one place and not move, causing the whole LED board to go wrong.
After testing countless times (very painful), I came up with the final solution: reduce the beat points (not too many beat points in the Processing, otherwise the Arduino part will have problems); limit the song length. We also wanted to let the board make other instructions in the whistle part (such as changing the LED color), but we couldn’t do it due to coding problems. However, the final work is nice (to me).
The circuit:
The codes: (Arduino)
#include <FastLED.h>
#include "SerialRecord.h"
#define NUM_LEDS 64 // How many leds in your strip?
#define DATA_PIN 10
int state[83];
CRGB leds[NUM_LEDS]; // Define the array of leds
int prems[83];
int direction[83];
int num = 0;
int start;
int pre_state1 = LOW;
int pre_state2 = LOW;
int pre_state3 = LOW;
int pre_state4 = LOW;
int buttonState1 = 0;
int buttonState2 = 0;
int buttonState3 = 0;
int buttonState4 = 0;
SerialRecord reader(1);
SerialRecord writer(1);
//button a-d:led button1-4:switch
int buttona = 2;
int buttonb = 3;
int buttonc = 4;
int buttond = 5;
int button1 = 6;
int button2 = 7;
int button3 = 8;
int button4 = 9;
void setup() {
Serial.begin(115200);
pinMode(button1, INPUT_PULLUP); //switch
pinMode(button2, INPUT_PULLUP);
pinMode(button3, INPUT_PULLUP);
pinMode(button4, INPUT_PULLUP);
pinMode(buttona, OUTPUT); //led
pinMode(buttonb, OUTPUT);
pinMode(buttonc, OUTPUT);
pinMode(buttond, OUTPUT);
FastLED.addLeds<NEOPIXEL, DATA_PIN>(leds, NUM_LEDS); // GRB ordering is assumed
for (int i = 0; i < 63; i++) {
leds[i] = CRGB::Black;
}
FastLED.show();
for (int i = 0; i < 83; i++) {
prems[i] = -10000;
}
}
void loop() {
buttonState1 = digitalRead(button1);
buttonState2 = digitalRead(button2);
buttonState3 = digitalRead(button3);
buttonState4 = digitalRead(button4);
if (!buttonState1 == HIGH && buttonState1 != pre_state1) { //if button pressed, send value to processing
writer[0] = 1;
writer.send();
}
if (!buttonState2 == HIGH && buttonState2 != pre_state2) {
writer[0] = 2;
writer.send();
}
if (!buttonState3 == HIGH && buttonState3 != pre_state3) {
writer[0] = 3;
writer.send();
}
if (!buttonState4 == HIGH && buttonState4 != pre_state4) {
writer[0] = 4;
writer.send();
}
pre_state1 = buttonState1;
pre_state2 = buttonState2;
pre_state3 = buttonState3;
pre_state4 = buttonState4;
if (reader.read()) { // if receive value, LED light up accordingly
if (reader[0] == 1) {
digitalWrite(2, HIGH);
}
if (reader[0] == 2) {
digitalWrite(3, HIGH);
}
if (reader[0] == 3) {
digitalWrite(4, HIGH);
}
if (reader[0] == 4) {
digitalWrite(5, HIGH);
}
if (reader[0] == 5) { // when the ball reach center, all LED put out
digitalWrite(2, LOW);
digitalWrite(3, LOW);
digitalWrite(4, LOW);
digitalWrite(5, LOW);
}
if (reader[0] != 5) {
prems[num] = millis();
Serial.println(num);
//void PinReaderMap[] = {0, 0,1,3,2}
if (reader[0] == 1) {
direction[num] = 0;
}
if (reader[0] == 2) {
direction[num] = 1;
}
if (reader[0] == 3) {
direction[num] = 3;
}
if (reader[0] == 4) {
direction[num] = 2;
}
num++;
}
}
int now = millis();
for (int i = 0; i < 83; i++) { // to make neopixel 8*8 work with the screen
if (now - prems[i] >= 0 && now - prems[i] < 150 && state[i] == 0) {
// Serial.println("?");
switch (direction[i]) {
case 0:
{
leds[35] = CRGB::Red;
FastLED.show();
break;
}
case 1:
{
leds[36] = CRGB::Red;
FastLED.show();
break;
}
case 2:
{
leds[27] = CRGB::Blue;
FastLED.show();
break;
}
case 3:
{
leds[28] = CRGB::Blue;
FastLED.show();
break;
}
}
state[i]++;
} else if (now - prems[i] >= 150 && now - prems[i] < 300 && state[i] == 1) {
switch (direction[i]) {
case 0:
{
leds[35] = CRGB::Black;
leds[45] = CRGB::Red;
FastLED.show();
break;
}
case 1:
{
leds[36] = CRGB::Black;
leds[42] = CRGB::Red;
FastLED.show();
break;
}
case 2:
{
leds[27] = CRGB::Black;
leds[21] = CRGB::Blue;
FastLED.show();
break;
}
case 3:
{
leds[28] = CRGB::Black;
leds[18] = CRGB::Blue;
FastLED.show();
break;
}
}
state[i]++;
} else if (now - prems[i] >= 300 && now - prems[i] < 450 && state[i] == 2) {
switch (direction[i]) {
case 0:
{
leds[45] = CRGB::Black;
leds[49] = CRGB::Red;
FastLED.show();
break;
}
case 1:
{
leds[42] = CRGB::Black;
leds[54] = CRGB::Red;
FastLED.show();
break;
}
case 2:
{
leds[21] = CRGB::Black;
leds[9] = CRGB::Blue;
FastLED.show();
break;
}
case 3:
{
leds[18] = CRGB::Black;
leds[14] = CRGB::Blue;
FastLED.show();
break;
}
}
state[i]++;
} else if (now - prems[i] >= 450 && now - prems[i] < 600 && state[i] == 3) {
switch (direction[i]) {
case 0:
{
leds[49] = CRGB::Black;
leds[63] = CRGB::Red;
FastLED.show();
break;
}
case 1:
{
leds[54] = CRGB::Black;
leds[56] = CRGB::Red;
FastLED.show();
break;
}
case 2:
{
leds[9] = CRGB::Black;
leds[7] = CRGB::Blue;
FastLED.show();
break;
}
case 3:
{
leds[14] = CRGB::Black;
leds[0] = CRGB::Blue;
FastLED.show();
break;
}
}
state[i]++;
} else if (now - prems[i] >= 600 && state[i] == 4) {
switch (direction[i]) {
case 0:
{
leds[63] = CRGB::Black;
FastLED.show();
break;
}
case 1:
{
leds[56] = CRGB::Black;
FastLED.show();
break;
}
case 2:
{
leds[7] = CRGB::Black;
FastLED.show();
break;
}
case 3:
{
leds[0] = CRGB::Black;
FastLED.show();
break;
}
}
state[i]++;
}
}
}
(Processing)
Instruction:
import processing.sound.*; import processing.serial.*; import osteele.processing.SerialRecord.*; Serial serialPort; SerialRecord serialRecord; SoundFile sound; boolean[] flag=new boolean[18]; boolean[] flaga=new boolean[18]; boolean[] flagb=new boolean[18]; boolean[] flagc=new boolean[18]; boolean[] sendstate=new boolean[18]; int playStart; boolean success=false; int premil=0; int value=0; int num=0; int[]beats= { 752, 863, 979, 1090, 1486, 1543, 1598, 1653, 2337, 2506, 2677, 3004, 3059, 3116, 3172, 3229, 3284, 3339, }; int[] xpos=new int[18]; int[] ypos=new int[18]; int x; int y; int[] ball=new int[18]; int a=0; void star(float x1, float y1, float radius1, float radius2, int npoints) { //ball=start float angle = TWO_PI / npoints; float halfAngle = angle/2.0; beginShape(); for (float a = 0; a < TWO_PI; a += angle) { float sx = x1 + cos(a) * radius2; float sy = y1 + sin(a) * radius2; vertex(sx, sy); sx = x1 + cos(a+halfAngle) * radius1; sy = y1 + sin(a+halfAngle) * radius1; vertex(sx, sy); } endShape(CLOSE); } void keyPressed() { println(frameCount+","); //println(millis()); } void drawBigHeart() { // if player press the button correctly, the heart becomes bigheart fill(255, 0, 0); rectMode(CENTER); rect(height/2, width/2, 10, 10); rect(height/2-10, width/2, 10, 10); rect(height/2+10, width/2, 10, 10); rect(height/2-20, width/2, 10, 10); rect(height/2+20, width/2, 10, 10); rect(height/2-30, width/2, 10, 10); rect(height/2+30, width/2, 10, 10); rect(height/2+10, width/2-10, 10, 10); rect(height/2-10, width/2-10, 10, 10); rect(height/2+20, width/2-10, 10, 10); rect(height/2-20, width/2-10, 10, 10); rect(height/2-15, width/2-20, 10, 10); rect(height/2+15, width/2-20, 10, 10); rect(height/2, width/2+10, 10, 10); rect(height/2-10, width/2+10, 10, 10); rect(height/2+10, width/2+10, 10, 10); rect(height/2-20, width/2+10, 10, 10); rect(height/2+20, width/2+10, 10, 10); rect(height/2, width/2+20, 10, 10); rect(height/2-10, width/2+20, 10, 10); rect(height/2+10, width/2+20, 10, 10); rect(height/2, width/2+30, 10, 10); } void drawHeart() { // normal heart fill(255, 0, 0); rect(height/2, width/2, 10, 10); rect(height/2-10, width/2, 10, 10); rect(height/2+10, width/2, 10, 10); rect(height/2-20, width/2, 10, 10); rect(height/2+20, width/2, 10, 10); rect(height/2+10, width/2-10, 10, 10); rect(height/2-10, width/2-10, 10, 10); rect(height/2, width/2+10, 10, 10); rect(height/2-10, width/2+10, 10, 10); rect(height/2+10, width/2+10, 10, 10); rect(height/2, width/2+20, 10, 10); } void setup() { size(800, 800); frameRate(60); delay(1000); sound = new SoundFile(this, "g.mp3"); playStart = millis(); sound.play(); noStroke(); // the text textSize(30); textAlign(CENTER, CENTER); String serialPortName = SerialUtils.findArduinoPort(); serialPort = new Serial(this, serialPortName, 115200); serialRecord = new SerialRecord(this, serialPort, 1); for (int w=0; w<18; w++) { sendstate[w]=true; } } void key(int k) { //key is controlled by buttons //println((frameCount - playStart)+","); for (int i=0; i<18; i++) { //judgement if (abs(frameCount-beats[i])<10&&ball[i]==k&&!flaga[i]) { success=true; //if press button correctly, the heart become bigger //println("success=true"); premil=millis(); a=a+1; flaga[i]=true; } } } void draw() { background(0); fill(255, 100); ellipse(400, 400, 180, 180); fill(0); ellipse(400, 400, 160, 160); rectMode(CENTER); drawHeart(); //heart if (serialRecord.read()) { value = serialRecord.get(); key(value); //println(frameCount); } if (success==true) { if (millis()-premil<300) { fill(255); ellipse(400, 400, 180, 180); fill(0); ellipse(400, 400, 160, 160); drawBigHeart(); } else { success=false; } } else drawHeart(); int now = millis() - playStart; push(); if (now > 2000 && now < 5000) { clear(); textSize(96); fill(255, 255, 255); text("instruction", height/2, width/2); } pop(); if (now > 5000 && now < 8783) { clear(); fill(255, 255, 255); text("Pressed Button when the star hits White Line", height/2, width/2); } if (now > 20066 && now < 23850) { clear(); fill(255, 255, 255); text("You can also look at the pixel LED in the middle of the box", height/2, width/2-32); text("LEDs and the stars are moving in sync", height/2, width/2); text("Press button when LED Reach the Edge", height/2, width/2+32); } if (now > 29400 && now < 34500) { clear(); fill(255, 255, 255); text("If you press correctly,", height/2, width/2-32); text("the heart becomes bigger after each pressing", height/2, width/2); text("Your score is displayed in the top left corner", height/2, width/2+32); } if (now > 32500) { fill(255); text("Your score is:", 125, 65); text(a, 100, 100); } if (now >34500 && now < 37800) { clear(); fill(255, 255, 255); text("If the star is", height/2, width/2-32); fill(#FFF534); text("YELLOW", height/2, width/2); fill(255); text("Use Whistle and Press Button at the same time! ", height/2, width/2+32); } push(); if (now>46533&&now<48350) { clear(); fill(255); textSize(96); text(" ENJOY PLAYING !", height/2, width/2); } pop(); if(frameCount>3500){ clear(); fill(255); text("This is the end of the tutorial section",height/2, width/2); } //ball int cfc=frameCount; for (int i=max(0, num-2); i<18; i++) { if (cfc-beats[i]>-31&&!flag[i]) { fill(255); if (abs(cfc-(beats[i]-31))<2) { xpos[i]=800*floor(random(0, 2)); ypos[i]=800*floor(random(0, 2)); if (xpos[i]==0&&ypos[i]==0) { ball[i]=1; if (sendstate[i]==true) { serialRecord.values[0] = 1; serialRecord.send(); //println("1"); sendstate[i]=false; num++; } } if (xpos[i]==800&&ypos[i]==0) { ball[i]=2; if (sendstate[i]==true) { serialRecord.values[0] = 2; serialRecord.send(); //println("2"); sendstate[i]=false; num++; } } if (xpos[i]==0&&ypos[i]==800) { ball[i]=3; if (sendstate[i]==true) { serialRecord.values[0] = 3; serialRecord.send(); //println("3"); sendstate[i]=false; num++; } } if (xpos[i]==800&&ypos[i]==800) { ball[i]=4; if (sendstate[i]==true) { serialRecord.values[0] = 4; serialRecord.send(); //println("4"); sendstate[i]=false; num++; } } } if (xpos[i]<400 && ypos[i]<400) { xpos[i] = xpos[i] + 10; ypos[i]= ypos[i] + 10; } if (xpos[i]>400&&ypos[i]<400) { xpos[i] = xpos[i]-10; ypos[i]=ypos[i]+10; } if (xpos[i]>400&&ypos[i]>400) { xpos[i] = xpos[i]-10; ypos[i]=ypos[i]-10; } if (xpos[i]<400&&ypos[i]>400) { xpos[i] = xpos[i]+10; ypos[i]=ypos[i]-10; } noStroke(); if (ypos[i]<=400) { fill(255, 0, 0); } if (ypos[i]>=400) { fill(#30CEFC); } if (2300<cfc&&cfc<2700) { fill(#FFF534); //serialRecord.values[0]=6; // serialRecord.send(); //flagb[i]=true; } star(xpos[i], ypos[i], 40, 20, 5); if (xpos[i]==400&&ypos[i]==400) { flag[i]=true; serialRecord.values[0] = 5; serialRecord.send(); } } } } Normal:
import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;
Serial serialPort;
SerialRecord serialRecord;
SoundFile sound;
boolean[] flag=new boolean[83];
boolean[] flaga=new boolean[83];
boolean[] sendstate=new boolean[83];
int playStart;
boolean success=false;
int premil=0;
int value=0;
int num=0;
int[]beats= {
199,
265,
330,
397,
463,
530,
599,
667,
731,
801,
869,
935,
1004,
1071,
1106,
1141,
1175,
1210,
1244,
1277,
1310,
1344,
1379,
1414,
1445,
1479,
1513,
1548,
1580,
1615,
1647,
1680,
1714,
1750,
1783,
1816,
1851,
1884,
1918,
1951,
2018,
2052,
2085,
2119,
2152,
2187,
2221,
2254,
2286,
2319,
2353,
2388,
2422,
2456,
2489,
2523,
2557,
2591,
2623,
2657,
2691,
2724,
2758,
2791,
2825,
2859,
2894,
2927,
2959,
2977,
2995,
3013,
3029,
3046,
3063,
3080,
3097,
3113,
3129,
3146,
3163,
3181,
3196,
};
int[] xpos=new int[83];
int[] ypos=new int[83];
int x;
int y;
int[] ball=new int[83];
int a=0;
void star(float x1, float y1, float radius1, float radius2, int npoints) { //ball=start
float angle = TWO_PI / npoints;
float halfAngle = angle/2.0;
beginShape();
for (float a = 0; a < TWO_PI; a += angle) {
float sx = x1 + cos(a) * radius2;
float sy = y1 + sin(a) * radius2;
vertex(sx, sy);
sx = x1 + cos(a+halfAngle) * radius1;
sy = y1 + sin(a+halfAngle) * radius1;
vertex(sx, sy);
}
endShape(CLOSE);
}
void keyPressed() {
//println(frameCount+",");
//println(millis());
}
void drawBigHeart() { // if player press the button correctly, the heart becomes bigheart
fill(255, 0, 0);
rectMode(CENTER);
rect(height/2, width/2, 10, 10);
rect(height/2-10, width/2, 10, 10);
rect(height/2+10, width/2, 10, 10);
rect(height/2-20, width/2, 10, 10);
rect(height/2+20, width/2, 10, 10);
rect(height/2-30, width/2, 10, 10);
rect(height/2+30, width/2, 10, 10);
rect(height/2+10, width/2-10, 10, 10);
rect(height/2-10, width/2-10, 10, 10);
rect(height/2+20, width/2-10, 10, 10);
rect(height/2-20, width/2-10, 10, 10);
rect(height/2-15, width/2-20, 10, 10);
rect(height/2+15, width/2-20, 10, 10);
rect(height/2, width/2+10, 10, 10);
rect(height/2-10, width/2+10, 10, 10);
rect(height/2+10, width/2+10, 10, 10);
rect(height/2-20, width/2+10, 10, 10);
rect(height/2+20, width/2+10, 10, 10);
rect(height/2, width/2+20, 10, 10);
rect(height/2-10, width/2+20, 10, 10);
rect(height/2+10, width/2+20, 10, 10);
rect(height/2, width/2+30, 10, 10);
}
void drawHeart() { // normal heart
fill(255, 0, 0);
rect(height/2, width/2, 10, 10);
rect(height/2-10, width/2, 10, 10);
rect(height/2+10, width/2, 10, 10);
rect(height/2-20, width/2, 10, 10);
rect(height/2+20, width/2, 10, 10);
rect(height/2+10, width/2-10, 10, 10);
rect(height/2-10, width/2-10, 10, 10);
rect(height/2, width/2+10, 10, 10);
rect(height/2-10, width/2+10, 10, 10);
rect(height/2+10, width/2+10, 10, 10);
rect(height/2, width/2+20, 10, 10);
}
void setup() {
size(800, 800);
frameRate(60);
delay(1000);
sound = new SoundFile(this, "t.mp3");
playStart = millis();
sound.play();
noStroke(); // the text
textSize(90);
textAlign(CENTER, CENTER);
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 115200);
serialRecord = new SerialRecord(this, serialPort, 1);
for (int w=0; w<83; w++) {
sendstate[w]=true;
}
}
void key(int k) { //key is controlled by buttons
//println((frameCount - playStart)+",");
for (int i=0; i<83; i++) { //judgement
if (abs(frameCount-beats[i])<10&&ball[i]==k&&!flaga[i]) {
success=true; //if press button correctly, the heart become bigger
//println("success=true");
premil=millis();
a=a+1;
flaga[i]=true;
}
/*if (success==false) { //draw cross
fill(255, 0, 0);
push();
translate(400, 400);
rotate(PI/4);
rect(0, 0, 400, 100);
pop();
push();
translate(400, 400);
rotate(-PI/4);
rect(0, 0, 400, 100);
pop();
}*/
}
}
void draw() {
background(0);
fill(255, 100);
ellipse(400, 400, 180, 180);
fill(0);
ellipse(400, 400, 160, 160);
rectMode(CENTER);
drawHeart(); //heart
// we add a text
int now = millis() - playStart;
if (now > 1000 && now < 1500) {
fill(255);
text("3", height/2, width/4);
}
if (now > 1500 && now < 2000) {
fill(255);
text("2", height/2, width/4);
}
if (now > 2000 && now < 2500) {
fill(255);
text("1", height/2, width/4);
}
if (now > 2500 && now < 3000) {
fill(255);
text("Start!", height/2, width/4);
}
if (now > 17000 && now < 18500) {
fill(255, 255, 255);
text("It will be faster", height/2, width/4);
}
if (now > 48500 && now < 49500) {
fill(255, 255, 255);
text("Even more faster", height/2, width/4);
}
push();
textSize(32);
fill(255);
text("Your score is:", 125, 65);
text(a, 100, 100);
text("/83", 135, 100);
pop();
if (serialRecord.read()) {
value = serialRecord.get();
key(value);
//println(frameCount);
}
if (success==true) {
if (millis()-premil<300) {
fill(255);
ellipse(400, 400, 180, 180);
fill(0);
ellipse(400, 400, 160, 160);
drawBigHeart();
} else {
success=false;
}
} else drawHeart();
//ball
int cfc=frameCount;
for (int i=max(0, num-2); i<83; i++) {
if (cfc-beats[i]>-31&&!flag[i]) {
fill(255);
if (abs(cfc-(beats[i]-31))<2) {
xpos[i]=800*floor(random(0, 2));
ypos[i]=800*floor(random(0, 2));
if (xpos[i]==0&&ypos[i]==0) {
ball[i]=1;
if (sendstate[i]==true) {
serialRecord.values[0] = 1;
serialRecord.send();
//println("1");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==800&&ypos[i]==0) {
ball[i]=2;
if (sendstate[i]==true) {
serialRecord.values[0] = 2;
serialRecord.send();
//println("2");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==0&&ypos[i]==800) {
ball[i]=3;
if (sendstate[i]==true) {
serialRecord.values[0] = 3;
serialRecord.send();
//println("3");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==800&&ypos[i]==800) {
ball[i]=4;
if (sendstate[i]==true) {
serialRecord.values[0] = 4;
serialRecord.send();
//println("4");
sendstate[i]=false;
num++;
}
}
}
if (xpos[i]<400 && ypos[i]<400) {
xpos[i] = xpos[i] + 10;
ypos[i]= ypos[i] + 10;
}
if (xpos[i]>400&&ypos[i]<400) {
xpos[i] = xpos[i]-10;
ypos[i]=ypos[i]+10;
}
if (xpos[i]>400&&ypos[i]>400) {
xpos[i] = xpos[i]-10;
ypos[i]=ypos[i]-10;
}
if (xpos[i]<400&&ypos[i]>400) {
xpos[i] = xpos[i]+10;
ypos[i]=ypos[i]-10;
}
noStroke();
if (ypos[i]<=400) {
fill(255, 0, 0);
}
if (ypos[i]>=400) {
fill(#30CEFC);
}
if (1024<=cfc&&cfc<=1072||1526<=cfc&&cfc<=1578||2100<=cfc&&cfc<=2148||2600<=cfc&&cfc<=2648) {
fill(#FFF534);
}
star(xpos[i], ypos[i], 40, 20, 5);
if (xpos[i]==400&&ypos[i]==400) {
flag[i]=true;
serialRecord.values[0] = 5;
serialRecord.send();
}
}
}
if(millis()>58500){
clear();
fill(255);
text("Your score is:", height/2,width/2-50);
text(a, height/2-50,width/2+50);
text("/83", height/2+50,width/2+50);
}
}
Harder:
import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;
Serial serialPort;
SerialRecord serialRecord;
SoundFile sound;
boolean[] flag=new boolean[74];
boolean[] flaga=new boolean[74];
boolean[] sendstate=new boolean[74];
int playStart;
boolean success=false;
int premil=0;
int value=0;
int num=0;
int[]beats= {
186,
210,
232,
255,
279,
302,
324,
348,
373,
388,
419,
435,
464,
487,
509,
532,
557,
579,
602,
626,
650,
672,
695,
720,
744,
761,
789,
804,
833,
857,
880,
904,
926,
950,
973,
997,
1022,
1045,
1068,
1090,
1113,
1136,
1159,
1185,
1208,
1231,
1255,
1274,
1300,
1323,
1346,
1368,
1392,
1417,
1439,
1463,
1484,
1502,
1531,
1548,
1576,
1601,
1627,
1650,
1672,
1690,
1719,
1735,
1762,
1786,
1811,
1835,
1858,
1951,
};
int[] xpos=new int[74];
int[] ypos=new int[74];
int x;
int y;
int[] ball=new int[74];
int a=0;
void star(float x1, float y1, float radius1, float radius2, int npoints) { //ball=start
float angle = TWO_PI / npoints;
float halfAngle = angle/2.0;
beginShape();
for (float a = 0; a < TWO_PI; a += angle) {
float sx = x1 + cos(a) * radius2;
float sy = y1 + sin(a) * radius2;
vertex(sx, sy);
sx = x1 + cos(a+halfAngle) * radius1;
sy = y1 + sin(a+halfAngle) * radius1;
vertex(sx, sy);
}
endShape(CLOSE);
}
void keyPressed() {
println(frameCount+",");
//println(millis());
}
void drawBigHeart() { // if player press the button correctly, the heart becomes bigheart
fill(255, 0, 0);
rectMode(CENTER);
rect(height/2, width/2, 10, 10);
rect(height/2-10, width/2, 10, 10);
rect(height/2+10, width/2, 10, 10);
rect(height/2-20, width/2, 10, 10);
rect(height/2+20, width/2, 10, 10);
rect(height/2-30, width/2, 10, 10);
rect(height/2+30, width/2, 10, 10);
rect(height/2+10, width/2-10, 10, 10);
rect(height/2-10, width/2-10, 10, 10);
rect(height/2+20, width/2-10, 10, 10);
rect(height/2-20, width/2-10, 10, 10);
rect(height/2-15, width/2-20, 10, 10);
rect(height/2+15, width/2-20, 10, 10);
rect(height/2, width/2+10, 10, 10);
rect(height/2-10, width/2+10, 10, 10);
rect(height/2+10, width/2+10, 10, 10);
rect(height/2-20, width/2+10, 10, 10);
rect(height/2+20, width/2+10, 10, 10);
rect(height/2, width/2+20, 10, 10);
rect(height/2-10, width/2+20, 10, 10);
rect(height/2+10, width/2+20, 10, 10);
rect(height/2, width/2+30, 10, 10);
}
void drawHeart() { // normal heart
fill(255, 0, 0);
rect(height/2, width/2, 10, 10);
rect(height/2-10, width/2, 10, 10);
rect(height/2+10, width/2, 10, 10);
rect(height/2-20, width/2, 10, 10);
rect(height/2+20, width/2, 10, 10);
rect(height/2+10, width/2-10, 10, 10);
rect(height/2-10, width/2-10, 10, 10);
rect(height/2, width/2+10, 10, 10);
rect(height/2-10, width/2+10, 10, 10);
rect(height/2+10, width/2+10, 10, 10);
rect(height/2, width/2+20, 10, 10);
}
void setup() {
size(800, 800);
frameRate(60);
delay(1000);
sound = new SoundFile(this, "t.wav");
playStart = millis();
sound.play();
noStroke(); // the text
textSize(90);
textAlign(CENTER, CENTER);
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 115200);
serialRecord = new SerialRecord(this, serialPort, 1);
for (int w=0; w<74; w++) {
sendstate[w]=true;
}
}
void key(int k) { //key is controlled by buttons
//println((frameCount - playStart)+",");
for (int i=0; i<74; i++) { //judgement
if (abs(frameCount-beats[i])<10&&ball[i]==k&&!flaga[i]) {
success=true; //if press button correctly, the heart become bigger
//println("success=true");
premil=millis();
a=a+1;
flaga[i]=true;
}
}
}
void draw() {
background(0);
fill(255, 100);
ellipse(400, 400, 180, 180);
fill(0);
ellipse(400, 400, 160, 160);
rectMode(CENTER);
drawHeart(); //heart
// we add a text
int now = millis() - playStart;
if (now > 1000 && now < 1500) {
fill(255);
text("3", height/2, width/4);
}
if (now > 1500 && now < 2000) {
fill(255);
text("2", height/2, width/4);
}
if (now > 2000 && now < 2500) {
fill(255);
text("1", height/2, width/4);
}
if (now > 2500 && now < 3000) {
fill(255);
text("Start!", height/2, width/4);
}
push();
textSize(32);
fill(255);
text("Your score is:", 125, 65);
text(a, 100, 100);
text("/74", 135, 100);
pop();
if (serialRecord.read()) {
value = serialRecord.get();
key(value);
//println(frameCount);
}
if (success==true) {
if (millis()-premil<300) {
fill(255);
ellipse(400, 400, 180, 180);
fill(0);
ellipse(400, 400, 160, 160);
drawBigHeart();
} else {
success=false;
}
} else drawHeart();
//ball
int cfc=frameCount;
for (int i=max(0, num-2); i<74; i++) {
if (cfc-beats[i]>-31&&!flag[i]) {
fill(255);
if (abs(cfc-(beats[i]-31))<2) {
xpos[i]=800*floor(random(0, 2));
ypos[i]=800*floor(random(0, 2));
if (xpos[i]==0&&ypos[i]==0) {
ball[i]=1;
if (sendstate[i]==true) {
serialRecord.values[0] = 1;
serialRecord.send();
//println("1");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==800&&ypos[i]==0) {
ball[i]=2;
if (sendstate[i]==true) {
serialRecord.values[0] = 2;
serialRecord.send();
//println("2");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==0&&ypos[i]==800) {
ball[i]=3;
if (sendstate[i]==true) {
serialRecord.values[0] = 3;
serialRecord.send();
//println("3");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==800&&ypos[i]==800) {
ball[i]=4;
if (sendstate[i]==true) {
serialRecord.values[0] = 4;
serialRecord.send();
//println("4");
sendstate[i]=false;
num++;
}
}
}
if (xpos[i]<400 && ypos[i]<400) {
xpos[i] = xpos[i] + 10;
ypos[i]= ypos[i] + 10;
}
if (xpos[i]>400&&ypos[i]<400) {
xpos[i] = xpos[i]-10;
ypos[i]=ypos[i]+10;
}
if (xpos[i]>400&&ypos[i]>400) {
xpos[i] = xpos[i]-10;
ypos[i]=ypos[i]-10;
}
if (xpos[i]<400&&ypos[i]>400) {
xpos[i] = xpos[i]+10;
ypos[i]=ypos[i]-10;
}
noStroke();
if (ypos[i]<=400) {
fill(255, 0, 0);
}
if (ypos[i]>=400) {
fill(#30CEFC);
}
star(xpos[i], ypos[i], 40, 20, 5);
if (xpos[i]==400&&ypos[i]==400) {
flag[i]=true;
serialRecord.values[0] = 5;
serialRecord.send();
}
}
}
if(frameCount>2040){
clear();
fill(255);
text("Your score is:", height/2,width/2-50);
text(a, height/2-50,width/2+50);
text("/74", height/2+50,width/2+50);
}
}
Wendy did the Neopixel strip part. In this part, the Neopixel strip changes its color according to the loudness of the music. We also add a microphone function code in the Processing to sense and analyze the frequency of the whistle, so that every time we blow the whistle, the LEDs will change their color obviously, which looks very cool.
The circuit:
The code: (Arduino)
#include "SerialRecord.h"
#include <FastLED.h>
#define NUM_LEDS 60 // How many leds in your strip?
#define DATA_PINA 3
#define DATA_PINB 4
#define DATA_PINC 5
// Which pin are you connecting Arduino to Data In?
CRGB leds[NUM_LEDS];
// Change this number to the number of values you want to receive
SerialRecord reader(4);
void setup ()
{
Serial.begin(9600);
FastLED.addLeds<NEOPIXEL, DATA_PINA>(leds, NUM_LEDS); // Initialize
FastLED.addLeds<NEOPIXEL, DATA_PINB>(leds, NUM_LEDS);
FastLED.addLeds<NEOPIXEL, DATA_PINC>(leds, NUM_LEDS);
FastLED.setBrightness(10); // BEWARE: external power for full (255)
}
void loop() {
if (reader.read()) {
int n = reader[0];
int r = reader[1];
int g = reader[2];
int b = reader[3];
leds[reader[0]] = CRGB(reader[1], reader[2], reader[3]); // Prepare the color information using CRGB( Red, Green, Blue
FastLED.show(); // Pass the information of color to the LED
}
}
(The processing: For different songs, we just change the music name)
import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;
import processing.sound.*;
// declare an AudioIn object
AudioIn microphone;
// declare an Frequency analysis object to detect the frequencies in a sound
FFT freqAnalysis;
// declare a variable for the amount of frequencies to analyze
// should be a multiple of 64 for best results
int frequencies = 1024;
// Define the frequencies wanted for our visualization. Above that treshold frequencies are rarely atteigned and stay flat.
int freqWanted = 128;
// declare an array to store the frequency anlysis results in
float[] spectrum = new float[freqWanted];
// Declare a drawing variable for calculating the width of the
float circleWidth;
float a;
Serial serialPort;
SerialRecord serialRecord;
SoundFile sample;
Amplitude analysis;
int NUM = 60; //amount of pixels
int[] r = new int[NUM]; //red of each tile
int[] g = new int[NUM]; //red of each tile
int[] b = new int[NUM]; //red of each tile
int startTime;
void setup() {
fullScreen();
// load and play a sound file in a loop
sample = new SoundFile(this, "t.wav");
sample.loop();
// create the Amplitude analysis object
analysis = new Amplitude(this);
// analyze the playing sound file
analysis.input(sample);
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 9600);
serialRecord = new SerialRecord(this, serialPort, 4);
serialRecord.logToCanvas(false);
rectMode(CENTER);
startTime = millis();
circleWidth = width/float(freqWanted);
// create the AudioIn object and select the mic as the input stream
microphone = new AudioIn(this, 0);
// start the mic input without routing it to the speakers
microphone.start();
// create the Frequency analysis object and tell it how many frequencies to analyze
freqAnalysis = new FFT(this, frequencies);
// use the microphone as the input for the analysis
freqAnalysis.input(microphone);
}
void draw() {
println(analysis.analyze());
long t = millis() - startTime;
if (t < 1500) {
background(125, 255, 125);
} else if ((t>=1500) && t<3000) {
background(255);
} else {
background(55);
}
noStroke();
fill(255, 0, 150);
// analyze the audio for its volume level
float volume = analysis.analyze();
// volume is a number between 0.0 and 1.0
// map the volume value to a useful scale
float diameter = map(volume, 0, 1, 0, width);
float light = map(volume, 0, 1, 0, 255);
//int n = int(random(60));
a = random(0, 255);
int n = frameCount % 60;
r[n] = floor(light/2);
g[n] = floor(255-light);
b[n] = floor(a);
serialRecord.values[0] = n; // which pixel we change (0-59)
serialRecord.values[1] = r[n]; // how much red (0-255)
serialRecord.values[2] = g[n]; // how much green (0-255)
serialRecord.values[3] = b[n]; // how much blue (0-255)
serialRecord.send();
// draw a circle based on the microphone amplitude (volume)
circle(width/2, height/2, diameter);
// analyze the frequency spectrum and store it in the array
freqAnalysis.analyze(spectrum);
//printArray(spectrum);
background(0);
fill(200, 0, 100, 150);
noStroke();
float loudest = 0;
int loudestIdx = 0;
for (int i=0; i<freqWanted; i++) {
if (spectrum[i] > loudest) {
loudest = spectrum[i];
loudestIdx = i;
}
}
if ((loudestIdx == 19 || loudestIdx == 20) && loudest > 0.02) {
background(255,255,255);
int n1 = frameCount % 60;
r[n1] = floor(255);
g[n1] = floor(255);
b[n1] = floor(255);
serialRecord.values[0] = n1; // which pixel we change (0-59)
serialRecord.values[1] = r[n1]; // how much red (0-255)
serialRecord.values[2] = g[n1]; // how much green (0-255)
serialRecord.values[3] = b[n1]; // how much blue (0-255)// how much blue (0-255)
serialRecord.send(); // send it!
}
if (loudest > 0.01) {
println(loudestIdx + " @ " + loudest);
}
// draw circles based on the values stored in the spectrum array
// adjust the scale variable as needed
float scale = 600;
for (int i=0; i<freqWanted; i++) {
circle(i*circleWidth, height/2, spectrum[i]*scale);
}
}
The music we use:
(The music of the harder part is “.wav” and cannot be attached)
The video:
D. CONCLUSIONS:
The goal of our project is to make an interactive rhythm game. I think our project has achieved this goal from the player’s playing process and feedback. The player presses the button according to the rhythm of the music and can adjust their own speed according to the output (whether they get a score/hit correctly), which constitutes a kind of interaction. And my previous definition of interaction is between the user and artifact, the user gives an input to the artifact and receives an output from the artifact, and this output can change the behavior of the user. So this project aligns with my definition of interaction.
If I have more time, I will find a better way to program the Neopixel board, which can run more LED movements at the same time, and also make more changes, such as LED color changes, making specified patterns, etc.
The programming part of this project was difficult, but I learned more programming languages, such as boolean, using defining a flag to make a true/false judgment for start/stop a loop, and I also explored and learned some Neopixel usage by myself. In fact, in the beginning, I couldn’t even imagine that I could make a rhythm game by myself. From this project, although it took a lot of time and effort, I gained a lot, and I feel a sense of accomplishment when I look at the final work! The code we finished has some versatility, if you want to add another song to play, you can just change the beat point and play it. I also understood and learned some new functions of Processing more deeply, and learned how to make the Processing and Arduino deliver value and cooperate with each other. And through this project, I also have a deeper understanding of the rhythm game, I understand some of its logic and the way it is written, and also broaden my understanding of interaction.
E. Some other interesting projects…
