Floating Keys Documentation
Our project, Floating Keys, aimed to create a combined experience of Guitar Hero and Piano Tiles. Both games use instruments to create a fast pace, high-score aiming game experience. And we wanted to incorporate that same idea in a more physically interactive way.
This documentation will be split into three parts: Assembling the sensors on the Arduino, Designing the physical model of the project, and coding it.
Before you read the documentation here is how the project worked:
Assembling the Sensors
This part was the simplest and supposed to be the quickest since it was just performing skills we learned earlier in the semester. We had four infrared distance sensors, so we connected each’s power and ground to the corresponding places on the breadboard, and each analog line into A0, A1, A2, and A3. It looked like:
Assembling this was supposed to be the quickest part, but it was not as we spent around an hour asking teaching assistants and other professors on what was wrong. The problem was that the Arduino, when connected to the computer, did not show up in the “ports” list, and the green light would turn off after 20 seconds. Also, the Arduino was really hot. Turns out for one of the sensors, the ground and power cables were swapped. This was a very small, easy-to-fix mistake that ended up costing us an hour.
Furthermore, on our final presentation day (Thursday, December 12) the Arduino was not connecting to the computer for the same reason: the ground and power lines were swapped for one sensor. It was working on Tuesday, but suddenly not on Thursday and I got really worried questioning if the Arduino broke or the computer had issues. Turns out, it was just a simple mistake.
The coding part for the Arduino was fairly simple, as we just added “int sensor (1-4) = analogRead (each’s analog input). Next, we mapped each sensor value from 0-1023 to 0-60. This was because we wanted a smaller range of numbers to fit in the processing conditionals (to correspond in the game).
This was our code for Arduino:
// IMA NYU Shanghai
// Interaction Lab
// For sending multiple values from Arduino to Processing
void setup() {
Serial.begin(9600);
}
void loop() {
int sensor1 = analogRead(A0);
int sensor2 = analogRead(A1);
int sensor3 = analogRead(A2);
int sensor4 = analogRead(A3);
sensor1 = map(sensor1, 0, 1023, 0, 60);
sensor2 = map(sensor2, 0, 1023, 0, 60);
sensor3 = map(sensor3, 0, 1023, 0, 60);
sensor4 = map(sensor4, 0, 1023, 0, 60);
// keep this format
Serial.print(sensor1);
Serial.print(",");
Serial.print(sensor2);
Serial.print(",");
Serial.print(sensor3);
Serial.print(",");
Serial.print(sensor4);
// Serial.print(","); // put comma between sensor values
// Serial.print(sensor2);
Serial.println(); // add linefeed after sending the last sensor value
// too fast communication might cause some latency in Processing
// this delay resolves the issue.
delay(100);
}
Physical Design
This was what we discussed first since we thought the fabrication lab would be filled throughout the week after Thanksgiving break. We started out with different ideas and spend around half an hour debating how the project should look. Our first idea was to have a really large box (which was around 125×60 cm). However, this project would use 10 pieces of wood, which we and the TA thought were too much. We later both agreed on similar ideas for how to revise the project design, which was to have four individual boxes on a platform. Each box was separated with a wall, which was what Eric suggested in class. We drew the dimensions of how it should look like on a piece of paper:
Our original idea and sketches:
And this is our design of each box on makercase.com:
We had holes on the back of each box to allow the sensor’s wires in, and it was an open-top box because we wanted to glue the top of the box (without finger joints) to the platform. The bottom of the box became the top of the box since it had finger joints and a lid.
Fay later added the Chrismas and holiday decorations onto the top of the box, since Christmas is coming.
Our main challenge here was that the assistant told us we were using too much material. The first time we went, which was the Monday after Thanksgiving, we needed 10 pieces of wood, which even we thought was a lot. So the TA gave us ideas to save space and told us to come back after we finished our blueprints on Adobe Illustrator.
So we redesigned everything to only use six pieces of material, including the platform which took two. And that was still too much, so we had to use our back-up plan of just not laser-cutting the platform, and only cut the boxes. We tried to portray what our idea looked like and how important the platform was to our project, but six pieces still seemed like a lot so we had no other choice. So instead of the wood platform, we found a box in the cardboard room. The length of the box was more than half of our projects, which was perfect since we could use both sides of the box to cover the whole length. Using the boxes, we glued the top side of them to the cardboard, and glued the rest of the finger joints, other than the lid, to make the box stable. After that, we just put the Christmas designed bottom (or now the top) of the boxes on, and the sensors through the hole in the back. Our physical design looked like:
Coding
This was by far the most time-consuming and complicated part of the project. We were helped by lots of people, including professors, TAs, and multiple friends (some which I woke up late at night). In my first attempt at the code, I created a grid using a for loop, and I tried to have the grid move down one every-time a key was pressed.
ArrayList<Integer> blocks = new ArrayList<Integer>();
int a=0;
int t=0;
int y;
void setup(){
for( int i =0; i<100; i++){
blocks.add(int(random(0,4)));
}
System.out.println(blocks);
size(400,400);
}
void draw(){
for(int m = 0; m<8; m++){
for(int n = 0; n<4; n++){
noFill();
rect(n*30+150,m*30,30,30);
}
for(int j = 0; j<100; j++){
a = blocks.get(j);
//println(a);
fill(255,0,0);
y = 240-j*30;
if(key =='1'){
y= y+30;
}
rect(a*30+150,y,30,30);
}
}
}
That is the code and this was what happened every time I pressed the key 1.
We couldn’t get the tiles to move down continuously after changing it for around 2 hours. I asked two TAs in the lab, but they were busy with other students. So I thought it would be better if I asked a friend to help me code the concept. I didn’t like the feeling of only using someone else’s code for the main part of my project, so I changed and added a lot to the code.
We added a title screen so the players would not be immediately immersed in the game, a time bar that decreased to the left (with Eric’s help), and modified the score code adding in a high score feature. The high score feature would be there until the game was closed. For the title screen, we added snow that fell down continuously, which was used by the class we created. And to combine the title screen and the game, we changed the whole void draw into a conditional, using frameCount to change from the title screen into the game. I will copy and paste the code at the very end of the documentation.
Our biggest challenge was connecting the sensors to the piano tiles on the game. Before the connection, the game was using a modified version of the keyPressed function. And the main problem with sensors was, that it did not work when using for example sensorValues[0] > 15, inside the keyPressed conditionals. Eric helped us by changing the keyPressed function into sensorPressed, and calling out sensorPressed in the void draw. That allowed the sensor to press the tiles. However, ever time the sensor sensed something within its distance, the tiles would keep getting pressed nonstop. So we added if conditionals to the sensorPressed function saying that if the sensorValues are less than 15 (if they were greater than 15 the tile would be pressed) basically don’t trigger the tile. This can be seen in the code later on under void sensorPressed.
Our second challenge, which kind of ended up hurting our presentation was the inclusion of the jingle bells melody. On Tuesday, we thought we would be picked, but we weren’t. And we just had each tile play a music note, so the most left tile would play the C note, and the next one playing D, etc. We wanted a melody to match our theme of the project, which was Christmas time. So we added in the notes C, D, E, F, G and incorporated them using the Sound library. We created an array and an if function that played the next note in the array. However, when it came to the end of the array the game would crash. It was a very simple fix, but I thought it would be overly complicated and I ran out of time to fix it on Thursday, which was presentation day (I also was really busy with presentations in other classes on Wednesday). Eric helped us fix it after our presentations by adding a conditional, so that when the array list ended start back at spot 0 in the array.
Although our project was not perfected on the presentation day, I think in conclusion the whole process was very creative and fun working as a team. We created our vision of the project, with the help of Eric and many of our other friends. I think the project had many challenges and our way of finding how to surpass them or go around them was very creative, now that I think of it.
———————————PROCESSING CODE————————————
Snow Class:
class Snow {
float x, y;
int c;
float spdX, spdY;
float r;
boolean isDone;
Snow() {
r = random(5,8);
spdX = random(-2,2);
spdY = 5;
c = int(random(30,255));
x = random(width);
y = 0;
isDone = false;
}
void checkEdges(){
if(x<0 || y>height){
isDone = true;
}
}
void move() {
x += spdX;
y += spdY;
}
void display() {
fill(255,c);
ellipse(x,y,r,r);
}
}
Piano Sketch
Snow theboi;
int hi=0;
ArrayList<Snow> snowList = new ArrayList<Snow>();
PImage img;
import processing.sound.*;
SoundFile sound1;
SoundFile sound2;
SoundFile sound3;
SoundFile sound4;
SoundFile sound5;
SoundFile soundlist[] = {sound1, sound2, sound3, sound4, sound5};
ArrayList<SoundFile> soundList = new ArrayList<SoundFile>();
import processing.serial.*;
String myString = null;
Serial myPort;
int NUM_OF_VALUES = 4;
int[] sensorValues;
int blockSize = 125;
boolean[][] pTiles;
int score = 0;
int highScore = 0;
boolean record = false;
float time = 40;
float startTime = 0;
int gameMode = 0;//0=not start 1=start 2=over
boolean keyCTrigger = false;
boolean keyDTrigger = false;
boolean keyETrigger = false;
boolean keyFTrigger = false;
int songList[] = {3,3,3,3,3,3,3,5,1,2,3,4,4,4,4,3,3,5,5,3,2,1,3,3,3,3,3,3,3,5,1,2,3,4,4,4,4,3,3,5,5,3,2,1,3,3,3,3,3,3,3,5,1,2,3,4,4,4,4,3,3,5,5,3,2,1};
int a = 0;
void setup() {
fullScreen();
img = loadImage("floatingKeys.jpg");
sound1 = new SoundFile(this, "Cnote.mp3");
sound2 = new SoundFile(this, "Dnote.mp3");
sound3 = new SoundFile(this, "Enote.mp3");
sound4 = new SoundFile(this, "Fnote.mp3");
sound5 = new SoundFile(this, "Gnote.mp3");
soundList.add(sound1);
soundList.add(sound2);
soundList.add(sound3);
soundList.add(sound4);
soundList.add(sound5);
setupSerial();
pTiles = new boolean[4][ height - 50 / blockSize + 1];
for (int i = 0; i < height - 50 / blockSize + 1; i++) {
int t = int(random(0, 4));
for (int j = 0; j < 4; j++) {
if (j == t) {
pTiles[j][i] = true;
} else {
pTiles[j][i] = false;
}
}
}
}
void draw() {
snowList.add(new Snow());
if(frameCount<200){
background(0);
textSize(35);
fill(255);
text("Floating Keys Loading!",width/2 - 155,height/2,400,400);
for(int i = 0; i<snowList.size(); i++) {
Snow temp = snowList.get(i);
temp.display();
temp.checkEdges();
temp.move();
}
for(int j = snowList.size()-1; j>=0 ; j--){
Snow p = snowList.get(j);
if(p.isDone){
snowList.remove(j);
}
}
}
else {
background(255);
fill(255);
fill(50, 205, 50);
sensorPressed();
design();
for (int i = 0; i < 4; i++) {
for (int j = 0; j < height - 50/blockSize+1; j++) {
if (pTiles[i][j]) {
fill(177,18,38);
} else {
noFill();
}
rect(i *100 + width/3.275, height - 50 - blockSize - j*blockSize, 100, blockSize);
}
}
fill(0);
textSize(18);
text(score, width - 250, height/2);
text("Current Score:", width - 375, height/2);
if(record){
fill(255,0,0);
}else{
fill(128);
}
text("High Score:", width - 350, height/2 + 50);
text(highScore,width - 250,height/2 + 50);
//text("-10", 50, 750);
fill(0);
if(gameMode == 0){
textSize(35);
text("Quick! \n Put your hand \n in the box!", width/4 - 250, height/2 - 100);
}else if(gameMode == 1){
text((startTime + time * 1000 -millis())/1000,50,750);
rect(300,150 + 675 - 675 * ((startTime + time * 1000 -millis())/1000) / time,40,675 * ((startTime + time * 1000 -millis())/1000) / time);
if(startTime + time * 1000 -millis() <= 0){
gameMode = 2;
if(highScore < score){
record = true;
highScore = score;
}
}
}else if(gameMode == 2){
text("Game Over",50,750);
text("Please Press R To Start Over",width / 2 + 280, height - 145);
}
}
}
void playSound() {
soundList.get( songList[a]-1 ).play();
a++;
}
void sensorPressed() {
fill(0, 50);
updateSerial();
// printArray(sensorValues);
if (gameMode != 2) {
if (sensorValues [0] >= 15 && keyCTrigger) {
keyCTrigger = false;
rect(100, height - 50, 100, 25);
pushTile(0);
playSound();
}
if (sensorValues [1] >= 15 && keyDTrigger) {
keyDTrigger = false;
rect(200, height - 50, 100, 25);
pushTile(1);
playSound();
}
if (sensorValues [2] >= 15 && keyETrigger) {
keyETrigger = false;
rect(300, height - 50, 100, 25);
pushTile(2);
playSound();
}
if (sensorValues [3] >= 15 && keyFTrigger) {
keyFTrigger = false;
rect(400, height - 50, 100, 25);
pushTile(3);
playSound();
}
if(sensorValues [0] < 15 && keyCTrigger==false){
keyCTrigger=true;
}
if(sensorValues [1] < 15 && keyDTrigger==false){
keyDTrigger=true;
}
if(sensorValues [2] < 15 && keyETrigger==false){
keyETrigger=true;
}
if(sensorValues [3] < 15 && keyFTrigger==false){
keyFTrigger=true;
}
}
}
void pushTile(int n) {
if (gameMode == 0) {
gameMode = 1;
startTime = millis();
}
if (pTiles[n][0]) {
step();
score++;
} else {
background(255, 0, 0);
score -= 1;
}
}
void keyPressed() {
if (key == 'r'){
reset();
}
}
void design() {
image(img,width/1.5,height/4);
img.resize(400,250);
fill(0,255,0);
rect(width/2 - 250, height - 50, 100, 25);
fill(255);
rect(width/2 - 230, height - 50, 60, 15);
fill(255, 255, 0);
rect(width/2 - 150, height - 50, 100, 25);
fill(255);
rect(width/2 - 130, height - 50, 60, 15);
fill(255, 140, 0);
rect(width/2 - 50, height - 50, 100, 25);
fill(255);
rect(width/2 - 30, height - 50, 60, 15);
fill(50, 82, 123);
rect(width/2 + 50, height - 50, 100, 25);
fill(255);
rect(width/2 + 70, height - 50, 60, 15);
line(width/2 - 250, 0, width/2 - 250, height-50);
line(width/2 - 150, 0, width/2 - 150, height-50);
line(width/2 - 50, 0, width/2 - 50, height-50);
line(width/2 + 50, 0, width/2 + 50, height-50);
line(width/2 + 150, 0, width/2 + 150, height-50);
}
void step() {
for (int i = 1; i < height - 50 / blockSize + 1; i++) {
for (int j = 0; j < 4; j++) {
pTiles[j][i-1] = pTiles[j][i];
}
}
int t = int(random(0, 4));
for (int j = 0; j < 4; j++) {
if (j == t) {
pTiles[j][height - 50 / blockSize] = true;
} else {
pTiles[j][height - 50 / blockSize] = false;
}
}
}
void reset() {
score = 0;
gameMode = 0;
record = false;
for (int i = 0; i < height - 50 / blockSize + 1; i++) {
int t = int(random(0, 4));
for (int j = 0; j < 4; j++) {
if (j == t) {
pTiles[j][i] = true;
} else {
pTiles[j][i] = false;
}
}
}
}
void setupSerial() {
// printArray(Serial.list());
myPort = new Serial(this, Serial.list()[3], 9600);
myPort.clear();
// Throw out the first reading,
// in case we started reading in the middle of a string from the sender.
myString = myPort.readStringUntil( 10 ); // 10 = '\n' Linefeed in ASCII
myString = null;
sensorValues = new int[NUM_OF_VALUES];
}
void updateSerial() {
while (myPort.available() > 0) {
myString = myPort.readStringUntil( 10 ); // 10 = '\n' Linefeed in ASCII
if (myString != null) {
String[] serialInArray = split(trim(myString), ",");
if (serialInArray.length == NUM_OF_VALUES) {
for (int i=0; i<serialInArray.length; i++) {
sensorValues[i] = int(serialInArray[i]);
}
}
}
}
}