1. Soaring Over the Sound City – Junhao Zhu & Jiaxi Zhang – Gottfried Haider
Here is a picture of our project.
2. Conception And Design
In this project, we tried to experiment with the methods of players interacting with a game. We conducted research on the forms of interactivity that go beyond the monotonous keyboards and mice input. We found some amateur projects from profound companies including Kinect from Microsoft, a project that features multiple sensors for gesture and motion recognition, which add immersion for users playing games;
From Reliance Digital [@reliancedigital]. (2011, December 20). Xbox 360 Kinect. Youtube. https://www.youtube.com/watch?v=oyjNqksc-m8
and Labo, a toys-to-life concept developed by Nintendo. This project encourages users to invent new ways of interacting with their gaming devices with cardboards and imaginations.
Create new ways to play with Nintendo Labo! (n.d.). Nintendo of Europe AG. Retrieved May 8, 2024, from https://www.nintendo.com/en-gb/Nintendo-Labo/Nintendo-Labo-1328637.html
In conventional game interactions, players usually passively receive feedbacks from the game itself and then give out their responses. In comparison, the two projects above actively inviting the players to contribute to the interaction process. This gives us the thought of utilizing the user’s voice as the medium of interaction.
After making the decision of designing a voice-based multiplayer games, we looked up to possible applications of sounds. “YASUHATI / With your voice!” caught our attention.
YASUHATI / With your voice! (n.d.). Google.com. Retrieved May 9, 2024, from https://play.google.com/store/apps/details?id=jp.ne.freem.YASUHATI&hl=en&gl=US
Above is a game developed by Freedomcrow, an indie game developer in Japan. Players can bypass the need for traditional, complex control schemes. Instead, the game simplifies interaction by allowing players to use their voices to direct a musical note character. As the note navigates through various levels, players control its movements by altering the pitch and volume of their voices, enabling it to jump across a series of platforms and avoid obstacles.
Based on the previous researches, we developed our project of “Soaring Over the Sound City”. In this project, two players will act as the character of “Navigator” and “Builder”. The “Builder” will build the scene with his or her voice, the heights of all the blocks will be decided based on their vocal inputs. The “Navigator” will try to navigate through this vocally-built landscape with a controller. The “Builder” will have certain kinds of special skills to create obstacles in the scene and make the game more challenging.
Considering the game physics, there are two decisions that needed us to address in the design. The first one is how to interact with this game with sound, and the second one is finalizing the gameplay.
Sound can be measured with verious criterias including velocity, pitch, energy and loudness. In our initial design, we want to design a “pitch” based game to encourage users play with music pieces by tuning the pitchs and pausing the pieces. However, the existing pitch detecting algorithms in Processing is not powerful enough to support our idea, and in addition, lots of players are too shy to sing out in their own voice. Instead, we change it into a “loudness” control game. It enables us to develop a more inclusive and fun gaming experience.
For the gameplay, there were lots of existing games that inspired us. Besides “YASUHATI / With your voice!”, which is mentioned above, another game called “Line Rider” was also included in our early gameplay design. Below is a map design in Line Rider.
DoodleChaos [@DoodleChaos]. (2017, October 7). Line Rider – Mountain King. Youtube. https://www.youtube.com/watch?v=RIz3klPET3o
We originally proposed that the “builders” could draw these lines by changing their pitches, and generate blanks if they stop making sounds. The “navigators” need to control the “rider” to navigate through the scene with the controllers, and the controllers would have four functions: rotate clockwise, rotate counterclockwise, jump up and dive down.
Here is an early demo with the pitchs from a piece of music. This demo indicated a major drawback in the original design: the lines drawed with pitchs were too dense that the “rider” were impossible to ride on.
This is the user test version of “Soaring Over the Sound City.”
In the user testing, we recieved positive feedbacks in the gameplay and the multiple-player ideas. Testers also gave us suggestions on the gaming experience and guidance. They hope we can control the difficulties of the game as in certain circumstances, the blocks will be too close to the boundaries for the “navigator” to travel through. Another improvement that we adopted was a better instruction in the tutorial. In this version we only implemented some simple guidances on the functions of each key, and the players might be confused about the actual gameplay if we did not inform them. Our change is adding a full tutorial section for users to freely explore the interactions assisted by our minimal guidances.
Below is the video of us interacting with our project:
3. Fabrication And Production
Our project is a coding based project, and our fabrication is focused on the controller with a 3D printer.
In this project, our objective was to pay tribute to retro arcade games, which hold a special place in the history of gaming and in the hearts of many enthusiasts. Its nostalgia looking fits the 8-bit theme inside our game comparing to other designs like a modernized joystick. By building a controller that mimics the style and functionality of retro arcade game controllers, we aim to capture the essence of classic gaming experiences.
Below are the sketches for the arcade controller:
Case
Cover
3D-Printed Cover
When assembling the controller, we started by integrating buttons into the pre-drilled holes on the cover. Each button is carefully aligned with a corresponding hole on the cover, ensuring a snug and secure fit. This prevents any wobbling or misalignment during gameplay, which is essential for precise control.
After placing the buttons into their designated spots on the controller’s cover, the next step involves soldering and wiring these buttons to ensure they are functionally integrated with the controller’s electronics. The arduino is used to detect whether each button is pressed down or not.
The controller with the circuits
This is the code for Arduino:
void setup() {
Serial.begin(9600);
pinMode(2,INPUT);
pinMode(3,INPUT);
pinMode(4,INPUT);
pinMode(5,INPUT);
pinMode(6,INPUT);
pinMode(7,INPUT);
}
void loop() {
int button1State = digitalRead(2);
int button2State = digitalRead(3);
int button3State = digitalRead(4);
int button4State = digitalRead(5);
int button5State = digitalRead(6);
int button6State = digitalRead(7);
Serial.print(button1State);
Serial.print(",");
Serial.print(button2State);
Serial.print(",");
Serial.print(button3State);
Serial.print(",");
Serial.print(button4State);
Serial.print(",");
Serial.print(button5State);
Serial.print(",");
Serial.print(button6State);
Serial.println();
delay(20);
}
Then I will give an overview of key components and functionalities within the code as we adopted the methodology of object-oriented programming.
Main Game Loop (void draw())
This function is the heart of the game loop, responsible for updating all aspects of the game state:
Handles different game modes using a switch-case structure (modes for intro, tutorial, game start, map loading, and game over).
Calls specific functions based on the current game mode to handle game logic, drawing, and interactions.
Game Modes
Each mode in the game has a specific function:
Intro Mode (void Intro()): Displays initial instructions and allows the player to select a game mode.
Tutorial Mode (void Tutorial()): Guides the “builder” and “navigator” through game mechanics using a step-by-step tutorial.
Constructor Mode (void constructor()): Allows the “builder” to influence the game map using sound input, creating platforms of varying heights based on the volume detected.
Navigator Mode (void Navigator()): The “navigator” navigates the character through the map, attempting to collect coins and avoid obstacles.
Map Loading (void Map_load()): Handles selection and loading of pre-defined or random maps.
Interaction with Arduino
The script reads data from the Arduino via serial communication to get input from physical buttons:
Serial Input (void getSerialData()): Reads serial data to get button presses from Arduino, influencing game state for jumping and state selection.
Utility Functions
Several helper functions manage game dynamics:
Map Generation (void Map_generate()): Dynamically generates the game map based on sound input or pre-loaded configurations.
Character States (void state0(), void state1(), etc.): Manage different states of the character such as moving, jumping, or falling.
This is the code for Processing:
Minim minim;
AudioInput in;
SoundFile soundFile;
SoundFile music;
AudioIn input;
SoundFile sound;
import processing.sound.*;
import ddf.minim.*;
import ddf.minim.analysis.*;
import processing.serial.*;
PImage bgImage, cImage, treeImage, rbImage, transparentImage, vmImage, bdImage, coinImage; // Background, character
PImage warnImage;
PGraphics pg, pg2;//pg2 for coin
float volume;
Serial serialPort;
//Game initialization
int mode = 0;//1 for tutorial, 2 for direct start, 3 for load map, 4 for gameover(win)
int state = 0;//For Mode2/Mode3-Gamer1
int cur_width = 1200;//should be changed accordingly
int cur_height = 800;//should be changed accordingly
boolean gameStarted = false;
boolean gameOver = false;//mode 0-3 false, mode 4 true
boolean sound_exist = false; //For (from gamer0_0 to gamer0_1)
int move_state = 1;//1 for moving right, 2 for moving left in Intro
int death_count = 0;
//Variable initialization for the characters
int size = 40; // character size
int speed = 15; // character speed
int speed_fall = 18; // character fall speed
float x, y; // The character's x and y position
int lastStateChange;
int cur_i;
int new_cur;
//Map initialization with rectangle
int rec_x = 50; //The width of each rectangle on the x axis
int hole_size = 5;
int numRects = cur_width / rec_x; // Calculate the total number of rectangles
float[] rect_height = new float[numRects]; //The height for each rectangle
float[] rect_starting_x = new float[numRects];
float[] rect_height_tuto = new float[numRects];
float[] rect_starting_x_tuto = new float[numRects];
int j = 0;
int mini_height = 2;
//Map initialization with images
boolean pg_drawn = false;
boolean pg2_drawn = false;
//Map Memory
int map_numbers = 6;
int map_idx = 0;
int check_idx = 100;//For checking the available index
boolean loading_map = false;
float[][] Memo_rect_height = new float[map_numbers][numRects];
float[][] Memo_rect_starting_x = new float[map_numbers][numRects];
PImage[][] Image_for_x = new PImage[map_numbers][numRects];
PImage[] Image_for_cur_map = new PImage[numRects];
boolean[][] Memo_coin = new boolean[map_numbers][numRects];
int[] Memo_current_coin = new int[map_numbers];
int current_coin = 0;
boolean[] coin_for_cur_map = new boolean[numRects];
boolean[] Map_saved = new boolean[map_numbers];
boolean map_loaded = false;
//Other Initialization
int NUM_OF_VALUES_FROM_ARDUINO = 6;
int arduino_values[] = new int[NUM_OF_VALUES_FROM_ARDUINO];
boolean[] keys = new boolean[128];//For Key Press
int time = 1; //It is useless right now
float perc = 0.3; //Fault tolerance Percentage
int gamer = 0; //0 is constructor, 1 is navigator
int cons_tutor_state = 1;
boolean special_ability_used = false;
int force_exit= 0;
long force_exit_startTime = 0;
boolean music_play = true;
boolean[] press_button_exit = new boolean[3];
boolean exit_botton_pressed = false;
boolean exit_botton_previous =false;
//red:a[0] or keys[H]
void setup() {
String[] portNames = Serial.list();
for (String port : portNames) {
println(port);
}
serialPort = new Serial(this, "/dev/cu.usbmodem11201", 9600);
size(1200, 800);
pg = createGraphics(1200, 800);
pg2 = createGraphics(20, 20);
cur_width = width;
cur_height = height;
numRects = cur_width / rec_x;
bgImage = loadImage("bg3_2.jpg");
cImage = loadImage("character.png");
treeImage = loadImage("tree.png");
rbImage = loadImage("Roadblock.png");
transparentImage = loadImage("transparent.png");
vmImage = loadImage("VendingMachine.png");
bdImage = loadImage("bd.png");
warnImage = loadImage("warning.png");
coinImage = loadImage("Coin.png");
music = new SoundFile(this, "music.mp3");
music.play();
music.loop();
x = 0;
y = height - size;
minim = new Minim(this);
in = minim.getLineIn(Minim.MONO);
//Tutorial Design For Navigator
for (int i = 0; i < numRects; i++) {
Memo_current_coin[0] = 12;
Memo_rect_height[0][i] = 0;
Memo_rect_starting_x[0][i] = rec_x * i;
if (i % 4 == 0) {
Memo_rect_height[0][i] = i*20;
Memo_rect_height[0][i] = constrain(Memo_rect_height[0][i], 10, height*0.5);
if (Memo_rect_height[0][i]<=mini_height) {
Image_for_cur_map[i] = transparentImage;
} else if (Memo_rect_height[0][i]<=30) {
Image_for_cur_map[i] = rbImage;
} else if (Memo_rect_height[0][i]<=100) {
Image_for_cur_map[i] = vmImage;
} else if (Memo_rect_height[0][i]<=200) {
Image_for_cur_map[i] = treeImage;
} else if (Memo_rect_height[0][i]<=300) {
Image_for_cur_map[i] = bdImage;
} else {
Image_for_cur_map[i] = treeImage;
}
Image_for_x[0][i] = Image_for_cur_map[i];
} else {
Memo_rect_height[0][i] = 0;
Image_for_cur_map[i] = transparentImage;
}
Image_for_x[0][i] = Image_for_cur_map[i];
}
Map_saved[0] = true;
}
void draw() {
frameRate(60);
stroke(0);
image(bgImage, 0, 0, width, height);
getSerialData();
if (force_exit == 1) {
mode = 0;
return;
}
println(force_exit);
println(music_play);
if (music_play) {
if ((mode == 1 && gamer == 0) || (mode == 2 && gamer == 0))
{
music.stop();
music_play = false;
} else {
}
} else if (!music_play) {
if (!((mode == 1 && gamer == 0) || (mode == 2 && gamer == 0)))
{
music.play();
music.loop();
music_play = true;
} else {
}
}
//Intro
if (mode == 0) {
Intro();
return;
}
//Tutorial
else if (mode == 1) {
Tutorial();
} else if (mode == 2) {
if (gamer == 0) {
constructor();
} else if (gamer == 1) {
Map_generate();
Navigator();
}
} else if (mode == 3) {
if (!map_loaded) {
Map_load();
return;
}
if (!loading_map) {
frameRate(60);
Map_generate();
Navigator();
}
} else if (mode == 4 || gameOver) { //When the Navigator wins
win();
}
}
void Intro() {
//Initialization
pg = createGraphics(1200, 800);
pg2 = createGraphics(20, 20);
j = 0;
state = 0;
special_ability_used = false;
pg_drawn = false;
loading_map = false;
map_loaded = false;
pg_drawn = false;
gameStarted = false;
gameOver = false;
sound_exist = false;
pg2_drawn = false;
cons_tutor_state = 1;
current_coin = 0;
gamer = 0;
map_idx = 0;
death_count = 0;
//Move of the character
image(cImage, x, height-100, size, size);
if (move_state % 2 == 1) {
x = x + speed;
if (x + size >= width-1) {
move_state ++;
x = width - size;
}
} else {
x = x - speed;
if (x <= 0) {
x = 0;
move_state ++;
}
}
//Mode choosing
if ((keys['T']||keys['t']) ) {
mode = 1;
}
if ((keys['K']||keys['k']) ) {
gameStarted = true;
mode = 2;
}
if ((keys['L']||keys['l']) ) {
gameStarted = true;
mode = 3;
}
//Instruction Text
fill(255, 255, 255, 200);
stroke(255);
textSize(46);
textAlign(CENTER);
String text = "Press T to Begin Tutorial for Beginners!\nPress K to Directly Start a New Game!\nPress L to Load Maps!";
float textWidth = textWidth(text);
float textHeight = 46;
float textx = width / 2;
float texty = height / 2;
rect(textx - textWidth / 2-10, texty - textHeight / 2-20, textWidth+20, textHeight*3+25);
fill(255, 0, 0);
text(text, textx, texty);
return;
}
void Tutorial() {
if (gamer == 0) {
if (cons_tutor_state == 1) {
constructor_tuto1();
} else if (cons_tutor_state == 2) {
constructor_tuto2();
return;
}
} else if (gamer == 1) {
coin_for_cur_map[12] = true;
current_coin = 12;
map_idx = 0;
mode = 2;
if (!map_loaded) {
map_idx = 0;
for (int i = 0; i < numRects; i++) {
rect_height[i] = Memo_rect_height[map_idx][i]; //The height for each rectangle
rect_starting_x[i] = Memo_rect_starting_x[map_idx][i];
Image_for_cur_map[i] = Image_for_x[map_idx][i];
}
map_loaded = true;
}
Map_generate();
Navigator();
}
}
void constructor_tuto1() {
volume = in.left.level() * 100;
float each_rectHeight_tuto = 800 / 20;
float rect_height_tuto = volume * each_rectHeight_tuto;
fill(255);
rect_height_tuto = constrain(rect_height_tuto, 0, height*0.5);
rect(575, height-rect_height_tuto-50, 50, rect_height_tuto);
fill(255);
textSize(32);
text("This is the block generated by your current volume, continue by pressing RED BUTTON", 590, 790);
if (keys['M']|| keys['m']||arduino_values[0]==1) {
cons_tutor_state = 2;
}
}
void constructor_tuto2() {
//println(gamer);
if (keys['N']|| keys['n']||arduino_values[3]==1) {
gamer = 1;
return;
}
if (keys[ENTER]) {
cons_tutor_state = 1;
j = 0;
sound_exist = false;
current_coin = 0;
}
image(bgImage, 0, 0, width, height);
frameRate(4);
fill(255);
textSize(40);
textAlign(CENTER);
text("As a Constructor, try to make the map by changing your volume.\nYou can put a coin (for the navigator) by pressing RED BUTTON\n(Press BUTTON 4 if you are Navigator)", 590, 100);
volume = in.left.level() * 100;
if ((volume <= 0.5) && (!sound_exist)) {
return;
} else {
sound_exist = true;
if (j >= numRects) {
int j_ = numRects-1;
//gamer = 1;
for (int i = 0; i <= j_; i++) {
fill(255);
//Attention0
image(Image_for_cur_map[i], rect_starting_x_tuto[i]+hole_size, height - rect_height_tuto[i], rec_x-hole_size*2, rect_height_tuto[i]);
if (coin_for_cur_map[i]) {
image(coinImage, rect_starting_x_tuto[i]+hole_size+0.5*size, height - rect_height_tuto[i]-0.5*size, 0.5*size, 0.5*size );
}
}
} else {
float each_rectHeight = 800 / 20;
int m = j * rec_x;
rect_height_tuto[j] = volume * each_rectHeight;
rect_starting_x_tuto[j] = m;
fill(255);
rect_height_tuto[j] = constrain(rect_height_tuto[j], 0, height*0.5);
if ((!special_ability_used) && (keys['B']||keys['b']||arduino_values[0] == 1)&&rect_height_tuto[j] > mini_height) {
coin_for_cur_map[j] = true;
current_coin = j;
special_ability_used = true;
}
if (rect_height_tuto[j]<=mini_height) {
Image_for_cur_map[j] = transparentImage;
} else if (rect_height_tuto[j]<=30) {
Image_for_cur_map[j] = rbImage;
} else if (rect_height_tuto[j]<=100) {
Image_for_cur_map[j] = vmImage;
} else if (rect_height_tuto[j]<=200) {
Image_for_cur_map[j] = treeImage;
} else if (rect_height_tuto[j]<=300) {
Image_for_cur_map[j] = bdImage;
} else {
Image_for_cur_map[j] = treeImage;
}
println(j);
//Attention1
image(Image_for_cur_map[j], rect_starting_x_tuto[j]+hole_size, height - rect_height_tuto[j], rec_x-hole_size*2, rect_height_tuto[j]);
if (coin_for_cur_map[j]) {
image(coinImage, rect_starting_x_tuto[j]+hole_size+0.5*size, height - rect_height_tuto[j]-0.5*size, 0.5*size, 0.5*size );
}
//rect(rect_starting_x_tuto[j]+hole_size, height - rect_height_tuto[j], rec_x-hole_size*2, rect_height_tuto[j]);
//println(height -rect_height[j]);
for (int i = 0; i <= j; i++) {
println(j);
println(i);
fill(255);
//Attention2
image(Image_for_cur_map[i], rect_starting_x_tuto[i]+hole_size, height - rect_height_tuto[i], rec_x-hole_size*2, rect_height_tuto[i]);
//rect(rect_starting_x_tuto[i]+hole_size, height - rect_height_tuto[i], rec_x-hole_size*2, rect_height_tuto[i]);
if (coin_for_cur_map[i]) {
image(coinImage, rect_starting_x_tuto[i]+hole_size+0.5*size, height - rect_height_tuto[i]-0.5*size, 0.5*size, 0.5*size );
}
}
j ++;
}
}
}
void constructor() {
//j = 0;
frameRate(4);
fill(255);
textSize(32);
textAlign(CENTER);
text("Use Your Voice to Create Blocks", width / 2, height / 2);
textAlign(CENTER);
text("Press RED BUTTON to Generate A Coin", width/2, height/2+30);
fill(255, 0, 0, 100);
rect(0, 0, width, height*0.34);
image(warnImage, 0, 0, width, height*0.34);
volume = in.left.level() * 100;
if ((volume <= 0.5) && (!sound_exist)) {
return;
} else {
sound_exist = true;
if (j == numRects) {
gamer = 1;
return;
}
float each_rectHeight = 800 / 20;
int m = j * rec_x;
rect_height[j] = volume * each_rectHeight;
rect_starting_x[j] = m;
fill(255);
rect_height[j] = constrain(rect_height[j], 0, height*0.5);
if ((!special_ability_used) && (keys['B']||keys['b']||arduino_values[0] == 1)&&rect_height[j] > mini_height) {
coin_for_cur_map[j] = true;
current_coin = j;
special_ability_used = true;
}
if (rect_height[j]<=mini_height) {
Image_for_cur_map[j] = transparentImage;
} else if (rect_height[j]<=30) {
Image_for_cur_map[j] = rbImage;
} else if (rect_height[j]<=100) {
Image_for_cur_map[j] = vmImage;
} else if (rect_height[j]<=200) {
Image_for_cur_map[j] = treeImage;
} else if (rect_height[j]<=300) {
Image_for_cur_map[j] = bdImage;
} else {
Image_for_cur_map[j] = treeImage;
}
//Attention3
//rect(rect_starting_x[j]+hole_size, height - rect_height[j], rec_x-hole_size*2, rect_height[j]);
image(Image_for_cur_map[j], rect_starting_x[j]+hole_size, height - rect_height[j], rec_x-hole_size*2, rect_height[j]);
if (coin_for_cur_map[j]) {
image(coinImage, rect_starting_x[j]+hole_size+0.5*size, height - rect_height[j]-0.5*size, 0.5*size, 0.5*size );
}
for (int i = 0; i <= j; i++) {
fill(255);
//Attention4
image(Image_for_cur_map[i], rect_starting_x[i]+hole_size, height - rect_height[i], rec_x-hole_size*2, rect_height[i]);
if (coin_for_cur_map[i]) {
image(coinImage, rect_starting_x[i]+hole_size+0.5*size, height - rect_height[i]-0.5*size, 0.5*size, 0.5*size );
}
}
j ++;
}
}
void Map_load() {
image(bgImage, 0, 0, width, height);
String trueIndices = "Map Saved With Specific Index:";
String falseIndices = "Generate Random Map: 6";
for (int i = 1; i < map_numbers; i++) {
if (Map_saved[i]) {
trueIndices += (trueIndices.endsWith(":") ? "" : ", ") + i;
} else {
falseIndices += (falseIndices.endsWith(":") ? "" : ", ") + i;
}
}
fill(255, 255, 255, 200);
stroke(255);
textSize(46);
textAlign(CENTER);
String text = trueIndices + "\n" + falseIndices + "\nPlease Press The Corresponding Button To Load The Map!";
float textWidth = textWidth(text);
float textHeight = 46;
float textx = width / 2;
float texty = height / 2;
rect(textx - textWidth / 2-10, texty - textHeight / 2-20, textWidth+20, textHeight*3+25);
fill(255, 0, 0);
text(text, textx, texty);
loading_map = true;
for (int i = 0; i <= 5; i++) {
if (arduino_values[i] == 1) {
map_idx = i+1;
}
}
if (map_idx != 0 && loading_map) {
if (!(map_idx == 6)) {
if (Map_saved[map_idx]) {
for (int i = 0; i < numRects; i++) {
current_coin = Memo_current_coin[map_idx];
rect_height[i] = Memo_rect_height[map_idx][i]; //The height for each rectangle
rect_starting_x[i] = Memo_rect_starting_x[map_idx][i];
Image_for_cur_map[i] = Image_for_x[map_idx][i];
coin_for_cur_map[i] = Memo_coin[map_idx][i];
coin_for_cur_map[current_coin] = true;
}
} else if (!Map_saved[map_idx]) {
for (int i = 0; i < numRects; i++) {
float cur_height = (int) random(0, height*0.5);
if (cur_height<=mini_height) {
Image_for_cur_map[i] = transparentImage;
} else if (cur_height<=30) {
Image_for_cur_map[i] = rbImage;
} else if (cur_height<=100) {
Image_for_cur_map[i] = vmImage;
} else if (cur_height<=200) {
Image_for_cur_map[i] = treeImage;
} else if (cur_height<=300) {
Image_for_cur_map[i] = bdImage;
} else {
Image_for_cur_map[i] = treeImage;
}
current_coin = (int)random(0, numRects);
coin_for_cur_map[current_coin] = true;
Memo_rect_height[map_idx][i] = cur_height;
Memo_rect_starting_x[map_idx][i] = i * rec_x;
Image_for_x[map_idx][i] = Image_for_cur_map[i];
Memo_coin[map_idx][i] = coin_for_cur_map[i];
Map_saved[map_idx] = true;
rect_height[i] = cur_height; //The height for each rectangle
rect_starting_x[i] = i * rec_x;
}
check_idx = map_idx;
}
loading_map = false;
map_loaded = true;
gamer = 1;
} else {
for (int i = 0; i < numRects; i++) {
float cur_height = (int) random(0, height*0.5);
if (cur_height<=mini_height) {
Image_for_cur_map[i] = transparentImage;
} else if (cur_height<=30) {
Image_for_cur_map[i] = rbImage;
} else if (cur_height<=100) {
Image_for_cur_map[i] = vmImage;
} else if (cur_height<=200) {
Image_for_cur_map[i] = treeImage;
} else if (cur_height<=300) {
Image_for_cur_map[i] = bdImage;
} else {
Image_for_cur_map[i] = treeImage;
}
rect_height[i] = cur_height; //The height for each rectangle
rect_starting_x[i] = i * rec_x;
current_coin = (int)random(0, numRects);
coin_for_cur_map[current_coin] = true;
}
loading_map = false;
map_loaded = true;
}
}
}
void Map_generate() {
//println(coin_for_cur_map);
//Don't Jump into this area
if (!pg_drawn) {
pg.beginDraw();
pg.fill(255, 0, 0, 100);
pg.rect(0, 0, width, height*0.34);
pg.image(warnImage, 0, 0, width, height*0.34);
for (int i = 0; i < numRects; i++) {
fill(255);
//Attention5
pg.image(Image_for_cur_map[i], rect_starting_x[i]+hole_size, height - rect_height[i], rec_x-hole_size*2, rect_height[i]);
}
pg.endDraw();
pg_drawn = true;
}
if (!pg2_drawn) {
pg2.beginDraw();
for (int i = 0; i < numRects; i++) {
if (i == current_coin) {
pg2.image(coinImage, 0, 0, 0.5*size, 0.5*size);
}
}
pg2.endDraw();
pg2_drawn = true;
}
image(pg, 0, 0);
image(pg2, rect_starting_x[current_coin]+hole_size+0.5*size, height - rect_height[current_coin]-0.5*size, 0.5*size, 0.5*size);
}
void Navigator() {
//println(coin_for_cur_map[current_coin]);
if (coin_for_cur_map[current_coin]) {
image(pg2, rect_starting_x[current_coin]+hole_size+0.5*size, height - rect_height[current_coin]-0.5*size, 0.5*size, 0.5*size);
} else {
pg2 = createGraphics(20, 20);
}
frameRate(60);
//Intro
if (state == 0) {
state0();
}
//Wait
if (state == 1) {
state1();
}
//Jump
if (state == 2) {
state2();
}
//Falling Down
if (state == 3) {
state3();
}
if (state == 4) {
state4();
}
}
void state0() { //Intro for M2/3_G1
image(cImage, rect_starting_x[0], height- rect_height[0] - size, size, size);
x = rect_starting_x[0];
y = height- rect_height[0] - size;
fill(255);
textSize(24); // Set text size
textAlign(TOP); // Align text to center and bottom of the character image
fill(0, 0, 0, 90);
stroke(0, 0, 0, 90);
rect(5, 5, 450, 200);
fill(255);
text("Press YELLOW BUTTON To Start This Turn!", 10, 30); // Display the prompt above the character
text("Press LEFT BLUE BUTTON To Move Backward.", 10, 60);
text("Press RIGHT BLUE BUTTON to Jump Higher.", 10, 90);
text("Your Death Count is: "+ death_count, 10, 120);
text("Notice that you need to approach \nthe coin (if any) once to pass this map.", 10, 150);
if (keys['l']||keys['L']||(arduino_values[2]==1)) {
state = 1;
}
}
void state1() { //Game_Wait for M2/3_G1
int numRects = width / rec_x;
int cur_i = (int)(x / rec_x);
if (cur_i < numRects-2) {
cur_i ++;
}
coin_for_cur_map[cur_i-1] = false;
image(cImage, x, y, size, size);
if (keys['W']||keys['w']||arduino_values[1] == 1) {
state = 2;
}
}
void state2() {
//The charac1
image(cImage, x, y, size, size);
//Don't move forward
if (!(keys['A']||keys['a']|| arduino_values[5] == 1) ) {
x += 5;
} else {
x -= 1;
}
//Jump, or you will fall
if (keys['W']||keys['w']||(arduino_values[1] == 1)) {
y -= speed;
} else {
y += speed_fall;
}
cur_i = (int)(x / rec_x);
//Achieving the end
if (x+size >= width ) {
state = 0;
if (coin_for_cur_map[current_coin]) {
death_count ++;
} else if (!coin_for_cur_map[current_coin]) {
mode = 4;
gameOver = true;
return;
}
}
//This is Lose
if (y >= height || y <= height*0.3) {
death_count ++;
state = 0;
return;
}
new_cur = cur_i;
if (cur_i < (numRects-1)) {
new_cur = cur_i + 1;
}
if (
(x + size >= rect_starting_x[new_cur] + hole_size && height-rect_height[new_cur] <= y)
||
(x <= rect_starting_x[cur_i] - hole_size&& height-rect_height[cur_i] <= y)
||
(rect_height[new_cur] <= mini_height && y + size >= height-mini_height)
||
(rect_height[cur_i] <= mini_height && y + size >= height-mini_height)
)
{
state = 4;
lastStateChange = millis();
return;
}
boolean Condi_1 = x + perc * size >= rect_starting_x[new_cur]
&& x + size <= rect_starting_x[new_cur] + rec_x
;
boolean Condi_1_1 =
y + size >= height-rect_height[new_cur] - 10
;
boolean Condi_2 = x + size - perc * size <= rect_starting_x[cur_i] + rec_x
&&
x >= rect_starting_x[cur_i]
;
boolean Condi_2_1 =
y + size >= height-rect_height[cur_i] - 10
;
if ((Condi_1 && Condi_1_1)) {
state = 1;
y = height-rect_height[new_cur]-size;
;
}
if (Condi_2 && Condi_2_1) {
state = 1;
y = height-rect_height[cur_i]-size;
}
}
//Game_Fall
void state4() {
image(cImage, x+5, y+2, size, size);
image(Image_for_cur_map[cur_i], rect_starting_x[cur_i]+hole_size, height - rect_height[cur_i], rec_x-hole_size*2, rect_height[cur_i]);
image(Image_for_cur_map[new_cur], rect_starting_x[new_cur]+hole_size, height - rect_height[new_cur], rec_x-hole_size*2, rect_height[new_cur]);
if (millis()-lastStateChange > 500)
{
state = 3;
death_count++;
}
}
void state3() { //Game Fall for M2/3_G1
image(cImage, x+5, y+2, size, size);
image(Image_for_cur_map[cur_i], rect_starting_x[cur_i]+hole_size, height - rect_height[cur_i], rec_x-hole_size*2, rect_height[cur_i]);
image(Image_for_cur_map[new_cur], rect_starting_x[new_cur]+hole_size, height - rect_height[new_cur], rec_x-hole_size*2, rect_height[new_cur]);
y += 10;
if (y-40 >= height) {
state = 0;
return;
}
}
void win() {
//println(Map_saved);
String text;
float textHeight = 46;
float textx = width / 2;
float texty = height / 2;
fill(255, 255, 255, 200);
stroke(255);
textSize(46);
textAlign(CENTER);
if (check_idx >= 6) {
check_idx = (int)random(1, 6);
for (int i = 1; i < Map_saved.length; i++) {
if (!Map_saved[i]) {
check_idx = i;
break;
}
}
}
if (!(keys['S'] || keys['s'])) {
String dc = "You Win with a death count of: " + death_count;
text =dc + "!\nPress RED BUTTON to Play This Map Again!\nPress BUTTON 5 to Go Back to the Home Page\nPress S to Save this Map to Index " + check_idx;
float textWidth = textWidth(text);
rect(textx - textWidth / 2-10, texty - textHeight / 2-20, textWidth+20, textHeight*3+80);
fill(255, 0, 0);
text(text, textx, texty);
if (keys[ENTER]||arduino_values[0]==1) {
coin_for_cur_map[current_coin] = true;
pg2_drawn = false;
mode = 2;
gamer = 1;
state = 0;
gameOver = false;
death_count = 0;
}
if (keys['H'] || keys['h']||arduino_values[4]==1) {
mode = 0;
gamer = 0;
state = 0;
gameOver = false;
}
} else {
for (int i = 0; i < numRects; i++) {
Memo_rect_height[check_idx][i] = rect_height[i];
Memo_rect_starting_x[check_idx][i] = rect_starting_x[i];
Image_for_x[check_idx][i] = Image_for_cur_map[i];
Memo_current_coin[check_idx] = current_coin;
}
Map_saved[check_idx] = true;
check_idx ++;
mode = 0;
gamer = 0;
state = 0;
gameOver = false;
}
return;
}
//Additional
void keyPressed() {
if (key < 128) keys[key] = true; // When the key is pressed, update
if (key >= '1' && key <= '6') {
int index = key;
if (index >= 49) {
index = index - 48;
}
if (mode == 3 && loading_map) {
map_idx = index;
}
}
}
void keyReleased() {
if (key < 128) keys[key] = false; // When the key is released, update
}
void getSerialData() {
while (serialPort.available() > 0) {
String in = serialPort.readStringUntil( 10 );
if (in != null) {
//print("From Arduino: " + in);
String[] serialInArray = split(trim(in), ",");
if (serialInArray.length == NUM_OF_VALUES_FROM_ARDUINO) {
for (int i=0; i<serialInArray.length; i++) {
arduino_values[i] = int(serialInArray[i]);
}
}
}
}
}
4. Conclusions
Our goal is to experiment with the methods of players interacting with a game, and we want to build a game that everyone can enjoy. And we choosed sounds as the medium of interaction. I believe that from the feedbacks that we received from the IMA final show, our project attracted people of different ages. Their laughter and encouragement proves that we are successful.
Building upon the success and positive feedback from the IMA final show, our next steps are to refine and expand the game’s accessibility and interaction capabilities. The enthusiasm and enjoyment expressed by players from diverse age groups underscore the potential of our game. Here’s how we can move forward:
1. Gameplay
Navigator:
In the user test, some testers had proposed that same as the “builder”, the “navigator” can be voice-controlled. Mimicing your friends’ voices to jump through the map can be great fun.
Builder:
It can be better if we are able to engage more users, or even a whole audience to come and enjoy the game together.
I have an idea of building a bridge for a character to walk through by referring to the joint sounds from a large crowd, so everyone is engaged in the game and involved in the progress of the character. This interactive gameplay concept could transform individual audience reactions into a collective, real-time game control system. By using sound recognition technology, the game would analyze the volume and pitch of the crowd’s reactions to determine the actions of the character on the screen. For instance, when the crowd claps loudly, the character might jump, while laughter could cause the character to dance or perform a special move. This would not only make the gaming experience more immersive and entertaining but also foster a sense of community and collaboration among the participants. As the crowd learns how their actions directly influence the game, they become more engaged, creating a dynamic and exciting atmosphere that enhances the overall experience.
2. Devices
The controller that we used can be confusing as the instructions of it is not intuitive, we can minimize it by only saving the components for example four buttons for from directions and perform the rests with other creative devices.
To conclude, the journey of developing this interactive game, beginning with its inception and feedback at the IMA final show, has been a testament to the power of community engagement in gaming. Our innovative approach in using sound recognition to integrate audience interaction not only enhances the gameplay but also fosters a unique sense of friendship among players. The enthusiastic participation from people of various age groups underscores the universal appeal of our game, it breaks the bias that an experimental work is progressive and condescending, it can be something that everyone can enjoy.
5. Disassembly
6. Appendix
In the appendix I wanted to show some texture maps in our game.
