Category: Interaction Lab

A.PROJECT TITLE – YOUR NAME – YOUR INSTRUCTOR’S NAME

Re-walk the Journey of Little Prince-Jingyi Ma-Gottfried Haider

B.CONCEPTION AND DESIGN:

Our project’s concept is a “storytelling” experience by combining Little Prince’s journey in the novel with the visual and audio interaction with the users. Users will put the “Little Prince” on different planets, and the magnet sensor on Little Prince’s foot will trigger off the magnet sensor and switch the corresponding LED strip on. Meanwhile, the computer screen will play music and sound, showing what Little Prince has encountered on that planet. My definition of “interaction” is “The activity between two or more entities, with input, processing, output, feedback, and consideration.” So a single “showing” cannot be interaction, interaction needs mutual feedback. In our brainstorming period, we thought of making a Little Prince themed music box with sound visualization, but this did not provide any input from users, so we abandoned this opinion. After re-considering the concept of interaction mentioned above,  we implemented the concept of “点读机” or reading machine to our final project. Reading machine is an old-style electronic common in our childhood in China. This electronic can play music or sound if you put a particular book on it and use an electronic pen to tickle different areas. We think this is a very good example of interaction in daily life, and we decided to combine it with our project. We use magnet sensors as input signals and using sound and lights as output, a very simple but effective way of interaction. During the user testing session, the biggest problem was we lacked instructions to make the users tap the magnet on the sensor. Also, during that time, we lacked corresponding audio sounds, and Testors would like to have more sound effects rather than visual words. To make up these bugs, I used an A4 size hot-shrink sheet and marker to make a chess of Little Prince, stuck paper stars on his foot with a magnet, and put other stars around the magnet sensor to guide the users. Also, we used AI-generated voices (Narakeet) to read out the conversation of individuals. The AI voices worked well in the presentations and documents, but they didn’t reach our desired effect in the IMA show, for it was too noisy and it was hard for people to focus on the sound. 

C.FABRICATION AND PRODUCTION:

4/21:In the beginning, we thought of more keyboard and mouse interaction with the screen.

The professor suggested we use sequencers at first to make a “music game”, but we found that it’s not so good when implemented with other sounds as birds of planes, so we abandoned this plan. I built a prototype in Processing with various planets being shown at the same time. However, after discussing it with professor, he thought that it was more important to have more physical interaction, so we abandoned this plan. The first type of production is to build the prototype for our project. We originally wanted to make an interactive music box,  and we went to a musical box shop but it’s too hard to build so we changed it to the form of a book.

(The original design was more like a music box.)

4/23: We decided to use laser-cutting on 3mm wood to make the effect of a book. We used the cuttle.XYZ webpage to create SVG, and we combined the SVG of Little Prince found on the internet and some emoji from Cuttle to make the graph of each planet. I used cardboard to test the effect of Cuttle, but the first prototype sucked, for I misunderstood the relationship of “stroke” and “filling”, so some of the parts are engraved, while other parts are not cut through. After asking Professor Andy, we figured out how to change the “engrave” into “strokes”, by “create path” function. For our final woodcut, helped by our friend, the LA Sophia, we made our wooden board successfully.

4/24:Our friend Astroyd and Melaine’s Interaction Lab project last semester, “Portion Class”, inspired us greatly. Their project does not use LED strips to show sound visualization but makes it into a light source. We decided to make the whole lighting of the project with an LED strip. The project has a complex “landscape”, so we had to cut the LED strip into smaller parts.

We used the thinnest wire we could find in the IMA studio, but they were too thin so Jiaqi had to use nails to strip them by hand. We soldered each part of the LED strip, six points per strip, and it was a dizzy job.

After lighting up the LED strip, we found that some parts were not lighting the way we want-very frustrating. It turned out that some parts were, actually, shorted because of the wire attached to each other. So we used a glue gun to block these parts and stopped the short circuits.

The tester effect looks good.

4/25:After the LED strip works well, we want to implement sensors in the project. We agreed that magnet sensors are the most reliable ones, for they won’t be disturbed by sound and light in the show. Our original intention of interaction is to make a “magic pen” or something else. We found that the problem of failure in adding the magnet sensor is due to 1) We forgot to connect the magnet sensor with resistors, so the digital value was not stable. 2) no variables were storing the initial value of the sensor 0 and current value 1. After fixing this part, our project can turn on the LED strip when the magnet sensor is triggered. This is okay for user testing, but only for the Arduio part.

(The Arduino part was okay at user testing.)

4/27: After the user testing, we decided to make Serial Communication on Processing and also a figure for Little Prince to make his journey more “real”. I printed a graph from the internet, drew its sketch on hot-shrink sheet and Jiaqi colored it. I used the oven to make this shrink to a size comfortably held by hand. We used two magnets on that “Little Prince Chess” as they will stick to each other automatically.

4/30: As I was stuck in the coding with serial communication, Jiaqi and I communicated with Professor Gottfried and figured out what was wrong with our Serial Communication part. It turned out that we forgot to write the arrays storing Arduino values. After fixing this, we recorded the voices of each planet from the internet open-ended sources, like the sound of coins, birds, and wine. I used the loop() function to play these sounds, but I found that it wouldn’t stop playing once the music was played. After consulting our fellow Kevin, we used the stop() function to fix this problem. We used sound, picture, and text functions in Processing to mimic the effect of storytelling. Finally, during the presentation, we showed the effect of turning the LED on and playing corresponding music and voices at the same time. 

    coin.loop();
    globe.stop();
    lamp.stop();
    dove.stop();
    wine.stop();
    clap.stop();
    plane.stop();
    crown.stop();

5/2: We finished the presentation and things worked well on Arduino and Processing, the LED didn’t go wrong and music was played. My classmates and professors gave us some advice. They suggest making more direct directions on how to trigger the magnet sensors, as well as making more sound interaction and reducing the text. So we use the AI-generated text-to-speech tool Narakeet to generate the dialogue of Little Prince and other people. We found that it’s hard to make the dialogue being spoken separately(will involve millis() and more coding). We merged the audio and only played them once. This problem is solved.

    textFont(story, 40);
    fill(255,255,255);
    text("-Why are you drinking?",750,200);
    drinkall.play();
    fill(50, 205, 50);
    text("-So that I may forget.",750,250);
    fill(255,255,255);
    text("-Forget what?",750,300);
    fill(50, 205, 50);
    text("-Forget that I am ashamed.",750,350);
    fill(255,255,255);
    text("-Ashamed of what?",750,400);
    fill(50, 205, 50);
    text("-Ashamed of drinking!",750,450);

 

5/10:

During the IMA show, our project receives no less than 30 users! A lot of them give very good feedback on the visual and text effects. However, as it was too noisy in the studio, though we turned on the biggest volume, the sound effect was disturbed. But in all, people are deeply engaged in our project, and I think it’s a good ending for our semester.

D.CONCLUSIONS:

Our project achieved the goal of “interactive storytelling”. By putting the plastic “chess” of Little Prince on different planets, users can light up different planets and discover what happens between Little Prince and other people on a computer screen. Therefore, they can re-walk Little Prince’s journey and retrieve memories from the novel, reflecting on the relationship of childhood and adulthood, and learn to see the world with their heart. By implementing sound, text, and pictures, as well as the interaction of sensors, we create mutual communication between computers and humans, giving the user an immersive experience of reading a storybook.

If we have more time, I will try to refer to my friends from the communications lab, to improve the text and picture parts shown on the screen, to make a better visual effect. I also consider implementing a speaker to make a bigger voice in noisy situations. 

Due to the project being very big, it’s hard to capture the LED lights and the screen clearly at the same time. In our document, we shoot two videos, one is more focused on LED and one is more focused on screen.

We encountered many bugs and troubles in making this project. For example, when designing the model, we thought of many prototypes like a music box. We considered Chapter VIII in Monument Valley, also a design of a box. Unluckily, as it’s too complex to make a box, we abandoned this plan. Also, we initially wanted to use sequencers to make audio interactions, and we spent a lot of time building the sequencer, but after testing, we found its effects were no better than just looping the video. Also, during the documentation, a NullPointerExpectation happened and the whole processing failed, which nearly drove me to tears. After consulting our friend Jingran, we found the necessity to check the setup() and found this is an error of writing an mp3 audio in WAV format. We need to be more careful about details and be more eager to consult others when we can’t figure out the problem by ourselves. 

During the second half of this class, I am very glad that I learned to draw objects with code, as well as make screen-mechanism interactions by doing Serial Communications. My coding ability enhanced a lot in this semester, and I’m grateful to my professor, my teammate, and all the fellows, and LAs who helped me in this progress. 

E.DISASSEMBLY:

Because we cut the LED strips in this project and glued the sensors on the 3mm wood, we cannot recycle them for other projects. I disassembled the Adruino,  resistors, and other cables. We decided to keep the remaining parts of this project as decoration in my home to make full use of it. 

APPENDIX

(There were no magnet sensors in Tinkercard. We use light sensors in the graph to replace magnet sensors.)

Arduino Code:

PImage[] pic = new PImage[11];
boolean[] imageVisibility = new boolean[11]; // Array to track image visibility
int[] xc = new int[11];
int[] yc = new int[11]; // Array with center positions
String[]lines = new String[11];
float angle;
boolean click_other = false;
PFont title;
PFont story;
import processing.serial.*;
import processing.sound.*;

Serial serialPort;

int NUM_OF_VALUES_FROM_ARDUINO = 10;
int arduino_values[] = new int[NUM_OF_VALUES_FROM_ARDUINO];
int prev_arduino_values[] = new int[NUM_OF_VALUES_FROM_ARDUINO];
int AllOn = 0;

SoundFile start;
SoundFile beginning;
SoundFile coin;
SoundFile globe;
SoundFile lamp;
SoundFile dove;
SoundFile wine;
SoundFile clap;
SoundFile plane;
SoundFile crown;
SoundFile ending;
SoundFile flower;

SoundFile richall;
SoundFile geoall;

SoundFile lampall;
SoundFile pilot;
SoundFile birds;

SoundFile drinkall;

SoundFile admireall;
SoundFile kingall;

SoundFile thankyou;
void setup() {
  size(1600, 900);
  background(25, 25, 60);
  initializeImages();
  printArray(Serial.list());
  serialPort = new Serial(this, "/dev/cu.usbmodem21101", 9600);
  title = loadFont("ArimaMadurai-Bold-100.vlw");
  story = loadFont("Klee-Medium-50.vlw");
  textFont(title, 60);
  fill(255);
  text(" 'All grown-ups were once children...\n but only few of them remember it.'\n\nLet's re-walk the journey of the little prince.\n Our journey starts from planet B612 \nand ends at earth.", 100, 200);
  start = new SoundFile(this, "start.wav");
  globe = new SoundFile(this, "rotatingdiqiuyi-geologist.wav");
  coin = new SoundFile(this, "coin-businessman.wav");
  lamp = new SoundFile(this, "switch-lightingperson.wav");
  dove = new SoundFile(this, "gezi-earth.wav");
  wine = new SoundFile(this, "pouralchohol-drunk.wav");
  clap = new SoundFile(this, "applauding-vain.wav");
  plane = new SoundFile(this, "luoxuanjing-earth.wav");
  crown = new SoundFile(this, "drum.mp3");
  ending = new SoundFile(this, "ending.wav");
  beginning = new SoundFile(this, "beginning.wav");
  flower = new SoundFile(this, "Flower.wav");
  richall = new SoundFile(this, "richall.mp3");
  geoall = new SoundFile(this, "geoall.mp3");
  lampall = new SoundFile(this, "lampall.mp3");
  
  drinkall = new SoundFile(this, "drinkall.mp3");
  
  admireall= new SoundFile(this, "admireall.mp3");
  kingall= new SoundFile(this, "kingall.mp3");
  thankyou = new SoundFile(this, "Thankyou.mp3");
  birds = new SoundFile(this, "birds.mp3");
  pilot = new SoundFile(this, "pilotall.mp3");
  start.loop();
  beginning.play();
}

int lastImageTriggered = -1;

void draw() {
  getSerialData();
  if (arduino_values[0] == 1 && prev_arduino_values[0] == 0) {
    background(25, 25, 60);
    textFont(story, 40);
    fill(255);
    text("I have a flower,\n whom I water every day.\n I cultivate her, \nand I attend to her.\n Yet I have nothing to show for it.\n It's as though I had been nurturing her for nothing.", 100, 200);
    flower.play();
    textFont(title, 60);
    fill(255, 250, 205);
    text("Planet B612", 300, 700);
    lastImageTriggered = 0;
    AllOn +=1;
  } else if (arduino_values[1] == 1 && prev_arduino_values[1] == 0) {
    background(25, 25, 60);
    textFont(story, 40);
    fill(0, 255, 255);
    text("-Oh, look! Here is an explorer! Where do you come from?", 100, 100);
    fill(255);
    geoall.play();
    text("-Oh, where I live,it is not very interesting.\n I have three volcanoes. I have also a flower.", 100, 200);
    fill(0, 255, 255);
    text("-We do not record flowers,because they are ephemeral.", 100, 300);
    fill(255);
    text("-My flower is ephemeral,\nand she has only four thorns to defend herself against the world. \nAnd I have left her on my planet, all alone!", 100, 400);
    textFont(title, 60);
    fill(0, 255, 255);
    text("Planet of Geographer", 500, 750);
    globe.loop();
    coin.stop();
    lamp.stop();
    dove.stop();
    wine.stop();
    clap.stop();
    plane.stop();
    crown.stop();
    lastImageTriggered = 1;
    AllOn +=1;
  } else if (arduino_values[2] == 1  && prev_arduino_values[2] == 0) {
    background(25, 25, 60);
    lastImageTriggered = 2;
    AllOn +=1;
    textFont(title, 60);
    fill(0, 255, 30);
    text("Planet of Businessman", 600, 750);
    textFont(story, 40);
    fill(255, 255, 255);
    text("-What good does it do you to own the stars?", 100, 450);
    richall.play();
    fill(0, 255, 0);
    text("-It does me the good of making me rich.",100, 500);
    fill(255, 255, 255);
    text("-And what good does it do you to be rich?", 100, 550);
    fill(0, 255, 0);
    text("-It makes it possible for me to buy more stars, if any are discovered.",100,600);
    coin.loop();
    globe.stop();
    lamp.stop();
    dove.stop();
    wine.stop();
    clap.stop();
    plane.stop();
    crown.stop();
  } else if (arduino_values[3] == 1 && prev_arduino_values[3] == 0) {
    background(25, 25, 60);
    lamp.loop();
    coin.stop();
    globe.stop();
    dove.stop();
    wine.stop();
    clap.stop();
    plane.stop();
    crown.stop();
    lastImageTriggered = 3;
    AllOn +=1;
    textFont(title, 60);
    fill(255, 215, 0);
    text("Planet of Lamplighter", 300, 100);
    textFont(story, 40);
    fill(255, 250, 205);
    text("-From year to year the planet has turned more rapidly\n and the orders have not been changed!\nThe planet now makes a complete turn every minute,\n and I no longer have a single second for repose.\n Once every minute I have to light my lamp and put it out!", 50, 150);
    lampall.play();
    fill(255, 255, 255);
    text("-Your planet is so small that three strides will take you all the way around it.\n To be always in the sunshine, you need only walk along rather slowly.\nThat man is the only one of them all whom I could have made my friend.\n But his planet is indeed too small. There is no room on it for two people. . .", 50, 600);
  } else if (arduino_values[4] == 1 && prev_arduino_values[4] == 0) {
    background(25, 25, 60);
    dove.loop();
    coin.stop();
    globe.stop();
    lamp.stop();
    wine.stop();
    clap.stop();
    plane.stop();
    crown.stop();
    lastImageTriggered = 4;
    textFont(title, 60);
    fill(255, 215, 255);
    text("Birds", 700, 100);
    textFont(story, 40);
    fill(255,255,255);
    text("The king claims he rules over everything he sees,\n but he doesn't control over a single bird.", 200, 300);
    birds.play();
    AllOn +=1;
  } else if (arduino_values[5] == 1 && prev_arduino_values[5] == 0) {
    background(25, 25, 60);
    wine.loop();
    coin.stop();
    globe.stop();
    lamp.stop();
    dove.stop();
    clap.stop();
    plane.stop();
    crown.stop();
    textFont(title, 60);
    fill(50, 205, 50);
    text("Planet of Tippler", 50, 600);
    
    textFont(story, 40);
    fill(255,255,255);
    text("-Why are you drinking?",750,200);
    drinkall.play();
    fill(50, 205, 50);
    text("-So that I may forget.",750,250);
    fill(255,255,255);
    text("-Forget what?",750,300);
    fill(50, 205, 50);
    text("-Forget that I am ashamed.",750,350);
    fill(255,255,255);
    text("-Ashamed of what?",750,400);
    fill(50, 205, 50);
    text("-Ashamed of drinking!",750,450);
    lastImageTriggered = 5;
    AllOn +=1;
  } else if (arduino_values[6] == 1 && prev_arduino_values[6] == 0) {
    background(25, 25, 60);
    clap.loop();
    wine.stop();
    coin.stop();
    globe.stop();
    lamp.stop();
    dove.stop();
    plane.stop();
    crown.stop();
    textFont(title, 60);
    fill(255, 105, 180);
    text("Planet of Conceited Man", 600, 100);
    textFont(story, 40);
    text("-Ah! I am about to receive a visit\n from an admirer!",500,200);
    fill(255,255,255);
    text("-What does that mean--'admire'?",500,300);
    admireall.play();
    delay(1500);
    fill(255, 105, 180);
    text("-To admire means that you regard me as\n the handsomest, the best-dressed, the richest,\n and the most intelligent man on this planet.",500,350);
    fill(255,255,255);
    text("-But you are the only man on your planet!",500,600);
    lastImageTriggered = 6;
    AllOn +=1;
  } else if (arduino_values[7] == 1 && prev_arduino_values[7] == 0) {
    background(25, 25, 60);
    plane.loop();
    clap.stop();
    wine.stop();
    coin.stop();
    globe.stop();
    lamp.stop();
    dove.stop();
    crown.stop();
    textFont(title, 60);
    fill(238, 233, 233);
    text("The Pilot", 500, 100);
    textFont(story, 40);
    text("I have lived a great deal among grown-ups.\n I have seen them intimately, close at hand.\n And that hasn't much improved my opinion of them.",200,200);
    pilot.play();
    lastImageTriggered = 7;
    AllOn +=1;
  } else if (arduino_values[8] == 1 && prev_arduino_values[8] == 0) {
    background(25, 25, 60);
    crown.loop();
    plane.stop();
    clap.stop();
    wine.stop();
    coin.stop();
    globe.stop();
    lamp.stop();
    dove.stop();
    textFont(title, 60);
    fill(160, 32, 240);
    kingall.play();
    text("Planet of King", 500, 100);
    textFont(story, 40);
    text("-Ah! Here's a subjet.", 500, 200);
    fill(255,255,255);
    text("-May I sit down?", 500, 250);
    delay(1000);
    fill(160, 32, 240);
    text("-I order you to do so.", 500, 300);
    fill(255,255,255);
    text("But the little prince was wondering . . . \nThe planet was tiny. \nOver what could this king really rule?",500,350);
    kingall.play();
    lastImageTriggered = 8;
    AllOn +=1;
  } else if (arduino_values[9] == 1 && prev_arduino_values[9] == 0) {
    background(25, 25, 60);
    start.stop();
    crown.stop();
    plane.stop();
    clap.stop();
    wine.stop();
    coin.stop();
    globe.stop();
    lamp.stop();
    dove.stop();
    ending.play();
    lastImageTriggered = 9;
    textFont(title, 60);
    fill(0, 191, 255);
    text("Earth", 500, 100);
    fill(255,255,200);
    text("Thank you for re-walking my journey!\nGrowing up is not the problem; forgetting is.", 50, 200);
    textFont(story, 40);
    text("And now here is my secret, a very simple secret:\n It is only with the heart that one can see rightly;\n what is essential is invisible to the eye.", 450, 400);
    AllOn +=1;
    thankyou.play();
  }


  if (lastImageTriggered == 0) {
    image(pic[0], xc[0], yc[0]);
  } else if (lastImageTriggered == 1) {
    image(pic[1], xc[1], yc[1]);
  } else if (lastImageTriggered == 2) {
    image(pic[2], xc[2], yc[2]);
  } else if (lastImageTriggered == 3) {
    image(pic[3], xc[3], yc[3]);
  } else if (lastImageTriggered == 4) {
    image(pic[4], xc[4], yc[4]);
  } else if (lastImageTriggered == 5) {
    image(pic[5], xc[5], yc[5]);
  } else if (lastImageTriggered == 6) {
    image(pic[6], xc[6], yc[6]);
  } else if (lastImageTriggered == 7) {
    image(pic[7], xc[7], yc[7]);
  } else if (lastImageTriggered == 8) {
    image(pic[8], xc[8], yc[8]);//king
  } else if (lastImageTriggered == 9) {
    image(pic[9], xc[9], yc[9]);
  }
  for (int i=0; i < NUM_OF_VALUES_FROM_ARDUINO; i=i+1) {
    prev_arduino_values[i] = arduino_values[i];
  }
}


void getSerialData() {
  //while (serialPort.available() > 0) {
  String in = serialPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  if (in != null) {
    print("From Arduino: " + in);
    String[] serialInArray = split(trim(in), ",");
    if (serialInArray.length == NUM_OF_VALUES_FROM_ARDUINO) {
      for (int i=0; i<serialInArray.length; i++) {
        arduino_values[i] = int(serialInArray[i]);
      }
    }
  }
  // }
}//Arduion to Processing

void initializeImages() {
  pic[0] = loadImage("b612.png");
  xc[0]= 1050;//b612
  yc[0]= 450;//b612
  pic[0].resize(400, 400);

  pic[1] = loadImage("geo.png");
  xc[1]= 1100;
  yc[1]= 50;
  pic[1].resize(280, 280);//geographer

  pic[2] = loadImage("businessman.png");
  xc[2]= 740;
  yc[2]= 80;
  pic[2].resize(300, 300);//businessman

  pic[3] = loadImage("lamp.png");
  xc[3]= 500;
  yc[3]= 400;
  pic[3].resize(150, 150);//lamplighter

  pic[4] = loadImage("earth.png");//dove
  xc[4]= 0;//dove
  yc[4]= 400; //dove
  pic[4].resize(500, 500);

  pic[5] = loadImage("drunkard.png");
  xc[5]= 475;
  yc[5]= 30;
  pic[5].resize(250, 250);//drunkard

  pic[6] = loadImage("thevain.png");
  xc[6]= 300;
  yc[6]= 150;
  pic[6].resize(200, 300);//thevain


  pic[7] = loadImage("earth.png");
  xc[7]= 0;//plane
  yc[7]= 400; //plane
  pic[7].resize(500, 500);

  pic[8] = loadImage("king.png");
  xc[8]= 30;
  yc[8]= 50;
  pic[8].resize(250, 300);//king

  pic[9] = loadImage("earth.png");
  xc[9]= 0;//earth
  yc[9]= 400; //earth
  pic[9].resize(500, 500);
}

Background music: “Little Prince” from Rolling Sky 2 Original Soundtrack

Sound of coins, wine, cash, switch, doves, globe, plane: https://freesound.org/

Pictures of the planets: retrieved from Pinterest.com and cut background with https://www.remove.bg/zh/upload

Sincere gratitude to Professor Gottfried, Jiaqi, and other fellows and LAs.

上春山(Climb Mountain in Spring)！ – Jingyi Ma – Gottfried Haider

Here’s the video for winning this game!

 

• CONTEXT AND SIGNIFICANCE

  In the previous classes, my teammate Jiaqi and I made three projects together: “Race the LED”, “Roating Link” and “Seeing Stars”. “Race the LED” is a project that requires two players to push a button 10 times as quickly as possible to win the game. “Roating Link” requires a stepper motor and a link to transfer cycle movements into back-and-forth movements. “Seeing Stars” is a project that requires using a servo motor and a magnet sensor. By triggering the magnet sensor, 5 stars will pop out from the back of your head. These projects require users’ input and deliver output for interaction. The first project requires pressing the button, the second project requires sending commands through programming on the computer, and the third project requires a digital signal. So in the midterm project, we wish to create a fun and interactive project that can use digital inputs, analog inputs, and servo motors at the same time. The main idea of our project 上春山(Climb Mountain in Spring) is a speed racing game that requires three players. We implemented the logic of “Race the LED” for a speed game, this time with three players. We use 360-degree servo motors to draw the belt for racing. This project combines the latest meme on Chinese social media “上春山”, a stage accident that happened at CMG Spring Festival Gala. In this accident, the actor wearing black forgot to go off the top of the “mountain” and kept staying there, and this accident quickly became a meme.  We decided to make this into a “mountain climbing” game, giving everyone an equal chance to stay at the top of the mountain!

Our target audience is the young students and staff that are familiar with this meme. As this is a widespread meme on the Chinese internet, this project creates a dramatic comic effect for its players, and people can play this game to ease their minds during the midterm week.

(If you are interested in what happened at CMG Gala, click here.)

• CONCEPTION AND DESIGN

Our goal is to create a mountain climbing speed racing game with three players. To win the game, the players need to push two buttons one by one at the fastest speed possible. The first person arriving at the mountain will have to sing (or yell) at the microphone for two seconds to win the game, or they will lose the game and be pushed back to the foot of the mountain.

To mimic the effect of “Climbing Mountain in Spring”, after listening to Prof. Gohai’s suggestions, we decided to make two buttons for each player. Only when they push the buttons one by one will it activate the servo motor. This prevents players from clicking a button too fast and also creates a life-like experience of climbing a mountain step by step. We used cardboard and sticks to make the base of the mountain, for they are easy to retrieve and are very solid, providing good support for the belts and motors.

We use plastic wrap roll to make the rollers of the belt. There were no big rolls in the studio, and a friend of ours, Maggie Wang, has a plastic wrap out of use. So we cut the roll in six, wrapped it with black tape, and created six rollers. They were connected by one-time chopsticks, for the sticks in Inter Lab are too short. 

Professors initially suggested we use rubber bands from the Fashion Design Lab, but we were afraid that they were not strong enough to pull cardboard figures, so we bought fitness rubber bands. It turned out that they were too hard to be pulled by 360-degree servo motors on user testing, so we switched to rubber bands again for the final version.

I planned to paint the mountain in gradient light green and dark green, with white paper clay on top of it to make a snow-covered “mountain in spring”, and also hide the end of chopsticks.

(The colored sketch of the mountain)

(The circuit diagram)

• FABRICATION AND PRODUCTION

The first step is to draw a sketch based on our proposed ideas. I initially wanted to make a “climbing step” the same as the CMG gala, but Jiaqi considered this would be too hard to make the figures climb “step by step”, so she suggested building a mountain and letting people climb up by belt and rolls. After a discussion with Prof. Gohai, we think Jiaqi’s plan is more feasible.

(The first sketch of the mechanism)

3/4: At the beginning of the construction process, Jiaqi cut three 30*40*50 pieces of cardboard as the main bracket structure of the mountain. We initially wanted to put the track on the “ridge” of the mountain, but we found it wouldn’t be feasible, for rollers are too wide. Then we move the track to the “valley” of the mountain. The first stage of the mountain construction looks like this. We borrowed three 360-degree servos from the ER room. We use small sticks to strengthen the base. 

3/5 :

We continue to build the base of the mountain and it looks a bit like a mountain now. 

3/6:

In the second phase, we started coding and building circuits. Jiaqi first wrote a small program to test the feasibility of a 360-degree motor on fitness bands, and if yelling at the microphone could trigger the motor to move backwards.  She tested the code and found the rubber band couldn’t move because there existed too much resistance between it and the cardboard, so she cut the “valleys” open and made space for the rubber band.  After that, the project can run for the first time. Here’s the code for Jiaqi’s testing program, with only one button and one servo. 

#include 

Servo myservo1;
Servo myservo2;  // Create servo object to control a servo
int buttonPin1 = 2;
int buttonState = 0;
int prevbuttonState = LOW;  
int servoAngle = 90;       
int buttonPressCount = 0;  
int analogPin = A0;
int analogInput;
int check1 = 7;
int button = LOW;
int prevbutton = LOW;
bool reached1 = false;

unsigned long total;
unsigned int count;

unsigned long whenStarted;
const unsigned long INTERVAL = 5000;  



void setup() {
  // initialize the pushbutton pin as an input:
  pinMode(buttonPin1, INPUT);
  pinMode(check1, INPUT);

  myservo1.attach(11);
  myservo2.attach(9);
  Serial.println(F("Starting ..."));
  whenStarted = millis();

  // myservo2.write(servoAngle);  
  Serial.begin(9600);
}

void loop() {
  analogInput = analogRead(analogPin);
  // Serial.print("Analog Input: ");
  // Serial.println(analogInput);

  buttonState = digitalRead(buttonPin1);
  reached1 = digitalRead(check1);

  if (buttonState == HIGH && prevbuttonState == LOW)  
  {
    buttonPressCount++;
    servoAngle += 5;
    myservo2.write(servoAngle);  
  } else if (buttonState == LOW && prevbuttonState == HIGH) {
    myservo2.write(90);
  }
  prevbuttonState = buttonState;
  // Serial.println(reached1);

  if (button == HIGH && prevbutton == LOW) {
    reached1 = true;
  }
  prevbutton = button;

  // Serial.println(reached1);

  if (reached1) {
    total += analogRead(A0);
    count++;

    if (millis() - whenStarted >= INTERVAL)  
    {
      Serial.print(F("Average = "));
      Serial.println(total / count);
      total = 0;
      count = 0;
      whenStarted = millis();
    }
    if (total / count >= 65) {
      myservo2.write(90);
    } else {
      myservo2.write(90);
      servoAngle -= 5;
      myservo2.write(servoAngle);  
    }
  }
}

(The mountain was cut open.)

3/7:

I was mainly focused on creating the code, especially the “one step at a time” mode with two buttons. I encountered many difficulties in writing this program, and I asked Prof. Gohai for help. Unfortunately, this version of code cannot satisfy the “one step at a time” goal. I turned to ask Siwei and Kevin, and Kevin instructed me to add Boolean values and a variable “playerAlastbutton” to distinguish between button 1 and button 2 of each player. Kevin also helped me how to make the 360-degree servo move for 100 milliseconds with each push by writing myservo1.(135) and myservo1.(90). Here’s the example code of one-by-one steps:

if (playerAbutton1state == HIGH && playerAbutton1prev == LOW && playerAlastButton == 0 && playerAButtonEnabled == true) {
      whenStartedA = millis();
      playerAButtonEnabled = false;
      // go up
      playerAposition = playerAposition + 1;
      playerAlastButton = 1;
    } else if (playerAbutton2state == HIGH && playerAbutton2prev == LOW && playerAlastButton == 1 && playerAButtonEnabled == true) {
      whenStartedA = millis();
      playerAButtonEnabled = false;
      playerAposition = playerAposition + 1;
      playerAlastButton = 0;
    }

    if (millis() <= whenStartedA + 200) {
      myservo1.write(135);
    } else if (millis() <= whenStartedA + 300) {
      myservo1.write(90);
    } else {
      playerAButtonEnabled = true;
    }

    Serial.print("player 敬亭: ");
    Serial.println(playerAposition); 

By this method, the Arduino detects the buttons’ original state and changing state. The servo will only move for 100 milliseconds if you push the buttons one by one. So the biggest problem has been solved, at least for the time being 😮 (Harder ones are on the way.)

As the mountain was large and we needed to hide wires inside it, we needed to make the wires longer. Jiaqi did the work of soldering and connecting male and female jumpers to make the wires longer. Then she connected the circuits of the project to the Arduino and breadboard. This is 3 wires per button (*6) and 3 wires per servo (*3), 27 wires with no count of other jump cables. So it was indeed very messy.

I drew 3 cardboard figures and glued them on the fitness band. In case any wires drop off, I fetched two hinges from Fab lab and made a door on one side of the mountain. This enabled us to open the mountain and check the wires. 

So far, you can go up the mountain one step at a time, and sing at the microphone, but there will be no punishment for not singing. At the 3/8 user testing, it is just a speed racing game.

(Testing the speed function)
 

3/8: 

Our friend from the Humanities, Yuchen came to user testing! We found that as the fitness bands are heavy and smooth, it’s hard for servos to draw them, and we encountered rollers skidding from time to time. There’s also no indication of the start of the game and the end of the game. There’s plenty of things to be fixed. 

3/10:

We wanted to make a stage 2 that would draw the player off the mountain if they reached the top but didn’t sing. The mechanism was we would give them 2 seconds to prepare, and they had to sing for 2 seconds to stay at the top of the mountain. 

Jiaqi connected a microphone sensor to the top of the mountain by “twisting sticks.” We initially made a paper tube for holding the microphone, but now there were too many wires in the mountain that we couldn’t implement the tube. We initially wanted to use digital inputs, but as long as there’s some noise, the microphone reads 1. She then switched to analog mode. Luckily, it worked this time. She made the belts tighter and lighter to move more smoothly. 

Still, we’re struggling to connect the buzzer and push the player off the mountain. After testing, we found that it requires 23 steps for players B and C up the mountain, but 25 steps for player A. As the mountain has been fixed, we couldn’t do it again. This became a flaw in the project that player A needed to press two more times in stage 1. 

We were struggling in the “2 seconds to prepare and sing for 2 seconds” part of coding. I went to ask Rudi for help, but unfortunately, we had some misunderstandings in the communication process. As the code became very complicated, we thought there was nothing serious with the code but it reported errors all the time!!! At last, it was just a missing bracket, and the program could run successfully. Still, we are missing the music.

3/11:

There was still no indication of the start of the game and the end of the game. After a failure to transfer music into code, I decided to compose the music by myself. Luckily, it was a pentatonic scale and I only required 8 notes for two rhythms. 

(The starting music)

(The ending music)

The music “上春山” is really playful. I’m giving the code of the first and second pieces of music here if anyone wants:

int melody[] = {
  NOTE_E5, NOTE_CS5, NOTE_B4, NOTE_CS5, NOTE_E5, 0, NOTE_CS5, NOTE_B4, NOTE_GS4, NOTE_B4, NOTE_CS5, NOTE_GS4, NOTE_B4, NOTE_CS5, NOTE_B4, NOTE_GS4, NOTE_FS4, 0, NOTE_E4, NOTE_CS4,  NOTE_E4, 0,  NOTE_E4,  NOTE_FS4,  NOTE_E4,  NOTE_E4,  NOTE_E4
};

// note durations: 4 = quarter note, 8 = eighth note, etc.:
int noteDurations[] = {
  8, 16, 8, 8, 4, 8, 16, 16,8, 8, 8, 8, 2,4, 8, 8, 8, 8, 8, 8, 4, 8, 16, 16, 8, 8, 4
};

 

int melody[] = {
      NOTE_CS5, NOTE_B4, NOTE_CS5, NOTE_E5, 0, NOTE_GS4, NOTE_FS4, NOTE_GS4, NOTE_B4, 0, NOTE_CS5, NOTE_B4, NOTE_GS4, NOTE_FS4, NOTE_E4, NOTE_CS4, NOTE_CS5, NOTE_B4, NOTE_FS4, NOTE_GS4, 0, NOTE_CS5, NOTE_B4, NOTE_CS5, NOTE_E5, 0, NOTE_GS4, NOTE_FS4, NOTE_GS4, NOTE_B4, 0, NOTE_CS5, NOTE_B4, NOTE_GS4, NOTE_FS4, NOTE_E4, 0, NOTE_CS4, NOTE_E4, 0, NOTE_E5, NOTE_FS5, NOTE_E5, NOTE_E5, NOTE_E5
    };
    int noteDurations[] = {
      4, 8, 8, 4, 4, 4, 8, 8, 4, 4, 4, 8, 8, 4, 8, 8, 4, 8, 8, 4, 4, 4, 8, 8, 4, 4, 4, 8, 8, 4, 4, 4, 8, 8, 8, 8, 8, 8, 4, 8, 16, 16, 8, 8, 4
    };

When I wanted to input the music of the buzzer to the code, bugs appeared again and again, as the music would play without stopping. After discussing with Siwei, I set up stage 3 to make the music play only once.

At this stage, we calculated it would require 4 seconds to push the figure off the mountain. So we wrote the code to make the servo write (0) for 4000 milliseconds and stop. 

3/12:

Now we arrived at the final stage of the project for presentation. I didn’t know if my computer got nervous or not, but the project failed 2 times before it worked properly. Luckily, we survived the presentation and reached the wanted effects!!!

The full code of the project is here:

#include <Servo.h>

#include "pitches.h"
Servo myservo1;
Servo myservo2;
Servo myservo3;  // Create servo object to control a servo
int MicrophoneVal;
bool playerAButtonEnabled = true;
bool playerBButtonEnabled = true;
bool playerCButtonEnabled = true;
long state2startTime;
long xiashan1;
long xiashan2;
long xiashan3;

bool state2FirstRun = true;
int state1Winner = 0;

int state = 1;
int playerAbutton1pin = 2;
int playerAbutton2pin = 4;
int playerAbutton1state;
int playerAbutton2state;
int playerAbutton1prev;
int playerAbutton2prev;
int playerAlastButton = 0;
int playerAposition = 0;

int playerBbutton1pin = 6;
int playerBbutton2pin = 8;
int playerBbutton1state;
int playerBbutton2state;
int playerBbutton1prev;
int playerBbutton2prev;
int playerBlastButton = 0;
int playerBposition = 0;

int playerCbutton1pin = 10;
int playerCbutton2pin = 12;
int playerCbutton1state;
int playerCbutton2state;
int playerCbutton1prev;
int playerCbutton2prev;
int playerClastButton = 0;
int playerCposition = 0;

unsigned long total;
unsigned int count;
const unsigned long INTERVAL = 5000;
unsigned long whenStartedA = 0;
unsigned long whenStartedB = 0;
unsigned long whenStartedC = 0;
unsigned long Singing = 0;

void setup() {
  Serial.begin(9600);
  myservo1.attach(7);
  myservo2.attach(9);
  myservo3.attach(11);
  Serial.println(F("******************* 上春山 *******************"));  //开始
  delay(3000);
  //麦克风在3pin
  pinMode(A0, INPUT);
  int melody[] = {
  NOTE_E5, NOTE_CS5, NOTE_B4, NOTE_CS5, NOTE_E5, 0, NOTE_CS5, NOTE_B4, NOTE_GS4, NOTE_B4, NOTE_CS5, NOTE_GS4, NOTE_B4, NOTE_CS5, NOTE_B4, NOTE_GS4, NOTE_FS4, 0, NOTE_E4, NOTE_CS4,  NOTE_E4, 0,  NOTE_E4,  NOTE_FS4,  NOTE_E4,  NOTE_E4,  NOTE_E4
};

// note durations: 4 = quarter note, 8 = eighth note, etc.:
int noteDurations[] = {
  8, 16, 8, 8, 4, 8, 16, 16,8, 8, 8, 8, 2,4, 8, 8, 8, 8, 8, 8, 4, 8, 16, 16, 8, 8, 4
};
  for (int thisNote = 0; thisNote < 27; thisNote++) {

    // to calculate the note duration, take one second divided by the note type.
    //e.g. quarter note = 1000 / 4, eighth note = 1000/8, etc.
    int noteDuration = 1000 / noteDurations[thisNote];
    tone(3, melody[thisNote], noteDuration);

    // to distinguish the notes, set a minimum time between them.
    // the note's duration + 30% seems to work well:
    int pauseBetweenNotes = noteDuration * 1.30;
    delay(pauseBetweenNotes);
    // stop the tone playing:
    noTone(3);
  }
  state = 1; 
}




void loop() {
  if (state == 1) {
    state1();
  } else if (state == 2) {
    state2();
  } else if (state == 3) {
    state3();
  }
}

  void state1() {
    // ascent state
    //player A
    playerAbutton1state = digitalRead(playerAbutton1pin);
    playerAbutton2state = digitalRead(playerAbutton2pin);

    if (playerAbutton1state == HIGH && playerAbutton1prev == LOW && playerAlastButton == 0 && playerAButtonEnabled == true) {
      whenStartedA = millis();
      playerAButtonEnabled = false;
      // go up
      playerAposition = playerAposition + 1;
      playerAlastButton = 1;
    } else if (playerAbutton2state == HIGH && playerAbutton2prev == LOW && playerAlastButton == 1 && playerAButtonEnabled == true) {
      whenStartedA = millis();
      playerAButtonEnabled = false;
      playerAposition = playerAposition + 1;
      playerAlastButton = 0;
    }

    if (millis() <= whenStartedA + 200) {
      myservo1.write(135);
    } else if (millis() <= whenStartedA + 300) {
      myservo1.write(90);
    } else {
      playerAButtonEnabled = true;
    }

    Serial.print("player 敬亭: ");
    Serial.println(playerAposition);
    //player B 魏晨
    playerBbutton1state = digitalRead(playerBbutton1pin);
    playerBbutton2state = digitalRead(playerBbutton2pin);

    if (playerBbutton1state == HIGH && playerBbutton1prev == LOW && playerBlastButton == 0 && playerBButtonEnabled == true) {
      whenStartedB = millis();
      playerBButtonEnabled = false;
      // go up
      playerBposition = playerBposition + 1;
      playerBlastButton = 1;
    } else if (playerBbutton2state == HIGH && playerBbutton2prev == LOW && playerBlastButton == 1 && playerBButtonEnabled == true) {
      whenStartedB = millis();
      playerBButtonEnabled = false;
      playerBposition = playerBposition + 1;
      playerBlastButton = 0;
    }

    if (millis() <= whenStartedB + 200) {
      myservo2.write(135);
    } else if (millis() <= whenStartedB + 300) {
      myservo2.write(90);
    } else {
      playerBButtonEnabled = true;
    }
    Serial.print("player 魏晨: ");
    Serial.println(playerBposition);

    //player C
    playerCbutton1state = digitalRead(playerCbutton1pin);
    playerCbutton2state = digitalRead(playerCbutton2pin);

    if (playerCbutton1state == HIGH && playerCbutton1prev == LOW && playerClastButton == 0 && playerCButtonEnabled == true) {
      whenStartedC = millis();
      playerCButtonEnabled = false;
      // go up
      playerCposition = playerCposition + 1;
      playerClastButton = 1;
    } else if (playerCbutton2state == HIGH && playerCbutton2prev == LOW && playerClastButton == 1 && playerCButtonEnabled == true) {
      whenStartedC = millis();
      playerCButtonEnabled = false;
      playerCposition = playerCposition + 1;
      playerClastButton = 0;
    }

    if (millis() <= whenStartedC + 200) {
      myservo3.write(135);
    } else if (millis() <= whenStartedC + 300) { myservo3.write(90); } else { playerCButtonEnabled = true; } Serial.print("player 大勋: "); Serial.println(playerCposition); if (playerAposition >= 25) {
      myservo1.write(90);
      myservo2.write(90);
      myservo3.write(90);
      Serial.println("End of Climbing!!!");
      state = 2;
      state1Winner = 1;
      delay(4000);
      //Serial.println("Listening to microphone");
    } else if (playerBposition >= 23) {
      myservo1.write(90);
      myservo2.write(90);
      myservo3.write(90);
      Serial.println("End of Climbing!!!");
      state = 2;
      state1Winner = 2;
      delay(4000);
    } else if (playerCposition >= 23) {
      myservo1.write(90);
      myservo2.write(90);
      myservo3.write(90);
      Serial.println("End of Climbing!!!");
      state = 2;
      state1Winner = 3;
      delay(4000);
    }

    playerAbutton1prev = playerAbutton1state;
    playerAbutton2prev = playerAbutton2state;
    playerBbutton1prev = playerBbutton1state;
    playerBbutton2prev = playerBbutton2state;
    playerCbutton1prev = playerCbutton1state;
    playerCbutton2prev = playerCbutton2state;
  }

  // singing state
  void state2() {
    if (state2FirstRun == true) {
      state2startTime = millis();
      state2FirstRun = false;
    }
    int MicrophoneVal = analogRead(A0);
    Serial.println(MicrophoneVal);
    delay(10);

    if (millis() - state2startTime < 2000 && MicrophoneVal < 56) {
      if (state1Winner == 1) {
        myservo1.write(0);  // 敬亭下山
        delay(4000);
        myservo1.write(90);
        Serial.println("敬亭 lost.");
        state = 3;
      } else if (state1Winner == 2) {
        myservo2.write(0);  // 魏晨下山
        delay(4000);
        myservo2.write(90);
        Serial.println("魏晨 lost.");
        state = 3;
      } else if (state1Winner == 3) {
        myservo3.write(0);  // 大勋下山
        delay(4000);
        myservo3.write(90);
        Serial.println("大勋 lost.");
        state = 3;
      }
    } else {
      Serial.println("You win!!!");
      delay(2000);
      state = 3;
    }
  }

  void state3() {

    Serial.println("End of Game.");
    int melody[] = {
      NOTE_CS5, NOTE_B4, NOTE_CS5, NOTE_E5, 0, NOTE_GS4, NOTE_FS4, NOTE_GS4, NOTE_B4, 0, NOTE_CS5, NOTE_B4, NOTE_GS4, NOTE_FS4, NOTE_E4, NOTE_CS4, NOTE_CS5, NOTE_B4, NOTE_FS4, NOTE_GS4, 0, NOTE_CS5, NOTE_B4, NOTE_CS5, NOTE_E5, 0, NOTE_GS4, NOTE_FS4, NOTE_GS4, NOTE_B4, 0, NOTE_CS5, NOTE_B4, NOTE_GS4, NOTE_FS4, NOTE_E4, 0, NOTE_CS4, NOTE_E4, 0, NOTE_E5, NOTE_FS5, NOTE_E5, NOTE_E5, NOTE_E5
    };

    int noteDurations[] = {
      4, 8, 8, 4, 4, 4, 8, 8, 4, 4, 4, 8, 8, 4, 8, 8, 4, 8, 8, 4, 4, 4, 8, 8, 4, 4, 4, 8, 8, 4, 4, 4, 8, 8, 8, 8, 8, 8, 4, 8, 16, 16, 8, 8, 4

    };

    for (int thisNote = 0; thisNote < 45; thisNote++) {

      // to calculate the note duration, take one second divided by the note type.
      //e.g. quarter note = 1000 / 4, eighth note = 1000/8, etc.
      int noteDuration = 1000 / noteDurations[thisNote];
      tone(3, melody[thisNote], noteDuration);

      // to distinguish the notes, set a minimum time between them.
      // the note's duration + 30% seems to work well:
      int pauseBetweenNotes = noteDuration * 1.30;
      delay(pauseBetweenNotes);
      // stop the tone playing:
      noTone(3);
    }
    while (true);
  }

• CONCLUSIONS

Our project’s mechanism achieved the goal, but it still has some flaws. It’s a speed game with three stages. At the beginning setup() function, a piece of music will be played and enters the first stage. In the first stage, 3 players will race the speed of climbing up a mountain. When a player hits the required times, they reach the top of the mountain. They have two seconds to prepare and two seconds to sing at the microphone to win the game. If they don’t do as the requirements, the player who reaches the top first will be drawn back for 4 seconds to the foot of the mountain. Then, the program enters stage 3, another piece of music will be played and marks the end of the game.

In my Read and Analysis I, I said that “Interaction is the process of verbal or non-verbal “conversation” between the two entities, and as the conversations have differences from simple to complex, so as interactions.” This project requires players to push buttons after hearing the music and yell at the microphone after reaching the top of the mountains. So there’s rich interaction in this process. 

(A successful example of winning the game)

Still, there are some flaws in this project. The first one is that player A will require two more pushes to win, for their track is slightly longer than B and C. The second one is that there’s no indication of who’s the first to reach the top of the mountain and no indication of winners and losers. Also, the microphone can’t detect who’s yelling. If one player is drawn to the foot of the mountain, there will be no winners. 

 

(If you lose…you will go down, other people remain on the mountain, but there will be no winners.)

If I had more time, I would arrange 1 LED light and 1 Microphone for each player. If the player hits the top, the lights will be on and show the winner. And the player only needs to yell at the microphone in front of them. If one player fails to yell, the other player’s LED will be on and show the winner has been changed. (Though, I guess there will be soooooo many wires, I have no idea if the Arduino can hold them.

I learned that when I meet any bugs in my coding, I need to be patient. I should use the Auto format at first, and check for any missing } or ; before turning for help. I also need to be calmer about any possible flaws in the building process. I’m really glad that our project brings people plenty of joy in the midterm week, but I am also aware that I need to be better at time management when doing the final project text time. 

• DISASSEMBLY:

(We put all the cardboard into the trash bin.)

Special thanks to my teammate Jiaqi Qian, instructor Prof. Gohai, fellow Kevin, LA Siwei, and Prof. Andy.

• APPENDIX

“上春山” meme:

https://www.google.com/url?sa=i&url=https%3A%2F%2Fm.36kr.com%2Fp%2F2654994601017601&psig=AOvVaw2Om7Fk1atArnmN0IBQXQ-b&ust=1710677183151000&source=images&cd=vfe&opi=89978449&ved=0CBEQjRxqFwoTCMCroaff-IQDFQAAAAAdAAAAABAD