Final Project: 6. Report

A. Excalibur – Anita Luo & Kiana Ng – Inmi Lee

 



 

*Photographs taken by IXL Fellow, Shengli 

B.CONCEPTION AND DESIGN

Fiction offers creative reconstructions of events that narrate and organize behavior through the representation and implementation of space—shaping how the audience perceives and interacts with their environment. Games, for example, deviate from reality to make an illusion playable as they channel agency into new forms on the screen. Through actualizing fiction, “Excalibur” aims to alter the conventional game approach of reality-in-illusion by bringing illusion to reality and the physical space—reversing the roles. Taking inspiration from Star Wars’ lightsaber roleplay toy, “Excalibur” embeds the concept of “illusion-in-reality” by implementing familiar game strategies such as a timer, heart count (health tracker), competition, roleplaying, and sound and visual effects that observe the user’s real-time movements. Consequentially, the project also contributes to the theory that gamification, the application of elements of game-playing to other areas of activity, can enhance engagement and satisfaction. Additionally, the project is an extension of virtual games into the real world: by using the same types of manipulation adopted by virtual games, such as employing sound effects in a certain space, “Excalibur” constructs a virtual space without the need for a VR headset—introducing the user with convenience and innovation. The project is designed in a way that makes the game playable independent of lore, age, and physical fitness to fit a wider audience consisting of not just children. To do so, the option to skip or disregard elements in the game was welcome; players were given the option to skip the introduction explaining the character’s background and game instructions if applicable. Furthermore, disregarding game constraints, such as the game rules, did not put the project at a disadvantage as players could take pleasure in the system’s fast response to their actions through sound and visual effects accompanied by whimsical medieval background music. In terms of sound, the project used serial communication between Arduino and Processing to add sound effects based on the copper attached to two swords, two shields, and armor which allows electricity to flow to create a closed circuit—creating a switch. The simple and single use of switches as the foundational framework of the project ensures fast system response and easier troubleshooting. During user testing, it was suggested to add images of the characters, King Arthur and Mordred, in their respective positions to the players on the monitor to differentiate their roles. By using media objects as a representation of them, the players can position themselves within the context of a duel, to embody openness and experimentation in roleplaying. Secondly, it was recommended to incorporate game states with an open menu, instructions, the actual gameplay, and an ending screen to create an experience where the user sees the system’s response, and takes more action in response, in a continuous loop. Furthermore, including an instruction state stimulates the player to listen, think, and respond to the system as well as gives their gameplay a meaningful aim and purpose. Both suggestions from the user testing were effective. While analyzing the user’s experience during our end-of-semester IMA show, we found that most players would look at the monitor to make sense of the situation and formulate their next plan of action which made the system itself a great communicator.

Initial sketch

 

Revised sketch

*Note: Small changes were made to the final sketch

User’s Interactions




 

C. FABRICATION AND PRODUCTION

Materials

  • 1x Breadboard
  • 1x Arduino Uno
  • 1x USB Cable
  • 1x USB Protector
  • 4x 10 kOhm Resistors
  • A handful of M/M jumper cables
  • Jumper cables from the studio
  • 1x Ruler
  • 1x Hot glue gun
  • 1x Black cloth
  • 1x Brown cloth
  • 1x White cloth
  • 1x Silver cloth
  • 1x Tablecloth
  • 1x Strip of fake gemstones
  • 2x Copper tape
  • 1x Scissors
  • 1x Wire cutters
  • 1x Cutting board
  • Cardboard
  • 1x Masking tape
  • 1x Marker
  • 1x Electrical tape
  • Soldering set
  • 1x Wood for laser cutting

Not included anymore:

  • 3x Accelerometers

Project Plan

During the proposal phase, I created a Gantt Chart for the “Excalibur” Project so Kiana and I would have a better plan of action.

 

Lookback: One of the flaws of our midterm project was our lack of planning. As a result, many of the steps in the respective areas of coding, physical computing, and building did not work together. There were moments when we had to temporarily stop on one step and go back to working on another component of the project so that we could continue our last step because there was not a clear picture of how everything worked together. This stagnation affected our productivity and morale throughout our midterm project which caused us to have last-minute obstacles that we could not address before the final critique. To fix this mistake, we utilized a Gantt Chart for the final project to plan how every step correlated with the last having specific objectives, and how every step was building us towards a final goal. I believe the reason why we were able to finish the project on time with extra time to make additional adjustments was because of proper planning and staying on track; not much time was wasted which gave us extra time to do other work in other courses. By learning and proactively addressing past mistakes, the process of creating the project was much faster, and our experience was more rewarding and fun!

We also made a flow chart to understand the essential components of our game.

 

Building the Circuit (Anita and Kiana)

Step 1: Testing the accelerometer

 

Our initial idea was to use the accelerometer to detect changes to the sword orientation to make sword sheath sound effects to immerse the players in the idea of a manipulated reality. Additionally, we wanted to add a wand for Merlin, a wizard in the Excalibur story as an additional role for the audience if they would like to add special effects on the screen without affecting the game. Firstly, with the recommendation from Professor Lee, I consulted with the tutorial on how to use an accelerometer from the IXl tutorial repository. In this step, I was able to calibrate the accelerometers with xyz.

Secondly, the professor assisted me further by giving me a code on how we can send xyz values from the Arduino to Processing with the following:

void setup() {
  Serial.begin(9600);
}

void loop() {
  // to send values to Processing assign the values you want to send
  // this is an example:
  int sensor0 = analogRead(A0);
  int sensor1 = analogRead(A1);
  int sensor2 = analogRead(A2);

  // send the values keeping this format
  Serial.print(sensor0);
  Serial.print(",");  // put comma between sensor values
  Serial.print(sensor1);
  Serial.print(","); // add linefeed after sending the last sensor value
  Serial.print(sensor2);
  Serial.println();  // add linefeed after sending the last sensor value

  // too fast communication might cause some latency in Processing
  // this delay resolves the issue
  delay(100);

  // end of example sending values
} 

Lookback: While the code worked, we chose not to use the accelerometer because

1) its purpose was useful but not essential to the game design

2) there would be additional cautions to consider when attaching the sensor to the weapons

3) allowing single-player (one of the main reasons why we wanted to implement this sense) could be a distraction rather than a special addition to the game that would confuse the players with what the objective of the game is. When we initially wanted a single-player option, the idea of a health tracker and timer in the game rules had not been conceptualized so due to the recent changes, our old ideas were reevaluated and discarded.

4) No Merline anymore 🙁   (would be too distracting)

Lookback: Reevaluating every component in a project continuously throughout the project-making process is very valuable. Asking oneself what the significance and value of each component is a useful skill to adopt and probably the responsibility of the creator to ensure that all the elements of a project work together effectively to produce a meaningful and interactive product for the audience. Furthermore, as the making process progresses sometimes new elements can affect the old. Unnecessary or ineffective components could obscure the vision and mission of the product when the users interact with it: sometimes less is more.

Step 2: Prototype

For the prototype in user testing, I used a simple diagram from the internet to assist me in building the circuit. I consulted with an Arduino switch diagram because the mechanisms of the weapons were based on the idea of a switch creating a closed circuit when different weapons touched. For this, I used digital pins 2-4 to detect a 0 or 1 to send to Processing.

Digital Read Serial | Arduino Documentation

Source: docs.arduino

In the place of the switch would be the weapons. For the wires attached to the weapons, I used stranded wire which Kiana and I connected with solid wires for better connection with the breadboard.

 

https://drive.google.com/file/d/1Tb4Ia8wRYc3gnT6zFarGyT7_Mp6AMZOY/view 

Obstacle: It was difficult to differentiate the wires because the colors were the same and the wires were too long to trace back which object was which. Thus, we decided to label the wires. This step, along with Kiana’s help, was an effective step for me to quickly find wires in the case of troubleshooting.

 

Step 3: Final adjustments

 

 

We were also recommended to add the stone that King Arthur drew the sword from, in user-testing, because it would further illustrate the story of The Legend of Excalibur. This was a simple step in which I added one more digital pin value called “stonePin” on digital pin 5. 

We laser cute new objects and for this step, I attached solid wires to the copper on the weapons.

Obstacle: During user-testing, we found that the stranded wires would be tangled together because of their flexibility and material. To fix this issue, we used solid wires instead that would be more difficult to bend as well as cloth covering over it created even more friction. We found that the wires would not tangle up anymore and the cloth also added a professional touch to it.


 

Writing the code (Anita and Kiana)

 

I used Canva to create my video with voice narration by an AI voice generator website called ElevenLabs. Royalty-free sound effects were downloaded from Pixabay.

Obstacle: ElevanLabs required a subscription to continue the use of their service. Unfortunately, this meant that we could not fulfill our initial idea of having a storyline incorporated into the installation as we only had a limited amount of uses. If we had more time, we would have tried to consider a different narrator, maybe a real-life person, to do all the necessary voiceovers to make our game more complete, customizable, and unique to the audience. Furthermore, having voiceovers for the characters themselves would also have been a good addition to the game (especially in the ending state with different endings).

Images used:

Lookback: The midterm taught me a lot about the importance of synergy and teamwork. Therefore, for this project, I aimed to delegate as much of the codework as possible. Furthermore, I was more open to assistance from the learning assistants and fellow in Interaction Lab. Firstly, Kiana and I delegated the task of coding by being in charge of the coding for the different software respectively, namely Arduino and Processing. Because of clear communication and discussion, we both had a good idea of how the two codes would interact with each other; Arduino values were sent to Processing. Because Processing required more coding we compensated the difference between workload by giving Kiana more work in the building step. I felt a noticeable difference in my stress levels; I had a lot of pleasure writing the code this time. 

Obstacle: I encountered a multitude of issues while coding in Processing. Firstly, I had to learn game states which were not covered in our coursework. To do so, I looked at this YouTube tutorial. Independent learning is effective when the task is small. However, I encountered more problems with bigger issues like sound. The sounds would often overlap or play more than once. I struggled with the concepts of boolean, time (millis), and sound conditionals. Learning assistant, Daniel, and IXL fellow, Kevin, helped me extensively during this time. Additionally, Kevin helped me a lot on how to add videos to the “slide1” state as we did not learn how to add videos in class. This part took a long time (2+ hours) as we found the sound of the video continued after the state ended. Furthermore, we found that having void movieEvent(Movie m) was an essential part of playing the video on the screen (it had not been playing before adding this). Through experimentation, we were able to add a video that I created beforehand to the project. I found how effective it was to work together with others which I had not done in the midterm; I had more motivation and fun working on the project with others. I also found how supportive the IMA community is at NYU Shanghai!

 

Lookback: I also organized, formatted, and titled sections of my code for better viewing.

Arduino Code:

const int shieldPin = 2; // Digital pin for shield 1
const int swordPin = 3; // Digital pin for sword 1
const int heartPin = 4;
const int stonePin = 5;

void setup() {
 Serial.begin(9600); // Initialize serial communication
 pinMode(swordPin, INPUT); // Set sword  pin as input
 pinMode(shieldPin, INPUT); // Set shield  pin as input
 pinMode(heartPin, INPUT);
 pinMode(stonePin, INPUT); // Set shield  pin as input
}

void loop() {
 int swordValue = digitalRead(swordPin); // Read value from sword 1 pin
 int shieldValue = digitalRead(shieldPin); // Read value from shield 1 pin
 int heartValue = digitalRead(heartPin);
int stoneValue = digitalRead(stonePin); // Read value from shield 1 pin


// if ((swordValue == HIGH) || (shieldValue == LOW)) {
// )) Serial.println("1");

Serial.print(swordValue);
Serial.print(",");  // put comma between sensor values
Serial.print(shieldValue);
Serial.print(",");  // put comma between sensor values
Serial.print(heartValue);  // add linefeed after sending the last sensor value
Serial.print(",");
Serial.print(stoneValue);
Serial.println();

  // too fast communication might cause some latency in Processing
  // this delay resolves the issue
  delay(20);
} 

Processing Code:

String gameState;

import processing.serial.*;
import processing.sound.*;
import processing.video.*;

Serial serialPort;

int NUM_VALUES_FROM_ARDUINO = 4;
int arduino_values[]= new int[NUM_VALUES_FROM_ARDUINO];

//number of times the heart is attacked
int heartcounter = 0;

//images
PImage bg;
PImage heart;
PImage holyg;
PImage one;
PImage two;
PImage three;
PImage four;
PImage arthur;
PImage mordred;
PImage strike1;
PImage strike2;
PImage strike3;
int randomVal = 0;

//soundfiles
SoundFile bgm;
SoundFile countdown;
SoundFile hearts;
SoundFile sword;
SoundFile shield;
SoundFile start;
SoundFile startmenu;
SoundFile end;
SoundFile tick;

//video
Movie video1;
int previousValue;

//booleans
boolean playStartOnce = false;
boolean playEndOnce = false;
int starts = -1;

//fonts
PFont font;

//timer
String time = "60";
int t;
int interval = 60;
long startMillis;

//touching of swords and shields
int lastTouch;
int dist;

void setup() {
  fullScreen();

  gameState = "STARTMENU";
  //gameState values: STARTMENU, GAME, ARTHUR, MORDRED

  printArray(Serial.list());
  serialPort = new Serial(this, "COM5", 9600);

  //images
  bg = loadImage("Medieval Background.jpg");
  heart = loadImage("Pixel Heart.png");
  holyg = loadImage("Holy Grail.png");
  one = loadImage("1.png");
  two = loadImage("2.png");
  three = loadImage("3.png");
  four = loadImage("5.png");
  arthur = loadImage("King Arthur 1.png");
  mordred = loadImage("Mordred.png");
  strike1 = loadImage("Strike 1.png");
  strike2 = loadImage("Strike 2.png");
  strike3 = loadImage("Strike 3.png");
  //String[] strike = { "strike1", "strike2", "strike3"};
  //int index = int(random(strike.length));

  //fonts
  font = createFont("Arial", 30);

  //sound files
  bgm= new SoundFile(this, "Medieval Noble Music.mp3");
  countdown= new SoundFile(this, "Clock tick.mp3");
  hearts= new SoundFile(this, "Attack Heart.mp3");
  sword= new SoundFile(this, "Attack Sword.mp3");
  shield= new SoundFile(this, "Attack Shield.mp3");
  start= new SoundFile(this, "Starting sound.mp3");
  startmenu= new SoundFile(this, "Medieval Carnival Music.mp3");
  end= new SoundFile(this, "Endgame.mp3");
  tick = new SoundFile (this, "Clock tick.mp3");

  //video files
  video1 = new Movie(this, "Long version (online-video-cutter.com).mp4");
}

void draw() {

  getSerialData();

  if (gameState == "STARTMENU") {
    startmenu();
  } else if (gameState == "SLIDE1") {
    slide1();
  } else if (gameState == "GAME") {
    game();
  } else if (gameState == "ARTHUR") {
    arthur();
  } else if (gameState == "MORDRED") {
    mordred();
  } else {

    println ("something went wrong with gameState");
  };
  previousValue = arduino_values[1];
}

void startmenu() {
  image(one, 0, 0);
  if (startmenu.isPlaying()==false) {
    startmenu.loop();
  }
  if (arduino_values[3] == 0) {
    gameState = "SLIDE1";
  }
} //end startmenu

void slide1() {
  startmenu.pause();
  image(video1, 0, 0);
  video1.play();

  if (arduino_values[1] == 1 && previousValue == 0) {
    gameState = "GAME";
  }
  startMillis = millis();
}

void game() {
  video1.stop();
  if (start.isPlaying()==false && playStartOnce == false) {
    playStartOnce = true;
    start.play();
  }

  if (start.isPlaying() == false && bgm.isPlaying() == false && playStartOnce == true) {
    bgm.loop();
  }

  image(bg, 0, 0);
  //image(arthur, 30, 100);
  //image(mordred, 950, 170);
  image(heart, 1300, 5);
  image(heart, 1180, 5);
  image(heart, 1060, 5);

  //timer
  textSize(100);
  text(time, 650, 160);
  textSize(75);
  text("Timer", 630, 70);
  textSize(55);
  text("Arthur", 300, 900);
  text("Mordred", 1000, 900);
  t = interval-int((millis()-startMillis)/1000);
  time = nf(t, 3);
  
  if (t == 5 &&  tick.isPlaying() == false) {
    tick.play();
  }
  
  if (t == 0) {
    gameState = "MORDRED";
  }
  if (heartcounter==1 && end.isPlaying() == false) {
    image(bg, 0, 0);
    textSize(90);
    text(time, 650, 160);
    textSize(50);
    text("Timer", 660, 50);
    textSize(55);
    text("Arthur", 350, 900);
    text("Mordred", 1000, 900);
    image(heart, 1180, 5);
    image(heart, 1060, 5);
    //end.play();
  }

  if (heartcounter==2 && end.isPlaying() == false) {
    image(bg, 0, 0);
    textSize(90);
    text(time, 650, 160);
    textSize(50);
    text("Timer", 660, 50);
    textSize(55);
    text("Arthur", 350, 900);
    text("Mordred", 1000, 900);
    image(heart, 1060, 5);
    //end.play();
  }

  if (heartcounter==3 && end.isPlaying() == false) {
    image(bg, 0, 0);
    textSize(90);
    text(time, 650, 160);
    textSize(50);
    text("Timer", 660, 50);
    textSize(55);
    text("Arthur", 350, 900);
    text("Mordred", 1000, 900);
    //end.play();
  }

  if (heartcounter==3) {
    heartcounter = 0;
    gameState = "ARTHUR";
  }

  if (t==0) {
    heartcounter = 0;
    gameState = "MORDRED";
  }

  // play audio based on received value
  if (arduino_values[0] == 1 && shield.isPlaying() == false) {
    shield.play();
    lastTouch = millis();
    randomVal = floor(random(1, 4));
  }

  if (arduino_values[1] == 1 && sword.isPlaying() == false) {
    sword.play();
    lastTouch = millis();
    randomVal = floor(random(1, 4));
  }

  if (arduino_values[2] == 1 && hearts.isPlaying() == false) {
    hearts.play();
    heartcounter++;
  }



  dist = millis()-lastTouch;
  dist = constrain(dist, 50, 300);
  image(arthur, 500-dist, 100);
  image(mordred, 600+dist, 170);

  if (millis()-lastTouch<200) {
    if (randomVal == 1) {
      image(strike1, 0, 0);
    } else if (randomVal == 2) {
      image(strike2, 0, 0);
    } else if (randomVal == 3) {
      image(strike3, 0, 0);
    }
  }
}

void arthur() {
  //int startTime = millis();
  image(four, 0, 0);
  bgm.pause();
  tick.pause();
  if (playEndOnce == false) {
    playEndOnce = true;
    end.play();
  }
  //if (millis()-startTime >=4000) {
  if (arduino_values[3] == 1) {
    gameState = "STARTMENU";
  }
}

void mordred() {
  image(four, 0, 0);
  bgm.pause();
  tick.pause();
  if (playEndOnce == false) {
    playEndOnce = true;
    end.play();
  };
  if (arduino_values[3] == 1) {
    gameState = "STARTMENU";
  };
}

void movieEvent(Movie m) {
  m.read();
}

void getSerialData() {
  while (serialPort.available() > 0) {
    String in = serialPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
    if (in != null) {
      print("From Arduino: " + in);
      String[] serialInArray = split(trim(in), ",");
      if (serialInArray.length == NUM_VALUES_FROM_ARDUINO) {
        for (int i=0; i<serialInArray.length; i++) {
          arduino_values[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

Building the Installation (Anita and Kiana)

Step 1: Prototype

Kiana used Cuttle to laser-cut miniature weapons for user testing. 

 

Lookback: During the midterms, we put our attention too much on the visceral level of the project which left little time for coding and circuits. This time, we decided to leave external consideration until the end when the coding and circuit were done. We spent more time working on the functionality of the project which was a good choice because we were more confident that there were fewer obstacles during the installation stage (less time needed). We also decided to make the prototypes smaller for better transportation, but also to save resources at the IMA lab. More resources can be used for final touch-ups for both us and other students.

Initial “vest” design:

Final prototype:
 

Our first testers, Al and Henry:

 

Step 2: Final weapons

Sword

 

  • Kiana used Cuttle to create an SVG file which she then sent to the FabLab for lasercutting.
  • We then wrapped the top portion in copper tape.
  • We soldered solid wired to the base of the sword.
  • We wrapped the wire along with the grip of the sword in electrical tape.
  • We attached fake gemstones along the guard of the sword for design.
  • We hot glued the cloth covering the sold wire with hot glue onto the pommel of the sword.

 

We wanted to use silver as the color is more often associated with swords but the silver tape that the lab had was not conductive. 

Shield

  • Kiana used Cuttle to create an SVG file of two shields and two handles which she then sent to the FabLab for laser cutting.
  • Professor Lee recommended making triangular supporting material along the handle to ensure it doesn’t fall off. However, due to limited time, we used hot glue instead of wood which was sturdy nonetheless. 
  • We wrapped the entire shield in copper tape.
  • We soldered solid wire at the bottom of the shield.
  • We hot glued the cloth covering the sold wire with hot glue onto the bottom of the shield.

Armor

  • With cardboard from our midterm, we cut two identical pieces for the armor.
  • We cut two identical strips of brown cloth for the should piece.
  • We taped copper around the exterior parts of the armor. Electrical tape was attached on the inside for a better visual look and to prevent any possible electrocution (although highly unlikely).
  • We soldered solid wire at the back of the armor.
  • We hot glued the cloth covering the sold wire with hot glue onto the back of the armor.

Lookback: While observing the players at the IMA Show, I found that the wood material that we used could be hazardous. As a result, the users were either hurt or apprehensive during the duel. Although the wood was light, it was still hard enough to cause harm to the users if they put force in hitting their opponents. Thus, I propose changing the material of the weapons to a sturdy sponge so that it remains in shape but soft enough to avoid injury. This material would have been more child-friendly for more aggressive users. Due to the requirement of using digital fabrication, we could not follow up on this decision. Other things we noticed during the final IMA Show:

  • Due to the background noise, it was difficult for the users to hear the narrator. Thus, not many participants were invested in the lore/background of the game which confused them why there was an offensive and defensive role. This project is best suited in a quiet environment.
  • I wanted to allow players to skip slides in the introduction rather than the whole part so that they could see the instructions at the end of the game state. Because of this setback, I found that many players did not grasp the game rules because they skipped the introduction. I would like to fix this video issue if I had more time.

Lookback: Because we had more time, we once again delegated the task based on strengths and weaknesses (or more accurately familiarity with certain software). Kiana worked with Cuttle and I worked with Processing. Exposure to this step also required trust in each other because we did not work closely with each other which built a greater connection between us as teammates and I have become more open to working with others with different backgrounds and expertise.

Step 3: Stone

  • We used a big piece of cardboard from our midterm as the base of the stone.
  • Kiana created an SVG file of a box using a box generator which she then sent to the FabLab for laser cutting.

 

  • We made a smaller body with copper tape wrapped around it and solid wire soldered at the base. The solid wire was attached to pin 5.

 

  • We drilled a rectangular hole at the top of the big box and attached the smaller box to create a cave for the sword to slip in.
  • We hot glued bubble wrap at the top to create an organic shape for the stone.
  • We left one side open so we could access the inside; we plan to put the circuit in the box.
  • We covered the entire object in silver cloth to mimic a stone.
  • We hot glued some of the cloth onto the cardboard base to secure the position of the cloth.



 

Step 4: Instructions

 

  • Kiana engraved words on a shield that we planned to discard as additional instructions to the player. We noticed that people may not know which color, white or brown, belonged to which player. Thus we made a small riddle for the users to play before engaging with the game.

Step 5: Setup

  • A red tablecloth was placed on a table.
  • A monitor and speakers are placed on the table.
  • Mordred’s weapons are placed on the table.
  • Arthur’s rock and weapons are placed on the front left of the table.
  • The shield with additional instructions is placed on a chair on the right side. 
  • When everything is ready, the USB cable is attached to the laptop with the Processing sketch. The laptop is placed under the table, which will not be seen by the audience.

 

IMA Show:

 

D. CONCLUSIONS

The video below is a demonstration of the final project:

 

“Excalibur” is an interactive 2-player game that incites participants to explore their own movements within the intersection of technology and human perception, encouraging them to engage with the environment in a unique and surreal manner.  In doing so, the project becomes a derivative of reality by making an illusion playable as it channels agency into new forms of the conventional screen. Guided by user input and a basic set of rules, the project invites engagement and interaction, inviting people of all ages to play and create their unique patterns of sound and outcomes. Through this interactive experience, “Excalibur” successfully fosters a sense of community and shared creativity as participants collectively contribute to the game through the body as the channel for information. Furthermore, the connection is made unironically through the connection of the user’s weapons in a closed circuit that is signaled by the sounds of clash and force. This observation is evident from users’ engagement exemplified by the popularity and entertainment that brought laughter to the crowd at the IMA Show. Thus, the project resides in a new realm of creative expression, that transcends traditional boundaries between the human and technologychallenging the conventional notions of game. On the other hand, the project aligned with the dialectic definition of interaction as proposed by Chris Crawford: after receiving information from the user’s “extended organs”—their weapons—the game processes and interprets the data to produce variable results onscreen whether that is through sword slashes, character movement, or health reduction. In turn, the user sees the system’s response, and takes more action in response, in a continuous loop.  Furthermore, the dialogue is multi-faceted too: there is dialogue between the players too. To reinforce this interaction, Kiana and I would have liked to add more narrative and witty remarks to increase the humor of the game. To do so, we would either buy a subscription to continue using ElevenLab’s AI voice or hire a voice actor/actress to play the narrator role. Furthermore, to make the project more authentic, we would have drawn the media objects (images and video) ourselves or hired someone to do so instead of borrowing them from online. Overall, Kiana and I are both proud of our progress and dedication to this project. During our setbacks, we changed our old approach of troubleshooting independently to one that is open-minded: we asked for assistance from whoever whenever and wherever we needed it! By doing so, we’ve also come to appreciate the big social network structure that essentially holds our project together—a social network that comprises of user-testing, office hours, one-on-one assistance, and even the support Kiana and I gave each other. I plan to carry this sentiment with me in my next steps on this IMA journey.

E. DISASSEMBLY

 

F. APPENDIX









 

G. REFERENCES

Crawford, Chris. The Art of Interactive Design. San Francisco,  No Starch Press, 2002.

Norman, Don. The Design of Everyday Things. Cambridge, MA: The MIT Press, 2013.

 

Midterm Project: 5. Report

A. Behind the Screens – Anita Luo & Kiana Ng – Inmi Lee

Front view:

Source: Own Image

Side View:

Source: Own Image

 

Back view:

Source: Own Image

B. CONTEXT AND SIGNIFICANCE

Behind the Screens was impacted by Kiana and my experience of making Emote Gloves in which the user uses gloves with flex sensors to control a headband that displays emotes. How each finger is bent and its combination will determine where the servo motor will rotate. Firstly, we were inspired by the narrative behind the project. Interaction, according to Chris Crawford, involves the cyclic and collaborative interplay of stimulation between two or more things. It can be illustrated as the conversation between two actors that “alternately listen, think, and speak”— the degree of their interaction is determined by the quality of each of these communication steps (Crawford 3). Through our experience, quality in this context does not only refer to the effectiveness of different senses and perceptions but also the unique lesson and takeaway that these interactions bring: an interaction that has good quality has a compelling storytelling factor. For example, the Emote Gloves illustrate the translation of hand gestures to visual data which allows the user to contemplate the significance of visual communication–especially in the context of accessibility tools. On the other hand, Behind the Screens is built on the narrative that the user is controlling a Looney Tunes scene between Wile E. Coyote and the Roadrunner. What is significant about our current project is the intentions behind the design. In this new digital age where speed is encouraged, we wanted to focus on the frame-by-frame consciousness in which the user feels the energy, effort, and collaboration needed to produce a scene on screen–by taking them behind the scenes. To improve the interaction, we added another layer in which the project also becomes the means of allowing two actors to talk, think, and speak to each other and not necessarily operate as one of the interlocutor. This is encouraged by the game-like setup and the need for the input of both users to elicit a response in the project. The target audience is niche; the project is neither a tool nor an actual game but stands alone as a conceptual art piece. The users are those who are interested in the narrative of this artwork.

C. CONCEPTION AND DESIGN

Initial Conceptual Drawing:

After reading The Design of Everyday Things (2013) by Don Norman, we understood the importance of having an appealing design on the visceral level. Thus, we ensured that the design was characterized by many organic lines (such as the curved corners of the television) and cardboard. Organic lines, we believed, were more soothing and welcoming to the viewers’ eyes. We selected cardboard as our main material because the all-cardboard visual best suited our project purpose–the project aims to offer a glimpse into the 1950s and evoke nostalgia. Cardboard was also encouraged for the project by the professors so we did not consider other options. Furthermore, other visual design decisions included the subject matter: Looney Tunes’ Wile E. Coyote and the Roadrunner are well-known icons of the 1950’s retro scene. Through the subject matter, we wanted to draw the audience’s attention. Additionally, the television would spark familiarity within the viewers which elicits curiosity. Buttons tackled the behavioral component of the design as we believed that users could subconsciously press them through common knowledge and practice. Other options were rejected as they were not suitable to be placed in the context of a television. The overall experience is curated, through an interplay between the narrative and design, for the reflexive part of the project. 

*Above is the revised conception and design. Our initial idea in our Midterm Proposal adopted many adaptations which will be discussed in FABRICATION AND PRODUCTION.

D. FABRICATION AND PRODUCTION

Materials

  • 1x Breadboard
  • 1x Arduino Uno
  • 1x USB Cable
  • 2x Stepper motor (42STH33-0404AC)
  • 2x Stepper Motor Driver Module with Barrel Jack Connector pre-connected
  • 2x Shaft adaptor
  • 1x 12 Volt Power Supply
  • 1x USB Protector
  • 1x Buzzer (Speaker)
  • 2x 10 kOhm Resistors
  • 2x Arcade Buttons (one per person)
  • A handful of M/M and F/M jumper cables
  • 1x Box cutter knife
  • 1x Ruler
  • 1x Hot glue gun
  • 1x Cardboard sheets
  • An array of acrylic paint
  • Paintbrushes
  • 1x Scissors
  • 1x Wire cutters
  • 1x Cutting board
  • 1x Masking tape
  • 1x Marker

Not included anymore:

Building the Circuit (Anita)

Step 1: Race the Led

I first built the circuit following Race the Led. At first, the Roadrunner was set at a constant back-and-forth movement on a separate Arduino using a sketch with random(). Only the coyote determined the distance between the two characters in the main Arduino. However, during user testing, users were confused by the feedback they received. Thus, we decided to add both stepper motors in one Arduino and sketch so that both characters were responsive to the button presses.

Initial diagram followed:

Source: tinkercad

Step 2: Adjusted circuit with 2 stepper motors

For the new circuit, we incorporated both stepper motors and also removed the LEDs as we believed that this indication of a winner hindered the concept of the project–we did not want to suggest to the users that the project was solely a game. 

New circuit:

Made with circuit-diagram.org (stepper motor symbol may not be accurate due to the limited symbol library on the website)


Lookback: I used masking tape and marker to label each wire. When the circuit became more complex, this strategy assisted me to make sense of the circuit and its components.

I also soldered wires to connect both stepper motor driver modules to one 12V power supply. 

Obstacle: Later when Kiana and I started coding there were a few instances when the Arduino would abruptly disconnect. One reason was that we did not connect the circuit properly. We solved this problem by using the examples for analog and digital reading in Arduino to check each component’s connection. However, there were other reasons that we could not explain–the servo motor would not move despite being well-connected and coded. As a result, we removed the “Game Over” sign that would rotate down on the screen because it caused the Arduino to persistently disconnect.

Disconnecting problem:

Checking components using readings:

Servo motor not moving properly:

Solutions:

Building the prototype (Anita and Kiana)

Step 1: Building the Television

Following the diagram we drew in the proposal we cut out cardboard, painted a white base (for vibrant colors), painted color, and stuck the pieces together. For the antennae, we created a taco-shaped cardboard and twisted and soldered wire.

Lookback: Although we did not encounter any issues, it is recommended to indicate measurements in the sketches during the planning phase to prevent any potential miscalculations.






Step 2: Building the Stage

We were inspired by retro Hollywood sets with cardboard props. Thus, we cut and painted each layer of the background as well as the ground and sky.


Step 3: Making Wile E. Coyote and the Roadrunner

Using a frame from a Looney Tunes scene, we traced, cut, painted, and lined each character.



We stuck these on our stepper motors (explained later).

Step 4: Assemble

Obstacle: We decided not to add rotating legs (controlled by servo motors) as we were afraid that the wire attached to them would jam the track of each character.

Step 5: Button

Instead of putting the button on the television as seen in our initial sketch, we decided to extend it outwards to prevent potential damage to the circuit hidden at the base of the television as the user pushes the button.


Step 6: Adaptations after User Testing

Kiana and my initial idea was to create an interactive multiplayer game that introduces nostalgia and childhood recreation. Two users will recreate a Looney Tunes scene, in which Wile. E Coyotes chases the Road Runner, while simultaneously competing with each other to see who can press their respective buttons faster. However, after the user-testing session in class, we found many drawbacks to our design. Firstly, we followed the circuit and code of Race the Led as the basis of our game. Yet, we needed to code a game with a beginning, middle, and end, and incorporate music, a point system, and visual progress to enhance the gaming experience from feedback. Coding this was out of our scope in the course–for example, we had not learned the “for” loop yet to move characters forward in increments. Unfortunately, due to time conflicts and time constraints, we knew that asking for assistance would be problematic and time-consuming. Thus, we limited our design by transforming it into a conceptual work. However, we did incorporate sound and faster feedback as suggested in user testing.

To incorporate sound, we coded for a buzzer to play a shorter version of the Looney Tunes Main Theme Song to indicate the end of the scene. We also ensured the buzzer was protruding out of the television so that the sound could be heard clearly.

We ensured faster feedback to confirm the user’s expectations by changing the initial stepper motor mechanism to a belt mechanism. The belt mechanism ensured faster and bigger movement. Furthermore, we added hot glue grips so that the friction between the stepper motor and the cardboard belt could create precise movement. Alternatively, we could have bought a pre-made belt online. However, we rejected this decision because we wanted to maintain the cardboard style of the design. Almost all the visible components of the project are cardboard because we wanted to align the exterior design to the retro theme. The friction produced by the cardboard belt mechanism creates a non-mechanical sound which further supports this theme.

Initial stepper motor mechanism:

New cardboard belt mechanism:


Secondly, we included a sign that read ” 2 PLAYER/Recreate a Scene!” to suggest to the audience the co-op framework of the project. From the user testing, we noticed that people pressed both buttons. Although we used blue and red buttons as indications of the commonly known arcade buttons for player 1 and player 2, people may not have had this background to know this.

Thirdly, user testers recommended we add lights inside the television box to enhance the visuals. Hence, we stuck spotlights at the top of the inside as follows:

Writing the code (Anita and Kiana)

We coded with the guidance of Race the Led, toneMelody, and Recitation 5.

/* Behind the Screens
*/


/*************************************************
   Public Constants, defining the pin numbers used in Arduino
 *************************************************/
int buzzerPin = 8;
int buttonPin1 = 10;
int buttonPin2 = 9;
//int ledPin1 = 9;
//int ledPin2 = 10;


/*************************************************
   Public Constants, defining the notes in Hertz
 *************************************************/

#define NOTE_B0 31
#define NOTE_C1 33
#define NOTE_CS1 35
#define NOTE_D1 37
#define NOTE_DS1 39
#define NOTE_E1 41
#define NOTE_F1 44
#define NOTE_FS1 46
#define NOTE_G1 49
#define NOTE_GS1 52
#define NOTE_A1 55
#define NOTE_AS1 58
#define NOTE_B1 62
#define NOTE_C2 65
#define NOTE_CS2 69
#define NOTE_D2 73
#define NOTE_DS2 78
#define NOTE_E2 82
#define NOTE_F2 87
#define NOTE_FS2 93
#define NOTE_G2 98
#define NOTE_GS2 104
#define NOTE_A2 110
#define NOTE_AS2 117
#define NOTE_B2 123
#define NOTE_C3 131
#define NOTE_CS3 139
#define NOTE_D3 147
#define NOTE_DS3 156
#define NOTE_E3 165
#define NOTE_F3 175
#define NOTE_FS3 185
#define NOTE_G3 196
#define NOTE_GS3 208
#define NOTE_A3 220
#define NOTE_AS3 233
#define NOTE_B3 247
#define NOTE_C4 262
#define NOTE_CS4 277
#define NOTE_D4 294
#define NOTE_DS4 311
#define NOTE_E4 330
#define NOTE_F4 349
#define NOTE_FS4 370
#define NOTE_G4 392
#define NOTE_GS4 415
#define NOTE_A4 440
#define NOTE_AS4 466
#define NOTE_B4 494
#define NOTE_C5 523
#define NOTE_CS5 554
#define NOTE_D5 587
#define NOTE_DS5 622
#define NOTE_E5 659
#define NOTE_F5 698
#define NOTE_FS5 740
#define NOTE_G5 784
#define NOTE_GS5 831
#define NOTE_A5 880
#define NOTE_AS5 932
#define NOTE_B5 988
#define NOTE_C6 1047
#define NOTE_CS6 1109
#define NOTE_D6 1175
#define NOTE_DS6 1245
#define NOTE_E6 1319
#define NOTE_F6 1397
#define NOTE_FS6 1480
#define NOTE_G6 1568
#define NOTE_GS6 1661
#define NOTE_A6 1760
#define NOTE_AS6 1865
#define NOTE_B6 1976
#define NOTE_C7 2093
#define NOTE_CS7 2217
#define NOTE_D7 2349
#define NOTE_DS7 2489
#define NOTE_E7 2637
#define NOTE_F7 2794
#define NOTE_FS7 2960
#define NOTE_G7 3136
#define NOTE_GS7 3322
#define NOTE_A7 3520
#define NOTE_AS7 3729
#define NOTE_B7 3951
#define NOTE_C8 4186
#define NOTE_CS8 4435
#define NOTE_D8 4699
#define NOTE_DS8 4978


/*************************************************
   Public Variables
 *************************************************/

//button and count variables
int goal = 15;
int buttonState1 = LOW;
int previousState1 = LOW;
int buttonState2 = LOW;
int previousState2 = LOW;


int counter1 = 0;
int counter2 = 0;
boolean winner1 = false;
boolean winner2 = false;


long time = 0;        // the last time the output pin was toggled
long debounce = 200;  // the debounce time, increase if the output flickers

//stepper motor variables
#include 
int randNumber1;
int randNumber2;
int CDIR_PIN = 2;
int CSTEP_PIN = 3;
int CEN_PIN = 4;
int RDIR_PIN = 5;
int RSTEP_PIN = 6;
int REN_PIN = 7;
AccelStepper stepperC(AccelStepper::DRIVER, CSTEP_PIN, CDIR_PIN);
AccelStepper stepperR(AccelStepper::DRIVER, RSTEP_PIN, RDIR_PIN);

//servo motor variables
//#include 
//Servo servoL;
//Servo servoR;

// notes in the main theme melody and duration
int melody[] = {
  NOTE_G4,
  NOTE_A4,
  NOTE_E4,
  NOTE_G4,
  0,
  NOTE_G4,
  NOTE_A4,
  NOTE_E4,
  NOTE_G4,
  0,
  NOTE_G4,
  NOTE_A4,
  NOTE_E4,
  NOTE_G4,
  NOTE_A4,
  NOTE_B4,
  0,
};

int noteDurations[] = {
  16, 16, 8, 4, 8,
  16, 16, 8, 4, 8,
  16, 16, 4, 4, 4, 1
};



void setup() {
  Serial.begin(9600);
  pinMode(buzzerPin, OUTPUT);
  pinMode(buttonPin1, INPUT);
  pinMode(buttonPin2, INPUT);
  randNumber1 = random(0, 100);
  randNumber2 = random(100, 200);
  //pinMode(ledPin1, OUTPUT);
  //pinMode(ledPin2, OUTPUT);
  Serial.println("******************* LOONEY TUNES *******************");
  delay(1000);
  Serial.println("READY");
  tone(8, 440);
  delay(1000);
  noTone(8);
  delay(50);
  Serial.println("SET");
  tone(8, 440);
  delay(1000);
  noTone(8);
  delay(50);
  Serial.println("GO!!!!!!!!!!!!!!!!");
  tone(8, 880);
  delay(1500);a
  noTone(8);


  // Set initial speed & acceleration for stepper motors
  pinMode(CEN_PIN, OUTPUT);
  digitalWrite(CEN_PIN, LOW);
  stepperC.setMaxSpeed(500);
  stepperC.setAcceleration(300);

  pinMode(REN_PIN, OUTPUT);
  digitalWrite(REN_PIN, LOW);
  stepperR.setMaxSpeed(500);
  stepperR.setAcceleration(300);

  stepperC.setCurrentPosition(0);
  stepperR.setCurrentPosition(0);

  //set up servo motors
  //   servoL.attach(11);
  //   servoR.attach(12);
  // servoL.write(90);
  //  servoR.write(0);
}


void loop() {
  buttonState1 = digitalRead(buttonPin1);
  buttonState2 = digitalRead(buttonPin2);
  //Serial.println(buttonState1);

  //this checks the times player 01 has pressed the button
  if (counter1 < goal && winner2 == false) { if (buttonState1 != previousState1 && millis() - time > debounce) {
      if (buttonState1 == HIGH) {
        counter1++;
        Serial.print("player 01:  ");
        Serial.println(counter1);
        time = millis();
      }
    }
    previousState1 = buttonState1;
    if (counter1 == goal && winner2 == false) {
      winner1 = true;
      // digitalWrite(ledPin1, HIGH);
      Serial.println("PLAYER 01 WINS");
      // servoL.write(180);
      //  servoR.write(90);
      delay(500);
      playMelody();
      endGame();
    }
  }


  //this checks the times player 02 has pressed the button

  if (counter2 < goal && winner1 == false) { if (buttonState2 != previousState2 && millis() - time > debounce) {
      if (buttonState2 == HIGH) {
        counter2++;
        Serial.print("player 02:  ");
        Serial.println(counter2);
        time = millis();
      }
    }
    previousState2 = buttonState2;
    if (counter2 == goal && winner2 == false) {
      winner2 = true;
      //digitalWrite(ledPin2, HIGH);
      Serial.println("PLAYER 02 WINS");
      playMelody();
      endGame();
      // servoL.write(180);
      //  servoR.write(90);
    }
  }

  //this prompts the two characters to move based on the number of counts of each player
  if (counter2 > counter1) {
    stepperC.runToNewPosition(randNumber2);
    stepperR.runToNewPosition(randNumber1);
  } else {
    stepperC.runToNewPosition(randNumber1);
    stepperR.runToNewPosition(randNumber2);
  }
}


void playMelody() {
  for (int thisNote = 0; thisNote < 18; thisNote++) {
    int noteDuration = 1000 / noteDurations[thisNote];
    tone(8, melody[thisNote], noteDuration);
    int pauseBetweenNotes = noteDuration * 1.30;
    delay(pauseBetweenNotes);
    noTone(8);
  }
}

void endGame() {
  stepperC.runToNewPosition(0);
  stepperC.runToNewPosition(0);
  int counter1 = 0;
  int counter2 = 0;
  boolean winner1 = false;
  boolean winner2 = false;
  stepperC.stop();
  stepperR.stop();
  // servoL.write(180);
  // servoR.write(90);
}
 

E. CONCLUSIONS

The video below is a demonstration of the final project:

 

Behind the Screens is a conceptual interactive artwork aimed at inviting two players to recreate a scene from Looney Tunes. As a result, users have the cognizance of the handiwork and collaboration needed to create an animation. This narrative is noteworthy during an age of digital media in which speed is favored and appreciation of details and storytelling is lost in the process. The project aligned with the dialectic definition of interaction: after receiving information from the user’s button press, the machine processes and interprets the data to produce variable results “onscreen.” The users have the opportunity to interpret the feedback. However, due to limitations and adaptations (as discussed in FABRICATION AND PRODUCTION), the audience’s susceptibility to the narrative was restricted by the deceiving game design. If we had more time, we would a) add a start button, b) add “for” loops so that players see their progress in increments, and c) add continuous background music that increases in tempo as the players reach closer to the end.  Despite the short duration of the midterm project, the experience of making Behind the Screens taught me not only the practical and theoretical aspects of physical computing but also teamwork, time management, adaptation, and ambition. Firstly, I learned the importance of combined effort and cooperation to create synergy. At first, I did both the physical computing and the coding. There was continuous troubleshooting which was unmanageable to do alone. Thus, I was honest with my partner and asked for assistance–I saw a noticeable improvement in progress. Due to time constraints, delegating tasks based on strengths and weaknesses was unrealistic. Furthermore, I realized working on projects was a great way to tackle such weaknesses. Allowing my partner to attempt at previously unexposed skills also taught me patience and I believe this value is essential for all members of a team in the long run. Secondly, the execution process was nonlinear, involving a few modifications. The end product was different from the original design to a large degree. Through this journey, I understood the unpredictable but thrilling process of creativity. Likewise, the value of the end product did not outweigh the worth of the failures and successes I had along the way.

Source: https://alumni.sae.edu/2016/05/24/understanding-the-creative-process-inspiration-execution-release/ 

F. DISASSEMBLY:

Components:

Recycling cardboard:


G. APPENDIX














H. REFERENCES

Crawford, Chris. The Art of Interactive Design. San Francisco,  No Starch Press, 2002.

Norman, Don. The Design of Everyday Things. Cambridge, MA: The MIT Press, 2013.