Category: Marcela

Project Title
侬说啥 （What did you say?)

Conception and Design
When my partner and I were first designing the outline to our project, we knew it had to be simple and easy to understand. Our audience was targeted at young kids so they would be able to learn and understand their native Chinese dialect. We tried to think about how young kids approached learning. From my own experience growing up, I personally remembered loving to make and build things. Whether was with blocks of wood or legos, it was fun for me. Inspired by this, we eventually we decided that a puzzle of China would be a suitable method to help kids understand both the geography and dialect culture at the same time. To make a puzzle, we used Tinkercad online to manually piece together Chinese provinces and then laser cut the file on a wood board. Because the pieces were going to be taken out of the puzzle many times, we wanted a sturdy material so we avoided cardboard. During the process we thought about 3-D printing the map, but because we wanted it to be roughly 15 inches by 15 inches, it would’ve taken days to individually 3-D print separate parts and piece them together later.  Below are pictures of the initial base surrounding the puzzle and also the puzzle pieces fitted into the base.

Base of the puzzle by itself

Base with puzzle pieces inside

Fabrication and Production
In our initial plan, the puzzle was to be raised 3 inches off the ground and a motion sensor would be put under each province that we selected. It was meant to be a puzzle that one would put pieces into, but we changed it so that pieces would be taken out for the sake of the using sensors underneath. Simplified initial sketches of the design can be found below.

Sketch of the base

Sketch including  proposed sensors below

But, when we attempted to raise the puzzle, all of the pieces fell to the ground because they were not fastened in tight enough by themselves. We shifted the idea and placed the puzzle flat onto a cardboard background. Alongside this, we created a map online through processing and arduino which displayed the same physical map. We connected a joystick so that participants would be able to hover over provinces and hear a whisper of the dialect, and thus be incentivized to then pick up the puzzle piece. Unfortunately during user testing, we were not able to connect any sensors to the physical map, so our project was only displayed through the online map. One of the tips was received was to make the joystick easier to use and control our character on screen. So, we changed the initial code to support diagonal movements on screen and improve the fluidity of movement. Another crucial tip we received was a way to involve the physical puzzle pieces of our map in correlation with the online map. Using this advice, we poked holes in the bottom of the cardboard base corresponding underneath each province we had a recording for. We used a wire to connect from Arduino to the map and placed conductive tape over the wire once it was stuck through so just the foil tip ways displayed. We then placed conductive copper tape on the selected puzzle pieces to ensure that the energy from our body would be transferred through the tape on the piece to the wiring below and back into Arduino. This method made it possible to play the native dialect of the province when that particular piece was touched. It was essential to our project and half of it would’ve been incomplete without it. Because it was a last minute adjustment, many of the wires became tangled, but the attempt to beautify the project was our last priority at this design stage. Below are pictures of our project during user testing day.

Participants using the joystick

Conclusions
The goal of my project was to create an educational experience focused on exposing audiences to different Chinese dialects from separate provinces. I think we accomplished this and more by sparking a discussion on how important preserving these dialects are in user testing and presentation days. Within the context of interaction, I think it aligned with my original definition that “Interaction is a continuous conversation between two or more corresponding elements” because it was constant involvement either when using the puzzle, listening to the dialects, or controlling the joystick. The conversational element was representative when the participant would remove a puzzle piece and a dialect would play in response. This call and response constituted for me as a conversation between human and machine. Although, it may not necessarily be continuous because it only works when the participant picks up the puzzle piece. Thus, it isn’t interactive in the sense of a game where one is constantly playing. In our puzzle the subject is always thinking, but not necessarily interacting. Ultimately, our audience didn’t end up picking the pieces up, but rather just touching them. This was because they figured out that the sound would play as long as they touched it. It had the same educational effect as picking it up so this was just a minor diversion from our original intentions. When we concluded that it was not possible to use the light or motion sensors, I felt a sense of defeat. Through the help of professors and our classmates, we strayed from our comfort zone and used conductive tape, a material both my partner and I were not accustomed to working with. While it was indeed difficult to grasp at first, we became aware of the tape’s advantages and it eventually tied our entire project together. The main thing I’ve taken away from this project is that nothing is possible without the help and advice of those around you. Many of the notable alterations in structure came out of comments from our professors and classmates. For this, I am grateful to have such a unique community here at NYU Shanghai that is always willing to help and give feedback at almost any time of the day. Below is a picture of Kevin using our project on the final presentation day and also at the IMA fair. 

Presentation Day

Serial Communication

Although I was not present during this week’s observation, I choose to use the serial communication workshop with Young as a baseline of learning. I choose this workshop because I wanted to use past examples my work to further understand how I can use serial communication in the future. Alongside the workshop slides by Eszter and Jessica, I found both workshops to be very helpful by scanning through their notes and slides online.

Example of Serial Communication Using the Map Function

Because there is not a specific exercise involved with this workshop, I’ve chosen to properly showcase the mapping skills I’ve used throughout this class through the image tint I previously worked on.  The code for the image tinting is as follows:

My Code

Arduino

int a;
void setup() {
  Serial.begin(9600);
}


void loop() {
  int sensor1 = analogRead(A0);
  Serial.write(sensor1);
  int a = map(sensor1, 0, 1023, 0 , 255);
  Serial.write(a);
//  Serial.print(sensor1);
//  Serial.println();
  

  // too fast communication might cause some latency in Processing
  // this delay resolves the issue.
  delay(10);
}

Processing

import processing.serial.*;
PImage img;
Serial myPort;
int valueFromArduino;
int sensor1;


void setup() {
  size(600, 600);
  img = loadImage("HK.jpeg");
  imageMode(CENTER);
  background(255);
  
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[16], 9600);
}

void draw() {
  if (img.width > 0) {
    image(img, 300, 300, width, height);
    //float a = map(sensor1, 0, 1023, 0, 600); 
    
    while ( myPort.available() > 0) {
    valueFromArduino = myPort.read();
  }
  println(valueFromArduino);//This prints out the values from Arduino
}
  
  if ( valueFromArduino > 150) { 
    tint(0, 0, 255, 150);
    image(img, 250, 0);
  }
 else if (valueFromArduino < 149) {
  tint(0, 255, 0);
 }
  }

 Video example

Through this code, I created a mapping value in the Arduino code of “int a = map(sensor1, 0, 1023, 0 , 255)” and by doing so, I correlated the “int a” value with the values of sensor 1 (potentiometer). The reasoning for this was to create the lowest possible and highest possible values within the potentiometer in the second and third slots of the mapping. In the third and fourth slots I wrote in my lowest target value and highest target value. I also wrote in Sensor 1 so that all of the value in slots 1-4 would correlate with it. Through serial communication, these mapped out values were transferred to processing. The corresponding lines of code in processing were represented by “if ( valueFromArduino > 150) {     tint(0, 0, 255, 150);     image(img, 250, 0);   }  else if (valueFromArduino < 149) {   tint(0, 255, 0);” These values of 0-255 were the original target values I created in my map on Arduino. And thus by printing “println(valueFromArduino)” it was then possible to correlate the two programs to work together. 

For recitation 9, we were able to extend our learning from the transitions between Arduino and processing. Because we were free to choose what we wanted to do with the media, I decided to build the camera function that would be needed for my project. 

In order to make a camera in the phone screen, I took a screenshot of the phone camera screen and covered up the non-camera showing part with the image.

Whenever I was pressing the button connected to Arduino (it would have been cooler if the image could be taken with the actual camera button though- we needed Arduino bit), the processing would make a screenshot of the screen and save in the specific location. 

However, there was a shortcoming though.

As you can see from above, because the processing is taking the screenshot of the entire screen, the camera APP part is showing as well, which is different from the usual images inside the gallery. 

Below is the code.

Processing:

// IMA NYU Shanghai
// Interaction Lab
// This code receives one value from Arduino to Processing 
import processing.serial.*;
import processing.video.*; 

int photoNum = 0;

Serial myPort;
int valueFromArduino;

PImage camera;
Capture cam;

void setup() {
  size(335, 690);
  background(0);

  printArray(Serial.list());
  // this prints out the list of all available serial ports on your computer.

  myPort = new Serial(this, Serial.list()[5], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.
  camera = loadImage("images/camera.png");
  setCamera();
}


void draw() {
  // to read the value from the Arduino
  while ( myPort.available() > 0) {
    valueFromArduino = myPort.read();
  }
  showCamera();
  //println(valueFromArduino);
  if(valueFromArduino == 1){
    save("gallery/"+str(photoNum)+".png");
    photoNum++;
    //println("pressed");
  }
}

void setCamera() { 
  cam = new Capture(this, 640, 480);
  cam.start(); 
} 

void showCamera() { 
  if (cam.available()) { 
   cam.read(); 
  }   
  noStroke();
  for (int i=0; i<500; i++) {
    int size = int( random(10, 30) );
    int x = int( random(width) );
    int y = int( random(80, 480) );
    // get the pixel color
    color c = cam.get(x, y);
    // draw a circle with the color
    fill(c);
    ellipse(width-x, y+40, size, size);
 }
 image(camera, 0, 0);
}

Arduino:

// IMA NYU Shanghai
// Interaction Lab
// This code sends one value from Arduino to Processing

void setup() {
Serial.begin(9600);
}

void loop() {
int sensorValue = digitalRead(A0);
Serial.write(sensorValue);

// too fast communication might cause some latency in Processing
// this delay resolves the issue.
delay(10);
}

Below is the demo video.

While I was doing the recitation workshop and was proceeding with my final project, reading through Computer Vision for Artist and Designers inspired and motivated me to work. I grew up in a country which has very serious bullying that approximately 63% of the people have had experienced bullying (including cyber-bullying), and 19% of the people having tried suicidal attempt due to bullying. Reading through this week’s reading, I found two pieces really interesting. 

The first was Suicide Box by the Bureau of Inverse Technology. As previously mentioned, the country that I was born and raised is rated as a country with the most suicides amongst OECD countries. Though it is little sad to see this idea having to count people who are jumping off the bridge and is merely counting the numbers when the jumped off person is losing his or her life, I think by knowing the exact statistics, other people may be able to do something about the environment to lessen the number of people doing the suicidal attempt. 

The other was Stills from Cheese, an installation by Christian Möller. This was especially interesting because I was thinking to project the user’s face on the trash bin, giving the user the feeling that they were viewed as an (emotional) trash bin. Stills from Cheese, by analyzing the current photo showing up on the screen, is able to tell when the user is smiling or not. If I were to analyze a person’s face, I would be able to only portray the person’s face on the trash bin looking figure. Therefore this piece has gotten me a little sad and motivating at the same time.