Author: Zoe Liu

• CONCEPTION AND DESIGN:

For our final project “Chime”, we keep the design of the original traditional instrument. We use several popular brands such as Starbucks, Luckin Coffee, and waimai food containers, and paint them with metal material spray lacquer which makes these plastic products more relatable to audiences. Then, we hang nine of them on a clothes hanger using metal wires. There is a capacitive sensor attached to every item through wires. If you watch this project from a distance, it is more likely a set of metal bottles and containers hanging on a shelf. Since for the Chinese chimes, they are pretty big and heavy, hence, performers need sticks to hit it, and playing it is a collaboration work. In order to also mimic this detail, we also make a stick using a selfie stick which people can use to “hit” the “chime”. The idea of using all the modern element of consumerism culture is that we want to show audiences the huge contradiction between the past society and the current one. We hope that this project can build a connection between the past and the present. When people play this “instrument”, they will realize that they are playing a modern project but an ancient sound is played. They can feel how much our society has changed. Before we finally decide to use consumerism as the theme of our project, we also think about whether we should consider making it more aesthetically attractive. We originally plan to use plastic spoons to make water lilies and create an experience design. However, later we realize that aesthetic design cannot bring us the thoughtful interaction that we hope to achieve. The final design is settled with plastic bottles and containers.

 

When people use their hands to touch the items, there will be sounds played and also a piece of the picture which is a part of a Chinese chime’ pictures appear on the screen. People can also choose to use the selfie stick to touch hanging items, since we already use metal wires to connect stick and its handle, hence they can also make sound using the stick.

The process is shown in the testing video:

• FABRICATION AND PRODUCTION:

Two major challenges that we meet during the production are a coding and physical connection.

We split work before we start working, and I was mainly responsible for the code part. I spent a lot of time to make code works. Our original idea was that when people touch hanging items, the sound of chime would be played and a piece of the picture will appear which symbolizes a further understanding of this project. Pictures would work like a puzzle. In order to achieve this goal, I had to build the serial communication and sent the sensor value from the Arduino to Processing and make outcomes happen according to certain circumstances. However, when I finished coding, only the picture that people first touch would show up, and the rest of the pictures seem to get stuck in the loop function. I did not know why this was happening, so I turned a fellow for help. After testing my code, Professor Young said that I had an inefficient logic, and computer could not get clear instruction from this so it functioned in a wrong way.   

void draw() {

 background(0);

 updateSerial();

 

 //printArray(sensorValues);

 // state1

 if (sensorValues[0] == 1 && !h1) {

   //filef4.play();

   h1 = true;

 }

 if (h1) {

   if (filef4.isPlaying()==false) {

     filef4.play();

   }

   

   tint(255, i1);

   image(photo1, 0, 0);

   i1 = i1-step;

   if (i1 < 0) {

     h1 = false;

     i1 = 255;

   }

 }

Professor Young said that I was sending three different flags in the loop function and they were intertwined together. I need to figure out how to simplify them. For example, with the int h, how to make it be the signal for both image and sound function. Then Professor Young helped me fix the logic into this.

if (sensorValues[0] == 1 && !h1) {

   filef4.play();

   h1 = true;

 }

 if (h1) {

   //if (filef4.isPlaying()==false) {

   // filef4.play();

  

   //}

   

   tint(255, i1);

   image(photo1, 0, 0);

   i1 = i1-step;

   if (i1 < 0) {

     h1 = false;

     i1 = 255;

   }

 }

After this, I split the if condition into nine separate one. I used to put nine conditions in one if function. After splitting them, there was more than one picture can show up at the same time.

After user testing, we got a very important suggestion. People suggested that why not make pictures fade away so that people will continue playing them. In order to achieve this, I started working again. I found the function tint() and (isPlaying). For tint() function, it can help picture change its transparency which can create the feeling of fading away. I changed the variable of tint and mapped value in it so that the transparency could change by time. For the (isPlaying) function, it could make sure that every time the sound file will be completely played once before starting next round. However, the problem was that all sound files played more than once. I have tried the once() function but it did not work. Then I went to Professor Marcela, she helped me recognize my code and solve my problem.

• CONCLUSIONS:

In the final presentation, we found that people prefer to touch by hands and feel them in

the person instead of using a selfie stick. I realized that we should not limit our project within the definition of a mimic work of Chinese chime. People are m=not familiar with the traditional instrument anymore, and we need more new creativity to motivate them to interact with this project. Using their hands to touch the project and playing the sound is a very modern style of interaction, and when combined with a traditional element, people feel a sense of excitement, and this is what makes our project special. If I have more time, I hope that I can have an upgrade to the physical installation part. I want to have a better-decorated appearance for our project so that it is easier for audiences to dive into the atmosphere. I think my takeaway for this project is that we should find the trace of culture and tradition in modern life. It is always important to reflect on the process that we have gone through. If we simply look to the front, we will lose our cultural root, and what is more important is that we should recreate the connection between the past and the present. By giving new energy to the tradition, they can survive in this modern society using another characteristic.  

Code:

import ddf.minim.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;
import ddf.minim.signals.*;
import ddf.minim.spi.*;
import ddf.minim.ugens.*;

// IMA NYU Shanghai
// Interaction Lab
// For receiving multiple values from Arduino to Processing

/*
* Based on the readStringUntil() example by Tom Igoe
* https://processing.org/reference/libraries/serial/Serial_readStringUntil_.html
*/
Minim minim;
AudioInput in;
import processing.serial.*;
String myString = null;
Serial myPort;
int NUM_OF_VALUES = 9; /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues; /** this array stores values from Arduino **/

PImage photo1;
PImage photo2;
PImage photo3;
PImage photo4;
PImage photo5;
PImage photo6;
PImage photo7;
PImage photo8;
PImage photo9;

import processing.sound.*;
SoundFile filef7;
SoundFile filera;
SoundFile filexi;
SoundFile fileso;
SoundFile filela;
SoundFile filef4;
SoundFile filef5;

int i1 = 255;
int i2 = 255;
int i3 = 255;
int i4 = 255;
int i5 = 255;
int i6 = 255;
int i7 = 255;
int i8 = 255;
int i9 = 255;

boolean h1 = false;
boolean h2 = false;
boolean h3 = false;
boolean h4 = false;
boolean h5 = false;
boolean h6 = false;
boolean h7 = false;
boolean h8 = false;
boolean h9 = false;

boolean once1, once2, once3, once4, once5, once6, once7, once8, once9;
int step = 2;

void setup() {
size(1200, 802, P2D);
//frameRate(5);//
stroke(1000);

minim = new Minim(this);
in = minim.getLineIn();

setupSerial();
photo1 = loadImage(“1_01.jpg”);
photo2 = loadImage(“2_02.jpg”);
photo3 = loadImage(“3_03.jpg”);
photo4 = loadImage(“4_04.jpg”);
photo5 = loadImage(“5_05.jpg”);
photo6 = loadImage(“6_06.jpg”);
photo7 = loadImage(“7_07.jpg”);
photo8 = loadImage(“8_08.jpg”);
photo9 = loadImage(“9_09.jpg”);

filef4 = new SoundFile(this, “f4.mp3”);
filera = new SoundFile(this, “ra.mp3”);
filexi = new SoundFile(this, “xi.mp3”);
filef5 = new SoundFile(this, “f5.mp3”);
fileso = new SoundFile(this, “so.mp3”);
filela = new SoundFile(this, “la.mp3”);
filef7 = new SoundFile(this, “f7.mp3”);
}

void draw() {
background(0);

updateSerial();

//printArray(sensorValues);

// state1
if (sensorValues[0] == 1 && !h1) {
filef4.play();
h1 = true;
}
if (h1) {
//if (filef4.isPlaying()==false) {
// filef4.play();

//}

tint(255, i1);
image(photo1, 0, 0);
i1 = i1-step;
if (i1 < 0) {
h1 = false;
i1 = 255;
}
}

if (sensorValues[1] == 1 && !h2) {
filera.play();
h2 = true;
}
if (h2) {
tint(255, i2);
image(photo2, 400, 0);

i2 = i2-step;
if (i2 < 0) {
h2 = false;
i2 = 255;
}
}

// state3
if (sensorValues[2]==1 && !h3) {
fileso.play();
h3 = true;
}
if (h3) {
tint(255, i3);
image(photo3, 800, 0);
i3 = i3-step;
if (i3 < 0) {
h3 = false;
i3 = 255;
}
}

if (sensorValues[3] == 1 && !h4) {
filexi.play();
h4 = true;
}
if (h4) {
tint(255, i4);
image(photo4, 0, 267.3333);
i4 = i4-step;
if (i4 < 0) {
h4 = false;
i4 = 255;
}
}

if (sensorValues[4] == 1 && !h5) {
filela.play();
h5 = true;
}
if (h5) {
tint(255, i5);
image(photo5, 400, 267.3333);
i5 = i5-step;
if (i5 < 0) {
h5 = false;
i5 = 255;
}
}

if (sensorValues[5] == 1 && !h6) {
fileso.play();
h6 = true;
}
if (h6) {
tint(255, i6);
image(photo6, 800, 267.3333);
i6 = i6-step;
if (i6 < 0) {
h6 = false;
i6 = 255;
}
}

if (sensorValues[6] == 1 && !h7) {
filef7.play();
h7 = true;
}
if (h7) {
tint(255, i7);
image(photo7, 0, 534.6666);
i7 = i7-step;
if (i7 < 0) {
h7 = false;
i7 = 255;
}
}

if (sensorValues[7] == 1 && !h8) {
filef7.play();
h8 = true;
}
if (h8) {
tint(255, i8);
image(photo8, 400, 534.6666);
i8 = i8-step;
if (i8 < 0) {
h8 = false;
i8 = 255;
}
}

if (sensorValues[8] == 1 && !h9) {
filef5.play();
h9 = true;
}
if (h9) {
tint(255, i9);
image(photo9, 800, 534.6666);
i9 = i9-step;
if (i9< 0) {
h9 = false;
i9= 255;
}
}
float hue = random(360);
//fill(hue, 200, 100, 100);
//noStroke();
//int dropNum = int(map(random(1), 0, 1, 700, 1000));
//for (int z = 0; z < dropNum; z++) {
// float diameter = pow(random(1), 20);
// float distance = sq((1.0 – pow(diameter, 2)) * random(1));
// float scaledDiameter = map(diameter, 0, 1, 1, 30);
// float scaledDistance = map(distance, 0, 1, 0, 300);
// float radian = random(TWO_PI);
// ellipse(random(width),random(height), scaledDiameter ,scaledDiameter);
// //ellipse(scaledDistance * cos(radian) + 100, scaledDistance * cos(radian) +100 , scaledDiameter ,scaledDiameter);
//}

float a = 0;
float angle = (2*PI) /100;
int step = in.bufferSize() / 100;
for (int i=0; i < in.bufferSize()-step; i+=step) {
float x1 = width/2 + cos(a) * (1000 * in.mix.get(i) + 150);
float y1 = height/2 + sin(a) * (1000 * in.mix.get(i) + 150);
float x2 = width/2 + cos(a + angle) * (1000 * in.mix.get(i+step) + 150);
float y2 = height/2 + sin(a + angle) * (1000 * in.mix.get(i+step) + 150);
stroke(hue, 200, 100,100);
strokeWeight(5);
line(x1, y1, x2, y2);
a += angle;

}

}

void setupSerial() {
printArray(Serial.list());
myPort = new Serial(this, Serial.list()[ 4 ], 9600);
// WARNING!
// You will definitely get an error here.
// Change the PORT_INDEX to 0 and try running it again.
// And then, check the list of the ports,
// find the port “/dev/cu.usbmodem—-” or “/dev/tty.usbmodem—-”
// and replace PORT_INDEX above with the index number of the port.

myPort.clear();
// Throw out the first reading,
// in case we started reading in the middle of a string from the sender.
myString = myPort.readStringUntil( 10 ); // 10 = ‘\n’ Linefeed in ASCII
myString = null;

sensorValues = new int[NUM_OF_VALUES];
}

//void img1(){

//}

//void img2(){

//}

//void img3(){

//}

//void img4(){

//}

//void img5(){

//}

void updateSerial() {
while (myPort.available() > 0) {
myString = myPort.readStringUntil( 10 ); // 10 = ‘\n’ Linefeed in ASCII
if (myString != null) {
String[] serialInArray = split(trim(myString), “,”);
if (serialInArray.length == NUM_OF_VALUES) {
for (int i=0; i<serialInArray.length; i++) {
sensorValues[i] = int(serialInArray[i]);
}
}
}
}
}

For this recitation, I chose to attend the serial communication session. Because my project is about communication from the Arduino to the Processing, I want to learn more about this process. 

We first learned the communication from Arduino to Processing. I think the most important and basic step in the process is to write the exact number of Serial.print  into the code. 

Serial.print(state1);
Serial.print(“,”);
Serial.print(state2);
Serial.print(“,”);
Serial.print(state3);
Serial.print(“,”);
Serial.print(state4);
Serial.print(“,”);
Serial.print(state5);
Serial.print(“,”);
Serial.print(state6);
Serial.print(“,”);
Serial.print(state7);
Serial.print(“,”);
Serial.print(state8);
Serial.print(“,”);
Serial.print(state9);
Serial.println( );

This is important to send value from Arduino to Processing. This is really important when comes to build the communication process in a project.

 

We had several little exercises to practice the code sample.  They were all very helpful in terms of providing basic examples which helps us to challenge harder codes. 

The second thing that I learn from the recitation and think is very important is the process of setting value in the Arduino which is able to be read in the processing.  By doing this, we can actually convert the read in Arduino to the variable that can be read by Processing. This is very essential for a project to have this part to connect two projects which is exactly what we need to do for the final project. 

 

I benefited from my biggest two takeaways and then started working on my project. I use a capacitive sensor to send the value to Processing. After receiving all the data, the  Processing is able to act accordingly. 

This is a brief clip from my project using serial communication. 

During this recitation, we are required to manipulate the media in our Processing, and I choose to use the potentiometer to change the output on the screen. This video below is the final version of my exercise. Actually, I make use of this chance to do a little preparation for my final project and also explore the capacity of the Processing.

In this exercise, I use the image function and an image filter function. People can wist the potentiometer to change the value in the filter so that the color and shadow condition can change accordingly. (source: http://lolasbrush.com/painting-party/monet-water-lilies-jupiters/)

Like what we did in former recitations, I replace the fixed value in the filter function with the valueFromAudrino, so that I can connect the Audrino and Processing together. 

With only the potentiometer, what I can do is very limit. When I read the Computer Vision for Artist and Designer, I saw people now were combining audiences’ interactions with the process of digital computer functions.   What I have done in this exercise is only a changeable process that we can manipulate. If I have the chance to improve my exercise or make it more interactive, I hope that I can combine people’s emotion with the value of the filter. For example, if the sensor test users’ pupils extend among a certain range then the value will be changed. By doing this, I can actually replace my potentiometer with people’s engagement. This is how I feel an interactive project should achieve. From the reading, I realize that the reason why people use computer technology is its reliability in rational and authentic perspective. If we can make use of this characteristic, then the method we can apply to capture the emotion change of people will be more sensitive. I hope in the future where more and more high-tech projects will appear, people can develop a closer relationship with things that surrounds us. 

Works Cited

“Monet Water Lilies @ Jupiters.” Lola’s Brush, lolasbrush.com/painting-party/monet-water-lilies-jupiters/.

In this recitation, we discussion four projects in total including my own.

The first one is called “Put Down Your Phone and Plan a Tree”. This project basically designs a box or other form of the area where you can put your phone. If you do not move your phone, then there will be a tree growing on the screen. The designer of this project hopes that people can reduce the time people spend on their phones and focus on more of their lives. The feedback she got from the discussion are, firstly, where does the tree will appear? On the screen of the phone or an external screen? Secondly, is there a collaborating form which people can put multiple phones and have a different outcome? This project is very interesting for me because it does not require a lot of technological contribution. However, it requires more activities from users. How much will they think of when they are not allowed to use their mobile phones?

The second one is called “Your Pet Plant”. This project will include a physical plant and an electronic plant on the screen. When people touch the physical plant, the one on the screen will have different feedbacks, including sounds and animation changes. For this project, we think that designer should be specific about what feedback he will focus on. Because in order to start working on it, it is better for him to have a detailed plan so that he can start testing materials. This project, I think, it captures the ignored fact that we do not have a strong connection with the plant. Since plants remain silent all the time, hence we forget that they are also ling creatures. This is a very interesting idea.

The third project is called “E-Maze”. This is a maze, and people can control the character to go through the maze. At the same time, there will be colored imaged appearing after the character passes by. Our questions towards this project is that which one is the focal point, the maze or image? If the designer wants to concentrate on the maze, then the image will not be painted completely since the character will not pass by every corner. However, if he wants to concentrate on the drawing image, then there is no need to set the function of playing a maze game. It will be better to tell users that there is a reset image hidden here, and they need to go through all every area to see what the image looks like.

My project is called “Lilyzone”. I want to design an experience. There will be artificial water lilies made by plastic spoons. When people touch them, there will be the sound of “bianzhong” comes out. Together with the processing, I hope that I can give audiences both vocal and visualized feedbacks. I originally plan to put the lilies on a shelf and attach the wires under the shelf. During the discussion, people say that why not put them back into the water since water lilies are supposed to grow in the water. Hence, I think maybe instead of applying a shelf, I can make a fake pool and decorate it to make it looks like a pool. This feedback is very helpful. By changing the surrounding decorations, it will be easier for audiences to dive into the atmosphere that I hope to create.

In this week’s recitation, I had two exercise practice. The first one was to send instruction from the Audrino to the Processing. By sending certain instructions, there will be outcomes appearing through the Processing. I chose to connect the potentiometer which is able to send instructions, then I can use it to draw circles on the screen.  And the second is the opposite process which is from Processing to the Audrino. 

Here is the basic schematic for each circuit. Actually, the basic theory is the same. From this diagram, we just need to replace the potentiometer to the buzzer. Then it becomes the second circuit.

And here is the video for the first exercise :

recitaton10

I can move the circle from twisting the potentiometer.

Here is the code for processing:

import processing.serial.*;

String myString = null;
Serial myPort;
int fromArduino;

int NUM_OF_VALUES = 2; /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues; /** this array stores values from Arduino **/

void setup() {
size(500, 500);
printArray(Serial.list());
myPort = new Serial (this, Serial. list() [0],9600);
setupSerial();
}

void draw() {
updateSerial();
while(myPort. available()>0){
fromArduino = myPort.read();
}
background(fromArduino);
ellipse (map(fromArduino,0,225,0,width),height/3,fromArduino,fromArduino);
printArray(sensorValues);

}

void setupSerial() {
printArray(Serial.list());

myPort.clear();

myString = myPort.readStringUntil( 10 ); // 10 = ‘\n’ Linefeed in ASCII
myString = null;

sensorValues = new int[2];
}

void updateSerial() {
while (myPort.available() > 0) {
myString = myPort.readStringUntil( 10 ); // 10 = ‘\n’ Linefeed in ASCII
if (myString != null) {
String[] serialInArray = split(trim(myString), “,”);
if (serialInArray.length == 2) {
for (int i=0; i<serialInArray.length; i++) {
sensorValues[i] = int(serialInArray[i]);
}
}
}
}
}

And for the second exercise, I chose to use the keyPress code to achieve the instruction. If people press the “UP,” “DOWN,” and “RIGHT,” the buzzer will have a different type of sounds.

Here is the video:

And here is the code in the processing:

import processing.serial.*;

int NUM_OF_VALUES = 2; /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/

Serial myPort;
String myString;

// This is the array of values you might want to send to Arduino.
int values[] = new int[NUM_OF_VALUES];

void setup() {
size(500, 500);
background(0);

printArray(Serial.list());
myPort = new Serial(this, Serial.list()[ 3 ], 9600);
myString = myPort.readStringUntil( 10 ); // 10 = ‘\n’ Linefeed in ASCII
myString = null;
}

void draw() {
background(0);

// changes the values
for (int i=0; i<values.length; i++) {
values[i] = i; /** Feel free to change this!! **/
if (keyPressed == true) {
if (keyCode == UP) {
values[0] = 1;
values[1] = 1;
}
if (keyCode == DOWN) {
values[0] = 1;
values[1] = 2;
}
if (keyCode == RIGHT) {
values[0] = 1;
values[1] = 3;
}
} else {
values[0] = 0;
}
}

sendSerialData();

echoSerialData(200);
}

void sendSerialData() {
String data = “”;
for (int i=0; i<values.length; i++) {
data += values[i];
//if i is less than the index number of the last element in the values array
if (i < values.length-1) {
data += “,”; // add splitter character “,” between each values element
}
//if it is the last element in the values array
else {
data += “n”; // add the end of data character “n”
}
}
//write to Arduino
myPort.write(data);
}

void echoSerialData(int frequency) {
if (frameCount % frequency == 0) myPort.write(‘e’);

String incomingBytes = “”;
while (myPort.available() > 0) {
//add on all the characters received from the Arduino to the incomingBytes string
incomingBytes += char(myPort.read());
}
//print what Arduino sent back to Processing
print( incomingBytes );
}

Since for this recitation, we got the sample code for most of our exercise, hence, when I saw the code, I first felt confused and did not know how to start. Fortunately, I figured out at the end. This is also the first time we learn how to build the connection between the Audrino and processing which means we can finally have both visual and physical tools for our study.