Author: Jiayan Liu

I just went to my favorite singer’s concert  and I thought the lighting in the concert is very suitable for the tint function. I set the blue and green value to random and use one potentiometer to control the red value. An important thing is to map the values from the potentiometer to  (0,255).

This is the final result:

This is my processing code:

import processing.serial.*;
import processing.video.*;
Movie myMovie;

String myString = null;
Serial myPort;


int NUM_OF_VALUES = 1;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/


void setup() {
  size(1280, 1080);
  //myMovie = new Movie(this, "dancing.mp4");
  myMovie = new Movie(this, "hcy.MOV");
  myMovie.play();
  setupSerial();
  
}


void draw() {
  updateSerial();
  printArray(sensorValues);

float hcy = map (sensorValues[0],0, 1023, 0,255);

  // use the values like this!
  // sensorValues[0] 

  // add your code
if (myMovie.available()) {
    myMovie.read();
  }
  tint(hcy, random(255), random(255)); 
  image(myMovie, 0, 0);
  //
}



void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 1 ], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}



void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = '\n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

this is my Arduino code

[code] // IMA NYU Shanghai
// Interaction Lab
// For sending multiple values from Arduino to Processing

void setup() {
Serial.begin(9600);
}

void loop() {
int sensor1 = analogRead(A0);
//int sensor2 = analogRead(A1);
//int sensor3 = analogRead(A2);

// keep this format
Serial.print(sensor1);
//Serial.print(“,”); // put comma between sensor values
//Serial.print(sensor2);
//Serial.print(“,”);
//Serial.print(sensor3);
Serial.println(); // add linefeed after sending the last sensor value

// too fast communication might cause some latency in Processing
// this delay resolves the issue.
delay(100);
}
[/code]

Reflection about the ways technology was used in your project:

After reading the texts, I realized that technology is just one approach to convey my meaning in the project, to enhance the interactive experience. technology itself is not the main part of the project, the most important things are first, the message behind. Second, the user’s experience. 

In the text, it introduced Rafael Lozano­ Hemmer’s installation Standards and Double Standards (2004). I like this project because first it involves full-body movement. Second, it’s thought-provoking. Are those belts a representative of humans? The first time encounter it, I was not impressed by how fancy technology is used in this installation but to think about the deeper meaning behind this project.

In our final project, we also want users to have full-body interaction so that we plan to use multiple sensors. Then, we want users to think about the relations between human and nature while experiencing this project.

1. PROJECT TITLE

Reconsider Human-Nature Relationship

PROJECT STATEMENT OF PURPOSE *

This project aims at addressing the question of “human-nature” relationship. The insights we’ve found are first an art work in our school’s gallery called the 72-relations with a goldenrod. And another internet interactive art project called “way to go”. These two works narrow our idea and make it tangible. Our project provides experience for users to explore different ways that one can interact with nature and then reflect back on our own behavior in daily life.

PROJECT PLAN *

Our project is a screen-based VR experience. Users interact with multiple sensors physically and it results in different visuals on the screen. We will gather our shooting footage by Nov.23 and edit it by Nov.24. On Nov.25-27, we will start to figure out the code in Arduino and processing with the help of professors and fellows as well as online tutorials. We need to know first, what kinds of sensors we need in terms of “relation between nature”. Second, how to add animations onto a pre-shot video. Third, how to switch between different scene we’ve shot. On Nov.28-30, the thanksgiving holiday, as we already have a basic structure in our code, we will try to modify it on our own. By Dec. 2, we will manage to run through the whole project, make the Arduino and processing communicating with each other. For the rest of the week, we need to design for which part we want to use fabrication tools to help us as well as user testing and further improvements.

CONTEXT AND SIGNIFICANCE *

In our preparatory research and analysis, we define interaction as a kind of conversation where two or more actors are involved. They listen (receive the information), think (process the information) and speak (give out the processed information). In the projects we’ve looked at, two things inspired us. First is the how to trigger user’s motivation to interact. We found out that users are more possible to interact when they see themselves in the screen (for example, using mirrors). So, we decided in the transition of different scene, the users can see themselves by involving web cam in our project. The second thing is that when there’s more people interacting together, it gets more fun. So, with different sensors, our project lets multiple users to explore together. after successful completion, our project leads the users to think about their relations with nature and to reflect on their everyday behavior. The idea of this project is inspired by our humanities course which the topic is “the question of the Anthropocene”. We begin to think about how we can better coexist with nature, to keep the harmony between human and the environment. I guess this is the biggest significance of our project. This project aims at users of all ages, because the question of human and nature is a question that involves all of us.

Exercise 1: Make a Processing Etch A Sketch

circles:

lines:

The Arduino code for the ellipses and the lines are the same.

First change the input value to 2 because we have two potentiometers, commented out  //Serial.print(“,”); //Serial.print(sensor3);

[code] // IMA NYU Shanghai
// Interaction Lab
// For sending multiple values from Arduino to Processing

void setup() {
Serial.begin(9600);
}

void loop() {
int sensor1 = analogRead(A0);
int sensor2 = analogRead(A1);
//int sensor3 = analogRead(A2);

// keep this format
Serial.print(sensor1);
Serial.print(“,”); // put comma between sensor values
Serial.print(sensor2);
//Serial.print(“,”);
//Serial.print(sensor3);
Serial.println(); // add linefeed after sending the last sensor value

// too fast communication might cause some latency in Processing
// this delay resolves the issue.
delay(100);
}
[/code]

For the processing code, first step is to comment out the line and to figure out the port, the second step is to map the value of the potentiometer to the canvas size.

Processing code for the ellipse: 

// IMA NYU Shanghai
// Interaction Lab
// For receiving multiple values from Arduino to Processing

/*
 * Based on the readStringUntil() example by Tom Igoe
 * https://processing.org/reference/libraries/serial/Serial_readStringUntil_.html
 */

import processing.serial.*;

String myString = null;
Serial myPort;


int NUM_OF_VALUES = 2;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/
float B;
float C;

void setup() {
  size(500, 500);
  //background(0);
  setupSerial();
}


void draw() {
  updateSerial();
  printArray(sensorValues);

float x = map(sensorValues[0],0,1023,0,500);
  float y = map(sensorValues[1],0,1023,0,500);
  printArray(sensorValues);
  fill(255);
  ellipse(x,y,100,100);

  // use the values like this!
  // sensorValues[0] 

  // add your code

  //
}



void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 1 ], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}



void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = '\n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

Processing code:

// IMA NYU Shanghai
// Interaction Lab
// For receiving multiple values from Arduino to Processing

/*
 * Based on the readStringUntil() example by Tom Igoe
 * https://processing.org/reference/libraries/serial/Serial_readStringUntil_.html
 */

import processing.serial.*;

String myString = null;
Serial myPort;


int NUM_OF_VALUES = 2;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/
float B;
float C;

void setup() {
  size(500, 500);
  //background(0);
  setupSerial();
}


void draw() {
  updateSerial();
  printArray(sensorValues);


  float x = map(sensorValues[0],0,1023,0,width);//these are values that are being calculated, 0 to width part is making it compressed 
  float y = map(sensorValues[1],0,1023,0,height);// different scale
  line (x,y,B,C);//makes the line change the position based on the sensor values
   B=x;// where we start the next time will be where "x" was
   C=y;//where we start the next time will be where "y" was 
   


  // use the values like this!
  // sensorValues[0] 

  // add your code

  //
}



void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 1 ], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}



void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = '\n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

The difference between two codes is in the void draw, for the first one I use ellipse function and the second one I use the line function.

Exercise 2: Make a musical instrument with Arduino

the key to this is the number of values (2: duration and frequency) and the tone function tone (pin number, frequency, duration)

Arduino code:

[code] // IMA NYU Shanghai
// Interaction Lab

/**
This example is to send multiple values from Processing to Arduino.
You can find the Processing example file in the same folder which works with this Arduino file.
Please note that the echo case (when char c is ‘e’ in the getSerialData function below)
checks if Arduino is receiving the correct bytes from the Processing sketch
by sending the values array back to the Processing sketch.
**/

#define NUM_OF_VALUES 2 /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/

/** DO NOT REMOVE THESE **/
int tempValue = 0;
int valueIndex = 0;

/* This is the array of values storing the data from Processing. */
int values[NUM_OF_VALUES];

void setup() {
Serial.begin(9600);
pinMode(11, OUTPUT);
}

void loop() {
getSerialData();
tone(11,values[0],values[1]);

}

//recieve serial data from Processing
void getSerialData() {
if (Serial.available()) {
char c = Serial.read();
//switch – case checks the value of the variable in the switch function
//in this case, the char c, then runs one of the cases that fit the value of the variable
//for more information, visit the reference page: https://www.arduino.cc/en/Reference/SwitchCase
switch (c) {
//if the char c from Processing is a number between 0 and 9
case ‘0’…’9′:
//save the value of char c to tempValue
//but simultaneously rearrange the existing values saved in tempValue
//for the digits received through char c to remain coherent
//if this does not make sense and would like to know more, send an email to me!
tempValue = tempValue * 10 + c – ‘0’;
break;
//if the char c from Processing is a comma
//indicating that the following values of char c is for the next element in the values array
case ‘,’:
values[valueIndex] = tempValue;
//reset tempValue value
tempValue = 0;
//increment valuesIndex by 1
valueIndex++;
break;
//if the char c from Processing is character ‘n’
//which signals that it is the end of data
case ‘n’:
//save the tempValue
//this will b the last element in the values array
values[valueIndex] = tempValue;
//reset tempValue and valueIndex values
//to clear out the values array for the next round of readings from Processing
tempValue = 0;
valueIndex = 0;
break;
//if the char c from Processing is character ‘e’
//it is signalling for the Arduino to send Processing the elements saved in the values array
//this case is triggered and processed by the echoSerialData function in the Processing sketch
case ‘e’: // to echo
for (int i = 0; i < NUM_OF_VALUES; i++) {
Serial.print(values[i]);
if (i < NUM_OF_VALUES – 1) {
Serial.print(‘,’);
}
else {
Serial.println();
}
}
break;
}
}
}
[/code]

Processing code:

// IMA NYU Shanghai
// Interaction Lab


/**
 * This example is to send multiple values from Processing to Arduino.
 * You can find the arduino example file in the same folder which works with this Processing file.
 * Please note that the echoSerialData function asks Arduino to send the data saved in the values array
 * to check if it is receiving the correct bytes.
 **/


import processing.serial.*;

int NUM_OF_VALUES = 2;  /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/


Serial myPort;
String myString;

// This is the array of values you might want to send to Arduino.
int values[] = new int[NUM_OF_VALUES];

void setup() {
  size(500, 500);
  background(0);

  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 1], 9600);
  // check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index of the port

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;
}


void draw() {
  background(0);
values [0]= mouseX;//frequency is changing based on moving the mouse around, sends to arduino
values [1]=int(mousePressed);// turn true or false (boolean) into an integer that can be sent 
values[0]=mouseX;
  values[1]=mouseY;
  ellipse(mouseX,mouseY,20,20);
  // sends the values to Arduino.
  sendSerialData();

  // This causess the communication to become slow and unstable.
  // You might want to comment this out when everything is ready.
  // The parameter 200 is the frequency of echoing. 
  // The higher this number, the slower the program will be
  // but the higher this number, the more stable it will be.
  echoSerialData(200);
}

void sendSerialData() {
  String data = "";
  for (int i=0; i<values.length; i++) {
    data += values[i];
    //if i is less than the index number of the last element in the values array
    if (i < values.length-1) {
      data += ","; // add splitter character "," between each values element
    } 
    //if it is the last element in the values array
    else {
      data += "n"; // add the end of data character "n"
    }
  }
  //write to Arduino
  myPort.write(data);
}


void echoSerialData(int frequency) {
  //write character 'e' at the given frequency
  //to request Arduino to send back the values array
  if (frameCount % frequency == 0) myPort.write('e');

  String incomingBytes = "";
  while (myPort.available() > 0) {
    //add on all the characters received from the Arduino to the incomingBytes string
    incomingBytes += char(myPort.read());
  }
  //print what Arduino sent back to Processing
  print( incomingBytes );
}

reflect on the interactions: for exercise one, the physical interaction is to turn the knobs and resulted in drawing on the screen. For exercise two, the interaction is to move the mouse and produce different tones. I prefer the first one because we now are always typing on key broad and click the mouse. 

1: rethink the human-nature relationship

We want to address the problem of environmental protection and let the audience think about the question: how closely man can interfere nature. The insight comes from one writing class both of us are taking, the topic of the class is called: the question of the Anthropocene. The reason to declare this new era, the Anthropocene, is because of the acceleration of environmental changes caused by humans are so profound. And we need to rethink the relation of human and nature. It’s relevant to all of us. In the researches, we found more and more contemporary art works start to look at nature and humanities, including the exhibition in our school’s gallery. Also, in my research, I think it’s better to involve more people in an interactive experience.

We want to shoot videos ourselves of scenes that represents nature, combine it to processing and add web cam interactions that the users can see themselves in the scene. In both our research and personal experience, we found that projects involves mirror or some kinds of reflections that we can place ourselves in can attract our attentions more. By seeing themselves on the screen realtime, the users will want to move farther of nearer, or have different poses. We want add some kind of sounds can visuals that will change according to users distance to the screen with the distance sensor. In this case, we want the users to think about their distance between nature and the power of one man and a group of them.

2: Project title: release your magic

As a fan of Harry Potter, I always imagine myself to have that kind of power: controlling things in a wave of the wand. For this project, we aim at children or Harry Potter fans of all ages, to interact a scene from the movie, seeing themselves on the screen. In our research, there’s a very playful interaction screen that the movement of objects on the screen is depending on the act of users. This interaction evolves huge physical movement and multiple people. The issue we want to convey a positive attitude towards our users: we all have magic power. Recently especially after midterm, many of my friends are under great pressure and keep looking down on themselves. So, I think it’s really important to send out positive messages towards our project.

3:project title: Listen to the non-human

We are inspired by a project called “Meaningful interactive audio experiences” created by Playtronica Studio Showreel which experimented with sounds from different objects such as plants, water, and our everyday objects. The project involves different kind of senses including watching, hearing and touching. In this artistic way, everyday objects take on new identities. In our project, we encourage users to experience the relationship between themselves and the objects around them by interacting with the objects in a totally different way from what we are used to: to touch and hear. In this kind of practice, we want to encourage people to be not that rush in everyday life and ignore the value of little object around us.

we are trying to enhance the interaction between human and non-human to make humans realize that non-human like static object has its own logic and even possesses some human traits, so as to raise humans’ respect for them and challenge assumption about them as disposable. In terms of the interaction, we will attribute human traits like generating sound (even language) or make movements so that the user will realize that the seemingly static and non-living thing has its own logic and life.