A. Feel the forest – Yongxin(Phoebe) Li – Rodolfo Cossovich
B. CONCEPTION AND DESIGN:
I had decided to do something about forests as I started doing my project. I always think forests is full of knowledge and interest and make people feel a kind of the beauty of nature. However, since everyday we are living in the modern cities we seldom feel it (especially now we are in the lockdown!!), so I want to mimic some parts of the experience of people in the forest. Generally speaking, I put some inputs and outputs in a box and played some videos about different situations in the forest. When playing the video, I use my computer to share screen with my phone, and out the phone in the box, so people can see the image of forest directly in the box, which is more immersive than touching some sensors on the desk but staring at the computer screen in front of the participant.
The whole interaction process was like this: when the participants first open the lid of the box, they will see a piece of video showing the sunny days in the forest, this video will show the surrounding: sunlights, big trees, etc. Meanwhile, the computer will play the sound of birds in the forest, to make the interaction more immersive. But this is just a transition video. There is a leaf in the box, which is actually a steam sensor. Participants can touch the sensor. As they first touch the sensor, the video zoom in to show the image of many leaves, and sunlight shines through them. When participants holding and touching the leaves, they will see the leaves in the video moving because wind is blowing, and hearing the sound of wind blow through the leaves ( as well as the birds’ sound, this one is always played as long as the weather is good); when they release the leaf they move of leaves in the video stop and so does the sound of wind. At this stage, if participants want to stop the interaction, they can close the lid and the screen will become black and the sound will stop. However, if participants keep interacting for a very long time, the weather in the screen will become bad. A video about leaves washing by the rain will be played, as well as the sound of rains and thunders. It is a storm. Meanwhile, in the box, there will be a LED blinking, and a vibration output vibrating when there is thunder in the sound. I designed to make the LED keep blinking because I think this can make the atmosphere more like a storm weather. The box is small and portable, maybe participants use it in a very bright room it is hard to feel immersive in the bad weather. However, the LED is blue, I think it will add some blue color and shadows to the participants’ visual areas, or at least it can create a nervous atmosphere. Also, participants can feel the vibration if they touch the box, or at least they can hear the sound made by the vibration. During this process, participants cannot interaction by touching the sensor, nor can they close the lid to stop the sensor. Because I want to mimic the experience in the forest and in the forest when there is a storm, we human beings’ power is very small and can do few things. Participants only need to wait and the storm(video+sound) will stop automatically.
I actually learnt a lot from the first project in my preparatory research. In that project, participants can do different things to trigger different effect. By decomposing it, I think it use different input to trigger different outputs, so I did this in my project as well. When participants open the lid, the light sensor will trigger the transition part, when they touch the leaf, the steam sensor will trigger the wind, when they close the lid, the interaction stop. Also, both that project and mine create a continuous but various interaction process. Participants will get different experience when they participating different part, however, of curse, they are highly recommend to participate the whole process.
What help me make great adaptation and improvement was actually not the user testing session, but my conversation with professor Rudi. He recommended me to use my phone to play the video and it is really a good idea.
C. FABRICATION AND PRODUCTION:
Firstly, I made a lot of efforts on the code. At first, when I was trying to make the processing play the video when I touching the sensor, I only use one if logic. It was like:
if(sensorVlues[0] == 1){
video.read();
sound.play();
}
However, it does not work well. Then I changed my strategy. I used more if logic and add boolean variables. So it was like:
if (sensorValues[0] > 15 && playbirds == false) {//light sensor
playbirds = true;
birds.loop();
stop = false;
}
if (sensorValues[0] < 15 && playbirds == true) {
playbirds = false;
birds.stop();
stop = true;
}
And this time it worked well.
I also tried a lot to make the weather changing part work well. At first, it either change too early or do not change, so I added many conditions and used a lot of variables to confine, and finally it worked. It was like:
if (millis() >= 25000 && raintime < 40.0 && raintime >= 0 && stop == false && playrain == false) {
wind.stop();
windsound.stop();
birds.stop();
playrain = true;
badweather.loop();
}
//print(playrain);
if (playrain == true) {
if (rain.available()) {
rain.read();
}
processing_values[0] = 1;
}
image(rain, 0, 0);
if (volume > 0.5) {
processing_values[1] = 1;
} else {
processing_values[1] = 0;
}
image(rain, 0, 0);
To make the storm stoped automatically, I also learned to set the time the video played as a variable. I use float raintime = rain.time();, so when I want the process stop, I only need to write:
if (raintime > 40.0) {
playrain = false;
processing_values[0] = 0;
badweather.stop();
As for the circuit part, what spent me most efforts was selecting the sensor. At first, my design about the leaf was to blow the leaf, but I did not have the sensor that can sense wind. I tried all the sensors in the kits but none of them are satisfactory, then I thought of using touching instead of blowing. So my criteria was: (1)when I touch, the value change considerably; (2) when I keep touching, the value keep relatively stable; (3) when I release, the value change back to the original one; (4) it is small and easy to hide; (5) it allow certain errors; (6) it does not affect by the light, since I use light to realize other interaction. So, finally, steam sensor is the only one that satisfied my criteria.
D. CONCLUSIONS:
My goal is to mimic the experience of human in the forests and to create a whole process of various interactions. Additionally, I want to make a balance between the interactivity and level of immersive experience. I think my project achieve these goals. What is a pity was that at first I want to create a fake book and when participants open the book they can begin the interaction process but due to the limitation of materials I finally use a box instead of a book, and I think it to some extent degrade the interactivity of my project. Moreover, as professor Marcela’s words, it will be more immersive and interactive is I embed my phone a layer of box instead of simply put it on it.
E. TECHNICAL DOCUMENTATION:
Code:
Processing
import processing.serial.*;
import processing.sound.*;
import processing.video.*;
Movie forest;
Movie wind;
Movie rain;
SoundFile birds;
SoundFile windsound;
SoundFile badweather;
Amplitude analysis;
boolean playbirds = false;
boolean playwind = false;
boolean playrain = false;
boolean stop = false;
int NUM_OF_VALUES_FROM_PROCESSING = 2; /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int NUM_OF_VALUES_FROM_ARDUINO = 2; /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
Serial myPort;
String myString;
int processing_values[]; /** this array stores values you might want to send to Arduino **/
int sensorValues[]; /** this array stores values from Arduino **/
void setup() {
//fullScreen();
size(800, 800);
background(0);
forest = new Movie(this, "forest2.mp4");
wind = new Movie(this, "leaves.mp4");
forest.play();
wind.loop();
wind.volume(0);
rain = new Movie(this, "rain.mp4");
rain.loop();
birds = new SoundFile(this, "birds.mp3");
windsound = new SoundFile(this, "wind.mp3");
badweather = new SoundFile(this, "badweather.mp3");
analysis = new Amplitude(this);
analysis.input(badweather);
setupSerial();
}
void draw() {
//println(analysis.analyze());
// varibles used in draw loop
float volume = analysis.analyze();
float raintime = rain.time();
print(stop);
//receive the values from Arduino
getSerialData();
//use the values from arduino
//beginning forest
if (sensorValues[0] > 15 && playbirds == false) {//light sensor
playbirds = true;
birds.loop();
stop = false;
}
if (sensorValues[0] < 15 && playbirds == true) {
playbirds = false;
birds.stop();
stop = true;
}
//print(playbirds);
if (playbirds == true) {
if (forest.available()) {
forest.read();
}
}
image(forest, 0, 0);
//touching leaves and wind blowing
if (sensorValues[1] > 100 && playwind == false) {//steam sensor
playwind = true;
windsound.loop();
}
if (sensorValues[1] < 100) {
playwind = false;
windsound.stop();
}
//print(playwind);
if (playwind == true) {
if (wind.available()) {
wind.read();
}
}
image(wind, 0, 0);
// give values to the variables you want to send here
//if the participant hold it for too long period, the weather goes bad
//println(millis());
if (millis() >= 25000 && raintime < 40.0 && raintime >= 0 && stop == false && playrain == false) {
wind.stop();
windsound.stop();
birds.stop();
playrain = true;
badweather.loop();
}
//print(playrain);
if (playrain == true) {
if (rain.available()) {
rain.read();
}
processing_values[0] = 1;
}
image(rain, 0, 0);
if (volume > 0.5) {
processing_values[1] = 1;
} else {
processing_values[1] = 0;
}
image(rain, 0, 0);
//bad weather stops automatically after 10s
//print(raintime);
if (raintime > 40.0) {
playrain = false;
processing_values[0] = 0;
badweather.stop();
//rain.stop();
}
if(stop == true && millis() < 25000){
print("black");
fill(0);
rect(0, 0, 2000, 2000);
}
if(stop == true && raintime > 40.0){
print("black");
fill(0);
rect(0, 0, 2000, 2000);
}
// send the values to Arduino.
sendSerialData();
}
void setupSerial() {
printArray(Serial.list());
myPort = new Serial(this, Serial.list()[2], 9600);
myPort.clear();
// Throw out the first reading,
// in case we started reading in the middle of a string from the sender.
myString = myPort.readStringUntil( 10 ); // 10 = '\n' Linefeed in ASCII
myString = null;
processing_values = new int[NUM_OF_VALUES_FROM_PROCESSING];
sensorValues = new int[NUM_OF_VALUES_FROM_ARDUINO];
}
void getSerialData() {
while (myPort.available() > 0) {
myString = myPort.readStringUntil( 10 ); // 10 = '\n' Linefeed in ASCII
if (myString != null) {
print("from arduino: "+ myString);
String[] serialInArray = split(trim(myString), ",");
if (serialInArray.length == NUM_OF_VALUES_FROM_ARDUINO) {
for (int i=0; i<serialInArray.length; i++) {
sensorValues[i] = int(serialInArray[i]);
}
}
}
}
}
void sendSerialData() {
String data = "";
for (int i=0; i<processing_values.length; i++) {
data += processing_values[i];
//if i is less than the index number of the last element in the values array
if (i < processing_values.length-1) {
data += ","; // add splitter character "," between each values element
}
//if it is the last element in the values array
else {
data += "\n"; // add the end of data character "n"
}
}
//write to Arduino
myPort.write(data);
print("to arduino: "+ data); // this prints to the console the values going to arduino
}
Arduino
// IMA NYU Shanghai
// Interaction Lab
//This example is to send multiple values from Processing to Arduino and Arduino to Processing.
#define NUM_OF_VALUES_FROM_PROCESSING 1 /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
#define ZD 3
#define LED 13
/** DO NOT REMOVE THESE **/
int tempValue = 0;
int valueIndex = 0;
/* This is the array of values storing the data from Processing. */
int processing_values[NUM_OF_VALUES_FROM_PROCESSING];
void setup() {
Serial.begin(9600);
pinMode(3, OUTPUT);
pinMode(LED, OUTPUT);
// this block of code is an example of an LED, a DC motor, and a button
/*
pinMode(13, OUTPUT);
pinMode(9, OUTPUT);
pinMode(2, INPUT);
*/
//end example
}
void loop() {
// to receive a value from Processing
getSerialData();
// add your code here
// use elements in the values array
//and print values to send to Processing
// this is an example:
/*
//example of using received values and turning on an LED
if (processing_values[0] == 1) {
digitalWrite(13, HIGH);
} else {
digitalWrite(13, LOW);
}
analogWrite(9, processing_values[1]);
// too fast communication might cause some latency in Processing
// this delay resolves the issue.
delay(10);
//end of example of using received values
//example of sending the values to Processing
int sensor1 = analogRead(A0); // a potentiometer
int sensor2 = digitalRead(2); // the button
// send the values keeping this format
Serial.print(sensor1);
Serial.print(","); // put comma between sensor values
Serial.print(sensor2);
Serial.println(); // add linefeed after sending the last sensor value
// end of example sending values
*/
// end of example
if (processing_values[0] == 1) {
digitalWrite(LED, HIGH);
delay(100);
}
if (processing_values[0] == 0) {
digitalWrite(LED, LOW);
delay(100);
}
if (processing_values[1] == 1) {
analogWrite(ZD, 200);
delay(50);
}
if (processing_values[1] == 0) {
analogWrite(ZD, 0);
delay(0);
}
//Serial.println(LED);
// to send values to Processing assign the values you want to send
int sensor1 = analogRead(A0);
int sensor2 = analogRead(A1);
// send the values keeping this format
Serial.print(sensor1);
Serial.print(","); // put comma between sensor values
Serial.print(sensor2);
// // Serial.print(","); // put comma between sensor values
// // Serial.print(sensor3);
Serial.println(); // add linefeed after sending the last sensor value
// too fast communication might cause some latency in Processing
// this delay resolves the issue.
delay(100);
// end of example sending values
}
//receive serial data from Processing
void getSerialData() {
while (Serial.available()) {
char c = Serial.read();
//switch - case checks the value of the variable in the switch function
//in this case, the char c, then runs one of the cases that fit the value of the variable
//for more information, visit the reference page: https://www.arduino.cc/en/Reference/SwitchCase
switch (c) {
//if the char c from Processing is a number between 0 and 9
case '0'...'9':
//save the value of char c to tempValue
//but simultaneously rearrange the existing values saved in tempValue
//for the digits received through char c to remain coherent
//if this does not make sense and would like to know more, send an email to me!
tempValue = tempValue * 10 + c - '0';
break;
//if the char c from Processing is a comma
//indicating that the following values of char c is for the next element in the values array
case ',':
processing_values[valueIndex] = tempValue;
//reset tempValue value
tempValue = 0;
//increment valuesIndex by 1
valueIndex++;
break;
//if the char c from Processing is character 'n'
//which signals that it is the end of data
case '\n':
//save the tempValue
//this will b the last element in the values array
processing_values[valueIndex] = tempValue;
//reset tempValue and valueIndex values
//to clear out the values array for the next round of readings from Processing
tempValue = 0;
valueIndex = 0;
break;
}
}
}