Bruce

Void Music Shapes:

 

Project Statement of Purpose:

 

The project “void music shapes” has been a product of Alex’s and my inspiration from the project Bubbles’ Cellar, where participants use the behavior of blowing bubbles in the reality to create projected bubbles in the cellar. We also gained inspiration from Mutual input sessions, where participants will use the projection of their hands to create different shapes that make music. The general concept of behaving in the reality to create something in the virtual world really inspired us a lot, and our project is like a combination of the two artifacts mentioned above. The ultimate purpose of Void Music Shapes is to function as a novel interactive music expression. The player will create shapes that fall and bounce on the processing sketch by waving their wrist, while the movement of the other hand will be used to decide the shapes of the object displayed. This project intends to create a new means to perform music, and the main challenge we seek to address now is to combine the Arduino part and the processing part efficiently to make a better music expression delivery. 

 

Project Plan:

 

Again, this project aims to create a novel way for music expression to users. Let me be more specific about the design of the artifact. The Arduino part will act as the input part of the project, using an acceleration sensor and the ultrasonic sensor. The acceleration sensor will be attached to the back of the user’s hand, detecting the movement which will be used to create different objects on the processing sketch, while at the same time, the ultrasonic sensor will detect its distance with the user’s hand and based on that, decide the shape of the objects created. The objects created will then fall from the top of the screen to the end of the screen and then gradually disappear, and during this process, where the objects hit the ground, different sounds will be generated based on different shapes. What users want, from my perspective, is to have an experience that they are really creating music instead of throwing stones with sound. Therefore, we have included the procedure that the sun will be random and based on the shapes of different objects, instead of creating a mechanic project that certain sounds will be generated in response to a certain input. To achieve this, Alex and I will see to different tasks. The detailed plan is that he will be in charge of the processing part, while I will be in charge of the Arduino part. These two parts will be finished separately before the end of Thursday, and before Friday, a prototype with two parts combined together will be finished for the user test on that day. Based on the feedback from the user test, we will make certain adjustments before 12.8. 

 

Context and Significance:

 

As mentioned in the first paragraph, Bubbles’ Cellar and Mutual input sessions are two artifacts that inspired me most because of their concept of creating something in the virtual world as a response to real-world behavior. The point is that, if these two actions are separated, they will both become meaningless, and if they are combined, they will be interactive enough to create a unique interactive experience for the users. In fact, if I have to name exactly what artifacts influenced me on the concept of my final project, I would say they are all the artifacts I have researched this semester because they all contributed to my definition of interaction. The interaction needs to be simultaneous between the participants of the interaction, and continuity, along with unpredictability, or in other words, the ability to create different interaction processes for different users are also important factors to be included in the interaction process. Similarly, in the article called Art, interaction and engagement written by Ernest Edmonds, he said that “Thus, activities external to the artwork itself, such as audience behavior, altered the generative process.” (7). The relatively spontaneous interaction from both sides drags the general interaction process out of the simple mind mode of input-output. Alex’s and my final project, Void Music shapes, satisfy these requirements because each user’s experience will be different because the music different people create will definitely different from each other, and once the interaction process has started, the user won’t need further instructions to inform him/her what to do next. The interaction process will then be simultaneous and continuous. This project, frankly speaking, doesn’t focus specifically on satisfying a certain group of people’s needs. Instead, Alex and I would like to create something that everyone can participate in and have fun with, generating music with a novel form and interface. The success of our project will potentially give inspirations to the artists who are interested in creating new interactive interfaces for music expression, which is also something we hope to achieve. Also, we wish that the idea of combing the virtual world and reality will also be passed on. 

 

Exercise 1:

Ellipse Movement

Line Movement

Arduino Code here:

// IMA NYU Shanghai
// Interaction Lab
// For sending multiple values from Arduino to Processing


void setup() {
  Serial.begin(9600);
}

void loop() {
  int sensor1 = analogRead(A0);
  int sensor2 = analogRead(A1);

  // keep this format
  Serial.print(sensor1);
  Serial.print(",");  // put comma between sensor values
  Serial.print(sensor2);
  Serial.println(); // add linefeed after sending the last sensor value
  // too fast communication might cause some latency in Processing
  // this delay resolves the issue.
  delay(100);
}

Processing Code here:

 

import processing.serial.*;

String myString = null;
Serial myPort;


int NUM_OF_VALUES = 2;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/
float X0 = width/2;
float Y0 = height/2;
float Xt;
float Yt;
void setup() {
  size(500, 500);
  background(255);
  setupSerial();
}


void draw() {
  updateSerial();
  printArray(sensorValues);

  Xt = map(sensorValues[0], 0, 1023, -500, 500);
  Yt = map(sensorValues[1], 0, 1023, -500, 500);
  fill(0);
  line(X0, Y0, Xt, Yt);
  X0=Xt;
  Y0=Yt;
}



void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 7 ], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}



void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = '\n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

 

Exercise 2:

Music

Arduino Code here:

/**
  This example is to send multiple values from Processing to Arduino.
  You can find the Processing example file in the same folder which works with this Arduino file.
  Please note that the echo case (when char c is 'e' in the getSerialData function below)
  checks if Arduino is receiving the correct bytes from the Processing sketch
  by sending the values array back to the Processing sketch.
 **/

#define NUM_OF_VALUES 2    /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/


/** DO NOT REMOVE THESE **/
int tempValue = 0;
int valueIndex = 0;

/* This is the array of values storing the data from Processing. */
int values[NUM_OF_VALUES];
float x;
float y;

void setup() {
  Serial.begin(9600);
  pinMode(11, OUTPUT);
//  pinMode(9, OUTPUT);
}

void loop() {
  getSerialData();
  x= map(values[0], 0, 500, 500, 1000);
  y= map(values[1], 0, 500, 100, 200);
  
  tone(11, x, y);
  delay(1);
  
  // add your code here
  // use elements in the values array
  // values[0]
  // values[1]
//  if (values[0] == 1) {
//    digitalWrite(13, HIGH);
//  } else {
//    digitalWrite(13, LOW);
//  }
//
//  if (values[1] == 1) {
//    tone(9, values[2]);
//  } else {
//    noTone(9);
//  }

}


//recieve serial data from Processing
void getSerialData() {
  if (Serial.available()) {
    char c = Serial.read();
    //switch - case checks the value of the variable in the switch function
    //in this case, the char c, then runs one of the cases that fit the value of the variable
    //for more information, visit the reference page: https://www.arduino.cc/en/Reference/SwitchCase
    switch (c) {
      //if the char c from Processing is a number between 0 and 9
      case '0'...'9':
        //save the value of char c to tempValue
        //but simultaneously rearrange the existing values saved in tempValue
        //for the digits received through char c to remain coherent
        //if this does not make sense and would like to know more, send an email to me!
        tempValue = tempValue * 10 + c - '0';
        break;
      //if the char c from Processing is a comma
      //indicating that the following values of char c is for the next element in the values array
      case ',':
        values[valueIndex] = tempValue;
        //reset tempValue value
        tempValue = 0;
        //increment valuesIndex by 1
        valueIndex++;
        break;
      //if the char c from Processing is character 'n'
      //which signals that it is the end of data
      case 'n':
        //save the tempValue
        //this will b the last element in the values array
        values[valueIndex] = tempValue;
        //reset tempValue and valueIndex values
        //to clear out the values array for the next round of readings from Processing
        tempValue = 0;
        valueIndex = 0;
        break;
      //if the char c from Processing is character 'e'
      //it is signalling for the Arduino to send Processing the elements saved in the values array
      //this case is triggered and processed by the echoSerialData function in the Processing sketch
      case 'e': // to echo
        for (int i = 0; i < NUM_OF_VALUES; i++) {
          Serial.print(values[i]);
          if (i < NUM_OF_VALUES - 1) {
            Serial.print(',');
          }
          else {
            Serial.println();
          }
        }
        break;
    }
  }
}

 

Processing Code here:

 

// IMA NYU Shanghai
// Interaction Lab


/**
 * This example is to send multiple values from Processing to Arduino.
 * You can find the arduino example file in the same folder which works with this Processing file.
 * Please note that the echoSerialData function asks Arduino to send the data saved in the values array
 * to check if it is receiving the correct bytes.
 **/


import processing.serial.*;

int NUM_OF_VALUES = 2;  /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/


Serial myPort;
String myString;

// This is the array of values you might want to send to Arduino.
int values[] = new int[NUM_OF_VALUES];

void setup() {
  size(500, 500);
  background(0);

  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 7 ], 9600);


  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;
}


void draw() {
  background(0);

  // changes the values
  for (int i=0; i<values.length; i++) {
    values[0] = mouseX;
    values[1] = mouseY;/** Feel free to change this!! **/
  }

  // sends the values to Arduino.
  sendSerialData();

  // This causess the communication to become slow and unstable.
  // You might want to comment this out when everything is ready.
  // The parameter 200 is the frequency of echoing. 
  // The higher this number, the slower the program will be
  // but the higher this number, the more stable it will be.
  echoSerialData(200);
}

void sendSerialData() {
  String data = "";
  for (int i=0; i<values.length; i++) {
    data += values[i];
    //if i is less than the index number of the last element in the values array
    if (i < values.length-1) {
      data += ","; // add splitter character "," between each values element
    } 
    //if it is the last element in the values array
    else {
      data += "n"; // add the end of data character "n"
    }
  }
  //write to Arduino
  myPort.write(data);
}


void echoSerialData(int frequency) {
  //write character 'e' at the given frequency
  //to request Arduino to send back the values array
  if (frameCount % frequency == 0) myPort.write('e');

  String incomingBytes = "";
  while (myPort.available() > 0) {
    //add on all the characters received from the Arduino to the incomingBytes string
    incomingBytes += char(myPort.read());
  }
  //print what Arduino sent back to Processing
  print( incomingBytes );
}

 

Homework:

Additional Homework

Arduino Code here:

// IMA NYU Shanghai
// Interaction Lab
// For sending multiple values from Arduino to Processing


void setup() {
  Serial.begin(9600);
}

void loop() {
  int sensor1 = digitalRead(4);
  int sensor2 = digitalRead(8);

  // keep this format
  Serial.print(sensor1);
  Serial.print(",");  // put comma between sensor values
  Serial.print(sensor2);
  Serial.println();
  // add linefeed after sending the last sensor value

  // too fast communication might cause some latency in Processing
  // this delay resolves the issue.
  delay(100);
}

 

Processing Code here:

 

import processing.serial.*;

String myString = null;
Serial myPort;


int NUM_OF_VALUES = 2;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/
int preButtonValue0=0;
int preButtonValue1=0;
boolean star1=false;
boolean star2=false;



void setup() {
  size(600, 600);
  background(0);
  setupSerial();
}


void draw() {
  updateSerial();
  printArray(sensorValues);
  background(0);
  if(sensorValues[0]==1 && sensorValues[0]!=preButtonValue0) {
    star1 = !star1;
  }
  if (sensorValues[1] ==1 && sensorValues[1] !=preButtonValue1){
    star2 = !star2 ;
  }
  if (star1){
    pushMatrix();
    translate(width*0.3, height*0.3);
    rotate(frameCount / 200.0);
    star(0,0,30,70,5);
    popMatrix();
  }
  if (star2) {
    pushMatrix();
    translate(width*0.7, height*0.7);
    rotate(frameCount / 400.0);
    star(0,0,80,100,40);
    popMatrix();
  }
  
    preButtonValue0= sensorValues[0];
    preButtonValue1= sensorValues[1];
    

  
}

void star(float x, float y, float radius1, float radius2, int npoints) {
  float angle = TWO_PI / npoints;
  float halfAngle = angle/2.0;
  beginShape();
  for (float a = 0; a < TWO_PI; a += angle) {
    float sx = x + cos(a) * radius2;
    float sy = y + sin(a) * radius2;
    vertex(sx, sy);
    sx = x + cos(a+halfAngle) * radius1;
    sy = y + sin(a+halfAngle) * radius1;
    vertex(sx, sy);
  }
  endShape(CLOSE);

}



void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[3], 9600);
  
  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}



void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = '\n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

 

 

 

 

PROJECT PROPOSAL:

1.  Sound Canvas

   

The project “Sound canvas” focuses on visualization of sound.  Different notes will be representing the x-axis of the processing display screen, and at the same time, the volume of the notes will be representing the y-axis of the screen. A point for each note has thus formulated and therefore using sound to draw lines becomes possible. Additionally, the shape of the lines will also change according to different tones. For example, songs from Guns N Roses will definitely “look” different from that from Chopin. Also, the screen will refresh itself after certain time to avoid overcapacity. This idea is inspirited by the bubble project I included in my previous blog post, which all focus on integrating something happening in the real life into something virtual. The biggest problem of this project, is to figure out a way to best visualize the sound without letting the screen being too crowded. Furthermore, This project, especially, is designed for deaf people, who can’t feel the sound with their ears. This project intends to offer them an opportunity, and another way, to perceive, see and feel the sound. 

    

2. Sand Art

   

Frankly speaking, this idea comes from my mid-term project, but, let me explain it again. Users will make sand drawings on a working pad where LDR will detect the intensity of the light and send in signals. The signals will be transferred to processing and what is drawing on the sanding painting working pad, will be displayed in pixels in the screen. This project, or more specifically, this game, requires at least two players. One person draws on the working pad and the other person tries to guess what he is drawing from the pixels displayed in the screen. The main challenge of this project is to find a way to use as fewer Arduino board as possible to control a large number of sensors. In other words, the challenge here is to avoid using countless boards (like my midterm project). This project is designed for kids mainly. I think the form of sand art can greatly strength children’s operational ability and the appreciation for art. 

 

 

3. Drawing Gloves 

“Drawing Gloves” is an idea suddenly posed up in my head, and it’s hard to tell where I get the inspiration of this form (maybe from the robotic hand of Luke Skywalker). The participant will use the glove to draw. Gestures will be used for change the color (The idea of using gestures comes from the MacOS). The second to the fourth finger will work together to control the direction of different lines. The first finger will be used to deicide how much the line will be curved. The interface of this project will eventually look like a kind of exoskeleton. The biggest problem of this project design is to optimize the process of using gestures and different fingers to draw. In other words, the problem is to find a best way for users to draw with this glove. This project is designed for kids. Different from the second project which is also designed for kids, this project aims to sparkle their enthusiasm towards the IoT and showing them novel ways to interact with the world. 

    

This week’s recitation task focuses on using arrays to control a large number of objects at the same time.

The project I created:

Done!

Code here:

 

int[] Yue = new int[100];
float[] x = new float[100];
float[] y = new float[100];
color[] c = new color[100];


void setup() {
  size(600, 600);
  background(255);
  for (int i=0; i < 100; i ++) {
    x[i] = random(30, width-30);
    y[i] = random(30, height-30);
    c[i] = color(int(random(255)), int(random(255)), int(random(255)));
    Yue[i] = int(random(0, 2))*4 - 2;
  }
}

void display(float x, float y, color c) {
  rect(x-20, y-20, 40, 40);
  strokeWeight(5);
  fill(255);
  circle(x, y, 40);
  fill(c);
  ellipse(x, y, 40, 20);
  fill(0);
  circle(x, y, 10);
}

void draw() {
  background(255);
  for (int i=0; i < 100; i ++) {
    display(x[i], y[i], c[i]);
    if ((x[i] + Yue[i] > width-30) || (x[i] + Yue[i] < 30)) {
      Yue[i] = -Yue[i];
    }
    if ((y[i] + Yue[i] > height-30) || (y[i] + Yue[i] < 30)) {
      Yue[i] = -Yue[i];
    }
    x[i] = x[i]+Yue[i];
    y[i] = y[i]+Yue[i];
  }
}

I turned the “+” in the last line into a “-” and the animation will eventually stop

Code here:

 

int[] Yue = new int[100];
float[] x = new float[100];
float[] y = new float[100];
color[] c = new color[100];


void setup() {
  size(600, 600);
  background(255);
  for (int i=0; i < 100; i ++) {
    x[i] = random(30, width-30);
    y[i] = random(30, height-30);
    c[i] = color(int(random(255)), int(random(255)), int(random(255)));
    Yue[i] = int(random(0, 2))*4 - 2;
  }
}

void display(float x, float y, color c) {
  rect(x-20, y-20, 40, 40);
  strokeWeight(5);
  fill(255);
  circle(x, y, 40);
  fill(c);
  ellipse(x, y, 40, 20);
  fill(0);
  circle(x, y, 10);
}

void draw() {
  background(255);
  for (int i=0; i < 100; i ++) {
    display(x[i], y[i], c[i]);
    if ((x[i] + Yue[i] > width-30) || (x[i] + Yue[i] < 30)) {
      Yue[i] = -Yue[i];
    }
    if ((y[i] + Yue[i] > height-30) || (y[i] + Yue[i] < 30)) {
      Yue[i] = -Yue[i];
    }
    x[i] = x[i]+Yue[i];
    y[i] = y[i]-Yue[i];
  }
}

Answer to question one:

If the “for loop” were removed to the draw section, the overall animation will become incredibly fast because the randomizing process is repeated every millisecond. 

Answer to question two:

Using arrays offered programmers the opportunity to simplify their work, enabling them to control a great number of objects at the same moment. Also, this helps to reduce the amount of the code. If I were to use arrays in a potential project,  I think I would program them in a random way, delivering to the audience a feeling of unpredictability, which is one key factor, from my perspective, of a successful interaction.

 

Additional Homework Here: (Really took me plenty of time to figure it out)

Matrix