Author: Russel Sy

Introduction

The purpose of this recitation was to use serial communication in order to communicate between processing and arduino with more than one variable.

Etch-a-sketch

We were tasked with creating an etch-a-sketch using values taken from potentiometers on arduino then translating it to a line being drawn on processing. I first started by creating a code on processing in which I could draw a line using my mouse, and clear the line by pressing the enter key.

  stroke(100);
  strokeWeight(4);
  line(mouseX,mouseY,pmouseX,pmouseY);
  if (keyPressed) {
    if (key == ENTER) {
      background(200);

Afterwards, I created a circuit on arduino with two potentiometers, and created code in order to print the values of both the potentiometers.

void setup() {
  Serial.begin(9600);
}

void loop() {
  int sensor1 = analogRead(A0);
  int sensor2 = analogRead(A1);

  Serial.print(sensor1); 
  Serial.print(",");
  Serial.print(sensor2);
  Serial.println();

  delay(100);
}

After creating the two separate codes, I used the serial communication template for communication between arduino to processing. I had to tweak the values on the line() function so that instead of reading the x and y values from the mouse in order to draw the line, the values from the potentiometers on arduino were used instead. 

// IMA NYU Shanghai
// Interaction Lab
// For receiving multiple values from Arduino to Processing

/*
 * Based on the readStringUntil() example by Tom Igoe
 * https://processing.org/reference/libraries/serial/Serial_readStringUntil_.html
 */

import processing.serial.*;

String myString = null;
Serial myPort;


int NUM_OF_VALUES = 2;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/
int previousSensorValues1;
int previousSensorValues2;


void setup() {
  size(1023, 1023);
  background(200);
  setupSerial();
}


void draw() {
  updateSerial();
  printArray(sensorValues);

  stroke(100);
  strokeWeight(4);
  line(previousSensorValues1,previousSensorValues2,sensorValues[0],sensorValues[1]);
  previousSensorValues1 = sensorValues[0];
  previousSensorValues2 = sensorValues[1];
  
   if (keyPressed) {
    if (key == ENTER) {
      background(200);
    }
   }
}

void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[7], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = '\n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

Musical Instrument

The coding for the second task was a little more straightforward. I started with the arduino coding and creating the circuit. Using the serial communication template, I created the code so that if the value of an array sent from processing was 1, then a tone would play from the pitches.h file, and if the the value was 0, then nothing would play. From processing, if the mouse were clicked, then a value of 1 would be sent.

// IMA NYU Shanghai
// Interaction Lab


/**
  This example is to send multiple values from Processing to Arduino.
  You can find the Processing example file in the same folder which works with this Arduino file.
  Please note that the echo case (when char c is 'e' in the getSerialData function below)
  checks if Arduino is receiving the correct bytes from the Processing sketch
  by sending the values array back to the Processing sketch.
 **/

#define NUM_OF_VALUES 2    /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
#include "pitches.h"

/** DO NOT REMOVE THESE **/
int tempValue = 0;
int valueIndex = 0;

/* This is the array of values storing the data from Processing. */
int values[NUM_OF_VALUES];


void setup() {
  Serial.begin(9600);
  pinMode(10, OUTPUT);
}

void loop() {
  getSerialData();

  // add your code here
  // use elements in the values array
  // values[0]
  // values[1]
if (values[0] == 1) {
  tone(10, values[1]);
}
else {
  noTone(10);
}

}


//recieve serial data from Processing
void getSerialData() {
  if (Serial.available()) {
    char c = Serial.read();
    //switch - case checks the value of the variable in the switch function
    //in this case, the char c, then runs one of the cases that fit the value of the variable
    //for more information, visit the reference page: https://www.arduino.cc/en/Reference/SwitchCase
    switch (c) {
      //if the char c from Processing is a number between 0 and 9
      case '0'...'9':
        //save the value of char c to tempValue
        //but simultaneously rearrange the existing values saved in tempValue
        //for the digits received through char c to remain coherent
        //if this does not make sense and would like to know more, send an email to me!
        tempValue = tempValue * 10 + c - '0';
        break;
      //if the char c from Processing is a comma
      //indicating that the following values of char c is for the next element in the values array
      case ',':
        values[valueIndex] = tempValue;
        //reset tempValue value
        tempValue = 0;
        //increment valuesIndex by 1
        valueIndex++;
        break;
      //if the char c from Processing is character 'n'
      //which signals that it is the end of data
      case 'n':
        //save the tempValue
        //this will b the last element in the values array
        values[valueIndex] = tempValue;
        //reset tempValue and valueIndex values
        //to clear out the values array for the next round of readings from Processing
        tempValue = 0;
        valueIndex = 0;
        break;
      //if the char c from Processing is character 'e'
      //it is signalling for the Arduino to send Processing the elements saved in the values array
      //this case is triggered and processed by the echoSerialData function in the Processing sketch
      case 'e': // to echo
        for (int i = 0; i < NUM_OF_VALUES; i++) {
          Serial.print(values[i]);
          if (i < NUM_OF_VALUES - 1) {
            Serial.print(',');
          }
          else {
            Serial.println();
          }
        }
        break;
    }
  }
}

I then went to make the processing code. I made it so that if the mouse was pressed, the x value of the mouse would be recorded and sent to arduino in order to play a pitch corresponding to that x value.

// IMA NYU Shanghai
// Interaction Lab


/**
 * This example is to send multiple values from Processing to Arduino.
 * You can find the arduino example file in the same folder which works with this Processing file.
 * Please note that the echoSerialData function asks Arduino to send the data saved in the values array
 * to check if it is receiving the correct bytes.
 **/


import processing.serial.*;

int NUM_OF_VALUES = 2;  /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/


Serial myPort;
String myString;

// This is the array of values you might want to send to Arduino.
int values[] = new int[NUM_OF_VALUES];

void setup() {
  size(500, 500);
  background(0);

  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[7], 9600);
  // check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index of the port

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;
}


void draw() {
  background(0);

if (mousePressed) {
  values[0] = 1;
}
else {
  values[0] = 0;
  }
  
  values [1] = mouseX;


  // sends the values to Arduino.
  sendSerialData();

  // This causess the communication to become slow and unstable.
  // You might want to comment this out when everything is ready.
  // The parameter 200 is the frequency of echoing. 
  // The higher this number, the slower the program will be
  // but the higher this number, the more stable it will be.
  echoSerialData(200);
}

void sendSerialData() {
  String data = "";
  for (int i=0; i<values.length; i++) {
    data += values[i];
    //if i is less than the index number of the last element in the values array
    if (i < values.length-1) {
      data += ","; // add splitter character "," between each values element
    } 
    //if it is the last element in the values array
    else {
      data += "n"; // add the end of data character "n"
    }
  }
  //write to Arduino
  myPort.write(data);
}


void echoSerialData(int frequency) {
  //write character 'e' at the given frequency
  //to request Arduino to send back the values array
  if (frameCount % frequency == 0) myPort.write('e');

  String incomingBytes = "";
  while (myPort.available() > 0) {
    //add on all the characters received from the Arduino to the incomingBytes string
    incomingBytes += char(myPort.read());
  }
  //print what Arduino sent back to Processing
  print( incomingBytes );
}

From my previous definition of interaction, all my projects focus on encouraging people to do something– encouraging people to engage in activities or to better themselves.

“Dance Visualizer”

The issue I try to tackle in this project is the lack of physical activity in some people. Rather than trying to encourage exercise in the traditional sense (going to the gym, playing sports, etc), I try to encourage it through something that everyone can do in the comfort of their own homes– dancing. Although everyone can dance or has tried to dance, it can often be frustrating if one cannot get the moves right. This can create a negative feedback loop and discourage people from continuing to dance. One way I thought to tackle this problem would be to create an immediate visual aid as one dances. 

The visuals would come from processing, and the sensing of the dancer’s movement would come from arduino. The way this project would work is that pressure sensors would be placed at the bottom of the dancers feet, inside a shoe, and when a dancer would step and execute a dance move, a visual representation would be displayed through processing. Because of this, even if the dancer were to make a wrong move, a nice visual would still be displayed, disregarding the fact that the dancer did a wrong move.

The way in which the visuals would work would rely on different combinations of steps from both the left and right feet of the dancer. For example, if the dancer were to step once on their right foot, a bouncing ball would show up, or if the dancer were to step once on the right and twice on the left, lines would move across the screen. Because of the multiple combinations of foot movement, the dancer could freestyle their dances and yield many different visualizations. This allows the dancer to get new feedback from the visualizer every time they dance, making dancing a new and refreshing experience every time.

“Veggie-table”

The issue I try to tackle in this project comes from people’s diets, and the lack of vegetables in some people’s daily food intake. The target audience would be people roaming around in grocery stores, buying their groceries. As some people walk by the vegetables section, they would not bat an eye, but my project would aim to draw people into the vegetable section, and encourage them to buy more vegetables.

The way that my project would work requires simple sensors from arduino, and visuals from processing. A person would walk through the vegetable isle and see a screen with a bunch of different vegetables laid in front of it. The screen would say “Pick one up and see what happens” and a sensor would sense when the if a certain vegetable is picked up. On the screen would display the vegetable’s nutrition facts, as will as a bar that would be filled up as the customer would pick up more and more vegetables. If the customer picks up a sufficient amount of vegetables, the screen would say “Congratulations! You now have a balanced diet!”

The visualizations can be customized, but the general idea is to inform the customer of the nutritional benefits of each of the vegetables they pick up, and then encourage the customer to pick up more vegetables. The screen could also show different recipes that the vegetable could be used in.

“See, Touch, Piano”

In this project, I try to encourage people to play the piano through visual representations of music. People can often be discouraged from playing the piano due to how they perceive the difficulty of starting to play, which makes them not want to start playing in the first place. My project would be placed in public places with a piano where anyone can walk up and start playing.

The way in which my project works is similar to that of my first idea “Dance Visualizer”. Touch sensors would be placed in the keys of a piano and detect when the key is pressed. Processing would then take these different combinations of notes and turn them into visuals in a screen. Different combinations of 1, 2, 3, or more notes would turn into moving visuals on the screen. In this way, the person playing the piano would not only get positive feedback from the harmony in the notes they play, but also nice visual feedback showing the harmonies. Because of this double positive feedback loop, the person would be more encouraged to start playing the piano because of this enjoyable experience.