Recitation 10: Media Controller by Sharon Xu

Video of My Work

Code

ARDUINO

void setup() {
Serial.begin(9600);
}

void loop() {
int ad1 = analogRead(A0);
int ad2 = analogRead(A1);
int ad3 = analogRead(A2);

Serial.print(ad1);
Serial.print(“,”);
Serial.print(ad2);
Serial.print(“,”);
Serial.print(ad3);
Serial.println();
delay(100);
}

PROCESSING

PImage photo1;

import processing.serial.*;

String myString = null;
Serial myPort;

int NUM_OF_VALUES = 3;
int[] sensorValues;

void setup() {
size(600, 400);
photo1 = loadImage(“hokusai.jpg”);
setupSerial();
image(photo1, 0, 0);
}

void draw() {
updateSerial();
printArray(sensorValues);
int ad1 = sensorValues[0];
int ad2 = sensorValues[1];
int ad3 = sensorValues[2];
float x= map(ad1,0,1023,0,255);
float y= map(ad2,0,1023,0,255);
float z= map(ad3,0,1023,0,255);
tint(x,y,z);
image(photo1,0,0);
}

void setupSerial() {
printArray(Serial.list());
myPort = new Serial(this, Serial.list()[ 28], 9600);

myPort.clear();

myString = myPort.readStringUntil( 10 );
myString = null;

sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
while (myPort.available() > 0) {
myString = myPort.readStringUntil( 10 );
if (myString != null) {
String[] serialInArray = split(trim(myString), “,”);
if (serialInArray.length == NUM_OF_VALUES) {
for (int i=0; i<serialInArray.length; i++) {
sensorValues[i] = int(serialInArray[i]);
}
}
}
}
}

Reflection

My work is to use three potentiometers to respectively control the R,G,B of the famous painting called The Great Wave created by Hokusai. For the Arduino part, I used what I learned in class to set three variable value parameters from three potentiometers. The Arduino part went smoothly. For the Processing part, I met some problems. First, I mistakenly put the code photo1 = loadImage(ā€œhokusai.jpg”); to void draw() part so that every time I run the code, the picture didn’t show up. After I fixed this problem, I found that modifying the potentiometers doesnā€™t work for changing R,G,B of the picture. I read the code in Processing again and again. Finally, I found that I didnā€™t type the code image(photo1, 0, 0); in void setup() part. Then I modified it. It worked!

After reading Computer Vision for Artist and Designers, I feel like the author mainly focuses on the ability of computer vision to track human activity. According to the argument that ā€œTechniques exist which can create realĀ­time reports about people’s identities, locations, gestural movements, facial expressions, gait characteristics, gaze directions, and other characteristics,ā€ in my opinion, my work does not fully conform to the technology of computer vision, because it only changes the nature of existing pictures through user input. It has no such technology for tracking people’s movements. The article inspires me that in the process of creating interactive artworks, we can add more computer functions related to tracking people’s activities to realize the innovation of traditional and modern art.

Recitation 9: Final Project Process by Sharon Xu

Group Mates: Karen Zhang, Celine Yu, Alice Frank

Karen Zhang:

The Project

Karenā€™s project is called ā€œPickmon,ā€ where users will be given different types of Pokemon (4 in total) based on their heartbeat as well as their answers to the questions. It is inspired by the sorting hat from Harry Potter and is interactive because the project involves input, processing, and output. The output given to the potential users will be presented through-3D printed boxes, inside which there would be a cute Pokemon. I find the Pokemon would be attractive to many users because Pokemon is the memory of childhood for many people. I find the project interesting because the type of Pokemon the user gets is unknown and personalized. But for me, a person who is not familiar with Pokemon, the project is not as much as attractive.

The Feedback

As for feedback, we came up with many ideas that we thought would help Karenā€™s project be more interesting and interactive. I suggest that the questions designed be Karen not too personal, but interesting, such as some quiz or NYUSH related questions. I found the most interesting idea given by Celine is that after the two players go through different ā€˜testsā€™ that would determine their Pokemon type for them, the two Pokemon the users were given would then ā€˜battleā€™ with each other. This idea would make the project more competitive and interactive. The users need to move quickly or use other methods to raise their heartbeat to win the game and the increasing heartbeat would make them excited. By realizing this idea, I think Karenā€™s project would be more attractive even to those who are not familiar with Pokemon and make users feel motivated to start a new round of input, which meets my definition of interaction.

Celine Yu:

The Project

Celineā€™s project is called ā€œBreak Freeā€, which is aimed to help people deal with stress and overcome their failures of fear. As the escape room, Celine’s project would use Processing to create a virtual scene. Users try to escape the room by interacting with real objects or cracking some clues. The goal of this project is to enable users to feel the pressure released by breaking the sense of imprisonment. I found the project interesting because itā€™s interactive and can create a sense of virtual reality. However, I think this project needs more thinking and improvement in terms of helping users release pressure because I didnā€™t feel itā€™s exactly a method for people to release pressure.

The Feedback

The advice my group members and I all had for Celineā€™s project was not to make it too difficult, because then users would feel the pressure to escape from the scene, rather than release the pressure. So the game can be divided into different difficulty levels, allowing users to choose according to their personal situation. I also suggested Celine not to make the interface too scary. Instead, try to make it fun and light, even with some interesting Easter eggs. In addition, the outcome of the game requires more design. We suggested designing some reward mechanism to make users feel encouraged once they finish the game, for example, showing the number of people who have escaped the room to tell the users that they are doing well, so as to make them feel confident about themselves. Karen suggested using a joystick instead of potentiometers, which are sometimes not so easy to control.

Alison Frank:

The Project

The name of Alisonā€™s project has not been decided, but it is basically a project based on a storyline.  Alison is in charge of the code, her partner is in charge of the storyline. By making different choices, users can get different storylines, which I thought was like my favorite game Detroit: Become Human. The focus of the interaction will be achieved by using a distance sensor, which will record the detected movement of the user’s hand. I feel the project inactive because users can become the protagonist of the story, leading the clean development. But I think it would be challenging for Alison and her partner to design a large number of storylines based on usersā€™ choices.

The Feedback

Based on what I thought about Alisonā€™s project, I suggested that she simplify the plot as much as possible by, for example, adding a few plot lines that can overlap. Alison’s storyline isn’t finished yet, so we can’t suggest anything more about the storyline. But as a plot-based project, the project requires higher quality sound effects and pictures, which requires Alison to give more consideration. Also, audio copyright issues also need to be considered.

My Own Project: Feedback

My final project is based on the traditional mask-changing performance from Sichuan Opera. OpenCV could help me recognize faces and add mask stickers. Users can change the mask by pushing the button connected through Arduino. I think the feedback I received is quite useful.

1.ā€œTo have text alongside each mask to provide additional information to the user about its cultureā€

I think this suggestion is useful because it meets my aim to design the project. I want users to learn more about the traditional mask-changing performance from Sichuan Opera by designing the project, so more background knowledge is needed to provide to users. I plan to add a small clip of video about the mask -changing performance at the beginning and more introduction information for each mask in a folder which can be opened by users.

2.ā€œ The point of interaction be changed and much more intuitive and creative by placing the sensor next to the cameraā€

This suggestion helped me a lot. Changing an input from a button to a distance sensor meets the usersā€™ performing process, which means users can alter their faces with performative gestures instead of mechanically pressing buttons. I plan to install a distance sensor next to the camera to detect changes in distance, triggering the computer’s output.

3.  ā€œTo implement different cultures into the project, we also suggested and asked if she could insert different forms of interactionā€

I understand this suggestion because cultural diversity is a very important factor. But at this stage, I want to focus on the mask-changing culture first and provide more information about face changing masks. After the initial completion of the project, based on the feedback from users, I will consider adding masks of other cultures to expand the cultural diversity.

Recitation 8: Serial Communication by Sharon

Exercise 1: Make a Processing Etch A Sketch (From A to P)

Schematic

Video

Code

PROCESSING// IMA NYU Shanghai

// Interaction Lab

// For receiving multiple values from Arduino to Processing

/*

* Based on the readStringUntil() example by Tom Igoe

* https://processing.org/reference/libraries/serial/Serial_readStringUntil_.html

*/

import processing.serial.*;

String myString = null;

Serial myPort;

int NUM_OF_VALUES = 2;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/

int[] sensorValues;      /** this array stores values from Arduino **/

int posX;

int posY;

void setup() {

  size(500, 500);

  background(0);

  setupSerial();

}

void draw() {

  updateSerial();

  printArray(sensorValues);

  // use the values like this!

  // sensorValues[0]

  // add your code

   posX=sensorValues[0];

   posY=sensorValues[1];

  posX=int(map(sensorValues[0],0,1023,0,500));

  posY=int(map(sensorValues[1],0,1023,0,500));

ellipse(sensorValues[0], sensorValues[1], 30, 30);

  //

}

void setupSerial() {

  printArray(Serial.list());

  myPort = new Serial(this, Serial.list()[4], 9600);

  // WARNING!

  // You will definitely get an error here.

  // Change the PORT_INDEX to 0 and try running it again.

  // And then, check the list of the ports,

  // find the port “/dev/cu.usbmodem—-” or “/dev/tty.usbmodem—-”

  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();

  // Throw out the first reading,

  // in case we started reading in the middle of a string from the sender.

  myString = myPort.readStringUntil( 10 );  // 10 = ‘\n’  Linefeed in ASCII

  myString = null;

  sensorValues = new int[NUM_OF_VALUES];

}

void updateSerial() {

  while (myPort.available() > 0) {

    myString = myPort.readStringUntil( 10 ); // 10 = ‘\n’  Linefeed in ASCII

    if (myString != null) {

      String[] serialInArray = split(trim(myString), “,”);

      if (serialInArray.length == NUM_OF_VALUES) {

        for (int i=0; i<serialInArray.length; i++) {

          sensorValues[i] = int(serialInArray[i]);

        }

      }

    }

  }

}

ARDUINO

// IMA NYU Shanghai

// Interaction Lab

// For sending multiple values from Arduino to Processing

void setup() {

  Serial.begin(9600);

}

void loop() {

  int sensor1 = analogRead(A0);

  int sensor2 = analogRead(A1);

  

  // keep this format

  Serial.print(sensor1);

  Serial.print(“,”);  // put comma between sensor values

  Serial.print(sensor2);

  Serial.println();

  

  // too fast communication might cause some latency in Processing

  // this delay resolves the issue.

  delay(100);

}

Reflection&Process

The interaction of this circuit is that the user controls the position of the small circle on the computer screen by controlling the two potentiometers. The input is the two potentiometers and the output is the position of the ellipse. To make this circuit, I first connected the potentiometer part on the breadboard and then modified part of the code according to the circuit based on the existing sample code, such as the number of values, the number of the port and so on. Finally, the code runs successfully.

Exercise 2: Make a musical instrument with Arduino (From P to A)

Schematic

Video

Code

PROCESSING

// IMA NYU Shanghai

// Interaction Lab

/**

* This example is to send multiple values from Processing to Arduino.

* You can find the arduino example file in the same folder which works with this Processing file.

* Please note that the echoSerialData function asks Arduino to send the data saved in the values array

* to check if it is receiving the correct bytes.

**/

import processing.serial.*;

int NUM_OF_VALUES = 2;  /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/

Serial myPort;

String myString;

// This is the array of values you might want to send to Arduino.

int values[] = new int[NUM_OF_VALUES];

void setup() {

  size(500, 500);

  background(0);

  printArray(Serial.list());

  myPort = new Serial(this, Serial.list()[ 4], 9600);

  // check the list of the ports,

  // find the port “/dev/cu.usbmodem—-” or “/dev/tty.usbmodem—-”

  // and replace PORT_INDEX above with the index of the port

  myPort.clear();

  // Throw out the first reading,

  // in case we started reading in the middle of a string from the sender.

  myString = myPort.readStringUntil( 10 );  // 10 = ‘\n’  Linefeed in ASCII

  myString = null;

}

void draw() {

  background(0);

  // changes the values

  //for (int i=0; i<values.length; i++) {

  //  values[i] = i;  /** Feel free to change this!! **/

  //}

  values[0]=mouseX;

  values[1]=mouseY;

  // sends the values to Arduino.

  sendSerialData();

  // This causess the communication to become slow and unstable.

  // You might want to comment this out when everything is ready.

  // The parameter 200 is the frequency of echoing.

  // The higher this number, the slower the program will be

  // but the higher this number, the more stable it will be.

  echoSerialData(200);

}

void sendSerialData() {

  String data = “”;

  for (int i=0; i<values.length; i++) {

    data += values[i];

    //if i is less than the index number of the last element in the values array

    if (i < values.length-1) {

      data += “,”; // add splitter character “,” between each values element

    }

    //if it is the last element in the values array

    else {

      data += “n”; // add the end of data character “n”

    }

  }

  //write to Arduino

  myPort.write(data);

}

void echoSerialData(int frequency) {

  //write character ‘e’ at the given frequency

  //to request Arduino to send back the values array

  if (frameCount % frequency == 0) myPort.write(‘e’);

  String incomingBytes = “”;

  while (myPort.available() > 0) {

    //add on all the characters received from the Arduino to the incomingBytes string

    incomingBytes += char(myPort.read());

  }

  //print what Arduino sent back to Processing

  print( incomingBytes );

}

ARDUINO

// IMA NYU Shanghai

// Interaction Lab

/**

  This example is to send multiple values from Processing to Arduino.

  You can find the Processing example file in the same folder which works with this Arduino file.

  Please note that the echo case (when char c is ‘e’ in the getSerialData function below)

  checks if Arduino is receiving the correct bytes from the Processing sketch

  by sending the values array back to the Processing sketch.

**/

#define NUM_OF_VALUES 2    /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/

/** DO NOT REMOVE THESE **/

int tempValue = 0;

int valueIndex = 0;

/* This is the array of values storing the data from Processing. */

int values[NUM_OF_VALUES];

void setup() {

  Serial.begin(9600);

}

void loop() {

  getSerialData();

  // add your code here

  // use elements in the values array

  // values[0]

  // values[1]

  int freq = map(values[0],0,500,31,1000);

  int dura = map(values[1],0,500,31,1000);

   tone(11, freq, dura);

//recieve serial data from Processing

void getSerialData() {

  if (Serial.available()) {

    char c = Serial.read();

    //switch – case checks the value of the variable in the switch function

    //in this case, the char c, then runs one of the cases that fit the value of the variable

    //for more information, visit the reference page: https://www.arduino.cc/en/Reference/SwitchCase

    switch (c) {

      //if the char c from Processing is a number between 0 and 9

      case ‘0’…’9′:

        //save the value of char c to tempValue

        //but simultaneously rearrange the existing values saved in tempValue

        //for the digits received through char c to remain coherent

        //if this does not make sense and would like to know more, send an email to me!

        tempValue = tempValue * 10 + c – ‘0’;

        break;

      //if the char c from Processing is a comma

      //indicating that the following values of char c is for the next element in the values array

      case ‘,’:

        values[valueIndex] = tempValue;

        //reset tempValue value

        tempValue = 0;

        //increment valuesIndex by 1

        valueIndex++;

        break;

      //if the char c from Processing is character ‘n’

      //which signals that it is the end of data

      case ‘n’:

        //save the tempValue

        //this will b the last element in the values array

        values[valueIndex] = tempValue;

        //reset tempValue and valueIndex values

        //to clear out the values array for the next round of readings from Processing

        tempValue = 0;

        valueIndex = 0;

        break;

      //if the char c from Processing is character ‘e’

      //it is signalling for the Arduino to send Processing the elements saved in the values array

      //this case is triggered and processed by the echoSerialData function in the Processing sketch

      case ‘e’: // to echo

        for (int i = 0; i < NUM_OF_VALUES; i++) {

          Serial.print(values[i]);

          if (i < NUM_OF_VALUES – 1) {

            Serial.print(‘,’);

          }

          else {

            Serial.println();

          }

        }

        break;

    }

  }

}

Reflection&Process

The interaction of this circuit is that the user can change the tone and frequency of a buzzer by controlling the position of the mouse. The input is the mouseā€™s x and y positions and the output is the tone and frequency of a buzzer. To make this circuit, I first connected the buzzer on the breadboard and then modified part of the code according to the circuit based on the existing sample code, such as using tone() function and so on. Finally, the code runs successfully.

Final Project Essay by Sharon

A. Project Title: A unique mask-changing performance
B. Project Statement of Purpose :

In my previous research, according to Crawfordā€™s article The Art of Interactive Design, I defined interaction as a cyclic process where two actors alternately receive and respond to the message, which in computer terms are called: input, processing, and output. Looking back on my previous research, I have a new understanding of interactivity: people should learn something meaningful while interacting with interactive devices. In the final research paper, I studied the anti-drawing machine. I think this project can not only enable users to create art but also reflect on the relationship between human and machine, to have a deeper understanding of this Information Age. At the same time, in the media and daily life, I find that people tend to ignore their traditional culture. As a Chinese, I am very interested in the art of Sichuan dramatic face change, but I find that many new generations of young people in China slowly forget this art. I intend to design an interactive device to attract people, especially the new generation of young people, to value and understand the traditional art of face changing. I want to use processing for face recognition, and add the traditional face mask to the face that appears in the picture. Through Arduino’s manipulation, I hope to achieve changes between different masks. With the user’s performance, this project could create a scene that the user is giving a mask-changing performance on the stage. I hope that through this project people, especially teenagers can find the pleasure and charm of changing faces through virtual performances.

C. Project Plan *
For my project, I plan to use openCV to realize facial recognition and add masks to human faces by programming. I plan to make as many mask models as possible to increase the diversity of change. Also, I will add appropriate music and provide face-changing costumes to create an atmosphere where users are performing. To give users a sense of artistic creation, the project will provide a video recording function. After using interactive devices, users can save their own performance videos and have an interesting and meaningful memory. For the Arduino section, I plan to use keys to manipulate face changes. Users can play their own performances freely according to music or other factors.
In the next three weeks, Iā€™m going to figure out these things:

In the next three weeks, we are going to figure out these things:

Week 1: Visual Coding

Week 2: Start working on connecting the buttons from Arduino to Processing

Week 3: User testing

D. Context and Significance *

In my group research paper, I defined interaction as a cyclic process where two actors alternately receive and respond to the message. In my final research paper, I put forward two new ideas: Firstly, interaction should be people-oriented. Secondly, interactive works should make people feel motivated to input next time. With the progress of learning, I have a new insight into the interactivity of final project: people should learn something meaningful from interactive works, and interactive works should have more profound significance. In the anti-drawing machine project mentioned above, besides interaction, users can feel the freedom of artistic creation and have a sense of reflection on the relationship between human and machine. The project ā€œA unique mask-changing performanceā€ is not only artistic but also can be used as a media to promote traditional culture.

Preparatory Research and Analysis by Sharon Xu

    In my previous research, according to Crawfordā€™s article The Art of Interactive Design, I defined interaction as a cyclic process where two actors alternately receive and respond to the message, which in computer terms are called: input, processing, and output. As the learning process goes on, I have a new understanding of the concept of interaction. First, one of the two actors in the interaction process should be human because the concept of interaction is human-centered. From the midterm projectā€™s user testing, I learned that the goal of our product design is to achieve a better user experience, so the concept of interaction must include a human role. Second, the output of the machine should not be limited to physical output to the user, but should also include emotional stimulation to facilitate the user’s next input.

    The first interactive project I researched is called Anti-Drawing Machine. The machine allows either a disrupted or collaborative drawing by manipulating the rotation and position of ordinary paper material. In the video of user testing, I found that users are interested in the response of the anti-drawing machine, which constantly stimulates their emotions and promotes their actions to compete with the machine. The output of machine stimulates usersā€™ new rounds of input, which lines up with my definition that interaction is a cyclic process which includes input, processing, and output. Also, we could see thereā€™re many pictures made by users on the back wall. The reader’s input and the machine’s output can produce visual images, which makes the project experience more memorable to users. This idea inspires my final project. Maybe I could I visualize the user’s input path, and gave certain artistic output, such as music, painting, etc. Also, creating a sense of competition for users makes interactive products more interesting.

 

WEBSITE: https://www.creativeapplications.net/arduino-2/anti-drawing-machine-whimsical-and-imperfectly-characteristic-collaborator/

   Another project I want to analyze is Open Data Cam. This project is a great work. It is a tool that helps to quantify the world. With computer vision ā€˜Open Data Camā€™ understands and quantifies what it sees. For example, in the video, the user can use this Open Data Cam to calculate the types and numbers of vehicles passing a certain street in a given period of time. Open Data Cam is a useful data collecting product, but it seems not applied to my definition of interaction. First, its input is not directly controlled by the user, but caused by the movement of the vehicle. So it doesn’t fit in with my definition that one side of my interaction should be human users. Second, used as a practical tool, this product does not have an interactive process in which output and input stimulate each other. Therefore, this project does not line up with my definition of interaction.

WEBSITE: https://www.creativeapplications.net/news/open-data-cam/

    After more than half a semester of study, I have a new understanding and definition of interaction: It is a cyclic process where users and another actor (usually digital devices) alternately input, process, and output. During the interaction, output and input stimulate each other, which leads to a circular interaction. According to the article Designing Interaction for Chatbot-Based Conversational Commerce with User-Centered Design, the interactive design product should fulfill the usability goal and the user experience goal, which means interaction is a user-centered process. Therefore, in the future learning process, I would pay more attention to the user experience in the interactive process, and strive to enhance the user’s input interest, which could stimulate a new round of output.