Month: December 2020

Stringless

—- by Alan Guo & Rainee Wang, Instructor: Andy Garica

 
 video edited by Rainee

 

CONCEPTION AND DESIGN

The basic concept of this project, Stringless, intends to create a Pipa without actual strings, when users touch the invisible laser strings, different pieces of music and patterns would be triggered. Combining the traditional Chinese culture with the modern technology nowadays, this project raises people awareness of Chinese traditional heritage and brings back the culture origins in our country.

The origin concept and idea of this project is raised by Rainee, my teammate (thanks Rainee so much for her marvelous great idea). She came up the idea from on exhibition in one Japan Museum, traditional Chinese Pipa from Tang dynesty are now perceived in Japan. The wars and a series of revolution happened in the latest several hundred years in China causes many precious traditional Chinese cultural relics to be kept away from their homeland and perceived in other countries all over the world. Thus, Rainee raised the idea that we could bring these ancient Pipa back to us with modern technology, and here comes our final project.

At the stage of designing, we thought about different artifacts and different ways to interact. We raised the idea to use a real Pipa, and while users are playing with the instrument, the sound would be recorded. Different patterns and images will then show up according to the pitch and rhythm the user is playing. However, we didn’t finally adapt this idea for most of our users are beginners to this specific instrument, so beautiful melody can’t be guaranteed and users may even don’t know how to play Pipa at all.

We also thought about how to present the project at first, we thought about different ways to show image. Putting the Pipa on the ground, suspended, horizontally or put it straight. We finally adapted the idea of letting the Pipa lean against a wall, which is the best way to present the image projected on it as well as to give users a better view of the whole piece.

Another part we struggled with was the interaction. We found the interaction of triggering music might be too simple, so we tried to add on more to the interaction. As inspired by Andy, after discussion with him, we decided that different strings can be triggered at the same time, so there exists a lot of combinations for four strings to play with. We also add more sets of music and patterns, at total we have three different sets of music and image to show.

Thus, we decided the plan of making a Pipa ourselves, and using laser strings to represent the real strings on Pipa for people to play with. Projectors would show different patterns to our instrument when users touch the laser strings.

FABRICATION AND PRODUCTION

For the producing procedure, there are mainly several parts as building the circuits, music projecting the images, laser cutting the Pipa and the coding process.

Building the Circuits:

During the whole process, we first built a prototype using cardboards, to try what is feasible and what is not. We cut the cardboard to roughly the shape of a Pipa and built part of the circuits on it to test. No doubt, there must be something happened beyond our expectation.

The first difficulty we met was that we used laser strings to represent the real string. We thought that we could see the red light of the laser, however, the lasers harps aren’t powerful enough to show the whole string, we can only see the dot it’s pointing. Only with smokes the string is visible. We tried to use the dry ice to generate smoke, but it didn’t work out so well.

The next hard part was to position the light dependent resistors  with laser harps. Andy encouraged us to use light dependent resistors to detect the laser strings. But it’s really really a hard job to put the resistors and laser harps exactly in places so that the small dot of laser can be located on the resistor.

After simple testing that both the laser harps and the resistors can work with each other, we decided to laser cut the Pipa out and then build the whole circuits on it. We cut with the shape of one ancient Chinese Pipa, along with the shape we put some traditional Chinese pattern on the wood, making it more like a traditional Chinese instrument.

After having the final vision for the Pipa, we started to built the circuits on it. For laser harps, we hot glued them on the board, and we cut small wood pieces to lift them up as well as making them more stable.

For the lightness resistors, as helped by Andy, we positioned them on a piece of wood by making four holes on the wood and insert the resistors on them. As the holes are exactly the position of the laser dots, each pair of resistor and laser harp finally matched perfected.

The final part of the circuits are the buttons, as we laser cut the Pipa with holes with the size of the buttons. It’s simple and easy to just put the buttons on.

Last but not least, the Arduino board along with the breadboard are positioned at the back of the instrument. We hide all the wires to the back, connecting the the Arduino board.

Music and Projecting the Images:

Rainee generated three sets of music from one famous piece of Pipa music “春江花月夜”, the three different sets of patterns are from real ancient Chinese Pipa. When playing different strings, different parts of the pattern would be projected to the surface of the instrument.

Coding Process:

The coding part of this project involves Arduino as input and Processing as the sound and visual outputs.

The Arduino reads the four lightness resistors as well as the three buttons. With the lasers, resistors would have a high number exceeding 800, and only about 500 without the laser. So the code reads the number of resistor and convert them into 0 and 1. 0 indicating no lights and 1 means the laser is pointing the resistor.

 int sensor1 = analogRead(A0);

  if (sensor1 > 800){
    values1 = 1;
  }else {
    values1 = 0;
  }

For the three buttons, we name their value as 1, 2, 3. Once a button is pressed, the button value would change correspondingly. Arduino detects whether a button is pressed and if its current state is not the previous one, meaning that the Arduino only detects the change of button from 0 to 1, the moment when the button is pressed. And the initial button is set to be 1, so the first set of music and pattern would be shown is default, only when pressing the second or the third button would the music and image alter.

  if (button1 == 1 && pb1 == 0) {
    values5 = 1;
  }
  else if (button2 == 1 && pb2 == 0){
    values5 = 2;
  }
  else if (button3 == 1 && pb3 == 0){
    values5 = 3;
  }

Thus, the four values of resistors and the one button value is detected by the Arduino and sent to Processing.

For the Processing part, the program detects whether the invisible string is triggered or not, once triggered, it would play a short piece of music and show one fragment of pattern on the screen.After the music stops, the image would gradually fade away on the screen and finally disappear. The code here detects whether the certain music is playing or not, if not, the image would become more transparent every loop until it disappears.

  //sound 0
  if (sensorValues[0] == 0 && p0 == 1 && sound0.isPlaying() == false){
    sound0.play();
    i0 = 0;
  }
  //image 0  
  if (sound0.isPlaying() == true){
    tint(255, i0);
    image(image0, 0, 0);
    i0 = i0 + 15;
  }
  else if (sound0.isPlaying() == false && i0 > 0){
    tint(255, i0);
    image(image0, 0, 0);
    i0 = i0 - 15;
  }

There is another function in Processing that reads the button value, and once is changes, the Processing would load a different set of music and pattern to show.

void button(int x){
  if (x != px){
    if (x == 1){
      sound0 = new SoundFile(this, "set101.wav");
      image0 = loadImage("1Flower.png");
    }
    else if (x == 2){
      sound0 = new SoundFile(this, "set201.wav");
      image0 = loadImage("1rich.png");
    }
    else if (x == 3){
      sound0 = new SoundFile(this, "set301.wav");
      image0 = loadImage("1click.png");
    }
  }
}

User Testing:

When user testing out project, we haven’t finished adding all the sounds and images to the processing, also because we just set up everything from the cardboard to the laser cut Pipa, one of the lightness sensor is not working so well. So we tested the basic functions of our project as well as part of the music images. Through only a prototype, the project received plenty of comments and feedbacks for us to further work on.

Suggestions and feedbacks:

1. Our project has four laser strings but only one piece of music could be placed at a time, and the sound might repeat a lot if the user frequently “touches” the same string. 

2. It’s one of the difficult part to accurately install the laser harps and lightness resistors so that they could match perfectly. Or otherwise the project can’t actually function well. One professor suggest us to glue the laser harp and resistor at the same time using low temperature glue guns so we can adjust them easily.

3. We are using lasers strings representing the real strings on the instrument, so the users can’t really see the lasers. We received one feedback that we don’t actually need to see the strings, since it’s part of our concept that the project represents the heritage currently not reachable.

cDuring the user testing process, we have received a lot of useful and helpful feedbacks. From the way to interact, how to install and build up everything to even further develop the concept and meaning of the whole project. We adapted a lot of the suggestions and feedbacks to improve our final work.

The feedbacks from user testing influenced a lot in our later work, we changed many details according to the suggestion from the users, they also gave us many useful and helpful instructions as building the circuits and enriching the concepts.

CONCLUSIONS:

This project, Stringless, tends to arise people’s awareness of the traditional Chinese culture. By building a Pipa without string, a combination of modern technology and Chinese heritage, this project shed lights on the Chinese history and its meaningful sediments.

We think our project accomplished this goal, as during the user testing and IMA show, a lot of users showed their interests in the pattern and music. By drawing audience’s attention to the ancient pattern and image of Pipa, this project strengthens Chinese history heritage and culture.

If having more time, we would make further improvements to our project from different angles. First, we were planning to cover the back of the Pipa, making it a belly to hold all the wires as well as making it more like a real Pipa. We would also work on to have more and variable interactions with the project more than three sets of music and images. As Andy told us, there is no limit for a project, we can always do something more and make the project even better.

The interaction part of this project has two angles, the first one is that users interact with the four invisible strings and three buttons, the project would react back with different music and pattern showing on the surface. It is the interaction between the user and the computer programming. 

However, the second part is the user interacting with traditional Chinese culture, the heritage facing the danger of loss. When audience play with the instrument, they listen, see and feel the ancient culture of Chinese, these images are calling back users to pay more attention and awareness to the protecting of them. This part of interaction comes along with my definition with “interaction” that the two parts may not be restricted to the user and the programming, the environment or something else can also be part of the interaction. In this project, the third part is the Chinese culture that connects users with the project Stringless.

Taking the project as a whole, the building process involves difficulties and problems, but it is overall a very fun and exciting experience. As there are so many parts in the projects, from generating ideas, narrowing downing thoughts, making minimum valuable progress, building up everything, encountering and solving numerous problems, receiving suggestions and feedbacks, having changes all the time and finally showing the project to all the audience, every single part of the experience worth ten thousand words recording. The most precious thing I get is the experience of walking through all these parts. All these experience would help me so much in future project making and also problem solving process, that everything I met in this project helps me to do the next project better.

Long story short, besides all my experience and joy, this project, Stringless, points at the heritage of traditional Chinese Pipa and its culture. By making this modern combined Pipa, we aim to arise people’s awareness of Chinese traditional culture and call on protection of them. Through the real Pipa is not here with us, our project can only be stringless, we hope that more social awareness could be drawn on the culture heritage.

Maybe on day in the future, our Pipa can have real strings back.

TECHNICAL DOCUMENTATION:

Circuits：

Arduino:

int values1;
int values2;
int values3;
int values4;
int values5 = 1;
int button1 = 0;
int button2 = 0;
int button3 = 0;
int pb1;
int pb2;
int pb3;

void setup() {
  Serial.begin(9600);
}

void loop() {
  pb1 = button1;
  pb2 = button2;
  pb3 = button3;
  
  int sensor1 = analogRead(A0);
  int sensor2 = analogRead(A1);
  int sensor3 = analogRead(A2);
  int sensor4 = analogRead(A3);
  
  button1 = digitalRead(9);
  button2 = digitalRead(10);
  button3 = digitalRead(11);
  
  if (sensor1 > 800){
    values1 = 1;
  }else {
    values1 = 0;
  }
  
  if (sensor2 > 800){
    values2 = 1;
  }else {
    values2 = 0;
  }
  
  if (sensor3 > 800){
    values3 = 1;
  }else {
    values3 = 0;
  }
  
  if (sensor4 > 500){
    values4 = 1;
  }else {
    values4 = 0;
  }

  if (button1 == 1 && pb1 == 0) {
    values5 = 1;
  }
  else if (button2 == 1 && pb2 == 0){
    values5 = 2;
  }
  else if (button3 == 1 && pb3 == 0){
    values5 = 3;
  }

  Serial.print(values1);
  Serial.print(",");
  Serial.print(values2);
  Serial.print(",");
  Serial.print(values3);
  Serial.print(",");
  Serial.print(values4);
  Serial.print(",");
  Serial.print(values5);
  Serial.println();

  delay(100);
}

Processing:

import processing.serial.*;
import processing.sound.*;

SoundFile sound0;
SoundFile sound1;
SoundFile sound2;
SoundFile sound3;

PImage image0;
PImage image1;
PImage image2;
PImage image3;

float x;
float y;

int p0;
int p1;
int p2;
int p3;
int px;

int i0 = 0;
int i1 = 0;
int i2 = 0;
int i3 = 0;

String myString = null;
Serial myPort;
boolean playSound = true;

int NUM_OF_VALUES = 5;
int[] sensorValues;

void setup() {
  size(600, 1000);
  setupSerial();
}


void draw() {
  p0 = sensorValues[0];
  p1 = sensorValues[1];
  p2 = sensorValues[2];
  p3 = sensorValues[3];
  px = sensorValues[4];
  
  updateSerial();
  printArray(sensorValues);
  background(0);
  button(sensorValues[4]);

  //sound 0
  if (sensorValues[0] == 0 && p0 == 1 && sound0.isPlaying() == false){
    sound0.play();
    i0 = 0;
  }
  //sound 1
  if (sensorValues[1] == 0 && p1 == 1 && sound1.isPlaying() == false){
    sound1.play();
    i1 = 0;
  }
  //sound 2
  if (sensorValues[2] == 0 && p2 == 1 && sound2.isPlaying() == false){
    sound2.play();
    i2 = 0;
  }
  //sound 3
  if (sensorValues[3] == 0 && p3 == 1 && sound3.isPlaying() == false){
    sound3.play();
    i3 = 0;
  }
    
  //image 0  
  if (sound0.isPlaying() == true){
    tint(255, i0);
    image(image0, 0, 0);
    i0 = i0 + 15;
  }
  else if (sound0.isPlaying() == false && i0 > 0){
    tint(255, i0);
    image(image0, 0, 0);
    i0 = i0 - 15;
  }
  //image 1
  if (sound1.isPlaying() == true){
    tint(255, i1);
    image(image1, 0, 0);
    i1 = i1 + 15;
  }
  else if (sound1.isPlaying() == false && i1 > 0){
    tint(255, i1);
    image(image1, 0, 0);
    i1 = i1 - 15;
  }
  //image 2
  if (sound2.isPlaying() == true){
    tint(255, i2);
    image(image2, 0, 0);
    i2 = i2 + 15;
  }
  else if (sound2.isPlaying() == false && i2 > 0){
    tint(255, i2);
    image(image2, 0, 0);
    i2 = i2 - 15;
  }
  //image 3
  if (sound3.isPlaying() == true){
    tint(255, i3);
    image(image3, 0, 0);
    i3 = i3 + 15;
  }
  else if (sound3.isPlaying() == false && i3 > 0){
    tint(255, i3);
    image(image3, 0, 0);
    i3 = i3 - 15;
  }
}

void setupSerial() {
  myPort = new Serial(this, Serial.list()[1], 9600);
  myPort.clear();
  myString = myPort.readStringUntil( 10 );
  myString = null;
  sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 );
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

void button(int x){
  if (x != px){
    if (x == 1){
      sound0 = new SoundFile(this, "set101.wav");
      sound1 = new SoundFile(this, "set102.wav");
      sound2 = new SoundFile(this, "set103.wav");
      sound3 = new SoundFile(this, "set104.wav");
      image0 = loadImage("1Flower.png");
      image1 = loadImage("2Flower.png");
      image2 = loadImage("3Flower.png");
      image3 = loadImage("4Flower.png");
    }
    else if (x == 2){
      sound0 = new SoundFile(this, "set201.wav");
      sound1 = new SoundFile(this, "set202.wav");
      sound2 = new SoundFile(this, "set203.wav");
      sound3 = new SoundFile(this, "set204.wav");
      image0 = loadImage("1rich.png");
      image1 = loadImage("2rich.png");
      image2 = loadImage("3rich.png");
      image3 = loadImage("4rich.png");
    }
    else if (x == 3){
      sound0 = new SoundFile(this, "set301.wav");
      sound1 = new SoundFile(this, "set302.wav");
      sound2 = new SoundFile(this, "set303.wav");
      sound3 = new SoundFile(this, "set304.wav");
      image0 = loadImage("1click.png");
      image1 = loadImage("2click.png");
      image2 = loadImage("3click.png");
      image3 = loadImage("4click.png");
    }
  }
}

Thanks^

User Testing For others project

One project I user tested in the recitation is Lane’s project “Embracing Sadness: End 2020”. This project concentrates on the sadness and negative emotions in the tough year 2020. When sitting down on the right position, plastic balls filled with feathers above my head would rotate, representing that the sadness would gradually flow away from us. Also, I could control two balls in front of me, and when rotating them, images on the screen would change. As I constantly rotate the balls in one direction, the sad emoji on the screen would be vague and finally fade away.

I really like the concept of this project as we should embrace our sadness and let go of them to enter the next brand new year. Especially after Lane explained to me her idea, I found the concept meaningful to all of us. Also, I really like the part that balls above me could rotate, it’s really fun to play with. The rotating balls add motion and engage to the project.

For  suggestions and feedbacks:

1. I think the concept of embracing sadness might be a little bit vague when first looking at the project, I don’t really get the idea that the project conveys to me. Maybe the project can represent the sadness in a more clear way.

2. If I stay too long time sitting, the balls would rotate too fast and hit the frame poles. The motors might be adjusted to a lower speed so that the balls could rotate smoothly.

3. The image on the screen contains only a sad face. I think the idea of embracing sadness can be better convinced if the screen shows something more than the fading of sadness emoji. Something like a laughing face shown when the sadness disappears can be better.

User Testing For my project

When user testing out project, we haven’t finished adding all the sounds and images to the processing, also because we just set up everything from the cardboard to the laser cut Pipa, one of the lightness sensor is not working so well. So we tested the basic functions of our project as well as part of the music images. Through only a prototype, the project received plenty of comments and feedbacks for us to further work on.

Suggestions and feedbacks:

1. Our project has four laser strings but only one piece of music could be placed at a time, and the sound might repeat a lot if the user frequently “touches” the same string. I would adjust the program codes so that different strings can be triggered at the same time so that all four strings can be used just like a real instrument. Also, music should be prevented from playing over again before it ends.

2. It’s one of the difficult part to accurately install the laser harps and lightness resistors so that they could match perfectly. Or otherwise the project can’t actually function well. One professor suggest us to glue the laser harp and resistor at the same time using low temperature glue guns so we can adjust them easily.

3. We are using lasers strings representing the real strings on the instrument, so the users can’t really see the lasers. We received one feedback that we don’t actually need to see the strings, since it’s part of our concept that the project represents the heritage currently not reachable.

most and least successful parts:

For the successful parts, users like the pattern and music of the project a lot. The images, pattern and music are part of  ancient Chinese culture, they are also the key points this project wants to convey. Through playing the instrument, the user can really interact with it and receive what the project wants to tell. However, the project we presented was still a prototype with only part of the functions, users can’t have a complete understanding and interaction with it, which are the parts we need to further work on.

During the user testing process, we have received a lot of useful and helpful feedbacks. From the way to interact, how to install and build up everything to even further develop the concept and meaning of the whole project. We would pay much attention to all these feedbacks and continually work on these specific parts to better complete and display our project.

Arduino and Processing Documentations

In this recitation, I tried to make some fun actions with the camera captured image. Inspired by the example codes given, I change the size and positions of each pixel of the camera captured image. These two values are controlled by the potentiometers from Arduino. By rotating the two potentiometers, the face of the user or whatever the camera is capturing would be formed by pixels of different size and position, making an interesting image to interact with.

This work is associated with the computer vision, since the computer camera inputs a image, and it process through different algorithms and output new images.

Codes for Arduino

void setup() {
  Serial.begin(9600);
}w

void loop() {
  int sensor1 = analogRead(A0);
  int sensor2 = analogRead(A1);

  Serial.print(sensor1);
  Serial.print(",");
  Serial.print(sensor2);
  Serial.println();

  delay(100);
}

Codes for Processing

import processing.serial.*;

import processing.video.*; 
String[] cameras = Capture.list();

String myString = null;
Serial myPort;

int NUM_OF_VALUES = 2;
int[] sensorValues; 

Capture cam;


void setup() {
  size(640, 480); 
  printArray(cam.list());
  
  cam = new Capture(this, cam.list()[0]);
  cam.start(); 

  setupSerial();
}

void draw() {
  updateSerial();
  clear();
  
  printArray(sensorValues);
  if (cam.available()) { 
   cam.read(); 
  } 
  if (sensorValues[0] < 10){
    image(cam, 0, 0);
  }else{
  float dotSpacing = map(sensorValues[0], 0, 1023, 1, 20);
  float dotSize = map(sensorValues[1], 0, height, 1, dotSpacing * 1.4);
  for (int x = 0; x < cam.width; x += dotSpacing) {
    for (int y = 0; y < cam.height; y += dotSpacing) {
      color c = cam.get(x, y);
      fill(c);
      noStroke();
      circle(x, y, dotSize);
    }
   }
  }
}

void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[1], 9600);
  myPort.clear();
  myString = myPort.readStringUntil( 10 );
  myString = null;
  sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 );
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}