Final Project

Digital and Physical Convergence

Deziree Harmon 

Christian Grewell

CONCEPTION AND DESIGN:

My project goal was to create a unique interaction between digital portrayals of reality and reality itself. With this in mind, I wanted the user to have a control over the presented display. My initial idea allowed the user to control how the audio and camera inputs sounded and appeared based on the user’s responses. This idea explored the question of how our world appears if it is displayed digitally. However, I ran into major roadblocks in developing this project. For example, I created code that allowed the user to control the speed of a moving ball that bounces across the screen. The speed input would then impact the speed of the camera images. When I combined the two factors, the program became slow and glitchy so that it wasn’t functioning. Other issues included the camera freezing and the arduino not working correctly. While working, I had the idea of changing my idea a bit so that it instead explored the different ways that the computer can understand us. I then started ideating and playing with code.

In the end, I decided to make an artistic portrayal of how the digital world is becoming so intertwined with reality to us. To do this I built a tangible setting representing reality and a digital setting. They interacted with each other but the user also interacts with the digital realm representing the ways that digitized settings are becoming just as interactive and potentially encompassing as real settings. 

FABRICATION AND PRODUCTION:

Processing

I used processing to create a realistic setting that is relatable, a classic living room with a Christmas Tree. The user is able to interact with different parts of the room.

Arduino

Most of my fabrication was done with the goal of making a setting that people can relate to in order to fully portray the way that settings we are familiar with are becoming present digitally. I used LED lights because of their familiarity in such a setting.

Building Process

I used a laser cutter to cut out the necessary shapes for the walls of the room that I built and then used other forms of fabrication (clay models and fabric) to make the setting seem like a home.

Final Working Product

CONCLUSIONS:

The goal of this project was to portray the shift that is occurring to reality being in the digital realm. My project achieved its goal but at a very basic level which could be expanded upon. If I had more time I would create more interactions between processing and arduino and also make the coded setting look more realistic.

MASSIVE LESSON LEARNED : SLOW DOWN

While working on my initial idea I broke down the parts of the project and tried to compile them too quickly so that it was much more difficult to understand what was going wrong. When my project idea shifted I put an emphasis on doing one step at a time, making sure things are working at a very basic level and then build upon them. This would have saved me a lot of trouble and time with the first idea.

It is important to reflect on the way that technology is influencing our realities. This project is a base for potential thorough reflection. The settings we see every day can be easily recreated digitally. This allows us to find connection, comfort, and interaction in a digital setting. It is fun and reminiscent and the dangers of such a shift are usually ignored because of that.

Code

import processing.serial.*;
//Song used : The Christmas Song - Nat King Cole (sax cover Graziatto)
//Youtube link https://www.youtube.com/watch?v=4b6MY9LmGG0
import processing.sound.*;
import processing.video.*;
Movie myMovie;
boolean playSound = true;
// declare a SoundFile object
SoundFile sound;
int NUM_OF_VALUES_FROM_PROCESSING = 3;  /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int processing_values[] = new int[NUM_OF_VALUES_FROM_PROCESSING]; /** this array stores values you might want to send to Arduino **/

Serial myPort;
String myString;
int led = 0;
int record = 0;
int sleigh = 0;
int fire = 0;
int x = 0;
int y1 = 610;
int y2 = 615;
int y3 = 630;
PImage photo;
PImage couch, puppy;

void setup() {
  size(1450, 900, P3D);
  background(#E3EDF0);
  noStroke();
  noCursor();
  setupSerial();
  sound = new SoundFile(this, "christmasSong.mp3");
  photo = loadImage("santa.png");
  couch = loadImage("sofa.png");
  puppy = loadImage("puppy.png");
}
void draw() {
  lights();
  background(#FFFBDB);
  fill(#FAE121);
  star(mouseX, mouseY, 10, 20, 5);
  //floor
  noStroke();
  fill(#6F4923);
  rect(0, 680, width, height);
  rug();
  tree();
  noStroke();
  star(150, 100, 10, 35, 5);
  presents();
  window();
 // snow();
  recordPlayer();
  playRecord();
  //santa();
    translate(-100, 250);
  fireplace();
  couch.resize(300, 350);
  image(couch, 850, 400);
  calendar();
  lightFire();
}


void setupSerial() {
  // printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[3], 9600);
  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
  myString = null;
}

void sendSerialData() {
  String data = "";
  for (int i=0; i<processing_values.length; i++) {
    data += processing_values[i];
    //if i is less than the index number of the last element in the values array
    if (i < processing_values.length-1) {
      data += ","; // add splitter character "," between each values element
    }
    //if it is the last element in the values array
    else {
      data += "\n"; // add the end of data character linefeed "\n"
    }
  }
  //write to Arduino
  myPort.write(data);
  print(data); // this prints to the console the values going to arduino
}

void tree() {
  fill(#46861C);
  triangle(150, 100, 250, 250, 50, 250);
  triangle(150, 200, 300, 380, 0, 380);
  triangle(150, 310, 360, 550, -60, 550);
  print(mouseX, mouseY);
  if (led == 1) {
    fill(#FA2121);
  } else {
    fill(#E9EAA3);
  }
  //translate(-100, -100);
  ellipse(114, 198, 10, 20);
  ellipse(206, 305, 10, 20);
  ellipse(87, 310, 10, 20);
  //  ellipse(146, 444, 10, 20);
  ellipse(272, 480, 10, 20);
  ellipse(136, 503, 10, 20);
  ellipse(30, 474, 10, 20);
  if (led == 1) {
    fill(#42C60A);
  } else {
    fill(#E9EAA3);
  }
  ellipse(181, 196, 15, 25);
  ellipse(155, 338, 15, 25);
  //ellipse(99, 419, 10, 20);
  //  ellipse(212, 402, 10, 20);
  ellipse(208, 500, 15, 25);
  ellipse(74, 493, 15, 25);

  if (led == 1) {
    fill(#FA2121, 90);
  } else {
    noFill();
  }
  //translate(-100, -100);
  ellipse(114, 198, 15, 25);
  ellipse(206, 305, 15, 25);
  ellipse(87, 310, 15, 25);
  //  ellipse(146, 444, 10, 20);
  ellipse(272, 480, 15, 25);
  ellipse(136, 503, 15, 25);
  ellipse(30, 474, 15, 25);

  if (led == 1) {
    fill(#42C60A, 90);
  } else {
    fill(#E9EAA3);
  }
  ellipse(181, 196, 20, 30);
  ellipse(155, 338, 20, 30);
  //ellipse(99, 419, 10, 20);
  //  ellipse(212, 402, 10, 20);
  ellipse(208, 500, 20, 30);
  ellipse(74, 493, 20, 30);

  //bulbs
  // ellipse(
  fill(#F2EC2C);
  stroke(0);
}

void lights() {
  if (mousePressed) {
    float d = dist(527, 585, mouseX, mouseY);
    if (d<20) {
      led = 1;
    }
  }
  if (led == 1) {
    processing_values[0] = 1;
  } else {
    processing_values[0] = 0;
  }
  sendSerialData();
}
void playRecord() {
  if (mousePressed) {
    float d = dist(680, 640, mouseX, mouseY);
    if (d<100) {
      record = 1;
    }
  }
  if (record==1) {
    if (playSound == true) {
      // play the sound
      sound.play();
      // and prevent it from playing again by setting the boolean to false
      playSound = false;
    }
  }
}

void santa() {
  translate(-100, 250);
  photo.resize(400, 150);
  float d = dist(705, 250, mouseX, mouseY);
  if (d<250) {
    sleigh = 1;
  }
  if (sleigh ==1) {
    image(photo, x, 100);
    x = x+3;
  }
}

void presents() {
  //presents
  stroke(0);
  outlet();
  stroke(0);
  strokeWeight(2);
  fill(#F2EC2C);
  translate(400, 730);
  box(50);
  translate(-400, -730);
  translate(300, 750);
  box(40, 100, 50);
  //cursor
}

void fireplace() {
  stroke(#402E22);
  fill(#67432C);
  translate(200, 0);
  quad(1350, -55, width-50, -55, width-50, 750, 1350, 630);
  quad(1350, -55, 1400, -55, width, -55, width, -55);
  quad(width-50, -55, width-50, 750, width, 780, width, -55);
  //translate(0,-55);
  fill(0);

  quad(1350, 630, 1350, 400, 1385, 440, 1385, 725);
}
void calendar() {
  fill(250);
  rect(1000, 130, 200, 150);
  rect(1000, 190, 200, 30);
  rect(1000, 250, 200, 30);
  rect(1000, 220, 200, 30);
  //  rect(100, 160, 200, 30);
  fill(#BC0823);
  rect(1000, 130, 200, 30);
  fill(0);
  textSize(24);
  text("DECEMBER", 1045, 155);
}

void star(float x, float y, float radius1, float radius2, int npoints) {
  float angle = TWO_PI / npoints;
  float halfAngle = angle/2.0;
  beginShape();
  for (float a = 0; a < TWO_PI; a += angle) {
    float sx = x + cos(a) * radius2;
    float sy = y + sin(a) * radius2;
    vertex(sx, sy);
    sx = x + cos(a+halfAngle) * radius1;
    sy = y + sin(a+halfAngle) * radius1;
    vertex(sx, sy);
  }
  endShape(CLOSE);
}

void outlet() {
  noStroke();
  fill(#89888B);
  rect(419, 465, 35, 40);
  fill(#212121);
  line(432, 470, 432, 482);
  line(441, 470, 441, 482);
  rect(434, 490, 7, 7);
  translate(-100, -100);
  if (led ==0) {
    curve(437, 687, 352, 683, 468, 712, 450, 600);
    curve(537, 577, 468, 712, 537, 687, 512, 720);
    rect(528, 663, 15, 30);
  }
  if (led == 1) {
    rect(529, 570, 15, 30);
    stroke(0);
    strokeWeight(5);
    line(437, 687, 535, 594);
    line(352, 683, 437, 687);
    strokeWeight(1);
  }
}

void window() {
  fill(#0C1383);
  strokeWeight(20);
  stroke(#FAF297);
  translate(-400, -950);
  rect(600, 300, 400, 300);
  line(800, 300, 800, 600);
  line(600, 450, 1000, 450);
  strokeWeight(1);
}

void recordPlayer() {
  stroke(#2E2303);
  strokeWeight(3);
  fill(#3E2F06);
  translate(800, 870);
  box(130, 130, 130);
  fill(#BC0823);
  translate(10, -90);
  box(80, 30, 130);
  fill(0);
  rect(680, 560, 20, 10);

  noStroke();
  translate(-810, -780);
  ellipse(810, 750, 60, 30);
  fill(255, 0, 0);
  ellipse(810, 750, 25, 10);
  fill(0);
  ellipse(810, 750, 10, 5);
  strokeWeight(5);
  stroke(100);

  if (record == 0) {
    line(773, 770, 798, 740);
    //stroke(250);
    //curve(802, 770, 795, 755, 818, 760, 815, 765);
  }
  if (record == 1) {
    line(773, 770, 793, 753);
    stroke(250);
    //pushMatrix();
    //translate(width*0.02, height*0.05);
    //rotate(frameCount / 200.0);
    //curve(802, 770, 800, 755, 818, 758, 815, 760);
    //popMatrix();
  }
}

void rug() {
  translate(100, 100);
  fill(#6F4923);
  noStroke();
  rect(100, 550, 100, 90);
  fill(#D12A00);
  stroke(#AF2502);
  ellipse(140, 600, 300, 60);
  ellipse(1150, 650, 200, 90);
  fill(#A22100);
  noStroke();
  ellipse(1150, 650, 140, 40);
  puppy.resize(100,75);
  image(puppy,1100,600);
  noStroke();
}

void lightFire() {
  fill(#291C04);
  // rotate(PI/4);
  quad(1350, 610, 1380, 645, 1380, 655, 1350, 620);

  quad(1350, 620, 1380, 640, 1380, 650, 1350, 630);
  line(1360, 620, 1386, 665);
  line(1360, 630, 1386, 685);
  line(1360, 635, 1386, 680);
  if (mousePressed) {
    float d = dist(1360, 500, mouseX, mouseY);
    if (d<200) {
      fire=1;
    }
  }
  if (fire == 1) {
    noStroke();
    float r = random(230, 255);
    float g = random(0, 180);
    float b = random (7, 10);
    float rad = random(10, 30);
    float large1 = random(10, 50);
    float large2 = random(10, 50);
    float large3 = random(10, 50);
    fill(r, g, b);
    ellipse(1360, y1, rad, large1);

    ellipse(1370, y2, rad, large2);

    ellipse(1375, y3, rad, large3);

    //y1 =y1-2;
    //y2 = y2-3;
    //y3 = y3-5;
  }
}

DIGITAL MUSIC BOX – MAKING DIGITAL ART AND MUSIC

PROJECT STATEMENT OF PURPOSE

Over the time in this course I have felt very inspired by the interactive artworks that innovate the way we experience audio. Some examples of this are this “Voice Array” which layers the recordings people make in an array and these Musical Swings. Looking into these projects arose questions for me about how music and space interact as well as how other people interact to make music. This project is intended for those that have a curiosity about the process of music creation (for this project specifically music in the Western world). This will be impactful because it will allow music listeners to understand the intention that goes behind musical creation. This project will be a music box which allows the user to alter multiple factors in the music. The resulting impact on the song will be displayed on the screen.

PROJECT PLAN

DUE DATE: DECEMBER 6

Components of a song which the user will be able to alter:

1. Notes

2. Key

3. Tempo

4. Octave

(this is my current plan I may add other factors such as echo and reverb)

Each component will have a method of interaction – potentiometer and clicking the screen – which will make a change to the song. This change will subsequently be displayed on the screen. First, I need to make the song play through processing. Next, I decide how the arduino can be used to change the song. Finally, I will complete the fabrication and animation of the screen depending on the changes. I have three weeks which means one week for each step in the process.

CONTEXT AND SIGNIFICANCE

The prior research I did inspired me to work with the elements of space, and music. My concept of this music box is unique because the user will be able to both be inspired by the music and visuals as well as alter them. The user can enjoy the rewarding process of music creation. After it’s creation, the project could be done on a larger scale. Similar to the New York City Artechouse which I wrote about in my blog-post titled “Interaction Is”. I imagine the music being played out of high quality speakers and the visuals being displayed on a much larger display with a projector. This could be an inspiring interactive art exhibit for its viewers.The project could meaningfully express the subjectivity and power of art and music.

“Today I Saw Something Beautiful”

The idea of this project comes from a conversation I had with a friend about graffiti. Is it meaningful, or meaningless? Harmful, harmless, or advantageous? I would like to make a statement about the beauty of graffiti as a means for communication and expression with this project. I researched graffiti and how it varies in its creation process. This means that my project must have multiple methods of use. This project will be made for anyone with something to express. The goal is to provide a creative outlet to send a message to someone that they may never see. Similar to graffiti, it allows for genuine expression and inspiration.

“I am”

The question of how to define our essence is one that cannot be answered entirely with words. Since being in New York City, I have seen many artistic expressions of the essence of people. This project would allow the user to create a digital expression of what they are by mixing several factors. It is up to each individual to define and interpret what they are. This is relevant for any person as it is a beautiful, inclusive form of reflection that has no fixed mass meaning. The impact is the materialization of that which is not seen in everyday life: how people truly feel about themselves and their circumstances.

Digital Music Box

This project will play a song digitally (a music box on the computer) and the user can alter the song using multiple methods. The audience it is made for is those that want to learn about the basics of how music is made. I have always loved creating music and am interested to try this in a unique way using different music making methods.

RESEARCH

While looking for a successful interactive project I was looking for two main characteristics.

1. The project has two or more actors that communicate with each other

2. Complex interactions 

By complex I mean multiple methods of interaction, multiple inputs or outputs etc.

I. Interactive Table

This project is interactive because of its ability to interact with the user and other objects by lighting up. Both actors can speak and respond to one another (ex. placement  of the hand = speaking to table, the table speaks by lighting up).  However, of the three projects I researched, this is relatively the least successful as an interactive experience because the interaction is somewhat limited. I would love to see a project like this with more outputs and inputs. An ideas I have would be a thermal input that produces a specific color. The table would be able to tell you when your drink is at a good temperature.

II. Interactive Installation

Of the three that I researched this was the most interesting to me. It is an installation with multiple interactions. By blowing into a box, in another space, wind creates a sort of tornado. There is a monitor that reflects movements as a shadow. There is another box which upon blowing into triggers a separate component to blow bubbles. In each of these interactions the actors, the user and the project, communicate with each other cyclically. I am especially fascinated by the way space between the actors is used in this project. It is interactive from a distance. In my blog-post titled “Interaction Is…” I explained how an interactive project allows the user to become a part of a story. This project is a perfect example of that. The user is able to create art in an indirect yet interactive way.

III. Rain Room

This project is a rain room that uses 3D motion sensors to locate objects or people within its domain. Rain is falling everywhere except for where the sensors identify a presence. This project is very successful to me because it interacts with multiple senses making it thoroughly immersive. Similar to the one before, this project uses the concept of space in an interesting way which inspires me to think what else could be done using the components  of nature by manipulating small portions of it.

Defining Successful Interaction

As I stated earlier, successful interaction is cyclical communication between two actors. The cyclical communication discussed in The Art of Interactive Design is a cycle of two actors encoding and decoding messages. In addition to this fundamental piece of successful interaction, I think a project can become even more successful by becoming more immersive. During both my group project and midterm project, I worked with the idea of immersion. In my group project we discussed the potential of a piece of technology that interacts with all five senses. Later, during my midterm, we built our project off of the idea of fully immersive games.

In “A Brief Rant on the Future of Interactive Design” ,Brett Victor writes, “We live in a three-dimensional world. Our hands are designed for moving and rotating objects in three dimensions, for picking up objects and placing them over, under, beside, and inside each other. No creature on earth has a dexterity that compares to ours.” It is for this reason that I think that interactive projects should have complex systems of interaction. Our lives are becoming more and more technology based. It is incredible what we can do with technology. With an intersection of the capabilities of technology and the human body, a very successful project can be created. I believe that an ideal interactive project combines 2D technology and the 3D world allowing for complex communication between the two.