CREATIVE MOTION – LILLIE YAO – IMNI

 CONCEPTION AND DESIGN:

We wanted to create an interactive project where users would get to see the reflections of themselves through a light matrix, instead of seeing their reflections on something obvious like a camera or mirror. Since we wanted to make it very obvious what the users were doing, we decided to put the matrix (our output source) right in front of the camera (our input source). In the beginning we were just going to have them side by side but we realized that would take the attention off of the matrix since people tend to want to look at their reflection more than the light, no matter how obvious the light may be. 

During our brainstorm period, we tried to research different light boards since we knew we wanted to have a surface of light displays instead of single LED lights. We also though that programming and using 64 or more LED lights would be very complicated. We ended up using the Rainbowduino and 8×8 LED Matrix Super Bright to create the light board as well as the Rainbowduino being able to be our Arduino/breadboard source. Since we researched a bunch of different light sources, the only one that was available to us was the Rainbowduino and LED matrix. I’m sure there would have been better options, especially because we hoped to have a bigger LED board.

FABRICATION AND PRODUCTION:

During user testing we got different feedback and suggestions from our classmates that they thought would make our project better. A lot of classmates wished that the display of the LED board was bigger so that we could make our interaction more of an experience rather than just a small board in front of their eyes. As an attempt to change that, we wanted to put together multiple LED boards to make a bigger screen. Soon after attempting that, we realized that you actually can’t put together multiple LED boards easily and make them work together. Each LED board actually operates separately from each other. As stated in our concept and design, we researched multiple different types of LED boards but a lot of the materials that were better suited for our project were not available to us in the short amount of time after user testing.

After realizing that we still had to use one LED Matrix board, we decided to make our fabrication product so that it would magnify the LED board. We decided to make a polygonal shape with the clear acrylic material where the LED would fit snug into a box on the end. We decided to go with the clear acrylic board for laser cutting is because we thought the opaque look would suit our design better than any other material. In my mind, I pictured that the LED lights would reflect off of different surfaces and it would appear more interesting if the product was see-through. We really didn’t think there would have been a better option because all of the other laser cutting materials were too dull and 3D printing wouldn’t have worked for a hollow design. After fabrication,  a fellow ( which I forgot his name…..) gave us the idea to actually put the matrix INSIDE our polygon so that the lights would reflect within our polygon. This truly changed our project because now, we were able to utilize our fabrication design in a different and better way than before.

Sketching for fabrication:

Laser Printing:

Original fabrication:

Changed fabrication so the light would shine through:

Another suggestion during user testing that we had was that users wished the LED board was facing them instead of facing up because it was hard to see the board when it is not facing the user. Therefore, we decided to make the fabrication product a polygon so that it would be easier to angle the polygon to turn to the side and face the user.

Lastly, we had a great suggestion to implement sound into our project to make it more interesting rather than just seeing light, users would also be able to trigger different sounds while they move. After getting this feedback, we decided to code different sounds into our project that would trigger when you moved in different places. This really changed our project because we got to use different sounds and lights to create art, which in my opinion made our project more well rounded.

Sketches for LED board and pixels on camera:

After presenting our final project, we got some feedback saying that some of the sounds were too much and it would be better if we used all musical instruments instead of animal noises, snapshots, etc. Since we both really wanted to present our product at the IMA show, we decided to change the sounds to all instrument sounds before the show, that way it would be lighter on the ears and the users would be less confused. I think this helped our project a lot because many people really loved our project at the IMA show, even Chancellor Yu got to interact and see our project!

Chancellor Yu interacting with our project!!!!:

CONCLUSIONS:

The goal of my project was to create something that users could interact with but at the same time, have fun with. We wanted to create something that has a direct input/output as well as something users can play with for fun. As a creator, I felt like it was really cool to create something that people can interact with and have fun with at the same time.

My product result aligns with my original definition of interaction because it has both an input and an output but it will keep running whether or not it has an input. The input being the camera, will still detect changes in motion whether or not something or some one is moving. At the same time, my definition of interaction stated that interaction is a continuous loop from input to output. So if there is an input, there will 100% be an output. Which in my project, if there is any change in motion, it will change the light on the matrix and trigger the sound at the same time.

My expectation of the audience’s response was pretty similar. The only thing my partner and I didn’t really think about was that once a user sees their own reflection, they tend to focus on that instead of the lights changing. I often found myself having to explain my project instead of them figuring it out and if they did figure it out, it took a bit of time for them to do so. Other than that, my expectation of audience reactions was pretty similar.

User Reactions:

If I had more time to improve my project, I would definitely take into consideration the “experience” aspect of the project I wanted to implement. During our final project, Eric said that if really wanted to make it an experience, we needed to factor in a lot of different things. If I could change some of the things about my project to make it more of an experience I would have speakers around to amplify the sound, project the camera input onto a bigger screen, and make the LED light board bigger. 

From the setbacks and failures of my project, I learned that theres always room for improvement, even if you think there isn’t enough time. I learned that there is always going to be projects and parts of other peoples work that will be better than yours but you should never compare what other peoples capabilities are to yours. After taking this class and seeing all of the work that I have done, I am very happy with all of my accomplishments. I would have never thought that this project would come to life during our brainstorming period but I’m really glad we could make it work! I’m really glad that we were able to create a fun and interactive work or art where users were able to see themselves and make art with light as well as music and sound!

Arduino/Rainbowduino Code:

#include <Rainbowduino.h>
char valueFromProcessing;

void setup()
{
Rb.init();
Serial.begin(9600);
}

unsigned char x, y, z;

void loop() {

while (Serial.available()) {
valueFromProcessing = Serial.read();

if (valueFromProcessing == ‘D’) {
Rb.fillRectangle(0, 0, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘d’) {
Rb.fillRectangle(0, 0, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘C’) {
Rb.fillRectangle(2, 0, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘c’) {
Rb.fillRectangle(2, 0, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘B’) {
Rb.fillRectangle(4, 0, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘b’) {
Rb.fillRectangle(4, 0, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘A’) {
Rb.fillRectangle(6, 0, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘a’) {
Rb.fillRectangle(6, 0, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘H’) {
Rb.fillRectangle(0, 2, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘h’) {
Rb.fillRectangle(0, 2, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘G’) {
Rb.fillRectangle(2, 2, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘g’) {
Rb.fillRectangle(2, 2, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘F’) {
Rb.fillRectangle(4, 2, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘f’) {
Rb.fillRectangle(4, 2, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘E’) {
Rb.fillRectangle(6, 2, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘e’) {
Rb.fillRectangle(6, 2, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘L’) {
Rb.fillRectangle(0, 4, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘l’) {
Rb.fillRectangle(0, 4, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘K’) {
Rb.fillRectangle(2, 4, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘k’) {
Rb.fillRectangle(2, 4, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘J’) {
Rb.fillRectangle(4, 4, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘j’) {
Rb.fillRectangle(4, 4, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘I’) {
Rb.fillRectangle(6, 4, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘i’) {
Rb.fillRectangle(6, 4, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘P’) {
Rb.fillRectangle(0, 6, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘p’) {
Rb.fillRectangle(0, 6, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘O’) {
Rb.fillRectangle(2, 6, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘o’) {
Rb.fillRectangle(2, 6, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘N’) {
Rb.fillRectangle(4, 6, 2, 2, random(0xFFFFFF));
}
if (valueFromProcessing == ‘n’) {
Rb.fillRectangle(4, 6, 2, 2, 0x000000);
}

if (valueFromProcessing == ‘M’) {
Rb.fillRectangle(6, 6, 2, 2, random(0xFFFFFF));
delay(1);
}
if (valueFromProcessing == ‘m’) {
Rb.fillRectangle(6, 6, 2, 2, 0x000000);
}
}
}

Processing Code:

import processing.video.*;
import processing.serial.*;
import processing.sound.*;
Serial myPort;
Capture cam;
PImage prev;
boolean p[];
int circleSize = 10;
SoundFile a4;
SoundFile b4;
SoundFile c4;
SoundFile c5;
SoundFile cello_slide;
SoundFile d4;
SoundFile e4;
SoundFile f4;
SoundFile g4;
SoundFile violin_slide;
SoundFile guitar;

void setup() {
size(800,600);
cam = new Capture(this, 800,600);
cam.start();
prev = cam.get();
p = new boolean[width * height];

myPort= new Serial(this, Serial.list()[ 2 ], 9600);

a4 = new SoundFile(this, “a4.wav”);
b4 = new SoundFile(this, “b4.wav”);
c4 = new SoundFile(this, “c4.wav”);
c5 = new SoundFile(this, “c5.wav”);
d4 = new SoundFile(this, “d4.wav”);
e4 = new SoundFile(this, “e4.wav”);
f4 = new SoundFile(this, “f4.wav”);
g4 = new SoundFile(this, “g4.wav”);
guitar = new SoundFile(this, “guitar.wav”);
}
void draw() {
if (cam.available()) {
cam.read();
cam.loadPixels();
}
translate(cam.width,0);
scale(-1,1);
image(cam,0,0);

int w = cam.width;
int h = cam.height;
for (int y = 0; y < h; y+=circleSize){
for (int x = 0; x < w; x+=circleSize) {
int i = x + y*w;
//fill( 0 );
if(cam.pixels[i] != prev.pixels[i]){
p[i] = true;
}
else{
p[i] = false;
}
}
for (int y = circleSize; y < h- circleSize; y+=circleSize){
for (int x = circleSize; x < w- circleSize; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
fill(cam.pixels[i]);
}
else{
fill( 0 );
}
rect(x,y,circleSize,circleSize);
}
}
int countD = 0;
for (int y = circleSize; y < 150; y+=circleSize){
for (int x = circleSize; x < 200; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
countD++;
}
}

println(countD);
if (countD > 100){
myPort.write(‘D’);
if(!guitar.isPlaying()) {
guitar.play();
}
}
else {
myPort.write(‘d’);
}

int countC = 0;
for (int y = circleSize; y < 150; y+=circleSize){
for (int x = 200+circleSize; x < 400; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;

if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){

countC++;
}
}

println(countC);
if (countC > 100){
myPort.write(‘C’);
if(!g4.isPlaying()) {
g4.play();
}
}
else {
myPort.write(‘c’);
}

int countB = 0;
for (int y = circleSize; y < 150; y+=circleSize){
for (int x = 400+circleSize; x < 600; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countB++;
}
}

println(“B: ” + countB);
if (countB > 100){
myPort.write(‘B’);
if(!f4.isPlaying()) {
f4.play();
}
}
else {
myPort.write(‘b’);
}

int countA = 0;
for (int y = circleSize; y < 150; y+=circleSize){
for (int x = 600+circleSize; x < 800; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countA++;
}
}

println(countA);
if (countA > 100){
myPort.write(‘A’);
if(!guitar.isPlaying()) {
guitar.play();
}
}
else {
myPort.write(‘a’);
}

int countH = 0;
for (int y = 150+circleSize; y < 300; y+=circleSize){
for (int x = circleSize; x < 200; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countH++;
}
}

println(countH);
if (countH > 100){
myPort.write(‘H’);
if(!a4.isPlaying()) {
a4.play();
}
}
else {
myPort.write(‘h’);
}

int countG = 0;
for (int y = 150+circleSize; y < 300; y+=circleSize){
for (int x = 200+circleSize; x < 400; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countG++;
}
}

println(countG);
if (countG > 100){
myPort.write(‘G’);
}
else {
myPort.write(‘g’);
}

int countF = 0;
for (int y = 150+circleSize; y < 300; y+=circleSize){
for (int x = 400+circleSize; x < 600; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countF++;
}
}

println(countF);
if (countF > 100){
myPort.write(‘F’);
} else {
myPort.write(‘f’);
}

int countE = 0;
for (int y = 150+circleSize; y < 300; y+=circleSize){
for (int x = 600+circleSize; x < 800; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countE++;
}
}

println(countE);
if (countE > 100){
myPort.write(‘E’);
if(!e4.isPlaying()) {
e4.play();
}
}
else {
myPort.write(‘e’);
}

int countL = 0;
for (int y = 300+circleSize; y < 450; y+=circleSize){
for (int x = circleSize; x < 200; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countL++;
}
}

println(countL);
if (countL > 100){
myPort.write(‘L’);
if(!b4.isPlaying()) {
b4.play();
}
}
else {
myPort.write(‘l’);
}

int countK = 0;
for (int y = 300+circleSize; y < 450; y+=circleSize){
for (int x = 200+circleSize; x < 400; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countK++;
}
}

println(countK);
if (countK > 100){
myPort.write(‘K’);
}
else {
myPort.write(‘k’);
}

int countJ = 0;
for (int y = 300+circleSize; y < 450; y+=circleSize){
for (int x = 400+circleSize; x < 600; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countJ++;
}
}

println(countJ);
if (countJ > 100){
myPort.write(‘J’);
}
else {
myPort.write(‘j’);
}

int countI = 0;
for (int y = 300+circleSize; y < 450; y+=circleSize){
for (int x = 600+circleSize; x < 800; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
int downLt = (x – circleSize) + (y+ circleSize) * w;
int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[down] && p[left] && p[right] && p[upRt] && p[upLt] && p[downRt] && p[downLt]){
//fill(cam.pixels[i]);
countI++;
}
}

println(countI);
if (countI > 100){
myPort.write(‘I’);
if(!d4.isPlaying()) {
d4.play();
}
}
else {
myPort.write(‘i’);
}

int countP = 0;
for (int y = 450+circleSize; y < 600; y+=circleSize){
for (int x = circleSize; x < 200; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
//int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
if(p[i] && p[up] && p[left] && p[right] && p[upRt] && p[upLt]){
//fill(cam.pixels[i]);
countP++;
}
}

println(countP);
if (countP > 100){
myPort.write(‘P’);
if(!c5.isPlaying()) {
c5.play();
}
}
else {
myPort.write(‘p’);
}

int countO = 0;
for (int y = 450+circleSize; y < 600; y+=circleSize){
for (int x = 200+circleSize; x < 400; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
//int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
if(p[i] && p[up] && p[left] && p[right] && p[upRt] && p[upLt]){
//fill(cam.pixels[i]);
countO++;
}
}

println(countO);
if (countO > 100){
myPort.write(‘O’);
}
else {
myPort.write(‘o’);
}

int countN = 0;
for (int y = 450+circleSize; y < 600; y+=circleSize){
for (int x = 400+circleSize; x < 600; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
//int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[left] && p[right] && p[upRt] && p[upLt]){
//fill(cam.pixels[i]);
countN++;
}
}

println(countN);
if (countN > 100){
myPort.write(‘N’);
}
else {
myPort.write(‘n’);
}

int countM = 0;
for (int y = 450+circleSize; y < 600; y+=circleSize){
for (int x = 600+circleSize; x < 800; x+=circleSize) {
int i = x + y*w;
int up = x + (y- circleSize) * w;
//int down = x + (y+ circleSize) * w;
int left = (x – circleSize) + y*w;
int right = (x + circleSize) + y*w;
int upLt = (x – circleSize) + (y- circleSize) * w;
int upRt = (x + circleSize) + (y- circleSize) * w;
//int downLt = (x – circleSize) + (y+ circleSize) * w;
//int downRt = (x + circleSize) + (y+ circleSize) * w;
//fill( 0 );
if(p[i] && p[up] && p[left] && p[right] && p[upRt] && p[upLt]){
//fill(cam.pixels[i]);
countM++;
}
}

println(countM);
if (countM > 100){
myPort.write(‘M’);
if(!c4.isPlaying()) {
c4.play();
}
}
else {
myPort.write(‘m’);
}

prev = cam.get();
cam.updatePixels();

}

Creative Motion – Yu Yan (Sonny) – Inmi

Conception and Design:

During the brainstorming phase, my partner Lillie and I tended to build an interactive project that allows users to create digital paintings only with their motions. The interaction of this project includes users’ movements as the input and the image displayed on the digital device as the output. Our enlightenment comes from Leap Motion interactive art exhibit. At this point, we thought about using multiple sensors on Arduino to catch the movements, and display the image on Processing. However, after we tried on several sensors and did some researches, we found that there is no sensor suitable for our needs and even if there is, it would take a huge amount of time to build the circuit and understand how to code. So we turned to our instructor for help and also did some further researches to see other alternative ways. Finally, we decided to use the webcam in Processing as our “sensor” to catch the input (users’ movements) and build an LED board on Arduino to display the output (painting). The reasons why we choose the webcam are that it’s easier to catch images from the camera than from the sensor, the color values detected from the camera are more accurate than from the sensor, and the code is not too difficult to learn with the help of other IMA fellows. However, when we were figuring out the Arduino part, we found it hard to build the circuit using single-colored LEDs and connect all of them on the breadboard. With our further researches, we managed to find that the 8*8 LED matrix can replace the single-colored LEDs and also generate more colors. But the first few pieces of LED matrix are not satisfactory because we don’t know how to connect them to the Arduino board and we were unable to find the solutions online (We found this video that we thought it would be helpful for us to understand how to connect the LED matrix to the Arduino, but it wasn’t). We also found a sample code to test the LED matrix, but since we were unable to connect it to the Arduino, this code became useless as well. Moreover, those pieces can only generate three colors that didn’t meet our needs.

Since we want to allow users to create paintings with more diversity, we tried to find the LED matrix that can display in rainbow colors. After consulting with other IMA fellows, we found that the Rainbowduino can work with one kind of LED matrix and display in rainbow colors. The code for this is also easy to comprehend. So eventually, we decided to use the Rainbowduino and the LED matrix in Arduino as our output device, and the webcam in Processing as our input detector. 

Fabrication and Production:

One of the most significant steps in our production process in terms of failures are the coding phase. Since when we chose materials for the output device, we tried quite a few kinds of LED matrix and also looked for their codes, we discovered that the code for previous LED matrix are too complex to comprehend. We needed to set different variables for different rows and columns of the LEDs, which is quite confusing sometimes. But after we decided to use the Rainbowduino, the code for Arduino became much easier because we can use the coordinate to code for each single LED. And with the help of IMA fellows, we managed to write the code that satisfied our needs. This experience tells us that choosing suitable equipment is very crucial to a project,  for choosing a good one can bring great convenience to the progress and save us a lot of time. Another significant step is the feedback we received during the user testing session. The good things are that many users showed their interests to our project and thought it’s really cool when displaying different colors. They all thought the interaction with the piece is quite intriguing and they liked that their movements can light up the LEDs in different colors. This feedback meets our initial goal of providing users with opportunities to create their own art using their motions. However, there were still some problems we can still improve. First of all, one of the users said that the way the LEDs lighted up could be a little confusing because it cannot well illustrate where the user moves. It’s because we didn’t separate the x-axis and the y-axis for each section of LEDs at first. The following sketches and video help explain the situation.

To solve this issue, we modified our code and separated the x-axis and the y-axis for each section so that it can light up without causing other sections of LEDs lighting up as well. After we showed our modified project to the user who gave us this comment, he said that the experience is better now and he can see himself moving in the LED matrix more clearly. Second, the experience of interaction could be too single and boring and it’s hard to convey our message to users through this experience. Since the interaction is only about moving their bodies and displaying different colors in the same position of their movements on the LED matrix, it might be too stuffless for an interactive project. Marcela and Inmi suggested that maybe adding some sounds to it can make it more attractive and more meaningful. So we took their advice. In addition to turning up a section of LEDs when moving to the relative area, we also added some sound files to each section of LEDs and made them play with the lighting up of the corresponding LEDs. The following sketches illustrate how we defined each section and different sound file.

Initially, we used several random sounds such as “kick” and “snare” because we wanted to bring more diversity into our project. But during the presentation, some users commented that the sound is too random and sounded chaotic when they were all turned on. One of them also mentioned that the sound of “snapshot” made her feel uncomfortable. So for the final IMA show, we adjusted all the sound files to different key notes of the piano. This change made the sound more harmonious and comfortable to hear when users are interacting with the project. Third, some users mentioned that the LED matrix is too small and sometimes they might neglect the LED and pay more attention to the computer screen instead. At first, we thought about connecting more LED matrixes together and making a bigger screen, but we didn’t manage to do that. So instead of magnifying the LED matrix, we made the computer screen more invisible and the LED matrix more outstanding by putting it into our fabrication box. The result turned out to be much better than before and we caught users’ attention to our LED matrix instead of the computer screen.

By contrast, the fabrication process is one of the most significant steps of our project in terms of success. Before we settled down the final polygon shape, we came up with a few other shapes as well. Similar to my midterm project, we also chose Laser-cut and glued each layer together to build the shape. Since we wanted to make something cool and make the most use of our material, we decided to choose transparent plastic board to make the shape. We also discovered that polygon can help build a sense of geometric beauty, so finally, we made our box into a polygon shape. At first, we tended to just put the LED matrix on the top of the polygon. But one of IMA fellows suggested that we can put the LED matrix at the bottom of the polygon so that the light can reflect through the plastic and make it prettier. Thanks to this advice, it turned out to be a really cool project!

Conclusions:

For our final project, our goal is always allowing people to create their own art using their motions and also encourage them to create art in different forms. Although we changed our single output (painting) to multiple outputs (painting and music), our goal of creating art with motions still remains the same. Initially, we defined interaction as a continuous communication between two or more corresponding elements, an iterative process which involves actions and feedbacks. Our project successfully aligned with our definition by creating a constant communication between the project and the users and providing immediate feedback to users’ motions. However, the experience of interacting with the piece is still not satisfactory enough because we could not magnify the LED matrix so that it’s too small to notice. We didn’t create the best experience to users. But fortunately, most of the users understood that they can change the image and create different sounds with their motions. They all thought that this is a really interesting and interactive project that they can play with for a long time. Some users even tried to play a full song after they discovered the location of each keynote. If we had more time, we would definitely build a bigger LED board to make it easier for users to experience the process of creating art with their motions. The setbacks and obstacles we’ve encountered are all seemed quite fair during the process of completing a project. But the most important thing is to learn some lessons from these setbacks and obstacles. What I learned from them are that we should humbly take people’s comments about our project and turn them into useful improvements and motivations. In addition, I noticed that I still didn’t pay enough attention to the experience of the project. Since experience is one of the most vital parts of an interactive project, it should always be the first consideration. However, I also learned that the reason why many people like our project is that it can display their existence and be controlled by them. Users are in charge of everything the project displays. This also shows that we have created a tight and effective communication between the project and users. Furthermore, making the most use of our materials is also very important. Sometimes it can make a big change to the whole project and turn it into a more complete version. Since nowadays many people still hold the concept that art can only be created in those limited forms, we want to break this concept by providing them with tools to create new forms of art and inspiring them to think outside of the box. Art is limitless and with great potential. By showing that motion can also create different forms of art, this project is not only a recreation but also an enlightenment to help people generate more creative ideas of new forms of art and free their imagination. It also helps make people be aware of their ability and their “power”, and let them control the creation of art. “Be bold, be creative, and be limitless.” This is the message we want to convey to our audience. 

The code for Arduino is here. And the code for Processing is here.

Now, let’s have a look at how our users interact with our project!