This is an illustration of the GPS System on earth that serves to introduce the technology to aliens. GPS system, as we all know, is a satellite-based navigation system that allows users to determine their precise location, speed, and time, anywhere on or near the Earth’s surface.
To better understand the illustration, the characteristics of our subject, known as “heptapods” in the story, must be understood. They are described as having a very different physiology and perception of time than humans. They do their communications using a complex system of visual symbols, which they call “semagrams”. These symbols are circular, with multiple rings that represent different layers of meaning. The heptapods can manipulate these symbols in various ways to convey different messages. Furthermore, they have a non-linear perception of time, which means that they experience time all at once rather than in a linear sequence as humans. They are able to perceive past, present, and future events simultaneously.
Honestly speaking, it is quite hard to imagine how they perceive the world without the causal mindset, which we, as human beings, have been mastering in using it for millions of years and regard as something indispensable to our way of thinking, just as air and water to our living. So it’s quite difficult to imagine how the alien will understand the illustration. Thus, we tried to make the illustration as clear-cut to be understood by aliens as possible.
In the middle of the picture to the up-right is the earth. The drawing we think is pretty representative because the first thing of our civilization that aliens driving their spaceship perceive must be the earth, and the drawing vividly depicts it. And on the right surface of it, there is an object on the mountain. Then, an artificial satellite scans the surface of the earth and creates a piece of positional information, which is externalized using graphs. The information could be internalized and used by human beings on earth. We use a circular mode to represent all of this, instead of using a linear one. This is because according to “Story of Your Life”, it matches aliens’ mindset of thinking, which is kind of circular.
On the left corner of the picture, there is an illustration of the earth whose equator is highlighted, signalizing aliens to orbit around the earth. Next to it is a map of the earth. The former two things signalize what a map is, which can further illuminate how to understand the maps created by the GPS system. This illustration of how to understand a map is crucial to aliens’ understanding of the information given by the GPS system because human beings have been so familiar with the language of maps but aliens possibly aren’t.
On the right corner of the picture, it shows how the satellite scans the area to make map information, which further illuminates the idea.
The three parts of the picture, as a human being would understand, should have a logical order to express the whole idea. If the picture were written for human beings, we would add some logical symbols such as arrows to represent it. However, they would be useless and even distracting to aliens in the fiction, who see the picture as a whole.
However, we still think that there’s room for improvement. For example, we still divided the illustration into three parts, which is also something human beings have accustomed to doing when illustrating something. It would be more understandable to aliens if we come up with another great idea without using parts and logic.
Name: Siwei Chen
Title of the music: 骤雨の狭間 (Shower)
Artist Name: Silentroom
When I first heard this music, the beginning of the piece gave me a heavy percussive feeling. It has a darker style, like a heavy and noisy rainstorm hitting the roof, striking the ground, and falling into the water. The drums and beats are very complex. But when I heard the middle section, I heard the lush, smooth sound of the saxophone, accompanied by the strings, as if the dark clouds had been cut away, the sun had been revealed, the sunlight had fallen, and all the scenery was bright. It was a sun shower.
So in my work, I am going to draw the ripples of rain hitting the ground and add random elements to express the complex drums of the piece. These ripple patterns, although composed of words, are still recognizable, and by Gestalt psychology, the viewer can perceive and recognize these elements and thus associate them with the scene of heavy rain. I also used approximate and closure principles to form the close text circles into a ripple pattern. These ripple patterns are independent and not connected, showing that although the music is complicated with drums, the drum beat does not affect each other and still proceed in a regular manner. From the previous One Black Square assignment, I learned how to use simple black and white to represent abstract objects, and also learned to use appropriate interspersed black and white to give the viewer a sense of visual impact. Therefore, in this assignment, I used black and white to fill the background in a symmetrical way to make a strong sense of contrast, which corresponds to the quick and strong rhythm of the song. The black and white background also corresponds to the contrast between the darkness in the first half of the song and the light and brightness in the second half.
In the mid-critique, I received some suggestions: the pattern of the ripples looked like “cells”; the pattern was not complex enough to show the complexity of the music; the font was a bit monotonous, and so on. So in the final work, I changed the arrangement of the original pattern and used different fonts to outline the ripples in a circle, and then changed the shape a little to create the feeling of raindrops falling. Then I used the letter “O” and the letter “I” to make water droplets and raindrops, which fell randomly in my work. I also used a string of randomly typed letters as a random element at the corner. That’s how I finished my final piece.
I received some suggestions such as the black background and the white background doesn’t need to be completely symmetry, so I consider that if I had more time, I will try to enrich the background, for example, putting the black background at the top and bottom of the work, and the white one in the middle to make the white background interspersed with black, which brings a better visual effect to the viewer.
The whole piece of fiction is interspersed with two stories. One is the story of the author’s communication with the aliens, conducted in a human writing style. The other is the story of the author and her child, recounted in Heptapod writing style.
When speaking, Heptapods’ pronunciation is different from humans. Humans cannot pronounce the word they said. So it is hard to communicate orally.
They write in a semagram style, their written language is a composition of complex symbols that convey meaning in a nonlinear fashion, which can be unordered. And as they can know the future, so they know how would the composition of their writing be before they write, which is impossible for humans.
The way Heptapod thought is also different from humans. When they said: “process create-endpoint inclusive-we”, means “let’s start”. When it comes to understanding the travel of light rays. They thought it was because the path that the light choose is the shortest and quickest for it to travel. They thought in a way that they have seen the future and the end before the action.
The physical structure of Heptapod is that they have seven limbs, which enable them to use a complex combination of symbolic characters. The structure also affects the way they speak, their vocal tract is substantially different from a human vocal tract so they pronounce differently from humans. In contrast, humans use nonverbal communication cues, such as facial expressions, body language, and tone of voice, to convey meaning and emotion.
A. Project Title: Rhythm Master
Group members: Siwei Fang (Wendy), Siwei Chen
Instructor: Gottfried Haider
B. CONCEPTION AND DESIGN:
In our previous research, we were inspired by the music games “Osu!” and “Dance revolution”, in which players perform actions with the rhythm of the music and earn points according to the accuracy and correctness of their play. This is a kind of game with a strong sense of participation. So we decide to design a rhythm game (it’s really interactive! ) In the game, the player presses the button according to the rhythm and the prompt on the screen to interact with the artifact. In order to increase the fun and engagement, we also designed an additional session where the use of a whistle at a specified rhythm point can change the effect of the lights (unfortunately, because of the epidemic this session could not be realized by the game).
(Our first proposal:)
In the user testing, the game was mostly recognized by the players, they thought the experience was great, the lighting was cool, and the difficulty was challenging. The feedbacks and suggestions we received include: adding a tutorial section, changing the lighting effect (such as covering a white paper on top of Neopixel), adding different songs, adjusting the speed of falling stars, etc. Based on feedback, we made adjustments accordingly. I made the project into three parts, the tutorial part, the normal part, and the harder part. We also pasted white paper on all the LEDs to make the overall look more beautiful. In the final presentation, the player experience and feedbacks were better.
C. FABRICATION AND PRODUCTION:
The most significant part of the project is the rhythm, judgment area in processing, how and when to send and receive value with Arduino, and the coding of the Neopixel part on Arduino.
The most important part of the music game is the rhythm, I originally intended to calculate the BPM of each song and then write a for loop to achieve automatic rhythm recording, so that the beat points can be very accurate. But the beat points are exactly the same, which may be too simple and dull. So we directly input the rhythm manually (use “println” to record the beat), which may lead to some errors in the rhythm, but can make beat points variable.
The screen part is the same as our previous idea, there is a red heart in the middle of the screen protected by a white circle, and four stars will appear from the four corners, when the stars touch the white line a moment, the player should press the button to protect the red heart.
Then came the judgment part, because when playing music in Processing, it will have a delay, and there is no convenient way to measure the delay. So I used the most direct method – listening to the music and looking at the beat point (when the star hit the white line) to manually adjust the delay. In order to improve the game experience, I wrote the judgment interval as one second, that is, if the player presses the button within one second before or after the beat point, it will be judged as a success, if pressed outside of one second, it will be judged as a failure. I first completed the Processing part, using the keyboard instead of buttons, and this part can already be played as a tiny rhythm game.
Then came the harder part: how to connect Processing with Arduino? Because the star has four directions, so correspondingly, we used four buttons. First, before each star hits the white line, we have to send a signal to the Arduino to make the corresponding button glow as a reminder to the player; second, if a button in the same direction as the star is pressed, the Arduino has to send a message to Processing, and this message will be put into the judgment interval to determine whether the player scores. I defined a lot of arrays to complete this part and finally succeed.
Then it occurred to us that if we let players who are not familiar with the operation and have never played a game before, it is difficult to play the game by looking at the computer screen and not at the buttons. So we wanted to make the LED board in the middle of the button synchronized with the display of the stars on the computer screen (only the direction of movement is opposite because the LED moves from the center to the button). This step is the most painful because I have not touched the Neopixel 8*8 board before at all. We first measured the LED numbering order, and then add some basic changes, such as blink, and fade. But I’m not clear on how to make the LED and the star make a synchronous movement. With the help of the professor and my friend, we listed 20 cases to divide the situation and used “case” and “break” to control the movement of the LEDs (this step took a long time). After many attempts, we basically finished the Arduino part of the programming. But we still have some problems. The first problem is in the Processing part, we use a red cross as a reminder to the player that he does not press accurately in the judgment interval, the red cross is completely controlled by buttons but it sometimes appears when we even haven’t connected the button. We have no idea but to delete the cross. Another problem is the Neopixel board, our code is built on the basis that the next LED will glow only after the previous one finishes glowing and goes off, sometimes the LED will somehow get stuck in one place and not move, causing the whole LED board to go wrong.
After testing countless times (very painful), I came up with the final solution: reduce the beat points (not too many beat points in the Processing, otherwise the Arduino part will have problems); limit the song length. We also wanted to let the board make other instructions in the whistle part (such as changing the LED color), but we couldn’t do it due to coding problems. However, the final work is nice (to me).
The circuit:
The codes: (Arduino)
#include <FastLED.h>
#include "SerialRecord.h"
#define NUM_LEDS 64 // How many leds in your strip?
#define DATA_PIN 10
int state[83];
CRGB leds[NUM_LEDS]; // Define the array of ledsint prems[83];
int direction[83];
int num = 0;
int start;
int pre_state1 = LOW;
int pre_state2 = LOW;
int pre_state3 = LOW;
int pre_state4 = LOW;
int buttonState1 = 0;
int buttonState2 = 0;
int buttonState3 = 0;
int buttonState4 = 0;
SerialRecord reader(1);
SerialRecord writer(1);
//button a-d:led button1-4:switchint buttona = 2;
int buttonb = 3;
int buttonc = 4;
int buttond = 5;
int button1 = 6;
int button2 = 7;
int button3 = 8;
int button4 = 9;
voidsetup() {
Serial.begin(115200);
pinMode(button1, INPUT_PULLUP); //switch
pinMode(button2, INPUT_PULLUP);
pinMode(button3, INPUT_PULLUP);
pinMode(button4, INPUT_PULLUP);
pinMode(buttona, OUTPUT); //led
pinMode(buttonb, OUTPUT);
pinMode(buttonc, OUTPUT);
pinMode(buttond, OUTPUT);
FastLED.addLeds<NEOPIXEL, DATA_PIN>(leds, NUM_LEDS); // GRB ordering is assumedfor (int i = 0; i < 63; i++) {
leds[i] = CRGB::Black;
}
FastLED.show();
for (int i = 0; i < 83; i++) {
prems[i] = -10000;
}
}
voidloop() {
buttonState1 = digitalRead(button1);
buttonState2 = digitalRead(button2);
buttonState3 = digitalRead(button3);
buttonState4 = digitalRead(button4);
if (!buttonState1 == HIGH && buttonState1 != pre_state1) { //if button pressed, send value to processing
writer[0] = 1;
writer.send();
}
if (!buttonState2 == HIGH && buttonState2 != pre_state2) {
writer[0] = 2;
writer.send();
}
if (!buttonState3 == HIGH && buttonState3 != pre_state3) {
writer[0] = 3;
writer.send();
}
if (!buttonState4 == HIGH && buttonState4 != pre_state4) {
writer[0] = 4;
writer.send();
}
pre_state1 = buttonState1;
pre_state2 = buttonState2;
pre_state3 = buttonState3;
pre_state4 = buttonState4;
if (reader.read()) { // if receive value, LED light up accordinglyif (reader[0] == 1) {
digitalWrite(2, HIGH);
}
if (reader[0] == 2) {
digitalWrite(3, HIGH);
}
if (reader[0] == 3) {
digitalWrite(4, HIGH);
}
if (reader[0] == 4) {
digitalWrite(5, HIGH);
}
if (reader[0] == 5) { // when the ball reach center, all LED put out
digitalWrite(2, LOW);
digitalWrite(3, LOW);
digitalWrite(4, LOW);
digitalWrite(5, LOW);
}
if (reader[0] != 5) {
prems[num] = millis();
Serial.println(num);
//void PinReaderMap[] = {0, 0,1,3,2}if (reader[0] == 1) {
direction[num] = 0;
}
if (reader[0] == 2) {
direction[num] = 1;
}
if (reader[0] == 3) {
direction[num] = 3;
}
if (reader[0] == 4) {
direction[num] = 2;
}
num++;
}
}
int now = millis();
for (int i = 0; i < 83; i++) { // to make neopixel 8*8 work with the screenif (now - prems[i] >= 0 && now - prems[i] < 150 && state[i] == 0) {
// Serial.println("?");switch (direction[i]) {
case 0:
{
leds[35] = CRGB::Red;
FastLED.show();
break;
}
case 1:
{
leds[36] = CRGB::Red;
FastLED.show();
break;
}
case 2:
{
leds[27] = CRGB::Blue;
FastLED.show();
break;
}
case 3:
{
leds[28] = CRGB::Blue;
FastLED.show();
break;
}
}
state[i]++;
} elseif (now - prems[i] >= 150 && now - prems[i] < 300 && state[i] == 1) {
switch (direction[i]) {
case 0:
{
leds[35] = CRGB::Black;
leds[45] = CRGB::Red;
FastLED.show();
break;
}
case 1:
{
leds[36] = CRGB::Black;
leds[42] = CRGB::Red;
FastLED.show();
break;
}
case 2:
{
leds[27] = CRGB::Black;
leds[21] = CRGB::Blue;
FastLED.show();
break;
}
case 3:
{
leds[28] = CRGB::Black;
leds[18] = CRGB::Blue;
FastLED.show();
break;
}
}
state[i]++;
} elseif (now - prems[i] >= 300 && now - prems[i] < 450 && state[i] == 2) {
switch (direction[i]) {
case 0:
{
leds[45] = CRGB::Black;
leds[49] = CRGB::Red;
FastLED.show();
break;
}
case 1:
{
leds[42] = CRGB::Black;
leds[54] = CRGB::Red;
FastLED.show();
break;
}
case 2:
{
leds[21] = CRGB::Black;
leds[9] = CRGB::Blue;
FastLED.show();
break;
}
case 3:
{
leds[18] = CRGB::Black;
leds[14] = CRGB::Blue;
FastLED.show();
break;
}
}
state[i]++;
} elseif (now - prems[i] >= 450 && now - prems[i] < 600 && state[i] == 3) {
switch (direction[i]) {
case 0:
{
leds[49] = CRGB::Black;
leds[63] = CRGB::Red;
FastLED.show();
break;
}
case 1:
{
leds[54] = CRGB::Black;
leds[56] = CRGB::Red;
FastLED.show();
break;
}
case 2:
{
leds[9] = CRGB::Black;
leds[7] = CRGB::Blue;
FastLED.show();
break;
}
case 3:
{
leds[14] = CRGB::Black;
leds[0] = CRGB::Blue;
FastLED.show();
break;
}
}
state[i]++;
} elseif (now - prems[i] >= 600 && state[i] == 4) {
switch (direction[i]) {
case 0:
{
leds[63] = CRGB::Black;
FastLED.show();
break;
}
case 1:
{
leds[56] = CRGB::Black;
FastLED.show();
break;
}
case 2:
{
leds[7] = CRGB::Black;
FastLED.show();
break;
}
case 3:
{
leds[0] = CRGB::Black;
FastLED.show();
break;
}
}
state[i]++;
}
}
}
(Processing)
Instruction:
import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;
Serial serialPort;
SerialRecord serialRecord;
SoundFile sound;
boolean[] flag=newboolean[18];
boolean[] flaga=newboolean[18];
boolean[] flagb=newboolean[18];
boolean[] flagc=newboolean[18];
boolean[] sendstate=newboolean[18];
int playStart;
boolean success=false;
int premil=0;
int value=0;
int num=0;
int[]beats= {
752,
863,
979,
1090,
1486,
1543,
1598,
1653,
2337,
2506,
2677,
3004,
3059,
3116,
3172,
3229,
3284,
3339,
};
int[] xpos=newint[18];
int[] ypos=newint[18];
int x;
int y;
int[] ball=newint[18];
int a=0;
void star(float x1, float y1, float radius1, float radius2, int npoints) { //ball=startfloat angle = TWO_PI / npoints;
float halfAngle = angle/2.0;
beginShape();
for (float a = 0; a < TWO_PI; a += angle) {
float sx = x1 + cos(a) * radius2;
float sy = y1 + sin(a) * radius2;
vertex(sx, sy);
sx = x1 + cos(a+halfAngle) * radius1;
sy = y1 + sin(a+halfAngle) * radius1;
vertex(sx, sy);
}
endShape(CLOSE);
}
voidkeyPressed() {
println(frameCount+",");
//println(millis());
}
void drawBigHeart() { // if player press the button correctly, the heart becomes bigheartfill(255, 0, 0);
rectMode(CENTER);
rect(height/2, width/2, 10, 10);
rect(height/2-10, width/2, 10, 10);
rect(height/2+10, width/2, 10, 10);
rect(height/2-20, width/2, 10, 10);
rect(height/2+20, width/2, 10, 10);
rect(height/2-30, width/2, 10, 10);
rect(height/2+30, width/2, 10, 10);
rect(height/2+10, width/2-10, 10, 10);
rect(height/2-10, width/2-10, 10, 10);
rect(height/2+20, width/2-10, 10, 10);
rect(height/2-20, width/2-10, 10, 10);
rect(height/2-15, width/2-20, 10, 10);
rect(height/2+15, width/2-20, 10, 10);
rect(height/2, width/2+10, 10, 10);
rect(height/2-10, width/2+10, 10, 10);
rect(height/2+10, width/2+10, 10, 10);
rect(height/2-20, width/2+10, 10, 10);
rect(height/2+20, width/2+10, 10, 10);
rect(height/2, width/2+20, 10, 10);
rect(height/2-10, width/2+20, 10, 10);
rect(height/2+10, width/2+20, 10, 10);
rect(height/2, width/2+30, 10, 10);
}
void drawHeart() { // normal heartfill(255, 0, 0);
rect(height/2, width/2, 10, 10);
rect(height/2-10, width/2, 10, 10);
rect(height/2+10, width/2, 10, 10);
rect(height/2-20, width/2, 10, 10);
rect(height/2+20, width/2, 10, 10);
rect(height/2+10, width/2-10, 10, 10);
rect(height/2-10, width/2-10, 10, 10);
rect(height/2, width/2+10, 10, 10);
rect(height/2-10, width/2+10, 10, 10);
rect(height/2+10, width/2+10, 10, 10);
rect(height/2, width/2+20, 10, 10);
}
voidsetup() {
size(800, 800);
frameRate(60);
delay(1000);
sound = new SoundFile(this, "g.mp3");
playStart = millis();
sound.play();
noStroke(); // the texttextSize(30);
textAlign(CENTER, CENTER);
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 115200);
serialRecord = new SerialRecord(this, serialPort, 1);
for (int w=0; w<18; w++) {
sendstate[w]=true;
}
}
void key(int k) { //key is controlled by buttons//println((frameCount - playStart)+",");for (int i=0; i<18; i++) { //judgementif (abs(frameCount-beats[i])<10&&ball[i]==k&&!flaga[i]) {
success=true; //if press button correctly, the heart become bigger//println("success=true");
premil=millis();
a=a+1;
flaga[i]=true;
}
}
}
voiddraw() {
background(0);
fill(255, 100);
ellipse(400, 400, 180, 180);
fill(0);
ellipse(400, 400, 160, 160);
rectMode(CENTER);
drawHeart(); //heartif (serialRecord.read()) {
value = serialRecord.get();
key(value);
//println(frameCount);
}
if (success==true) {
if (millis()-premil<300) {
fill(255);
ellipse(400, 400, 180, 180);
fill(0);
ellipse(400, 400, 160, 160);
drawBigHeart();
} else {
success=false;
}
} else drawHeart();
int now = millis() - playStart;
push();
if (now > 2000 && now < 5000) {
clear();
textSize(96);
fill(255, 255, 255);
text("instruction", height/2, width/2);
}
pop();
if (now > 5000 && now < 8783) {
clear();
fill(255, 255, 255);
text("Pressed Button when the star hits White Line", height/2, width/2);
}
if (now > 20066 && now < 23850) {
clear();
fill(255, 255, 255);
text("You can also look at the pixel LED in the middle of the box", height/2, width/2-32);
text("LEDs and the stars are moving in sync", height/2, width/2);
text("Press button when LED Reach the Edge", height/2, width/2+32);
}
if (now > 29400 && now < 34500) {
clear();
fill(255, 255, 255);
text("If you press correctly,", height/2, width/2-32);
text("the heart becomes bigger after each pressing", height/2, width/2);
text("Your score is displayed in the top left corner", height/2, width/2+32);
}
if (now > 32500) {
fill(255);
text("Your score is:", 125, 65);
text(a, 100, 100);
}
if (now >34500 && now < 37800) {
clear();
fill(255, 255, 255);
text("If the star is", height/2, width/2-32);
fill(#FFF534);
text("YELLOW", height/2, width/2);
fill(255);
text("Use Whistle and Press Button at the same time! ", height/2, width/2+32);
}
push();
if (now>46533&&now<48350) {
clear();
fill(255);
textSize(96);
text(" ENJOY PLAYING !", height/2, width/2);
}
pop();
if(frameCount>3500){
clear();
fill(255);
text("This is the end of the tutorial section",height/2, width/2);
}
//ballint cfc=frameCount;
for (int i=max(0, num-2); i<18; i++) {
if (cfc-beats[i]>-31&&!flag[i]) {
fill(255);
if (abs(cfc-(beats[i]-31))<2) {
xpos[i]=800*floor(random(0, 2));
ypos[i]=800*floor(random(0, 2));
if (xpos[i]==0&&ypos[i]==0) {
ball[i]=1;
if (sendstate[i]==true) {
serialRecord.values[0] = 1;
serialRecord.send();
//println("1");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==800&&ypos[i]==0) {
ball[i]=2;
if (sendstate[i]==true) {
serialRecord.values[0] = 2;
serialRecord.send();
//println("2");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==0&&ypos[i]==800) {
ball[i]=3;
if (sendstate[i]==true) {
serialRecord.values[0] = 3;
serialRecord.send();
//println("3");
sendstate[i]=false;
num++;
}
}
if (xpos[i]==800&&ypos[i]==800) {
ball[i]=4;
if (sendstate[i]==true) {
serialRecord.values[0] = 4;
serialRecord.send();
//println("4");
sendstate[i]=false;
num++;
}
}
}
if (xpos[i]<400 && ypos[i]<400) {
xpos[i] = xpos[i] + 10;
ypos[i]= ypos[i] + 10;
}
if (xpos[i]>400&&ypos[i]<400) {
xpos[i] = xpos[i]-10;
ypos[i]=ypos[i]+10;
}
if (xpos[i]>400&&ypos[i]>400) {
xpos[i] = xpos[i]-10;
ypos[i]=ypos[i]-10;
}
if (xpos[i]<400&&ypos[i]>400) {
xpos[i] = xpos[i]+10;
ypos[i]=ypos[i]-10;
}
noStroke();
if (ypos[i]<=400) {
fill(255, 0, 0);
}
if (ypos[i]>=400) {
fill(#30CEFC);
}
if (2300<cfc&&cfc<2700) {
fill(#FFF534);
//serialRecord.values[0]=6;// serialRecord.send();//flagb[i]=true;
}
star(xpos[i], ypos[i], 40, 20, 5);
if (xpos[i]==400&&ypos[i]==400) {
flag[i]=true;
serialRecord.values[0] = 5;
serialRecord.send();
}
}
}
}
Normal:
Wendy did the Neopixel strip part. In this part, the Neopixel strip changes its color according to the loudness of the music. We also add a microphone function code in the Processing to sense and analyze the frequency of the whistle, so that every time we blow the whistle, the LEDs will change their color obviously, which looks very cool.
The circuit:
The code: (Arduino)
#include "SerialRecord.h"
#include <FastLED.h>
#define NUM_LEDS 60 // How many leds in your strip?
#define DATA_PINA 3
#define DATA_PINB 4
#define DATA_PINC 5
// Which pin are you connecting Arduino to Data In?
CRGB leds[NUM_LEDS];
// Change this number to the number of values you want to receive
SerialRecord reader(4);
voidsetup ()
{
Serial.begin(9600);
FastLED.addLeds<NEOPIXEL, DATA_PINA>(leds, NUM_LEDS); // Initialize
FastLED.addLeds<NEOPIXEL, DATA_PINB>(leds, NUM_LEDS);
FastLED.addLeds<NEOPIXEL, DATA_PINC>(leds, NUM_LEDS);
FastLED.setBrightness(10); // BEWARE: external power for full (255)
}
voidloop() {
if (reader.read()) {
int n = reader[0];
int r = reader[1];
int g = reader[2];
int b = reader[3];
leds[reader[0]] = CRGB(reader[1], reader[2], reader[3]); // Prepare the color information using CRGB( Red, Green, Blue
FastLED.show(); // Pass the information of color to the LED
}
}
(The processing: For different songs, we just change the music name)
import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;
import processing.sound.*;
// declare an AudioIn object
AudioIn microphone;
// declare an Frequency analysis object to detect the frequencies in a sound
FFT freqAnalysis;
// declare a variable for the amount of frequencies to analyze// should be a multiple of 64 for best resultsint frequencies = 1024;
// Define the frequencies wanted for our visualization. Above that treshold frequencies are rarely atteigned and stay flat.int freqWanted = 128;
// declare an array to store the frequency anlysis results infloat[] spectrum = newfloat[freqWanted];
// Declare a drawing variable for calculating the width of thefloat circleWidth;
float a;
Serial serialPort;
SerialRecord serialRecord;
SoundFile sample;
Amplitude analysis;
int NUM = 60; //amount of pixelsint[] r = newint[NUM]; //red of each tileint[] g = newint[NUM]; //red of each tileint[] b = newint[NUM]; //red of each tileint startTime;
voidsetup() {
fullScreen();
// load and play a sound file in a loop
sample = new SoundFile(this, "t.wav");
sample.loop();
// create the Amplitude analysis object
analysis = new Amplitude(this);
// analyze the playing sound file
analysis.input(sample);
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 9600);
serialRecord = new SerialRecord(this, serialPort, 4);
serialRecord.logToCanvas(false);
rectMode(CENTER);
startTime = millis();
circleWidth = width/float(freqWanted);
// create the AudioIn object and select the mic as the input stream
microphone = new AudioIn(this, 0);
// start the mic input without routing it to the speakers
microphone.start();
// create the Frequency analysis object and tell it how many frequencies to analyze
freqAnalysis = new FFT(this, frequencies);
// use the microphone as the input for the analysis
freqAnalysis.input(microphone);
}
voiddraw() {
println(analysis.analyze());
long t = millis() - startTime;
if (t < 1500) {
background(125, 255, 125);
} elseif ((t>=1500) && t<3000) {
background(255);
} else {
background(55);
}
noStroke();
fill(255, 0, 150);
// analyze the audio for its volume levelfloat volume = analysis.analyze();
// volume is a number between 0.0 and 1.0// map the volume value to a useful scalefloat diameter = map(volume, 0, 1, 0, width);
float light = map(volume, 0, 1, 0, 255);
//int n = int(random(60));
a = random(0, 255);
int n = frameCount % 60;
r[n] = floor(light/2);
g[n] = floor(255-light);
b[n] = floor(a);
serialRecord.values[0] = n; // which pixel we change (0-59)
serialRecord.values[1] = r[n]; // how much red (0-255)
serialRecord.values[2] = g[n]; // how much green (0-255)
serialRecord.values[3] = b[n]; // how much blue (0-255)
serialRecord.send();
// draw a circle based on the microphone amplitude (volume)circle(width/2, height/2, diameter);
// analyze the frequency spectrum and store it in the array
freqAnalysis.analyze(spectrum);
//printArray(spectrum);background(0);
fill(200, 0, 100, 150);
noStroke();
float loudest = 0;
int loudestIdx = 0;
for (int i=0; i<freqWanted; i++) {
if (spectrum[i] > loudest) {
loudest = spectrum[i];
loudestIdx = i;
}
}
if ((loudestIdx == 19 || loudestIdx == 20) && loudest > 0.02) {
background(255,255,255);
int n1 = frameCount % 60;
r[n1] = floor(255);
g[n1] = floor(255);
b[n1] = floor(255);
serialRecord.values[0] = n1; // which pixel we change (0-59)
serialRecord.values[1] = r[n1]; // how much red (0-255)
serialRecord.values[2] = g[n1]; // how much green (0-255)
serialRecord.values[3] = b[n1]; // how much blue (0-255)// how much blue (0-255)
serialRecord.send(); // send it!
}
if (loudest > 0.01) {
println(loudestIdx + " @ " + loudest);
}
// draw circles based on the values stored in the spectrum array// adjust the scale variable as neededfloat scale = 600;
for (int i=0; i<freqWanted; i++) {
circle(i*circleWidth, height/2, spectrum[i]*scale);
}
}
The music we use:
(The music of the harder part is “.wav” and cannot be attached)
The video:
D. CONCLUSIONS:
The goal of our project is to make an interactive rhythm game. I think our project has achieved this goal from the player’s playing process and feedback. The player presses the button according to the rhythm of the music and can adjust their own speed according to the output (whether they get a score/hit correctly), which constitutes a kind of interaction. And my previous definition of interaction is between the user and artifact, the user gives an input to the artifact and receives an output from the artifact, and this output can change the behavior of the user. So this project aligns with my definition of interaction.
If I have more time, I will find a better way to program the Neopixel board, which can run more LED movements at the same time, and also make more changes, such as LED color changes, making specified patterns, etc.
The programming part of this project was difficult, but I learned more programming languages, such as boolean, using defining a flag to make a true/false judgment for start/stop a loop, and I also explored and learned some Neopixel usage by myself. In fact, in the beginning, I couldn’t even imagine that I could make a rhythm game by myself. From this project, although it took a lot of time and effort, I gained a lot, and I feel a sense of accomplishment when I look at the final work! The code we finished has some versatility, if you want to add another song to play, you can just change the beat point and play it. I also understood and learned some new functions of Processing more deeply, and learned how to make the Processing and Arduino deliver value and cooperate with each other. And through this project, I also have a deeper understanding of the rhythm game, I understand some of its logic and the way it is written, and also broaden my understanding of interaction.
In this recitation class, I used a potentiometer to build a circuit and sent its value to processing to control the speed of the video.
I connected one side of the potentiometer to pin2 and connected the other side to 5V and ground. And I use the example of “Send single value” to send the value of the potentiometer to processing. In the processing part, I use “map” and “myMovie.speed” to control the speed of the video. When the value of the potentiometer is smaller, the speed of the video will be slower; when the value is bigger, the speed will become faster.
Cuttle is almost a new thing for me. I followed the instructions given, doing the work of “student A” and used various functions like “Round corners” and “Boolean Difference” to make a sketch of the cutting of the wooden board.
Step 1.2: Pattern design
What’s unlucky is that I forgot how to draw the curve. I tried by myself many times and watch the video many times. And finally, I found out that I should drag the line instead of just “clicking”. Using rotational repeat and outline stroke, I drew my own pattern.
Step 2: Laser cut
We went to fabrication lab 823 to do laser cutting. Laser cutting is really amazing!
Genre: Music.
We searched for some projects about rhythm games (Reference: https://en.wikipedia.org/wiki/Rhythm_game). For example, arcade rhythm games: Dance Revolution, and mobile rhythm games: osu! Combining what have researched, we decided to make a rhythm game that not only requires the player to catch the pace of the music but also the player needs quick react and make movements accordingly (press the button and blow the whistle with the prop). The challenge for us is to figure out how to write and calculate the range of judgment (the input from the player can only be recognized in the range of judgment time) because we should leave some time for players to react. And we also should consider the player’s game experience, like how to make the instruction more obvious, and we need to figure out how much time should the indicator light flash in advance.
C. PROJECT PROPOSAL PLAN
During the first week, we are going to finish the external device connected to the Arduino. We need to set four small buttons on a large cardboard, install an LED board in the middle, and put 4 Neopixel Strips around it. We will select a piece of music with lyrics, and select and record part of the rhythm because it is very difficult to get the computer to analyze the music directly. We would then artificially program these rhythms into the computer, and four different Neopixel Strips would light up with the rhythm. When a specific Neopixel lights up, the player needs to press the corresponding button. We will pick a specific lyric in the song, and when the word is sung, the middle LED will light up, at this time, the player needs to blow the whistle with the prop.
In the second week, we will finish the code for Processing. When a specific light band is lit up, the computer screen will have a small ball in the corresponding direction smashing into the stickman in the middle. When the player presses the button, he will make the ball bounce, and if the player does not react in time, then a big red cross will appear on the whole screen, and then all the light strips will turn red together.
At last, to make the game more interactive, we hope to add a sound sensor again, when a specific word appears, there will be small balls in all four directions to smash people, so players need to whistle, at this time the sound sensor will receive the signal, and the stickman will be safe.
D. CONTEXT AND SIGNIFICANCE
The preparatory research of rhythm games shows a great combination of the music and the movements of the players and inspired us. So we decided to put this feature of rhythm game into our project, but change its form of expression and add some interactive elements like blowing the whisper into the game. The different instructions (press buttons and blow whistle) require players to think and react quickly, and the different inputs from players will create various outputs. If pressing the button correctly, the stickman will be safe, if miss the opportunity to press the button or whistle, the stickman will be hit by the ball. The Player will adjust his pace according to the output (the flash of LEDs). So the game aligns with my definition of interactive. In the processing part, the game was presented in the form of a mini-game to the audience, to make the player’s action visualized to the audience, and make the game more interesting.
Using the code from the example: Send Multiple Values and Receive Multiple Values, and building the circuit of the two potentiometers, I could draw an Etch-A-Sketch by twisting the potentiometers. After the code for drawing circles worked, I modified circles into the line and add “previous x” and “previous y”, using “previous x=x” and “previous y=y” to keep the previous tracks of lines.
In this Etch-A-Sketch, one potentiometer controls the circle’s x-axis movement, and the other controls the circle’s y-axis movement. It’s convenient to draw straight lines, but difficult to draw curves because it is hard to control. So I just wrote a “Hello” with the mouse.
The videos:
The code: (Arduino part)
#include "SerialRecord.h"// Change this number to send a different number of values
SerialRecord writer(2);
voidsetup() {
Serial.begin(9600);
}
voidloop() {
int Value1 = analogRead(A0);
int Value2 = analogRead(A1);
writer[0] = Value1;
writer[1] = Value2;
writer.send();
// This delay slows down the loop, so that it runs less frequently. This// prevents it from sending data faster than a Processing sketch that runs at// 60 frames per second will process it. It also makes it easier to debug the// sketch, because values are received at a slower rate.delay(20);
}
(Processing part:)
import processing.serial.*;
import osteele.processing.SerialRecord.*;
float prex;
float prey;
Serial serialPort;
SerialRecord serialRecord;
voidsetup() {
size(500, 500);
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 9600);
// If the Arduino sketch sends a different number of values, modify the number// `2` on the next line to match the number of values that it sends.
serialRecord = new SerialRecord(this, serialPort, 2);
size(640, 360);
background(102);
}
voiddraw() {
serialRecord.read();
int value1 = serialRecord.values[0];
int value2 = serialRecord.values[1];
float x = map(value1, 0, 1024, 0, width);
float y = map(value2, 0, 1024, 0, height);
circle(x, y, 20);
stroke(255);
//line(x, y, prex, prey);
prex=x;
prey=y;
}
Task #2:
My partner Jinnuo (Jim) Liu took charge of the processing part, which should make the ball “bounce” and send values when the ball reaches the edge of the screen. I wrote Arduino code and built the circuit with two servos. My thought is when the ball hits the edge and sends a value to Arduino, one servo will work (turn to 120 degrees and then turn back), and when the ball reaches another edge, another value will be sent to make another servo work. When we first combined our code and circuit, only one servo worked, then we changed some parts of the code. We put “else” between the servo turning to 120 degrees and turning to 60 degrees. After a few changes, the servo did work with the ball, but it seems like having a delay between them.
import processing.serial.*;
import osteele.processing.SerialRecord.*;
Serial serialPort;
SerialRecord serialRecord;
int x=75;
int direction=20;
int radius=75;
voidsetup() {
fullScreen();
background(0);
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 9600);
serialRecord = new SerialRecord(this, serialPort, 2);
ellipseMode(CENTER);
}
voiddraw() {
fill(0);
rect(0, 0, width, height);
noStroke();
// display instructionsfill(255);
ellipse(x, height/2, 2*radius, 2*radius);
if (x<radius||x>width-radius) {
direction*=-1;
}
x=x+direction;
if (x<radius) {
serialRecord.values[0] = 1;
} else serialRecord.values[0] = 0;
if (x>width-radius)
{
serialRecord.values[1] = 1;
} else serialRecord.values[1] = 0;
serialRecord.send();
}
We struggled for a long time to find ways to reduce the delay between the ball hitting the edge and the servo turns. With the help of professors, we changed the code to decrease the frequency that Processing sent signals to Arduino. And used “if(reader. read())” instead of “reader. read()”. Also, we deleted the “else” in the Arduino code to directly let the servo turn back. Finally, the servos worked as expected.
1. The Snake
Genre: Game
Snake is a video game genre where the player maneuvers a line that grows bigger or longer after eating something, typically apples, making the snake a primary obstacle to itself. The player controls a dot, square, or object on a bordered plane. As it moves forward, it leaves a trail behind, resembling a moving snake. In some games, the end of the trail is in a fixed position, so the snake continually gets longer as it moves. (Wikipedia, https://en.wikipedia.org/wiki/Snake_(video_game_genre))
I want to simplify the map and add some music functions to this game. Using 4 NeoPixel strips to build a square frame where the snake (made of LED lights) can automatically travel through this frame. The snake moves with the pace of the music. There will be apples randomly appearing in the frame, represented as a blinking LED. Different from the original game, players should press the button when the snake comes across the apple as “eat the apple”, else, the snake will bump into the apple and lose its length. It’s a game that exercises the player’s reaction speed because the apple will appear randomly and unexpectedly, and players also have to press the button according to the rhythm of the music. These functions will make the game more interactive. The application of led adds a sense of technology to the whole game. I also want to use processing to make a thumbnail image which let the player see the location of the snake more clearly, and the screen can also display some data, such as the length of the snake, the number of apples, etc.
2. Rhythm Game
Genre: Game, Music
Rhythm game or challenges a player’s sense of rhythm. It focuses on dance or the simulated performance of musical instruments and requires players to press buttons in a sequence dictated on the screen. (Wikipedia, https://en.wikipedia.org/wiki/Rhythm_game)
I want to use some sensors to do a rhythm game that requires hand-arm movement, and the player makes the specified action according to the rhythm of the music and the prompt on the screen. For example, I can use a photoresistor to detect the player’s hand position, and arm position and use the tilt switch to input the movement data. When the game begins, there may be some instruction words appearing on the screen, players should move and make gestures with instructions and follow the rhythm. Or, I can just use some buttons, the player should timely and accurately press the button according to the instructions (for example, falling squares to catch, circle to point), and pressing with the pace of the music increase the engagement of players. Imagining a bit more, the game could be designed with two players competing with their speed of reaction and accuracy of the time of pressing buttons, which is more interactive.
3. The painting
After learning how to use a photoresistor to control the patterns presented in processing, I think maybe we can use temperature sensors and photoresistors as well as some other sensors to draw a unique image through the control of light, changing of temperature, and some other data.
For example, if my hand covers the photoresistor, then the image will be darker, and if I move my hand, the image will be lighter. Also, if I put my hand closer to the temperature sensor, the image will add some red and orange color as a “warm” feeling, otherwise, it will show in a cold tone. With the distance between the sensor and users changing, the image will become either vague or clear. If there are sensors that can detect other things like the wind and the humidity, and forms a simulation through processing to control the movement of the image (for example, the image moves and shakes because of the wind, some blue points appear in the image as raindrops when the humidity is high), the work will be more interactive.
In the midterm project, my partner and I designed some new playing methods based on the game “Truth or Dare”. Two players are involved in the game, having a competition on their speed to shake the clapping toy and on their luck. In my opinion, interactive means communication between the artifact and the users, or between users and users. From the article, I know more about interactive and different levels of art: Dynamic-Passive: the simple reaction of the artifact, which its output is completely predictable; Dynamic-Interactive: the interactive artifact that can deliver its output to the users, making the user think and change their input actions; Dynamic-Interactive (Varying): the highest level is that the work is unpredictable and interactive. It will learn experience and adjust its output, which adds more random and interesting functions to the performance.
(Ernest Edmonds, Art, Interaction and Engagement).
I am inspired by the game I played named “Osu!”. It is a great rhythm game, combining the music with clicking, tapping keyboards, and moving the mouse. I like the “osu! mania” mode in this game. In the game, the key notes appear in the form of squares, and the moment the squares hit the line, the player should immediately press the corresponding button to get a “combo” (“perfect”). If the error between the key pressing and the rhythm of the original song is larger, the “perfect” will become “good”, “bad” or “miss” displayed (the error will reduce the accuracy rate and the song score).”The map of the music is then played with accompanying music, simulating a sense of rhythm as the player interacts with the objects to the beat of the music.” (Wikipedia. https://en.wikipedia.org/wiki/Osu!)
When playing osu, I am completely immersed in the music. If I got a “miss” or “bad” mistake, I will immediately adjust my pace, it is interactive. Although this game looks simple compared to other games, this simple style and interactive playing method inspired me. In the final project, I may also try adding some combinations of music and physical control to make the game more interesting.
(The video I recorded myself playing the game.)
The other project is the artifact designed by Venezuelan artist Jesús Rafael Soto (1923 – 2005), made of colorful linear materials, appears to be a monolith, but the interior is free to travel, allowing visitors to enter this magical space, move freely, play, and enjoy as if they could touch the golden sunlight. The combination of linearity forms different shapes visually. The visitors also can travel through it, which is very attractive. If the linear combination can change with people’s movement, and position, it will be more interactive as Dynamic-Interactive. If the artifact can use some algorithmic to learn experiences from the users, and depending on the history of interactions with users, performing and modifying its specification, it will be more interactive and interesting.
