The main idea of this project is to extract elements from the video capturing the user’s reactions when he/she listens to a piece of music, and adjust the music chord and rhythm with the movement elements to make the user feel in control of the body and the music. Technically, our project has two main parts, movement analysis and music generation. After a lot of research, reading, and practice, and with the help of professors, we successfully realized real-time analysis of movement emotion, as well as non-real-time analysis of movement rhythm and strength. In order to realize an integrated project, we finally decided to adopt all the non-real-time results as phased results. In terms of music, we adopted the EC2VAE model to adjust the chord, rhythm, and intensity of the original song. We use emotion as the input for the chord, and the output of visbeat3 model as the input of the rhythm. After inputting the two parameters, we would get a series of lists of numbers as the output music. We use Pygame to play the music flow. Until now, what we can achieve is that by inputting a video of movement with original music, we could get a new piece of music with adjusted chords, rhythm, and volume in accordance with the user’s movement.
This project right now is composed of three parts, emotion from movement, rhythm from movement, and music generation with emotion and rhythm. (These have corresponding connection annotations in the GitHub folder)
In what ways did your understanding of how your users were going to interact with your project inform certain design decisions you made? Account for several such decisions. In addition, what form, material or elements did you use? What criteria did you use to select those materials? Why was it/were they best suited to your project purpose, more than any other one/s? What might have been another option? But you rejected it—why?
Luckily, this project ran on the way of the proposal, though we finally changed our idea to make a larger, digital music box instead of simply use the one I bought online.
The supposed interactions between users and the project are:
First, they can try the sample score and send the paper throught the box by rotating the handle–there is an interation from movement to another motion and music effect and visualization on the Neopixel. Here is a hidden action that users can play a score in four different directions.
Then we will offer some empty scores and some partly finished scores with type and circle paper for them to compose their own notes.
During the User Testing Session,Almost everyone admired our “machine” but they were not active to try punch holes on white paper at that time. While also the connection of switches of notes worked not so well at that time. Besides, we gain much suggestions about the appearence. Therefore, we decided to improve the score making logic and seek for a better way to make 16 switches work better. Moreover, we started our 3D print and laser cutting work to build up a outer box.
FABRICATION AND PRODUCTION:
In this section, describe and assess the most significant steps in your production process, both in terms of failures and successes. Please clarify exactly what contributions and roles you personally made to the process and project along the way, and describe the process your group used to work together. What happened during the User Testing Session? How did your user testing process influence some of your following production decisions? What kind of adaptations did you make? Were they effective? In short, keeping in mind your project goals, how do you account for and justify the various production choices you made for your project? Include sketches and drawings.
[the interaction with notes–switches control]
We diceded to make each switches corresponding to one notes. It means we will have 16 conditions referring 16 different notes. To begin with, also to leave some space for debugging, we first let notes interacted with keyboard after spending much time on the proper and licensed notes series. Therefore, we can change the conditions into swithes’ digital data later.
2. later in the user test, keyboard test was included into a step of user intructions. But we got feedback from Prof. Godfrid to remove it from the main part of user test. That’s pretty important, reminding us of the focus of our project. At least, keyboard test should not be a step of using the project.
[the interaction with notes–switches control]
This is one of the most essential part of the project, including both physical structure designing, making, testing, and revising, and programming logic to interact.
The switches for notes: how to make paper holes be recognized by the project?
Initially, I wanted to use a photosensor to identify holes at different points in a slit — that is, the change from no light to light. However, However, after my test, the accuracy and sensitivity of the photoresistor in kit and the equipment room are relatively low, which cannot meet the our expectations. Because we want to realize that multiple holes on A4 are identified and monitored by different photoresistors. Due to the small width of the paper, the transverse perforation will result in a tight distance between the positions that need to be monitored in the same line. Plus A4 white paper itself is thinner. Ambient flooding between adjacent paper holes affects each other’s photoresistors. Therefore, we were thinking about other methods.
Then the materials in 826 reminded us of the cardboard switch taught to build in the first recitation. We thought we can make a line of seperate tiny Metal tape, and we made one on our prototype by soldering tape with wires.
It’s the left part of the video:
Taking advantage of the slightly curved elasticity of the cardboard, we used that as a support, and started thinking how do we make the other end of this switch–a cover, which was the hardest and most time-consuming part of this project.
I made serveral versions of the cover:
I found it’s hard to give the paper enoght power to make each branch naturally contacting with the every bottom panel well. From my observation, it was because we use cardboard to be the button supporter, and the surface of card board is not flat as we expect. Also, when we use a stick to deliver the power to the side line of this version of cover, the power delivered to the botton panels was not uniform.
And then I tried to wrap solder wire around the stick to make contact with the bottom plate, but failed. There still some note lost when we test it.
Later. just before the user test, I suddenly got inspirations when I wandering about 8F and thought it might be possible to use the soft wires to make brushes.
But rose gold flexible wire still has a certain hardness. So before the final presentation, we remade one with woodboard and the Silver flexible wire.
Finally, with some pressure from my hand this version can work. But without outside pressure, it can not show a senstive reflection of holed. That’s a pity to fail on this part.
[other Other memorable parts]
Instruction paper for User test and final presentation:
3D Printing:
My partner found a original file online and we together change the size, shape, and appearence (the text “LET’S PLAY”) by Tinkercad. Quite interesting to explore this website.
Laser cutting:
As attached in the Annex, all the files were measured and designed by myself, with the instrutions from Prof. Andy. Besides, I got a personal trainning of laser cutting, including the machine setting and file adjusting; also, I gained some basic knowledge and experience with materials and parameters–now I’m “a advanced user” of laser cutting, as my partner said haha.
Design detials:
I designed two physical steps(deviders actually) of the outer box to hold the circuits with arduino UNO and breadboard.
More ideas came up during the process and may be tried in the future: (About the structure of better switches)
CONCLUSIONS:
Begin your conclusion by restating the goals of your project. How do your project results align with your definition of interaction? How do your project results not align with your definition of interaction? Based on both your definition of interaction and your expectations of your audience’s response, how, ultimately, did your audience interact with your project? How would you improve your project if you had more time? What of value have you learned from any setbacks and/or failures? What do you take away from your accomplishments?
The wrap-up of your conclusion delivers your final say on the issues you have raised previously, demonstrates the importance of your ideas, and insightfully synthesizes your thoughts about key elements of the project experience. As such, your conclusion should close by reflecting on the crucial questions “So what?” and “Why should anyone care?”
The original idea was a paper tape punch music box. It allows users to make a physical music by hands, back to the olds and play with the old style of coding, typing, and music making. It’s a reflection of the fast development of technology and people’s laziness caused by it. The interaction is mostly on score editting, playing and machine running with hands.
For the interaction part, users can feel real-time reflection of the project by rotating the handle with mechanisml. They can change the speed of rotating to control the paper speed with observation of the existing speed of the paper. And for the paper score, they can see how sample score and holes on them be interacted with the project(actually the program and computer), so that they can change the position of notes to make different sounds. And one advanced and hidden function is waiting for users to explore, that there is no direction of the score so that they can see a inversed version of a common score; besides, a physical score means not only music but also graphic transformation into a piese of sounds.
The results of the project greatly made people enjoy the old style of pythical score making and machine running. And we heard some reflections of reminding people of the old typing machine or Turing cipher code. Also the internal mechanical structure and interaction attracted people to try this project.
Similar situaction to the user test–making or editting a phisical score takes time and our audience were not so active and patient on making this. We even got a complaint that the student was too lazy to do so and wanna offer more easy way to make a physical score. Actually, I thought it was absurd and ironic to our original goals to bring people back to the physical old styles. But it also made me confirmed that we are doing a right thing and really made some different to make the reflection of laziness and technology-dependence exposed. It somehow led some reflections on our audience, which was expected when we were developing this project. I did regret to make it this way to remind people of the old days!
#include
#include "SerialRecord.h"
#define LED_PIN 12
#define NUM_LEDS 60
#define BRIGHTNESS 64
#define LED_TYPE WS2812
#define COLOR_ORDER GRB
CRGB leds[NUM_LEDS];
#define UPDATES_PER_SECOND 100
// This example shows several ways to set up and use 'palettes' of colors
// with FastLED.
//
// These compact palettes provide an easy way to re-colorize your
// animation on the fly, quickly, easily, and with low overhead.
//
// USING palettes is MUCH simpler in practice than in theory, so first just
// run this sketch, and watch the pretty lights as you then read through
// the code. Although this sketch has eight (or more) different color schemes,
// the entire sketch compiles down to about 6.5K on AVR.
//
// FastLED provides a few pre-configured color palettes, and makes it
// extremely easy to make up your own color schemes with palettes.
//
// Some notes on the more abstract 'theory and practice' of
// FastLED compact palettes are at the bottom of this file.
CRGBPalette16 currentPalette;
TBlendType currentBlending;
extern CRGBPalette16 myRedWhiteBluePalette;
extern const TProgmemPalette16 myRedWhiteBluePalette_p PROGMEM;
SerialRecord writer(16);
int pinValue[16];
int preValue[16];
bool notePlay[16];
void setup() {
Serial.begin(9600);
delay(3000); // power-up safety delay
FastLED.addLeds<LED_TYPE, LED_PIN, COLOR_ORDER>(leds, NUM_LEDS).setCorrection(TypicalLEDStrip);
FastLED.setBrightness(BRIGHTNESS);
for (int i = 0; i < 16; i++) {
preValue[i] = 0;
notePlay[i] = false;
}
currentPalette = RainbowColors_p;
currentBlending = LINEARBLEND;
}
void loop() {
for (int i = 0; i < 16; i++) {
preValue[i] = pinValue[i];
// Serial.print(writer[i]);
}
pinValue[0] = digitalRead(A0);
pinValue[1] = digitalRead(A1);
pinValue[2] = digitalRead(A2);
pinValue[3] = digitalRead(A3);
pinValue[4] = digitalRead(A4);
pinValue[5] = digitalRead(A5);
pinValue[6] = digitalRead(10);
pinValue[7] = digitalRead(11);
pinValue[8] = digitalRead(2);
pinValue[9] = digitalRead(3);
pinValue[10] = digitalRead(4);
pinValue[11] = digitalRead(5);
pinValue[12] = digitalRead(6);
pinValue[13] = digitalRead(7);
pinValue[14] = digitalRead(8);
pinValue[15] = digitalRead(9);
for (int i = 0; i < 16; i++) {
writer[i] = pinValue[i];
// Serial.print(writer[i]);
}
// Serial.println("next");
writer.send();
// This delay slows down the loop, so that it runs less frequently. This can
// make it easier to debug the sketch, because new values are printed at a
// slower rate.
delay(17);
if (pinValue[0] == 1) {
for (int i = 1; i < 16; i++) {
if (pinValue[i] == 1 && preValue[i] == 0) {
notePlay[i] = true;
//int value2 = serialRecord.values[1];
}
}
}
if (pinValue[0] == 0 && preValue[0] == 1) {
for (int i = 1; i < 16; i++) {
notePlay[i] = !notePlay[i];
if (notePlay[i] == true) {
leds[i-1] = CRGB::Yellow;
leds[i+15*2-1] = CRGB::Yellow;
leds[i+15*3-1] = CRGB::Yellow;
leds[i+15*4-1] = CRGB::Yellow;
FastLED.show();
// delay(500);
// Now turn the LED off, then pause
} else {
leds[i-1] = CRGB::Black;
leds[i+15*2-1] = CRGB::Black;
leds[i+15*3-1] = CRGB::Black;
leds[i+15*4-1] = CRGB::Black;
FastLED.show();
// delay(500);
}
}
}
ChangePalettePeriodically();
static uint8_t startIndex = 0;
startIndex = startIndex + 1; /* motion speed */
FillLEDsFromPaletteColors(startIndex);
FastLED.show();
FastLED.delay(1000 / UPDATES_PER_SECOND);
}
void FillLEDsFromPaletteColors(uint8_t colorIndex) {
uint8_t brightness = 255;
for (int i = 0; i < NUM_LEDS; ++i) {
leds[i] = ColorFromPalette(currentPalette, colorIndex, brightness, currentBlending);
colorIndex += 3;
}
}
// There are several different palettes of colors demonstrated here.
//
// FastLED provides several 'preset' palettes: RainbowColors_p, RainbowStripeColors_p,
// OceanColors_p, CloudColors_p, LavaColors_p, ForestColors_p, and PartyColors_p.
//
// Additionally, you can manually define your own color palettes, or you can write
// code that creates color palettes on the fly. All are shown here.
void ChangePalettePeriodically() {
uint8_t secondHand = (millis() / 1000) % 60;
static uint8_t lastSecond = 99;
int isNotesPlaying = 0;
// while
// while isNotesPlaying=0
if (lastSecond != secondHand) {
lastSecond = secondHand;
if (secondHand == 0) {
currentPalette = RainbowColors_p;
currentBlending = LINEARBLEND;
}
if (secondHand == 10) {
currentPalette = RainbowColors_p;
// currentPalette = RainbowStripeColors_p;
currentBlending = NOBLEND;
}
if (secondHand == 15) {
currentPalette = RainbowColors_p;
// currentPalette = RainbowStripeColors_p;
currentBlending = LINEARBLEND;
}
if (secondHand == 20) {
SetupPurpleAndGreenPalette();
currentBlending = LINEARBLEND;
}
if (secondHand == 25) {
// SetupTotallyRandomPalette();
SetupGreenAndBluePalette();
currentBlending = LINEARBLEND;
}
if (secondHand == 30) {
SetupBlackAndWhiteStripedPalette();
currentBlending = NOBLEND;
}
if (secondHand == 35) {
SetupBlackAndWhiteStripedPalette();
currentBlending = LINEARBLEND;
}
if (secondHand == 40) {
currentPalette = CloudColors_p;
currentBlending = LINEARBLEND;
}
if (secondHand == 45) {
currentPalette = PartyColors_p;
currentBlending = LINEARBLEND;
}
if (secondHand == 50) {
currentPalette = myRedWhiteBluePalette_p;
currentBlending = NOBLEND;
}
if (secondHand == 55) {
currentPalette = myRedWhiteBluePalette_p;
currentBlending = LINEARBLEND;
}
}
}
// This function fills the palette with totally random colors.
void SetupTotallyRandomPalette() {
for (int i = 0; i < 16; ++i) {
currentPalette[i] = CHSV(random8(), 255, random8());
}
}
// This function sets up a palette of black and white stripes,
// using code. Since the palette is effectively an array of
// sixteen CRGB colors, the various fill_* functions can be used
// to set them up.
void SetupBlackAndWhiteStripedPalette() {
// 'black out' all 16 palette entries...
fill_solid(currentPalette, 16, CRGB::Black);
// and set every fourth one to white.
currentPalette[0] = CRGB::Green;
currentPalette[4] = CRGB::Green;
currentPalette[8] = CRGB::Green;
currentPalette[12] = CRGB::Green;
}
// This function sets up a palette of purple and green stripes.
void SetupPurpleAndGreenPalette() {
CRGB purple = CHSV(HUE_PURPLE, 255, 255);
CRGB green = CHSV(HUE_GREEN, 255, 255);
CRGB black = CRGB::Black;
currentPalette = CRGBPalette16(
green, green, black, black,
purple, purple, black, black,
green, green, black, black,
purple, purple, black, black);
}
void SetupGreenAndBluePalette() {
CRGB blue = CHSV(HUE_BLUE, 255, 255);
CRGB green = CHSV(HUE_GREEN, 255, 255);
CRGB grey =0x808080;
CRGB black = CRGB::Black;
currentPalette = CRGBPalette16(
blue,grey, green, black,
blue, grey, green, black,
blue, blue, grey, grey,
green, green, black, black);
}
// This example shows how to set up a static color palette
// which is stored in PROGMEM (flash), which is almost always more
// plentiful than RAM. A static PROGMEM palette like this
// takes up 64 bytes of flash.
const TProgmemPalette16 myRedWhiteBluePalette_p PROGMEM = {
CRGB::Red,
CRGB::Gray, // 'white' is too bright compared to red and blue
CRGB::Blue,
CRGB::Black,
CRGB::Red,
CRGB::Gray,
CRGB::Blue,
CRGB::Black,
CRGB::Red,
CRGB::Red,
CRGB::Gray,
CRGB::Gray,
CRGB::Blue,
CRGB::Blue,
CRGB::Black,
CRGB::Black
};
const TProgmemPalette16 myRedWhiteBluePalette_p2 PROGMEM = {
CRGB::Green,
CRGB::Gray, // 'white' is too bright compared to red and blue
CRGB::Blue,
CRGB::Black,
CRGB::Green,
CRGB::Gray,
CRGB::Blue,
CRGB::Black,
CRGB::Green,
CRGB::Green,
CRGB::Gray,
CRGB::Gray,
CRGB::Blue,
CRGB::Blue,
CRGB::Black,
CRGB::Black
};
// Additional notes on FastLED compact palettes:
//
// Normally, in computer graphics, the palette (or "color lookup table")
// has 256 entries, each containing a specific 24-bit RGB color. You can then
// index into the color palette using a simple 8-bit (one byte) value.
// A 256-entry color palette takes up 768 bytes of RAM, which on Arduino
// is quite possibly "too many" bytes.
//
// FastLED does offer traditional 256-element palettes, for setups that
// can afford the 768-byte cost in RAM.
//
// However, FastLED also offers a compact alternative. FastLED offers
// palettes that store 16 distinct entries, but can be accessed AS IF
// they actually have 256 entries; this is accomplished by interpolating
// between the 16 explicit entries to create fifteen intermediate palette
// entries between each pair.
//
// So for example, if you set the first two explicit entries of a compact
// palette to Green (0,255,0) and Blue (0,0,255), and then retrieved
// the first sixteen entries from the virtual palette (of 256), you'd get
// Green, followed by a smooth gradient from green-to-blue, and then Blue.
Sources:
The 16 notes of music box was recorded by Folkman on the wibsite of Free Sounds.
“I recorded 16 notes from a music box. I waited for the motor to loose juice then I wound it forward manually so I could record each separate note chromatically. There are two octaves plus one note. Begins with D and climbs up the scale ending with E. ” described by Folkman!
For this part, the circuit will sent two parameter to the port. One is analogRead() from Sliding rheostat, which will later control the process of the video like a slidebar! The other one is digitalRead() from the bottun, sending 1 or 0 to change the speed level.
/*
SendMultipleValues
Reads an analog input on pin 0, and sends a record that contains two values:
1. Value of `millis()`, modulo 1024.
2. The analog value that is read from pin 0.
If you attach a potentiometer to pin 0, you can control this value by moving
the pot.
This sketch pairs well with the RecieveMultipleValues example from the
Processing SerialRecord library
<https://osteele.github.io/Processing_SerialRecord/>.
Things to try:
- Connect a second potentiometer to the Arduino, and send the values from both
potentiometers, instead of send a value that is based on `millis()`.
- Send the value from another sensor, such as temperature or proximity.
by Oliver Steele, 2020-2022
This example code is in the public domain.
*/
#include "SerialRecord.h"
// Change this number to send a different number of values
SerialRecord writer(2);
void setup() {
Serial.begin(9600);
}
void loop() {
int sensorValue1 = analogRead(A0);
int sensorValue2 = digitalRead(A1);
writer[1] = sensorValue2;
writer[0] = sensorValue1;
writer.send();
// This delay slows down the loop, so that it runs less frequently. This can
// make it easier to debug the sketch, because new values are printed at a
// slower rate.
delay(10);
}
Processing:
Here is a typical bug I met before. But last time, I focused too much on debugging, so I forgot to record this problem. For the bug, I remember the first time I run the given sample code of “ReceiceMultipleValues”, I did no change on the code but it failed to run. I also failed to find a brief and proper solution by googling.
Therefore, I tried to study the instructions given by Processing itself. When I slided up to the head of console information, I found hints that Processing use [0] [1][2] with the name of ports. Then it suddenly dawned on me that since there are more than one serial ports connected with my laptop, they are automatically grouped as an array. The extra thing I have to do is add the index of the port I want to connect into the code.
Processing code: (the video comes from https://pixabay.com/zh/videos/ )
/**
* Example sketch for the SerialRecord library for Processing.
*
* Receives two integers from the serial port, and uses them to control the x
* and y position of a circle on the canvas.
*/
import processing.serial.*;
import osteele.processing.SerialRecord.*;
import processing.video.*;
Movie myMovie;
Serial serialPort;
SerialRecord serialRecord;
int preValue2=1;
float newSpeed=1.0;
int isPlaying=1;
float preValue1=0.0;
void setup() {
size(640, 360);
String serialPortName = SerialUtils.findArduinoPort(2);
serialPort = new Serial(this, serialPortName, 9600);
// If the Arduino sketch sends a different number of values, modify the number
// `2` on the next line to match the number of values that it sends.
serialRecord = new SerialRecord(this, serialPort, 2);
//myMovie = new Movie(this, "Jing'An.mp4");
myMovie = new Movie(this, "dog.mp4");
myMovie.loop();
}
void draw() {
background(0);
serialRecord.read();
int value1 = serialRecord.values[0];
int value2 = serialRecord.values[1];
float progress = map(value1, 0, 1024, 0, 1);
float bar=map(value1, 0, 1024, 0, width);
//float y = map(value2, 0, 1024, 0, height);
//circle(x, y, 20);
if (myMovie.available()) {
myMovie.read();
}
if (value1 != preValue1) {
//myMovie.jump(0);
myMovie.jump(myMovie.duration()*progress);
//myMovie.jump(random(myMovie.duration()));
}
if (value2==1 && preValue2==0) {
newSpeed=(newSpeed+1.0)%2.1+0.5;
}
preValue2=value2;
myMovie.speed(newSpeed);
preValue2=value2;
preValue1=value1;
image(myMovie, 0, 0, width, height);
float rectWidth=10.;
rect(0, height-rectWidth, width, rectWidth);
circle(bar, height-rectWidth, rectWidth);
}
void mousePressed() {
if (isPlaying==1) {
isPlaying=0;
myMovie.stop();
} else {
isPlaying=1;
myMovie.loop();
}
}
Additional:
There is an issue about delay that I failed to find a solution. Hope someone can help me deal with it!
After trying the basic functions of https://cuttle.xyz/dashboard , I found a wonderful pattern — a spiral. So I started from their. I can still remember the first time I saw this pattern, an image of a wing came to my mind. It has three spirals, and I decided to repeat the graphic third times–perhaps it was just a coincidence–hard to tell the feelings of inspirations.
Since we didn’t finish the design part before the end of the recitation, we had to continue it and use the laser cutting later. To be honest, that was not a bad thing–for the first time, we tried the fabrication lab’s website and booked an appointment with a lab assistant. Though we finally failed to find the assistant, we were so luck to get help from prof. Andy.
He also helped us adjust the size and of the graphics we designed. One thing worth mentioning is that, we should keep those inportant parameter of elements in mind, such as the center circle/rectangle’s size–it should be fixed however the size of the graphic changes.
Finally! Amazing work! Like a magic windmill! I love it!
This is a device for users to create or compose a physical melody by themselves. The melody can be a classic/famous one, created by following the given instructions; also, if you have an interest in composition, it’s a good place for you to explore your talent!
To operate the device, first, you need to play a melody on my taplop with keyboard(or can be another physical keyboard made by cardboards and sensors). At this point, you can try and listen to difference combinations of the basic 15 tones(the easiest mucis box inner machine, a good starting point), and then with a certain button, you can add a chord or simple tone into the virtual score on the laptop screen. Second, there will be buttons on the sketch for testing the whole score, testing a certain line/block, printing, and redoing. Once you think you finished composing, just click “test” (think about duplication and saving a file functions, if having enough time); If you are satisfied, click print to print holes on the score type paper. Finally, cut the score you printed and put it into the music box, and then, play it!
PROJECT STATEMENT OF PURPOSE *
The target is to explore and help more people explore amazing music boxes, coming up from my experience of observing music boxes and finding a special type–a handmake score player. After learning much on this type of music box, I notice it would be hard for a person who is not good at music knowledge but quite interested in it–professional players used to compose with professional instruments(electronic mostly) and compose with advanced software. Also, positioning and drilling the physical score takes quite a lot time and energy. What’s more, one of the original idea is use processing and arduino to create mucis, but I’m highly eager to make mechanical engineering(I’m so crazy about this). Therefore, I chosed this musical box and decised to simplify the exprience of using this composition tool and playing music in a classic and physical way.
PROJECT PROPOSAL PLAN *
Before the music box machine I bought from Taobao arrives, I can first start with the processing coding part. For this part, I need to create a skecht with serveral buttons and have a slideable area for score composition. And then I have to think about how to make the computer(or other sounds player) to play tones/chord interacting to keyplaying as a virtual music box. The logic of edit score is another great challenge here.
Then, after materials arrives, fabrication part can start. I need to design containers for music box and printer, leaving enough space for hiding the circuit. One thing should be held in mind is that, I want to make every part of this device movable, which is easier to make further changes or repair. The hardest part here may be how to slot the score paper with the hand punch and how to move it step by step in certain direction and in a little but well-proportioned distance.
Finally, for the player part, the simplist way is just putting it here; then the user can freely sent their printed score into the chachine, and start play it by hand. For further development, I will consider making it into a electronic autoplayer whose speed can be changed by a slider. Besides, I also want to add a visual decration on it, such as a windmill rotating when the music plays.
CONTEXT AND SIGNIFICANCE *
An interesting question：In what way does your project proposal align with your definition of interaction?
“interaction” has an essential element–retroaction, or iteration sometimes.
Assuming there is an artifact and a user, it means that the artifact will give
a reaction to an operation from the user(if the operation fits the instructions);
sometimes, the user can do something more based on the reaction from the artifact;
then the user will keep trying playing with the artifact and enjoy it.
Therefore, interaction is not within a short time/seconds;
instead, it is a process with changes and attampts. ”
As for the form of it, the project is mainly based on a hand-cranked music box with self-designed physical scores, as I said above:
“positioning and drilling the physical score takes quite a lot time and energy. What’s more, one of the original idea is use processing and arduino to create mucis, but I’m highly eager to make mechanical engineering(I’m so crazy about this). Therefore, I chosed this musical box and decised to simplify the exprience of using this composition tool and playing music in a classic and physical way.”
—
As for the users, this project is for everyone who interested in music to experience a special for of music composition. To narrow down the users, on the one hand, it’s used to introduce this niche interest and special music box to the public; on the other hand, it offered an opportunity for people without much knowledge and skills to start trying composing from a very simple stage(only 15 basic tones). Moreover, the melodious sounds of Metal clashing attracts audience and are part of childhood memory for almost everyone. Therefore, it’s a great opportunity to make this unchangable childhood sounds able to be signed by themselves.
Besides, what Prof. Minsky said really impressed me, that “many musicians liked to make their own music when they were children by making new holes in the paper rolls for big player pianos.”
Extra interesting facts:
Punch tape was the input and output medium of early computers, which converted programs and data into binary digital: 1 with hole, 0 without hole, and input to the computer by photoelectric scanning. Perforated paper tape was replaced by more advanced magnetic tape (as a computer storage device) in 1951. Punch cards were used as a recording medium for more than 100 years before computers were invented, in 1801, by Frenchman Joseph Marie Jacquard, to control the patterns produced by a loom. In the 1880s, U.S. Census Bureau clerk Herman Holley invented a punch card and machine for census data and used it in the 1890 U.S. Census, which was completed in just six weeks. The previous U.S. census, in 1880, was processed by hand and took seven years to produce.
Also, I expected to learn more about music composition during this project.
One more thing worth mentioning is that, when I searched more informations about music box, I found and met a group of people also interested in it and some of them have already made great fruits. I joined QQ group chat and learn from the materials shared there. That’s why I think it worth introducing it into the public.
* As you compose your Project Proposal, avoid merely making lists. Instead focus on writing. ** While aesthetic factors are generally relevant, a project governed by “It will be pretty” doesn’t go far.
This part everything went well. One thing worth mentioning is that LA asked me why I chose to use the slider-shape protentiometer instead of rotary knob one. Actually, we have enough rotary protentiometers; the reason I chose to use slider is that, the basic motions of the little white ball on screen is up, down, right, and left–it’s more direct to see the control direction of right and left.
What’s more, it’s interesting to comment out the background so that the bouncing ball will become the drawing sketch:D
Code from the Arduino:
/*
ReceiveMultipleValues
This sketch repeatedly receives a record that contains a single value, and
uses it to control the builtin LED. The value should be 0 or 1.
This sketch pairs well with the SendSingleValue example from the Processing
SerialRecord library <https://osteele.github.io/Processing_SerialRecord/>.
You can also interact with this sketch from the Serial Monitor. Enter
`100,200` into the text area at the top, and press "Send". Then enter `!e` to
ask the Arduino to send back the last values it received.
by Oliver Steele, 2020-2022
This example code is in the public domain.
*/
#include "SerialRecord.h"
// Change this number to the number of values you want to receive
SerialRecord reader(2);
void setup() {
Serial.begin(9600);
pinMode(A0,INPUT);
pinMode(A1,INPUT);
// pinMode(LED_BUILTIN, OUTPUT);
// pinMode(9, OUTPUT);
}
void loop() {
reader.read();
if (reader[0] > 512) {
digitalWrite(LED_BUILTIN, HIGH);
} else {
digitalWrite(LED_BUILTIN, LOW);
}
tone(9, reader[1]);
}
Code from processing:
/**
* Example sketch for the SerialRecord library for Processing.
*
* Receives two integers from the serial port, and uses them to control the x
* and y position of a circle on the canvas.
*/
import processing.serial.*;
import osteele.processing.SerialRecord.*;
Serial serialPort;
SerialRecord serialRecord;
void setup() {
size(500, 500);
String serialPortName = SerialUtils.findArduinoPort(2);
serialPort = new Serial(this, serialPortName, 9600);
// If the Arduino sketch sends a different number of values, modify the number
// `2` on the next line to match the number of values that it sends.
serialRecord = new SerialRecord(this, serialPort, 2);
}
void draw() {
background(0);
serialRecord.read();
int value1 = serialRecord.values[0];
int value2 = serialRecord.values[1];
float x = map(value1, 0, 1024, 0, width);
float y = map(value2, 0, 1024, 0, height);
circle(x, y, 20);
}
Task #2:
Partner: Morgan
During the coding part we first use the spacebar and keyPressed() function to test the arduino IDE and servo. However, there was an issue that the servo kept shaking after swaping once; and before it, the servo could not stop swapping.
Finally, with the help of Prof. Minsky and one of my friend, I tested the code with different servos. Then, the problem is–the servo! Perhaps, the original servo I used was somehow broken.
And for the latter problem, Rudi helped us to fix it, by making the reader.read() into if (reader.read()) {}
Although it’s hard to know what exactly happened inside the reader.read() and arduino, it’s interesting to learn reader.read() can be used as a boolean parameter.
Code from Arduino:
#include "SerialRecord.h"
#include
Servo myservo;
Servo myservo2;
int step;
int pos;
// Change this number to the number of values you want to receive
SerialRecord reader(2);
void setup() {
// Servos
myservo.attach(9);
myservo2.attach(8);
myservo.write(0);
myservo2.write(0);
Serial.begin(9600);
}
void loop() {
if (reader.read()) { //Rudi helped and suggested
if (reader[0] == 1) {
for (pos = 0; pos < 120; pos += 3) { myservo.write(pos); delay(3); } for (pos = 120; pos > 0; pos -= 3) {
myservo.write(pos);
delay(10);
}
}
if (reader[1] == 1) {
for (pos = 0; pos < 120; pos += 3) { myservo2.write(pos); delay(3); } for (pos = 120; pos > 0; pos -= 3) {
myservo2.write(pos);
delay(10);
}
}
}
}
Code from Processing:
/**
* Example sketch for the SerialRecord library for Processing.
*
* Receives two integers from the serial port, and uses them to control the x
* and y position of a circle on the canvas.
*/
import processing.serial.*;
import osteele.processing.SerialRecord.*;
Serial serialPort;
SerialRecord serialRecord;
int step=2;
int x=0;
void setup() {
fullScreen();
size(500, 500);
background(0);
noCursor();
String serialPortName = SerialUtils.findArduinoPort(2);
serialPort = new Serial(this, serialPortName, 9600);
serialRecord = new SerialRecord(this, serialPort, 2);
serialRecord.values[0] = 0;
serialRecord.values[1] = 0;
serialRecord.send();
}
void draw() {
background(0);
//float x = map(value1, 0, 1024, 0, width);
x+=step;
//if (x>=90|| x<=0){
//step=-1*step;
//}
float r=radians(x);
float move=sin(r)*width/2+width/2;
circle(move, height/2, 80);
serialRecord.values[0] = 0;
serialRecord.values[1] = 0;
if (x%360==180+step) {
serialRecord.values[0] = 1;
//serialRecord.send();
} else if (move==0) {
serialRecord.values[1] = 1;
//serialRecord.send();
}
serialRecord.send();
}
//void keyPressed() {
// if (keyCode==32) {
// background(100);
// serialRecord.values[0] = 1;
// serialRecord.send();
// //fill(
// }
//}
The title must be direct, short, decisive, engaging, and self-explanatory.
Consider: modifying the user interaction, meaningful & innovative use of electronics & code, as well as form and any story being told.
1. Pingpong Training
It is a device that people can experience playing pingpong alone without pingpong table. It allows those who are interested in pingpong but don’t have enough time or a partner to play on b1; besides, it avoid the ball rolling far and player taking to much energy to pick it back(especially on b1.
For the interaction part, when the mental card on the pat connects certain part of the ball, the arduino will recieve the data and control the servos slightly change the direction and motion of the ball to react.
2. Music drawing
I believe beauty is a universal sense. That is, enjoyable songs, like classical music pieces, can be somehow transform into a beautiful paintings(visual artifact) or even a smell.
So, in the device, the player can sing or play a melody to the mic and beat beats with the handle, to draw a beautiful picture both on the computer screen but also a mechanical printer. After playing they can take away the picture drawn by the machine.
3. Dancing partner
It’s a wearable device that has five tilt sensors to collect simple movement of users from body(upper part), two forearms, and two legs. Then Arduino Uno digitalRead() to get input(HIGH or LOW) which is executed by code and OUTPUT as the motion of the little creature(maybe a toy kid, or an animal).
With the project, user can feel the interaction (with large delay set) like looking at a magic mirror. It feels like the user is dancing with the toy as partners
Thanks to La Kaylee, I finally understood how processing communicates with Arduino programs. The data of the index and CRGB color of the LEDs are stored in an array of serialRead(serialRead.value[]=4 here) and sent by serialRead.send(), toward the program in Arduino(we copied it from IX lab page); then, with library fastLEDs, arduino control the Neopixel.
To visualize the music, I chose the lenth of lit LEDs to show frequency of sounds–higher tone reacted with more LEDs.
As is shown in the first version above, every second, every LED changed colors because of the default random() color setting, making the Neopixel a effect of flashing. Unfortunately, the flashing effect weakens the LEDs’ reaction to music changes. That is, changing color made the length change unobvious.
Therefore, in the second version, I default all colors into one, to that we can simply see the jumping LED lines.
/* This is a code example for Processing, to be used on Recitation 7
You need to have installed the SerialRecord library.
Interaction Lab
IMA NYU Shanghai
2022 Fall
*/
import processing.sound.*;
SoundFile sample;
Amplitude analysis;
import processing.serial.*;
import osteele.processing.SerialRecord.*;
Serial serialPort;
SerialRecord serialRecord;
int W; //width of the tiles
int NUM = 60; //amount of pixels
int[] r = new int[NUM]; //red of each tile
int[] g = new int[NUM]; //red of each tile
int[] b = new int[NUM]; //red of each tile
// declare an AudioIn object
//AudioIn microphone;
// declare an Frequency analysis object to detect the frequencies in a sound
FFT freqAnalysis;
// declare a variable for the amount of frequencies to analyze
// should be a multiple of 64 for best results
int frequencies = 1024;
// Define the frequencies wanted for our visualization. Above that treshold frequencies are rarely atteigned and stay flat.
int freqWanted = 128;
// declare an array to store the frequency anlysis results in
float[] spectrum = new float[freqWanted];
// Declare a drawing variable for calculating the width of the
float circleWidth;
void setup() {
size(600, 400);
W = width/NUM;
// You can use this syntax and change COM3 for your serial port
printArray(Serial.list());
serialPort = new Serial(this, "COM3", 9600);
// in MacOS it looks like "/dev/cu.usbmodem1101"
//or you can try to use this instead:
String serialPortName = SerialUtils.findArduinoPort();
//serialPort = new Serial(this, serialPortName, 9600);
serialRecord = new SerialRecord(this, serialPort, 4);
serialRecord.logToCanvas(false);
rectMode(CENTER);
sample = new SoundFile(this, "lanTingXu.mp3");
// create the Amplitude analysis object
analysis = new Amplitude(this);
// analyze the playing sound file
analysis.input(sample);
//freq
//size(512, 480);
// Calculate the width of the circles depending on how the bands we want to display.
circleWidth = width/float(freqWanted);
// create the AudioIn object and select the mic as the input stream
//microphone = new AudioIn(this, 0);
// start the mic input without routing it to the speakers
//microphone.start();
// create the Frequency analysis object and tell it how many frequencies to analyze
freqAnalysis = new FFT(this, frequencies);
// use the microphone as the input for the analysis
//freqAnalysis.input(microphone);
}
void draw() {
background(0);
backGround();
for (int i=0; i<NUM; i ++) {
fill(r[i], g[i], b[i]);
rect(i * W + W/2, height/2, 10, 10);
}
//if (mousePressed == true) {
// int n = floor(constrain(mouseX/W, 0, NUM-1));
// r[n] = floor(random(255));
// g[n] = floor(random(255));
// b[n] = floor(random(255));
// serialRecord.values[0] = n; // which pixel we change (0-59)
// serialRecord.values[1] = r[n]; // how much red (0-255)
// serialRecord.values[2] = g[n]; // how much green (0-255)
// serialRecord.values[3] = b[n]; // how much blue (0-255)
// serialRecord.send(); // send it!
//}
noStroke();
fill(255, 0, 150);
// analyze the audio for its volume level
float volume = analysis.analyze();
float freq=analysis.analyze();
// volume is a number between 0.0 and 1.0//but actually the max is 0.5
// map the volume value to a useful scale
float diameter = map(volume, 0, 1, 0, width);
int voln=int(map(volume, 0, 0.5, 0, 30));
float y=height/2;
// draw a circle based on the microphone amplitude (volume)
circle(width/2, y, diameter);
//freq
// analyze the frequency spectrum and store it in the array
freqAnalysis.analyze(spectrum);
printArray(spectrum);
//background(0);
fill(200, 0, 100, 150);
noStroke();
// draw circles based on the values stored in the spectrum array
// adjust the scale variable as needed
//float scale = 600;
//for (int i=0; i<freqWanted; i++) {
// circle(i*circleWidth, height/2, spectrum[i]*scale);
//}
//
//lenth-volumn//light frequent
if (sample.isPlaying()) {
for (int i=0; i<voln; i++) {
//int n = floor(constrain(mouseX/W, 0, NUM-1));
int n=i;
//r[n] = floor(random(255));
//g[n] = floor(random(255));
//b[n] = floor(random(255));
r[n] = 140;
g[n] = 190;
b[n] = 200;
serialRecord.values[0] = n; // which pixel we change (0-59)
serialRecord.values[1] = r[n]; // how much red (0-255)
serialRecord.values[2] = g[n]; // how much green (0-255)
serialRecord.values[3] = b[n]; // how much blue (0-255)
serialRecord.send(); // send it!
}
for (int i=voln; i<60; i++) {
//int n = floor(constrain(mouseX/W, 0, NUM-1));
int n=i;
r[n] = 0;
g[n] = 0;
b[n] = 0;
serialRecord.values[0] = n; // which pixel we change (0-59)
serialRecord.values[1] = r[n]; // how much red (0-255)
serialRecord.values[2] = g[n]; // how much green (0-255)
serialRecord.values[3] = b[n]; // how much blue (0-255)
serialRecord.send(); // send it!
}
//int n = floor(constrain(mouseX/W, 0, NUM-1));
//r[n] = floor(random(255));
//g[n] = floor(random(255));
//b[n] = floor(random(255));
//serialRecord.values[0] = n; // which pixel we change (0-59)
//serialRecord.values[1] = r[n]; // how much red (0-255)
//serialRecord.values[2] = g[n]; // how much green (0-255)
//serialRecord.values[3] = b[n]; // how much blue (0-255)
//serialRecord.send(); // send it!
} else {
for (int i=0; i<30; i++) { //int n = floor(constrain(mouseX/W, 0, NUM-1)); int n=i; r[n] = 200; g[n] = 140; b[n] = 190; serialRecord.values[0] = n; // which pixel we change (0-59) serialRecord.values[1] = r[n]; // how much red (0-255) serialRecord.values[2] = g[n]; // how much green (0-255) serialRecord.values[3] = b[n]; // how much blue (0-255) serialRecord.send(); // send it! if (n>6) {
n=n-1;
r[n] = 0;
g[n] = 0;
b[n] = 0;
serialRecord.values[0] = n; // which pixel we change (0-59)
serialRecord.values[1] = r[n]; // how much red (0-255)
serialRecord.values[2] = g[n]; // how much green (0-255)
serialRecord.values[3] = b[n]; // how much blue (0-255)
serialRecord.send();
}
serialRecord.values[0] = n; // which pixel we change (0-59)
serialRecord.values[1] = r[n]; // how much red (0-255)
serialRecord.values[2] = g[n]; // how much green (0-255)
serialRecord.values[3] = b[n]; // how much blue (0-255)
serialRecord.send();
}
}
}
void backGround() {
long t = millis();
if (t < 1500) { background(155); } else if ((t>=1500) && t<3000) {
background(255);
} else {
background(55);
}
}
void mousePressed() {
sample.play();
}
One aspect can be develop is about delay. As we can see on the taptop screen in my two versions, the graphic changing was not smooth. I guessed it’s probably because too many LED changed status and took times and made delays. This was comfirmed when I helped another students to shut up the LED last seconds.
‘Ruan Jue Xian Yin ‘(“阮觉咸音”) is a new media theater performance project created by Fn media lab, which integrates ethnic music, Ruan instrumental music, light and shadow mapping and mechanical institutions. The whole performance is divided into ‘ Ruan Jue ‘ audio-visual content and ‘ Xianyin ‘ live performance in two parts. In one of the areas of ‘ Ruanjue ‘, there are seven zhongruan(中阮, a traditional Chinese instrument) hanging, and their strings are plucked by mechanical devices to stimulate different rhythms. In another area, there are many headphones recording sound instruments, and the audience can choose music to listen by choosing different music. In the performance of ‘ Xianyin ‘, we played 10 pieces of music together with the performers. Multiple communication design with the performer in the early stage to determine the final presentation effect. In order to ensure the real-time collection of the rhythm of the live music, we have modified the instrument, enhanced the directional collection of the instrument, made the digital calculation more accurate, and visually more vivid and full to the audience.
This project takes great advantages of the original features and sounds of zhongruan(中阮), which makes the project unique and a cultural flavor over it.
Based on the unique material, visualization of sounds/playing the instrument was created and finally shown to audience. There, I see the interaction not only happens when audience choose a piece of music, but also when the sounds were played–there is no much different between human-playing and plectrum&servo-playing, which is on the second half of the video.
For the user interaction, it might not so strong. Because in the exhibition, audience can only see the plectrum moving, hear sounds playing, and observe the shake reacting to device’s motion. [refer to later discuss about “interaction”–a process of Iteration of reaction]
2. A homepage of https://www.interactiveartifact.com/dashboard.
It attracted me by the dynamic changing of graphics reacting to my mouse. Is there any reaction of me to the page’s reaction? Of course. When I moved my mouse, I found my cursor sucking white short lines like gathering a dandelion, so I kept moving my mouse and wanted to gather all lines together; later I found, once I moved too fast, some lines dropped and were back to wander, so I tried to slow down and carefully gathered lines.
What’s more, I really like the effect of the low weight of those lines. It makes the interaction, that high speed will lose some lines, better and real–immersive and interactable.
