Author: rl3546

Project Title 
Spacetime Symphony

Project Statement
For my project, my partner and I want to make an interactive experience in which groups of people could work together to make something creative. After over a day of brainstorming, we decided we wanted to create an audio and visual soundscape from Arduino to Processing. To narrow the scope of our project, we created a moodboard to pinpoint what our vision of the project was going to be. Some projects I was inspired by is Portugal, the Man’s Coachella animations, and teamlab website.  

 

I also explored mathematical patterns you can make in Processing such as Julia sets, Mandelbrot set, starfields, and fractals, as well as the kinds of hues.  We also researched into the kind of music we were inspired by such as Tame Impala and Beach House. This research influences my design choices for inspiration for visuals and sounds, how the visuals and sounds will behave, and the intended scale of our project. Through this research, it helped nail down the kind of environment our project will live in. 

This project seeks to address how to communicate with others beyond language but with sound, visuals, and movement. It seeks to create a visual soundscape that is unrepeatable like memory and living as easily and naturally as breathing — something everyone can do even without a musical or artistic background.

Project Plan
This project aims to create a visual soundscape that anyone with or without a musical or artistic background can create and in a natural and fun way. For the process, the project will begin in smaller, manageable portions. I plan to focus on 1 manipulable environment with a main background and background audio and then other smaller addable assets and sounds. I first want to create the background and background audio and test it on Processing. After, I want to test if we can manipulate the background or asset with the Arduino to Processing. Each of these steps will be taken one step at a time to ensure that every step has been completed correctly. The project will incorporate various sensors such as the distance sensor, tilt sensor, light sensor, and pressure sensors. Each of these sensors will have a particular behavior it will run when used. For example, some may change the tempo, some may change the pitch, and some will add additional sounds to the soundscape. Some of the visuals will sync with the audio changes which will be manipulated through Processing. These additional elements are dependent on whether the main elements — the background, background audio, and some kind of sound manipulation — are working. The final product will be displayed on a projector and the sensors will be hidden from view, somewhat blended into the environment of the project but visible in front of the projector as a long table that the audience can interact with. 

Week 1: Visual and Musical Assets and Pseudo/Demo Code
Week 2: Continue to work on assets, Working Code 
Week 3: Combine final assets, code, and test

In order for the project to empathize with the audience, I want to make sure the elements of the project are easy to use, fun to use, and encourage teamwork. Not only will the users be able to create art and music without much thought, but they will be able to create connections by working together to create it. Through this method, the project will be able to bring together meaningful human connection in the real world as well. 

Context and Significance 
Ultimately, I want to create a project that brings together individuals by creating something very easily and naturally like communication. My preparatory research and analysis made me consider these factors as a priority for my project proposal. Our target audience is most of the general public. I think now we are caught up in a world of fast information and fast interactions, and a lot of those experiences are through text, screens, and skimming. This project aims to bring people together and allow them to slow down and break those barriers of text and speech into something they can take time to create and experience together. Since the visual soundscape can never be replayed like the music and images we can play from Spotify, Youtube, or Apple Music, it also encourages one to experience things in the moment away from traditional communication into an environment that can surround you and allow you to digest moments.

This project might lead into more collaborations with visual art and music. While there is often collaboration with music, with this project, one could create music and visuals at two different places in the world at once. It also encourages creating visual art with more than one person, which is often an individual activity. It will also allow groups of people to be able to communicate they emotions into sound and create an entire environment with their own mere movements. It could also encourage individuals to explore music and art more as people usually have a hard time picking these hobbies up because they don’t know where to begin and think it’ll be difficult when they merely just need to start. 

Lab Date:  April 19, 2019
Instructor: Marcela

Aim of Today’s Lab: Create an Etch A Sketch and a musical instrument with Arduino and Processing with serial communication

Building Etch A Sketch

Materials Used: Jumper Cables, 2 potentiometers, Arduino Uno, Arduino -> Processing Files

Process:
I always approach projects by splitting them into small checkpoints so I can make sure every step I have tested has worked up to that point. For this Etch A Sketch, I first checked if my potentiometers were working. To do this, I used the AnalogSerial example to see if I can get one potentiometer working. Afterwards, I set up two potentiometers to see if I can get the serial information from them through the Arduino serial monitor. Once I got that working (by only supplying the serial monitor with 2 sensor values), I incorporated Processing. I ran into one issue which was forgetting to change the port index into Arduino’s point. Once I changed the num of values to the corresponding values, I checked if Processing could receive information from both elements of the sensorValues so I printed them out in the console. 

Then, I built the sketch. Initially, I used ellipses as the drawing tool, but I noticed how scattered it was, so I eventually decided to create a set of points and connect them. To do this, I decided to use PVector class, beginShape and endShape functions, and an ArrayList. I added points to my ArrayList by creating PVector objects and drawing the “shape” I created by iterating through the ArrayList by connecting the vertices (or points). 

 

Building a Musical Instrument

Materials Used: Arduino Uno, Jumper Cables, Speaker/Buzzer, Processing -> Arduino Files

Process:
Once I got the speaker working with toneMelody, I used the Processing -> Arduino files to create the musical instrument. This process was slightly tricky seeing how the values array correlated in Processing, so I first decided to see if I could get a single tone output from the speaker when I clicked the Processing canvas. Once I got that working, I attempted to see if I could get the single tone output to play on and off when I clicked my mouse. Then to create the instrument, I got the speaker to change pitch when I moved my mouse. 

Conclusion
This recitation was great now since I have combined both Processing and Arduino, connecting both hardware and software! Some things I learned to keep note: check if the size of the array is equal on both Arduino and Processing, check Port Index, and check if serial communication is occurring between both Arduino and Processing.

During the group project, my definition of interaction was mainly influenced by Zach Lieberman’s definition.  In the post, I defined interaction as “the occurrence of entities reacting and responding to one another.” After doing my midterm project as well as experiencing other people’s projects, my definition of interaction has become much more relaxed. While my midterm project fitted my group project definition by having two entities (the user and the game) respond to each other’s actions, it made me reflect about other ways of interacting, particularly internal experiences, as well as made me reflect why is my definition of interaction defined as plural, “entities.” My definition has evolved slightly to one that includes interaction that involves with the action between the internal and the external.

One project that doesn’t align with my definition of interaction is this haptic navigation device called momo. While it uses touch to guide you through the world, I think it is less interactive (to say the least) because it merely guides the user through touch. The user doesn’t necessarily communicate to the device, but instead the devices guides you to a destination, similarly to a G.P.S. but through touch. However, by changing one of the senses, I don’t think this aligns with my definition of interaction because there is not back and forth feedback directly through the external or internal. While it does tell you where to go via the vibrations, it doesn’t react very much to how it communicates to you, and you don’t necessarily need to react to it. 

Two more projects I looked into was the Scribb and the Thinking Cap. These two projects are much more dynamic than the previous project. Scribb is more interactive to me because you get to draw and use the cursor to move through worlds. You’re able to communicate it to different ways and the game will respond in different ways with how you draw and where you guide the mouse. This interaction is much more explicit and applies to my external idea of interaction. 

Another project I looked into was from MIT Media Lab. While this isn’t necessarily an interactive project, but a research study using a sorting hat to study the well-being, resilience, and self-esteem to the student wearing it, I think it still is an interactive project because of its dynamics. Like Scribb, there are many ways to change the way the hat and the child responds. It uses an internal experience (mental processes and brain patterns) to “tell” the student what they are thinking about and what kind of self-esteem boosting phrases to use.  

These two projects inspire my final project because of its dynamic manner in which one can interact with it, as well as the almost natural and personal quality each project can have. For my final project, I want to challenge myself to make something that has both meaning and naturalness in reality, but also challenging me to create a project that has flexibility in its interaction. I think, for me, interaction involves some kind of communication that can be very changeable and dynamic. Minimal change in the response of the device is less interactive to me and one sided if the user’s actions don’t affect the device very much. In my definition of interaction, I also believe one can interaction via internally and externally, but ultimately the results must be visible in reality (through the device, for example). All in all, interaction is the visible occurrence of reacting and responding dynamically between two entities, either external or internal.