Task:
Create a generative or interactive musical piece. It can be web-based, a physical device, or exist in space. In your presentation, discuss musical, design, and programmatic aspects of your project.
Video:
Inspiration and concept:
I am really inspired by the Chinese electronic music artist Howie Lee. In his music video titled “Tomorrow cannot be waited” there is a scene at around 1:30 – 2:05 where it seems like he is performing live, controlling the sounds we hear with the movement of his hands.
I thought this was interesting because one of the major downfalls of using a electronic based instrument is that it lacks real time control over timbre, making it less viable to be performed live. However, with more advanced human computer interaction interfaces, these live performances are starting to work.
I remembered one of my previous projects that I had been working on which I titled “ML Dance”, it is a rhythm game utilizing ML5 posenet pose recognition. I thought if I change this from a game to something more music oriented, this could be beneficial for musicians looking to do some real time audio visual performance. It did not require any extra hardware similar to the one seen in the video, no sensors, no depth sensing camera, etc.
Combining the concept of real time timbre control with body movement along with machine learning powered pose recognition, I came up with a new project that I named “Expression”. The idea is that by adding tone JS I can not control the sounds of each individual musical element, such as changing volume or lowpass. I can also do musically synced animation that adds to the visual aspect of the project.
A description of your main music and design decisions:
The music is a song that I produced, the vocal and lyrics is from a classmate who is taking the same music tech class with me. I chose this song because I really like the simplicity of my arrangement choices, the song can be broken down to just a few different instruments: drums, vocals, plucks, and bass. Also it is more convenient to use my own track since I will not have access to all the stem files if I were to use a song from the internet, even if I did download the stem tracks to another song I needed the sounds unprocessed, because I need the raw audio signal to go into my program where tone JS applies the new effects.(if I feed an already low passed sound and try to remove the lowpass within tone JS it wouldn’t work)
As for my design decisions, most of the work was already done when I originally planned this project to be a rhythm game. I spent a lot of time filtering the machine learning model outputs for a smooth interaction, as well as a lot of pose interaction specific design decisions which can be found here
Aside from work already implemented from that project, I added a lot more when I am converting this project from a game to a more artistic project. First of all, I added visuals to the lyrics and synced them by calculating the time stamp. I also added ripple visuals that is taken from my rhythm project(Luminosus), I synced the ripples to the music and it also responds dynamically to how much the user is low passing the pluck sound. Finally, I added a layer of white rect to match the rising white noise during a typical EDM build up and created a series of animation styled images during the drop to keep the user interested.
An overview of how it works:
By sending posenet my camera capture, it returns me values that corresponds with its estimated pose position. I then use that value to not only update your virtual avatar, but also use that value to control low pass filters from tone JS. I import my tracks stems individually by grouping sounds in to 4 main tracks, three of which can be controlled by low pass and one where everything else plays in the background. I also ran a Tone.scheduleRepeat function at the same time to sync the triggering time of animations, such as having the ripples spawn at a 16th note pace or having the lyric subtitles appear and disappear at the right time.
Challenges:
I believe the most challenging parts are already sorted out when I was initially creating this as a game, problems such as posenet being inaccurate, how to sync gameplay with music, and how to design for a interface that is not controlled with the classic mouse+keyboard. However, when translating things from my old project into what I have now, I still encountered many annoying errors when trying to add tone JS and changing some of my original codes to match this new way of working with audio files.
Future Work:
I originally envisioned this project to be a much more complex control over timbre, not just low pass. But as I worked more and more towards my goal, it just makes more sense to have simple lowpass controls over multiple different musical elements. Perhaps if I have the right music, where the main element of the song heavily relies on the change in timbre, I can change from controlling multiple elements to using all of your body to control one single sound. With a much more complicated way of calculating these changes, not just x y coordinates of your hand, but the distance between all your separate body parts and their relationships can all affect a slight change in timbre. I also plan on emphasizing on a more user controlled visual, similar to the ripple generated by the pluck sound, as opposed to pre made animations synced to the music. I want this project to be about the expression of who ever is using it, not the expression of me as the producer and programmer. Also incorporating more accurate means of tracking, with sensors or wearable technologies, could also greatly benefit this project and even make it a usable tool for musicians looking to add synthetic sounds.