Project Link (Temporary): http://thomastai.com:3001/
Project Code: https://github.com/thomastai1666/Open-Orchestra
Video Demo
Introduction
For this week’s project, we were asked to design and implement an interactive exploration of melody. I was inspired by the projects we saw in class, and other projects I have seen on the internet. Most notably, I like the strings example from Chrome’s music lab and the “In C” web performance of Terry Riley’s piece In C. Since we are all separated right now, I thought it would be a great time to make a networked application to make music together.
Design
When I think of music, it is often represented on a score like the one shown below. The higher the line, the higher the frequency of the note. The lower the line, the lower the frequency of the note is. This was my first choice for the interface.
Source: https://blog.flat.io/music-theory-ryhthm-measure/
I made mockups of my interface in Sketch, which later evolved and changed as I used predefined bootstrap styles and added buttons and features as I was designing the application. I drew inspiration from stringed instruments and decided on using an Orchestra (cello, bass, violin, viola) as my instruments.
Code
Due to time constraints, many of the components that are used were from other projects. Most of my work was to bring together to components to make the interface I desired. I defined an Instrument object which also inherits the Magenta Player and Tone Player. For networking, I used the socket.IO library which makes use of the shiny new web sockets introduced a few years ago. I send the note value and instrument name of each note being played to all the other users online. It is somewhat hacked together like some of the code, but it works just fine. When a player plucks the string, the data is sent to the Piano Genie model to predict a suitable note to play. It makes use of LSTM (Long short term memory) model to find a suitable sequence.
Future Work
There are some minor changes I would like to make if given the time. I find when too many notes are played at once, the chords tend to be in disharmony. This is a limitation of the Piano Genie model I used in the project. It wasn’t intended for use with string instruments, so it certainly isn’t optimized for this purpose. In fact, there are probably better ways to choose notes that don’t use a machine learning model. I found using a simple scale made more pleasing sounds.
Sources and Attribution
Chrome Experiments Strings: https://musiclab.chromeexperiments.com/Strings/
String Code:
https://experiments.withgoogle.com/jam-with-chrome
Clef Images:
https://en.wikipedia.org/wiki/Clef
Magenta Library and Demo Code:
https://magenta.tensorflow.org/pianogenie
Socket.IO
https://socket.io/