The sketch is based on the game named “Flappy bird” like the left picture below. The bird position will be changed by the upper and lower button to get through the pipes. Meanwhile, the height of the pipes are related to the pitch of each note. The right one is my original sketch.
This is the link to the p5js sketch. Users can press “a,s,d,f,g” to add notes to the music, but to add each note, users need to get through a pipe, which height is determined by the pitch of the notes. The y position of the bird can be changed by the up and down arrow, which will also change the play back rate of the music.
The final sketch works like below.
I overcome the problem that I can not detect whether the bird meet the pipe to test whether the user is playing in the right way, which may be developed in the future. I also tried to use the waveform data of sound from mic to control the y position of the bird, but it is hard to control the bird because it will fall and rise randomly, so I finally changed to the current idea.
Resources: