Link to Project: https://glitch.com/edit/#!/final project-kathryn-madi-2020 

Link to presentation (for reference): https://docs.google.com/presentation/d/1J_2KONIbirVdi5zC-REqesl784PnIs-pNd0vfl1EKcU/edit#slide=id.ga59597a349_0_119 

• Preparation

In preparation for this project we looked for inspirations and began experimenting with implementing Posenet into our first sketches.  We first drew inspiration from games such as Just Dance and Beat Saber, since these games include both motion interaction and music.  As we were trying to make sound play in response to hand movements across the screen, we found an example called “Motion Music” by Atharva Patil and used his code as reference.  

Motion Music: https://medium.com/disintegration-anxiety-1/icm-final-project-53b624770bb6

Our first example included a basic skeleton where a sound would be played if the user’s nose position was in a certain region of the canvas.  Because the p5.sound.min.js was commented out in the index, the sound wouldn’t load at first and only using objects like rectangles would display in response to the user’s hand movements.  The message “loadSound is not defined” would also show up in the console.  However, once we fixed this error the sound started to work and we moved on to adding our own sounds and creating several designated regions for each sound.

Code: https://glitch.com/edit/#!/posenet-project1 

One thing we could have done differently is to draw more sketches of our ideas in order to have more creative visuals along with our technical ideas. 

• Process

Posenet & Sound

In this project, we used Posenet to detect the poses of the users and let them interact with the buttons on the screen. We first loaded Posenet in setup and added the camera video on the canvas so that the users can see their movements on the screen. 

By creating the functions: drawKeypoints(poses), drawSkeleton(poses), drawHand(), we showed the key points and the skeleton on the canvas. However, we noticed that Posenet is so sensitive and the key points are very jittery, so we applied the if statement (rightHand.score>0.5) in order to increase the certainty of each point. 

In trying out how to incorporate Posenet and sound into one sketch, we created a sample that included four different instruments and the user would need to move their left or right hands to make a song.  We experimented with different visual effects as well and started experimenting with using buttons.

Sketch: https://glitch.com/edit/#!/final-project-planarian 

Visuals

In order to let the users see the movements of their hands more clearly, we added a hand image at the position of the right hand using a for loop. 

We also added dots and lines for the Posenet keypoints using an array to help the user visualize where exactly the keypoints are on their bodies.

Buttons

We created the buttons called “play music” on the top left corner for each level so that the players can play as many times as they want. By having this button here, we thought it would be more user-friendly so that they replay the music when they forget it. 

With createButton(), button.position(), button.mousePressed(playlevel1), the button will play the music example when it is clicked. 

At first, we encountered the problem that the button wouldn’t disappear on the start screen and end screens. 

Later, we successfully solved the problem by using button.hide() and button.show() in different screens.

In order to create buttons for the individual notes C, E, D, and F, we created global variables button1 to button4 and used createButton() in setup() to make each button.  In order to use these buttons’ individual coordinates (x, y, width, height) for the handInButton() function, we created global variables buttonx, buttony, buttonwidth, and buttonheight for each button.  Additionally, to change the color of the buttons we used the function button.style(“background-color”, col) referenced from elinsterz p5.js sketch “Button style changes” and created global variables col and col2 for the colors green and red respectively.  In the final result, we ended up not making any red buttons, though. 

“Button style changes”: https://editor.p5js.org/elinsterz/sketches/NvK3RRihj 

Sound Sequencing

After creating the buttons and adding sound, we began coding the sound sequence.  At first, we tried making the colors of the buttons change whenever the user’s hand was in a button and added this to the code with the sound.play() functions.  However, adding the color change to the code for when the notes should play failed to create a specific sequence since the computer wouldn’t be recording where the user’s hand was previously.  

With help from Professor Steele and Linda, we were able to create functions buttonIndex, previousButtonIndex(), detectWristPosition(), targetButtonIndex(), findButtonEntryIndex(), and buttonEntryIndex() so that we could record the user’s previously touched button while also having the computer expect the user to touch the next button in the sequence.  This way, we were able to include color changes for each button whenever the user touches it if in the correct order, and change the mode for when the player passes a level.

We expected to create drastically different music samples for each level, but considering every time we changed the values for return in the findButtonEntryIndex() function or the targetButtonIndex values in the detectWristButton() function the sequence would not work properly.  So, we kept a similar sequence but changed the amount of required notes/buttons played to have some variety in the sequences for each level. 

Transitions & Key Interaction

We wanted to make the game more interesting by making different levels. For level 1, the music example contains only two notes, and in level 2 there are four. Therefore, with the start page and end pages, we basically created this logic: first start page, then press enter to go to level 1, and only if the user pass level 1, the project will move to end page 1, which says”go the the next level”, then level 2, and only if the player pass level 2, the end page 2 which says “you win” will show up. 

To achieve this result, we used setMode() and keyPressed functions so that the player can switch the screens by pressing “enter”. In draw(), we linked each mode with the corresponding page (we made functions for each page).However, we needed to avoid that when the players hit “enter” in level 1,  they will go to level 2directly, so we removed mode 4 and mode 5 in the keypressed function, and put them into the detectWristPosition(), so that only when player pass level 1, they can go to level 2.

Class

Since we sacrificed some of the visual aesthetics in order to figure out the technical difficulties, we wanted to add some visualizations in the end. By using class, we created some circles with random sizes scattered in the background. To relate it with our theme: music, we replaced them with the image of the music notes. 

• Successes & Failures

1. Success: Learning how to do a sequence of sound in addition to using posenet
2. Success: the transitions between each screen is smooth + able to replay the game as many times as you want

Some things we were not able to accomplish were using a class for the buttons and making wrong notes in the sequences red.  We tried to add a class for the buttons at the end, which forced us to try to change every button into an object instead.  This would further complicate our code considering the function button.style(“background-color”,col) would only work for buttons, not rectangles, and much of the other code like handInButton() was based around the details of the buttons created through createButton().  This also means we could not use the function Button.containsPoint() in the findButtonEntryIndex() function since we kept getting an error stating it was “undefined”.   

We tried to create a sequence where if the wrong button was touched at all, all the buttons would turn red and only the right ones would turn green.  However, every time the buttons turned red it would be very difficult for the right buttons to turn and stay green since the rightHand keypoint would jitter in and out of the button.

Figuring out the sequence was the most difficult part and was very time consuming up until the deadline.  For future projects, it would be beneficial to start on the difficult tasks earlier instead of leaving them for later. 

 

•  Reflection

In the first week, we planned to figure out how to get posenet and sound working together but couldn’t.  So, we started looking for more inspirations and codes to reference in learning how to incorporate posenet and sound together.  At first we didn’t plan to make a sound sequence game with specific goals and were thinking of doing an interactive experience, but later changed this idea. 

In week two, we created a sample to experiment with Posenet and sounds effects and we succeeded in getting posenet to work with sound. Then, we changed the instrument sounds in the sample into a sequence of piano notes because we decided to make it like a perfect pitch test. We didn’t plan to have a start screen or end screen, but we made it when we integrated our codes together.

In week three and four, we experimented with using buttons as the music notes.  One thing we did not expect was how difficult it would be to figure out the sequence of the buttons and how they change color in response to hand movements.  Initially we thought it might be too simple, but it proved to be a challenging task despite how the game looks.  One downfall because of this was that our project still had many issues during User Testing and that interfered with how our classmates interacted with the game.  We didn’t plan to learn about things like the difference between previousButtonIndex, buttonIndex, or targetButtonIndex, but learning these broadened our understanding of how these functions work to create a sequence.

After meeting with the professor and Linda, we got the basic idea of the sequence and looked into many tutorials. We finally made the sequence work and achieved our stretch goal: add another level so that the game would be more interesting. Building on this, we planned to make level 3, but we didn’t have enough time. Also, another experiment we made is to use the class function to create buttons instead of using the function createButton() directly. Unfortunately, we didn’t have enough time to change the whole structure.

Some things we could have done differently were to make the game more visually appealing and add another level.  We spent much of the time on creating a sequence and completing the technical aspects of the game and less time on making creative visuals.  Some ways we could change that is to make the game fill the user’s whole screen and to make different arrangements and colors for the notes on every level.  If we figured out the sequencing sooner and had more time we could add another level, possibly make the sequence use different notes than C, E, D, and F, and figure out how to make the wrong notes red.

• References & Sources Used:

1. Note Image: http://clipart-library.com/clipart/qTBoRXqEc.htm 
2. Hand Image: emoji
3. Notes/Music Files: https://pan.baidu.com/s/11SSUm (License: Personal Use)
4. Sequence: https://www.openprocessing.org/sketch/1039079 
5. Buttons Style Change: https://editor.p5js.org/elinsterz/sketches/NvK3RRihj 
6. “Motion Music” by Atharva Patil: https://medium.com/disintegration-anxiety-1/icm-final-project-53b624770bb6
7. Images for instruments in preliminary sketches” (License: Personal Use)
   1. Drums: http://clipart-library.com/clipart/ki8oMegdT.htm 
   2. Bass Guitar: http://clipart-library.com/search2/?q=bass%20guitar#gsc.tab=1&gsc.q=bass%20guitar&gsc.page=1 
   3. Electric guitar: http://clipart-library.com/clipart/8TAEyq8dc.htm 
   4. Saxophone: http://clipart-library.com/clipart/Saxophone-Free-Download-PNG.htm 
8. Sounds for preliminary sketches: Garageband sound library (License: Personal Use)