Category: Machine Learning for New Interfaces

Posted on

Midterm Project Documentation —— Lishan Qin

Overview 

For the midterm project, I utilized PoseNet and p5.js to develop an interactive dance game that requires players to move their whole body according to the signs showed up on screen, thus making their body dance to the rhythm of the music, as well as imitating the move of the Pikachu on screen. I named this project “Dance Dance Pikachu”.

Process of Development 

Different signs have different meaning in the game: red means left wrist, blue means right wrist, and when the circle is solid, it requires the user to clap hands. In order for players to dance to the rhythm of the music, the signs show up on the screen must always follow the beats of music. To accomplish this, I used “dia” to get new p5.AudioIn() to get the volunm of the music playing. I used this value as condition for the signs to show up at different times but always according to the beats of the music. Only when this value is higher than a certain number will signs show up. By doing so, the dancer can always dance to the rhythm of the music if he/she follows the signs correctly. I also used the frameCount value when arranging the coordination for the signs to show up. For instance, in the first 10 seconds of the song, during which the frameCount is always under 650, the coordination of the signs showing up would be different than those when frameCount equals 700. By making the coordination of the signs show up different, I’m able to diversify the dance moves in the game.

I used PoseNet to get the coordination of players’ wrists and built a counting score system. The more the player’s movement match with the signs, the higher he/her score will be. I also rely on the AudioIn() value to see if the player claps hands when he/she is supposed to. The player will also get a “Perfect” signal when he/she is doing correctly. I also tried my best to make the moves of the dancer identify with the moves of Pikachu in the screen. I did this by manipulating the condition (the frameCount) for signs to show up as the positions for the signs to make them match the posture of the Pikachu Dancing in the middle of the screen. By doing so, the player is able to dance like the dancing Pikachu.

(dancer: Ziying Wang)                                                               

Technical issue & Reflection

As is pointed out by the guest critics during my project presentation, the audioIn() mic value sometimes messes up with the time of the sign showing up. I should indeed try to catch the beats of the music with other value to make these signs more stable and matches the music. Other main technical issues I met when developing this project are all due to the instability of PoseNet. The coordination it provides is sometimes desultory, which brings difficulty for the game to return output timely. Sometimes even if the player’s hands are in the correct position, due to the model’s failure to recognize the wrist (sometimes because of the players’ quick movement), the game still shows that the player fails to meet the sign. In addition, the model also takes an unfixed number of time to load. I added a countdown session before the game because I don’t want to start the game when the model isn’t even loaded for the player to play. Originally, I want to use uNet to make the player’s figure behind the Pikachu image, so that the players can see if their movement match with the sign. However, when I use both the PoseNet and uNet, it takes even longer time for the model to load, the game also gets stuck more often, and the voice in uNet also messes with the AudioIn() value for the game to work properly.

If given more time, I will try to solve these problems mentioned above. I still think it would be more fun and easier for players if they could see their figure behind the Pikachu image and know where their hands are in the screen, and uNet creates a sense of mystery that allows players to not see their true identity. In addition, I will also try to make the interface look better. I will also try to diversify the dancing moves to make more dancing posture available.

Posted on

Midterm Project Documentation–Crystal Liu

My plan

My plan is to create an interactive painting. The user is a butterfly in the painting, and the coordinates of the butterfly is the location of the user’s left eye. I want to add some surprising part to let users explore. For example, there is a tree in the picture. If the butterfly gets close enough to it or even touches it, the leaves will fall down to the ground. Also, the user can do some specific poses to add the object to the picture. For example, if the user puts his or her hand in the air, there will be a cloud in the sky. To achieve this effect, I used poseNet as the core model since it can get the coordinates of the human’s body parts.

My process

First, I searched for some beautiful paintings about natural scenes and set the following picture as the background. And then I observed the painting to find the surprising part.

I found that there were plenty of trees. So maybe I could add a tree in the painting and when the butterfly approached, the leaves would fall down to the ground. Then I searched online for the GIF about trees, and I’ve found this gif. And I used screenshot to get a still PNG file.

 

First, I put the still image into the draw function and added an “if statement” so that when the butterfly approaches the still tree, the dynamic tree will show up.

 I also applied Same method to this group:

 

To achieve the other idea, I used two body parts and used the distance between them to trigger things, since the model couldn’t recognize poses directly. I used left eye and left wrist. For example, if the hand is a certain amount above the eye, a cloud will come out. In this way, the user needs to lift his/her hand to trigger the cloud.

These are about visual part. I also noticed that there were deer in the painting. So I wanted to add audio output. If the butterfly get close to the deer, the user will hear the cry of the deer. 

Difficulties

At first I used the coordinates of left wrist to control the position of butterfly but it was too tired to keep my hand in the air for such a long time. I also tried to use nose position, but it was not good as well. At last I decided to use the position of left eye to move the butterfly.

Also, the butterfly couldn’t fly smoothly. The professor suggested me I can use lerp () to make it smooth.  

Here, the number is 0.05, so the butterfly will cover 5% of the distance from (bfX, bfY) to (x, y) and then cover 5% of the rest part. Finally, it will get really close to the destination but never reach it. 

Another difficulty for me is to differentiate the coordinates of left wrist and left eye. At first I used x1=x, y1=y in the section of left wrist and x2=x, y2=y in the section of left eye. I didn’t declare x1 and x2, but it worked. But that was not a right way to get it. Professor told me I can declare x1 or other name and give the variable a value first. And then use the distance function to calculate the result.

Reflection

I have received so many excellent  suggestions in the presentation. I am really interested in Professor Moon’s idea. And my thought is that I can set different thesis and give users the corresponding filters. For example, in the map of Christmas thesis, if the user finds the “present” in the painting and collects it, there will be a red hat and white beard on his or her face (since I don’t want to hide the camera, the filter can appear in the camera canvas). Also, I like the suggestion provided by Tristan. I can make the interaction more smooth by making it continuous and gradual. For example, as he said, the user can hear a slight sound of running water when he gets close to the waterfall and the sound will become louder and louder as he gets closer to the waterfall. Also if the user leaves the waterfall, the sound will gradually fade away. Besides, Shenshen’s project inspired me a lot. Since I want to enrich my paintings into various pictures with some connections (storytelling part). And Shenshen’s project provides a way for the users explore or observe my project. They can use their eyes to “walk” in the painting. I can add these thought to my final project. And I really appreciate anyone who helped me promote my project or test it, especially Professor Moon! 

Posted on

MLNI: Blocking Out Noise – Sebastian Lau (Midterm Project)

I went through a lot of iterations with my project. At first, I wanted to base something off Avatar: The Last Airbender, but after working with the poseNet I realized that wrist detection was spotty and anything based on it would be difficult regardless of what I did. 

My next step was to try something a bit less ambitious. Inspired by Eszter’s initial flower idea, as well as the #TeamTrees thing on Youtube, I wanted to try making trees that would sprout from my shoulders. I was able to make the trees and the shoulder detection was good, unfortunately, I couldn’t find a way to combine the two. I had been looking at someone else’s code for making the trees and they were using an HTML canvas. I think with enough effort I would be able to combine the two, but at that point, I didn’t really have a lot of time. 

In fact, what I will most likely use as my final product (the picture you see above) was originally meant to be a backup just in case I couldn’t finish the tree idea. I wanted to combine a couple of things that we learned in class so that I wouldn’t have to worry so much about meshing them together as I did with the trees. 

At first, I wanted to make letters spill out of my ears, a visualization for me trying to retain information in classes. However, we had been using particles so I wasn’t able to make the shift from particles to characters (since much of the code seemed to be reliant on the fact that they were working on particles). 

With my final idea, I used the letters to make the picture, as we had done in class. Except for this time, black particles would be streaming out of my ears. Part of the inspiration was me realizing that poseNet was able to detect my ears even when they were covered by big headphones. Although serendipitous I think the visual works very well and fits with how I feel when I listen to music: I’m blocking out all the noise around me.