Tag: aiarts03

Among the examples we learned during the class, Posenet seems the most interesting because of its wide range of possibilities. I tried to create a Snapchat-like filter by combining both Posenet examples 1 and 2, drawing with nose and particles generator, respectively. Mosquito just randomly came into my mind so I wanted to visualize it (I’ve never seen a mosquito filter and probably that’s why).

First, I added a couple codes to detect both eyes of a user, and let the draw function can generate a circle on each eye; a line on a nose. One thing I noticed is that, motions of the shapes seem quite unnatural.  I improved this part by two steps: I applied lerp function “noseX = lerp(noseX, nX, 0.5)” to make the motion smoother, and set the diameter of circles as the distance between the two eyes so if a user gets closer to the screen, the size of eyes become larger accordingly. I tried to add more interactions on the nose part, but it seemed somewhat distracting after I added some effects. So I just kept the nose part as a constant line.

For the next step I implemented the particle generator part to the project. As it endlessly creates a number of randomly colored particles from both ears of a user, I modified the code a little bit to combine it with the mosquito filter. I changed the spot where particles are generated to the end of the mosquito nose, by setting the location to “noseX + d, noseY + d.” So now the particles are coming out from the tip of the nose. As the original color scheme was entirely random, I set the green and blue values to 0 and thus all the new particles have become red-tinted—which resembles drops of blood.

(if video doesn’t work, please click this link mosquito)

Overall, it was pretty fun to play around with the given examples. It was my first time trying to code with p5.js so I had struggled a bit, but still it was fruitful experience for me. I’m so excited to explore more about the possibilities that deep learning will bring to us. You can find my code here.

References:

https://github.com/aaaven/aiarts2019/tree/master/week03-5-PosenetExamples-1

https://github.com/aaaven/aiarts2019/tree/master/week03-6-PosenetExamples-2

https://youtu.be/EA3-k9mnLHs (Hour of Code with p5.js and PoseNet by CodingTrain)

Brief

https://www.youtube.com/watch?v=ZD2yjnwMOO

The above is a quick demo of what I have done for this week’s assignment. Basically, what I have done is to make a Pong-game controlled by your nose rather than a keyboard. And below is the github link:

https://github.com/WenheLI/AIArts/tree/master/week3

Tech

The core thing here is how to get the user’s nose position from the webcam. And, here, I use PoseNet to do the detection. Every frame, the posenet will output the keypoints(nose, eyes, ears and so on). In our case, we only need to know where the nose is and we map the nose position to our game space position.

Apart from that, I was planning to use BodyPix rather than posenet. While it turns out that the BodyPix in ml5 can not output the exact position of each body but a mask, which makes it impossible to use in my case.  Moreover, there is a chance to get some detailed output using purely tensorflow.js, but for this assignment, the time is limited.

Code: https://drive.google.com/drive/folders/1iYzpoaUQ30HNqOw04meyBtWoBm53cihh?usp=sharing

I really liked the Posenet examples we went through in class so I wanted to experiment a bit with those. I especially liked the week03-5-PosenetExamples-1 example where you could draw with your nose, and initially wanted to adapt this example so that you could draw with a different body part. I started off searching for the word “nose” within the ml5.min.js code in order to see what the other options are. However, after experimentation with other body parts, the nose was still the most fun to create shapes with.

So, I decided to create a very simple game where the user gets to use this nose movement but in a game format:

Essentially, there are a set of five circles, and the user must “boop” each circle with their nose to change the circle’s color.

Of course, I ran into a couple of challenges while doing this assignment. The main challenge was getting the ellipse to change once the nose reached it. I initially constructed the if statement like this:

However, nothing was happening when I touched my nose to the ellipse. Eventually, I ended up widening the range of the positioning values for the nose to be in in order to get a result:

After broadening the range, it started working properly. I ended up narrowing it by half later, as the above screenshot had too wide of a range. Once this essential part of the game was working, I worked on the user interaction part of the game. This included adding more ellipses and having the ellipses change color once the user touches them in order to indicate they have been touched.

If I were to build upon this project, I would want to add more UI components such as a timer or some kind of score-keeping system as well as prompts to start and finish. It would also be cool if the ellipses were randomly generated as the game goes along.

MyCode：https://drive.google.com/open?id=1Vci8NRnUh9j7PCPs_xzOobpghGQjaXq-

Project Name: “How to Train Your Dragon with Machine Learning”

Intro

For this week’s assignment, I utilize the ml5.js model PosenetBasics to develop an entertaining interactive project called “How to Train Your Dragon with Machine Learning”. My idea is to create an interactive AR app that allows users to interact with the virtue creature in screen using their whole body in the physical world. With the help of the PosenetBasics example, I’m able to make the users appear to be wearing a virtual Viking hat in the screen and allow them to hang out with the virtual creature “Toothless” by making different gestures and actions in real physical world to interact with it. 

Overview & Demo

The PosenetBasics model is able to recognize people’s nose, eyes and wrists, and provides data of the positions of these organs. Thus, I’m able to program “Toothless” to make different reactions according to the users’ action by changing the image and the position of “Toothless”. This project allows users to become a dragon trainer and interact with “Toothless” through different hand gestures. Firstly, the program recognizes the users’ face, and puts a Viking helmet on the users’ head. Then, the users can make different action to interact with “Toothless”, such as petting its head, poking its face or raising the right wrist to ask “Toothless” to fly.

Inspiration

The work that inspires me to build this project is Pokémon Go developed by Niantic. Pokemon Go allows users to use GPS to locate, capture, battle, and train virtual creature Pokemons, which appear in the game as if they’re in the player’s real world. The recent update of Pokémon Go brings a new function that allows users to take photos with the virtual Pokemon in the real world. Nonethless, even though I love this game and this update so very much, I still find the interaction between trainers and pokemons this game provides to be limited. The players of Pokemon Go can only interact with these virtual creatures through phones with their taps on the screen to switch the pokemon’s posture rather than using the users’  physical movement in the real world as input. Therefore, I wish to create a deeper and more direct ways of interaction between game players and those virtual creatures in the real world.

Technical Issues & Future Improvement

My original plan was to make Toothless react differently to both the right hand and left hand of the user. However, I found the model’s collected data of the wright and left wrists to be highly unstable. It often mistook the right wrist for the left, and when there was only one wrist in the screen, it’s not able to tell whether it’s a left wrist or a right wrist. Therefore, in my final work “Toothless” can only react to the users’ right hand’s movement. Also, the size of the Viking helmet that appears in the user’s head in the screen is not able to match the size of the users’ head automatically. I believe there is an algorithm that can make it work but I can’t figure it out. In addition, due to the limitation of time to finish this project, there are also a lot of different output I want to try that I haven’t finished. For example, if given more time, I’d like to add more different sound effects of Toothless to create more diverse output.

Source: https://giphy.com/stickers/howtotrainyourdragon-httyd-toothless-httyd3-1yTgtHaRuuaFWUWk7Q