Brief
https://www.youtube.com/watch?v=ZD2yjnwMOO
The above is a quick demo of what I have done for this week’s assignment. Basically, what I have done is to make a Pong-game controlled by your nose rather than a keyboard. And below is the github link:
https://github.com/WenheLI/AIArts/tree/master/week3
Tech
The core thing here is how to get the user’s nose position from the webcam. And, here, I use PoseNet to do the detection. Every frame, the posenet will output the keypoints(nose, eyes, ears and so on). In our case, we only need to know where the nose is and we map the nose position to our game space position.
Apart from that, I was planning to use BodyPix rather than posenet. While it turns out that the BodyPix in ml5 can not output the exact position of each body but a mask, which makes it impossible to use in my case. Moreover, there is a chance to get some detailed output using purely tensorflow.js, but for this assignment, the time is limited.