Machine Learning Pong
DATE: March 3RD, 2019
Introduction
P5.js has many interesting functions and models to choose from. I was very interested in PoseNet when I learned about it in class, because it has a very high accuracy and is relatively easy to use. Many of the other models had issues with performance and required more programming experience. The program can label and classify different parts of the body along with the location on the screen. One of my favorite classics while growing up was pong, which often included a single player mode with an AI. One of the most interesting implementations that I have seen is an augmented reality rock climbing wall. Although I cannot try it in person, I decided to make a machine learning version that could be played at home. For my version of pong, I could use the Y position of my nose to control the paddle.
Process
While brainstorming, I was stuck on a lot of different ideas that were either too difficult to make or weren’t very interesting. I am very new to machine learning and I don’t understand how a lot of it works. First, I thought about training the model to detect facial recognition and use it to like and dislike photos on tinder using a javascript API but this would take too long. Another idea I had was to use the YOLO model and LTSM text generator to give dialogue to people based on facial emotions. In the end, I decided to pair machine learning with a game to make it more modern.
I used the PoseNet example in class and modified it to only capture nose information. I watched a YouTube tutorial to learn how pong works and modified the code to overlay on top of the video output. The Y position of the nose is sent to the player paddle class to move to a specific X and Y location on the screen. Finally, I created a computer paddle class so that the computer controls the movement, similar to a rudimentary AI. I had to change the difficulty so that the user would have a chance to win every once in a while.
Demo
Possible Improvements
In the future, I want to add multiplayer support so that you can play with your friends. The only problem is that it may be difficult to identify whose nose belongs to which paddle. I intend to add sound effects and music before class on Monday. Also, I had to tweak the difficulty of the AI because it does not behave like a human. A single variable controls whether it is easy or impossible to beat. Facial recognition works surprisingly well, but sometimes it glitches when the user goes off the screen. This could be fixed with some maxY conditionals. All in all, I think the project turned out pretty well. I wish to make some minor refinements but I am happy with the overall concept.
Sources
Pong Code: https://www.youtube.com/watch?v=IIrC5Qcb2G4
Flip Video: https://forum.processing.org/two/discussion/22546/how-do-i-flip-video-in-canvas-horizontally-in-p5js
Posenet: https://ml5js.org/docs/PoseNet
Pong AI Info: http://jsfiddle.net/kHJr6/2/