Posenet Input Fruit Ninja and Dino Jump
Introduction
For my midterm project, I wanted to make a complete game using the machine learning models discussed in class. I used Posenet.js and Handtracking.js as control inputs for my games. These models were a fun way to experiment with new input methods; I think there are many possibilities that can be paired with these models in real life. I wanted to recreate flappy bird in 2D or 3D; I changed my mind and tried making different games that were more suitable to the Posenet model I was using. I used the three.js library recommended by moon to make the Fruit Ninja game more realistic and interactive. I chose these games because they had simple controls and were recognizable by people of all ages.
Demo
Public Link: thomastai.com/dinojump
(My computer was not fast enough to run this game at real time with screen capture so it is slowed down significantly)
Public Link: thomastai.com/fruitninja
Process
My inspiration for this project was this article on gesture based games using hand tracking. I love flappy bird, so I built it and tested my concept, but I found out pretty early that this would be impossible since flappy bird is very difficult to play, just with touch controls alone. The user would get a concussion at the rate it took to get the bird in the air. I thought of Chrome’s dinosaur game as a better alternative.
To make a simple game, you can use an infinite loop to show and update the location of each object. For this game, I used p5.js to draw rectangles and video output, as well as render images and text. I used an array of obstacles to store the address of each of the obstacles so that they could be updated as game progressed. Every 100 frames, the game would call new object and dispose of older objects that have gone past the screen. To find when a player is jumping in the camera, we can calculate the velocity between two points using (prevx – currentx) / 2. To make the dinosaur jump, I added the acceleration of the jump to the velocity, and added the velocity to the y coordinate every frame. To check for collisions, I used hitboxes for cacti with rectangles.
I experimented with Fruit Ninja before my presentation but I was unable to get it to work despite hours of research. The game is built using three layers – a transparent p5 layer, the game canvas using three.js, and a background/flash layer. Later, I went back and modified my code to use raycasting thanks to Professor Moon. This worked miracles, as I was able to find the 2D point in a 3D space without too much difficulty. I did have to convert it to a Vector2 object which can be found with the formula (screenX / window.innerWidth * 2 – 1).
However, the more challenging part was defining the physics since three.js does not have a built in physics engine. I used what I learnt from the dinosaur jump game to calculate the position of objects. When the raycaster detected a collision, it would remove it from the screen and call another fruit object to be launched. Fruits are randomly generated each time this function is called. If this is a bomb and the user hits it, the game ends.
Challenges
I faced several challenges when making these two games. I was unable to get more than one object on the screen at one time with a single Collada loader. I used a Constructor to call a new Fruit() object every time, but only one would update at a time. I should implement some type of input smoothing function, as posenet’s inputs are very unreliable and jump all over the place. As for the dinosaur game, the hitboxes can be somewhat inaccurate and it needs to be changed based on the obstacle size.
Conclusion
If I could make changes, I would use handtrack.js instead of posenet for hand tracking as it is more specialized, more efficient, and more accurate for my project. These games can take up a lot of resources on the computer, and it needs to be optimized.
I learned a lot about how three.js works and hope to use it in my projects for the future. Game design is quite interesting and I really respect game developers for their work now. Even with a high level library like three.js, it takes a lot of work to make even a simple game. Posenet has many applications, and I think we will see it a lot more in the future.
Code
Fruit Ninja – https://github.com/thomastai1666/Interactive-Machine-Learning/tree/master/ML%20Dino%20Game
Dino Jump – https://github.com/thomastai1666/Interactive-Machine-Learning/tree/master/ML%20Fruit%20Ninja
Sources and References
Game Development with Three.js
Importing 3d Models for Three.js
ML5.js Pose Tracking Reference