Link: https://github.com/z71258847/NOC2019SPRING/tree/master/final
Presentation Link: https://docs.google.com/presentation/d/1SvCdFDqJGAawD-RPhG5s5skkjLj71m1LQ9cEG98M3wY/edit#slide=id.p
Introduction:
I implemented a real-time “mage” simulator with skeleton tracking using p5.js and ml5.js for my final project. The program uses PoseNet integrated into ml5.js to recognize the body skeleton captured by web camera in real-time. It then uses the key-point information to render a “ghosty undead mage” figure on the screen, and the users may interact with the program by changing their pose to see several “magical” effect.
Inspiration:
As we have spent plenty of time in experimenting the particle systems in the course, I find that particle is capable of simulating various magical effects, especially with the Add BlendMode that makes the particles shinier. Therefore, as a fan of mages in video games and movies, I decided to explore the use of particle systems together with skeleton tracking to let the user feel the sense of magic through my project.
Implementation:
Skeleton:
I use the PoseNet integrated into ml5.js to process and estimate the key-point information from frames captured by the web camera. The API returns the “pose” data structure that contains 17 key-points of the human body with the confidence level of the estimation. A threshold of the confidence level is applied to the estimation to filter out the wrong predictions. Through a grid search of the threshold hyper-parameter, I find that setting the threshold at 0.1 gives a balanced performance in terms of detection accuracy and detection sensitivity.
Then, I cluster the key-points into several body structures: Face, Body, and Arms. The skeleton part will only be shown on the screen if all the key-points of that part is detected. With some hard coding of the skeleton position, the program is able to present the skeleton by drawing lines and circles.
Spell Orb:
When mages spell some magic, there will certainly be some effects on their hands! I choose to implement this effect by a particle system called spell orb. This particle system consists of two kinds of particles: the center particle and the attracted particles. These particles are instances of the same class Particle, giving maximal reuse of the code.
In general, the spell orb particle system is designed as follow: both kinds of particles have the same color; the center particle has a larger size; there is always a set number of attracted particles in the system; the center particle attracts the attracted particles; when an attracted particle is too near or too far from the center, it will be removed. With this logic, we can adjust the color of the system to get the following effect:
In the project, I mainly use two colors: orange to represent Fire, and blue to represent Frost. Also, BlendMode Add creates a shiny sense.
Wave:
When fire and frost elements are too close to each other, they will certainly repel each other! I use several wave-like effects to present a sense of repelling. I thought about using flowfields to generate a particle wave, but it is too smooth to show the “intensiveness”. Therefore, I use another particle system consists of particles and springs. A series of particles are consequently connected with springs. The two ends of the spring are set to the position of two hands. Therefore, when hands move, the spring will bounce frequently to create the wave effect. Also, when the hands come closer, the wave will fluctuate more because the springs are compressed more. This is also a great effect.
By computing the color of each spring according to the distance to two hands, the wave can have a smooth color transaction. Also, with a BlendMode Add, the waves can also achieve a shiny effect.
Fire Effect:
Tutorial Link: https://www.youtube.com/watch?v=X0kjv0MozuY
This is my favorite effect in this project. I learned it from the tutorial mentioned above. Basically, this is implemented using the following steps. First is to apply a blurring stencil to the whole canvas. This simulates the fire expanding. Then we scroll up the canvas for one pixel, and this is like lifting up the fire. Finally, a perlin noise is subtracted according to the position. This is called apply cooling map, which makes the fire “cools down”.
Actually, there are several implementation obstacles using p5.js. First, generating a full canvas of perlin noise in each frame is computationally expensive. Therefore, I preprocess the full canvas of perlin noise. Then at each frame, I scroll the perlin noise up for one pixel and only generate the last row. This greatly improves the computation speed. Second, the drawing APIs of p5.js don’t directly operate on pixels. Therefore, we have to write our own draw_line and draw_circle functions for manipulating the pixels.
I use this in two different parts of my project. First is the actual fire effect when a magic is spelled in the bottom of screen. Second is that I applied this effect to the skeleton, this can create a very interesting ghosty effect!
Interaction Design:
The interaction mainly depends on the arm position of the user. When the user lift their wrist over the elbow, the spell orb will appear on their hand. When there are two spell orbs existing and the distance between two hands are small, the wave will appear. When either of the spell orbs is over the shoulder, the corresponded color of fire will appear in the bottom of the screen.
Future Improvements:
Maybe use Kinect and Java instead of PoseNet will increase both execution time and detection accuracy. Also, more interesting magic effects can be applied. Translating this project into VR mode should also be an excellent idea.