Author: Lishan Qin

Overview

For the midterm project, I developed an interactive two player combat battle game with a “Ninjia” theme that allows players to use their body movement to control the actions of the character and make certain gestures to trigger the moves and skills of the characters in the game to battle against each other. This game is called “Battle Like a Ninjia”.

Demo

Code:

https://drive.google.com/open?id=1WW0pHSV2e-z1dI86c9RLPnCte3GflbJE

Methodology

Originally, I wanted to use a hand-tracking model to train a hand gesture recognition model that is able to recognize hand gestures and alter the moves of the character in the game accordingly. I spent days searching for an existing model that I can use in my project, however, I later found that the existing hand tracking models are either too inaccurate or not supported for CPU. Given that I only have one week to finish this project, I didn’t want to spent too much time on searching for models and ended up with nothing that can work. Thus, I then turned to use the PoseNet model. But I still do hope I can apply a hand-tracking model in the project in the future.

（hand gesture experiments ⬆️ didn’t work in my project）

In order to track the movement of the players’ body and use them as input to trigger the game, I utilized the PoseNet model to get the coordination of each part of the player’s body. I first constructed the conditions each body part’s coordination needs to meet to trigger the actions of the characters. I started by documenting the coordination of certain body part when a specific body gesture is posed. I then set a range for the coordination and coded that when these body parts’ coordinations are all in the range, a move of the characters in the screen can be triggered. By doing so, I “trained” the program to recognize the body gesture of the player by comparing the coordination of the players’ body part with the pre-coded coordination needed to trigger the game. For instance, in the first picture below, when the player poses her hand together and made a specific hand sign like the one Naruto does in the animation before he releases a skill, the Naruto figure in the game will do the same thing and  release the skill. However, what the program recognize is actually not the hand gesture of the player, but the correct coordination of the player’s wrists and elbows. When the Y coordination of both the left and right wrists and elbows of the player is the same, the program is able to recognize that and gives an output.

(use the correlation of coordination of wrists as condition to “train” the model to recognize posture)

As is shown in the above pictures, when the player does one specific body gesture correctly, and the program successfully recognize the gesture, the character in the game will release certain  skills, and the character’s defense and attack (the green and red chart) will also change accordingly. The screen is divided into half. PlayerA standing in front of the left part of the screen can control Naruto’s action with his body movement and PlayerB standing in front of the right part can control Obito. When the two characters battles against each other, their HP will also change according to the number of their current attack and defense.

Experiments

During the process of building this game, I encountered many difficulties. I tried using the coordination of ankles to make the game react to the players’ feet movement. However due to the position of the web cam, it’s very difficult for the webcam to get the look of the players’ ankle. The player would need to stand very far from the screen, which prevents them from seeing the game. Even if the model got the coordination of the ankles, the numbers are still very inaccurate. The PoseNet model also proves to be not very good at telling right wrist from right wrist. At first I wanted the model to recognize when the right hand of the player was held high and then make the character go right. However, I found that when there is only one hand on the screen the model is not able to tell right from left so I have to programmed it to recognize that when the right wrist of the player is higher than his left wrist, the character needs to go right…A lot of compromise like these were made during the process. If given more time, I will try to make more moves available to react to player’s gesture to make this game more fun.I will also try to build an instruction page to inform the users of how to interact with the game, such as exactly which posture can trigger what moves. 

Overview

For the midterm project, I’m planning on building an interactive 2-players battle game that requires each player to make certain specific body moves and hand gestures to control the action of his/her character, and the characters of the players in the game will make different moves to attack basing on the specific moves and gestures the players do. The outcome of each move will also be different accordingly. I called this project “Battle Like a Ninjia”.

Background & Inspiration

My idea for this form of interaction between the players and the game is to a large extent inspired by the cartoon “Naruto”. In Naruto, most of the Ninjias need to do a series of hand gestures before they launch their powerful Ninjitsu skills. In most of the existing battle games of Naruto today, players launch the character’s skills simply by pushing different button on joystick. I however want to put more emphasis on all the hand signs these characters do in the animation. Google’s pixel 4 that features with hand-gesture interaction also inspires me.

       

Motivation

I’ve always found that in most of the games today, the physical interaction between players and games is limited. Even though with the development of VR and physical computing technology, more games like “Beat Saber” and “Just Dance” are coming up, still, the number of video games that can give people the feeling of physical involvement is limited. Thus, I think it will be fun to explore more possibilities of diversifying the ways of the interaction between the game and players by riding of the keyboards and joysticks and having the players to use their body to control a battle game.

How to build it & Potential Technical Problem

Most of the PoseNet library don’t include detailed hand gesture recognition so I think I need another model to recognize those specific hand signs that will trigger the character to use different moves to attack. After research, I did find some projects that also involves hand gestures recognition like this and this. I’m probably going to use them as reference while developing this project. I might still be using PoseNet to detect the body movement of the players. One of the major difficulties I’m anticipating now is that there are two players in the game. I’m not sure if the trained model can tell which hand is which player. I think the coding might get a bit messy…

       

Ideal Final Output

The ideal result is that the model can successfully recognize the hand gesture and body movement of each player, thus allowing the character to battle accordingly in the game. The characters can use different skills, attack or dodge basing on the input the player gives, their HP will also change accordingly. And finally the game will give a winner.

Intro

For this week’s research, I’ve run the program 8 times in total to experiment on how different machine specs and training setup influence the accuracy of the classification and the time spent. I experimented on the influences the valuables such as epoch (number of iterations), the batch size (number of samples per gradient update) and the pool size (the size of the pooling) respectively exert on the performance of the program.

The Epoch Variable

Firstly, I only changed the variable epochs and make the other settings the same. The results are as follows.

I first changed epochs to 10. It took almost 18 minutes for the program to finish the 10 iterations, yet still the test accuracy is only 0.4711.

           

I then changed the epochs to 5 and 3, and left other settings unchanged. The test loss is now 1.83 and the accuracy is 0.4074 when epochs equals to 5, and the time required for the program to finish iterations is 9.4 minutes. And when epoch equals to 3, it took 5.9 minutes for the program to finish and the test loss is only 1.73 and test accuracy is 0.3977.

       

Discovery

According to the comparison between the result when epochs equals 10 and the results when epochs equals 5 and epochs equals 3, it appears that the smaller epochs is, the lower the test accuracy is and the higher the test lost is. There seems to be a positive correlation between the number of epochs and the test accuracy and a negative correlation between epochs and the test loss. But whether such pattern is definite or not is questionable. However, one thing for sure is that the larger the epochs, the more the time is spent for the program to finish because it has to iterate for more times.

The Batch Size Variable

I then only changed the variable epochs and make the other settings the same. The results are as follows.

.   

       

     

Discoveries

It seems that the smaller the batch size is, the higher the test accuracy is. I actually thought the result should be the other way around…But I’m guessing the reason behind this is that since the batch size is smaller, it took longer time for the program to finish examining all the samples, which requires more times to finish all samples, so the time is longer and the accuracy is higher…….?

The Pool Size Variable

Finally, I decided to research on how the pool size variable influences the accuracy of the estimation. I only changed the pool size value and left other values the same. The results are as follows.

.      .     

Discoveries

In terms of the pool size variable, it seems that the larger the pool size is, the less time is needed for the program to finish and the lower the accuracy is. I guess it’s because when the pool size is large, it took fewer times for the program to compare the sample one pool by one pool. And when the times of comparison is less, the estimation becomes rough too, thus lowering the accuracy.

Thoughts and Questions

In conclusion, this experiment showed me that there exists three correlations between the performance of the program and its training set up (epoch, batch size, pool size). it seems that the larger the epoch is, the more the time spent is and the higher the accuracy is. The larger the batch size is, the faster the program finishes and the lower the accuracy is. Same with the pool size. The larger the pool size is, the less time is required and the lower its accuracy is. But due to the limited data collected from my experiment, I’m still not sure to what extent the influence these variables exert on the program’s performance. I’m even not sure if these correlations are definite at all time. Because when in class, I encounter one situation where the program gets higher accuracy when the epoch equals to 2 than when it equals to 3. Thus, I believe that in order to better understand the deeper relation behind the variables and the performance, I still need to do more research about the structure of such program and the purpose of its.