I decided to work on my physical computing project by using p5js and ml5 model as input. I am primarily working on BodyPose’s MoveNet instead of PoseNet, because of its advantage in accuracy and speed. P5js would detect a certain value using MoveNet and send a serial output to Arduino. According to Danny Rozin and Professor Shiffman, compared to Xbox’s Kinect, this is an easier and more efficient way to detect human motion. For these few weeks, I will be experimenting with ml5 and serial communication with Arduino.
PComp Project Idea
We are making a kinetic sculpture that mimics the behavior of a living organism. The sculpture would be built with 25 servo motors each moving consecutively in a slight delay. As for the interaction, we are going to depict the natural response of any organism when interacting with humans.
Proximity Detection using ml5 MoveNet
Detecting the proximity of the person to the webcam by measuring the distance of the left eye and right eye using MoveNet. So if the person is closer to the webcam, the distance would be larger, and vice versa. Ideally, when the person gets too close to the camera, for example when the distance is greater than 100, it would send a serial output to Arduino, thus changing the behavior of the motors(speed and paste).
Limitation:
This method provides an estimate and can be influenced by the angle of the head, the quality of the pose estimation, and other environmental factors. It might not be precise for exact measurements but can be good for approximate estimations or for triggering interactions when someone gets ‘close enough’.
In this sketch, I noticed that the keypoints detected are super unstable, so we need to utilize the confidence score. Also, when the person tilts his head, the distance also shrinks, while his proximity remains the same. Other environmental factors for example another person walking by would undesirbly fluctuate the value and the behavior of the sculpture.
Pose Detection using ml5 MoveNet and ml5.NeuralNetwork()
Pose detection and classification would be much easier to work with. Ideally, the audience would be able to change the movement of the sculpture by posing a specific pose showing signs of intimidation or calmness. By training a ml5.NeuralNetwork with predetermined poses, p5 is able to easily send a serial output to Arduino and switch the state of the motors.
For this week I am experimenting with pose classification with Movenet using the YMCA poses. I have worked with pose classification with PoseNet by following the coding train tutorial. However, converting the code to MoveNet is quite tricky and I encountered an error in the model training phase. After I imported the pre-trained json data to the model training sketch and started training, the console got an error saying:
“data must be a json object containing an array called “data”
Error: TypeError: Cannot convert undefined or null to object
at Object.loadData (webpack://ml5/src/NeuralNetwork/NeuralNetworkData.js:748:12)