W10 Interactive Object Using ML
Interactive Wall
Demo
ScreenRecord
Hand Interaction
Process
Constructing the Wall:
I constructed 800 wall pieces of the same size and arranged them into a wall. However, the position is not applied to each wall piece at first:
I found that I didn’t call the update function, which set the mesh position to this.pos, and I accidentally apply an empty vector to the scale of the mesh, which caused the mesh to disappear.
Applying Gravity:
I apply a vertical force (gravity) and a random direction force (push force) to the wall.
Wrong direction of gravity force:
Direction corrected:
However, the force seems very unrealistic. I adjusted the force parameters to test out different forces. Here’s an example of floating wall pieces😂:
I also added bounce codes and friction codes following prof Moon’s example to the wall pieces to let them bounce around:
I adjusted the parameters of the friction force as well as the random push force until it looks somewhat realistic:
Teachable Machine:
I trained a model with “handpush” and “nonhandpush” images recorded from my desktop.
After successfully getting the model, I copied the link and code to my p5js sketch to connect my project.
I successfully console.log the label:
And I declared a boolean value to detect the handpush, and used it in the method wall[i].fall(push);
Discarded Attempt: Postnet hand detection:
At first, my idea is to use particles to construct walls and Posenet for hand movement detection.
However, the framerate dropped to 1-2 and the hand interaction never functions correctly. Prof Moon suggested that I should use Movenet or other ml libraries for the interaction since Posenet is a little tricky for precision of detection. And the effect I want to achieve is pretty complicated. I then switch the goal to a more reasonable one: to push the wall and only detect “whether pushed”.
Failed Attempt on Rotation, Collision, and Shadow:
I tried to apply rotation to the mesh using
rotate() {
if (this.ifFall) {
this.rot.add(this.rotVel);
this.mesh.rotation.set(this.rot.x, this.rot.y, this.rotate.z)
} else{
this.rot.mult(0);
this.mesh.rotation.set(0, 0, 0);
}
}
However, the code didn’t work and the meshes disappeared upon interaction.
I also tried to figure out how to apply collision to the wall pieces. I found solutions online (links are put in the reference) but I decided to focus on the ml part first within the limited time. To polish the physics effect will be the next step of the project.
The shadow doesn’t work for directional light. I’ve turned on the receive-shadow and cast-shadow for the mesh, and also adjusted the renderer, light setting, and light camera position. However, there is no shadow shown on the ground despite the moving of the directional light:
Reflection
In this project, I played with teachable machine and physics in Threejs. My next step is to develop collision and rotation for the falling wall pieces to make the effect more realistic.
Reference
https://threejs.org/examples/#physics_oimo_instancing
https://discourse.threejs.org/t/collisions-two-objects/4125
https://github.com/MOQN/IMA-Web-Page-To-Web-Space/tree/main/w02_forces/forces_23_collision_easy
https://github.com/MOQN/IMA-Web-Page-To-Web-Space/tree/main/w02_forces/forces_12_friction_easy