Git Repo
Demo
Story
“耳语”
This project is inspired by the concept of quiet, intimate, non-evident communications entangled with a public atmosphere. With ml5.js PoseNet and p5 visuals, this projects aims at visualizing whispers between private minds in a public space.
Program Design
When a person appears in the webcam captured video, the system looks for “leftEar” and “rightEar” keypoints in the pose that poseNet recognizes. Following a common saying “左耳朵进右耳朵出” (Left ear in, right ear out) describing insouciant communication, “rightEar” are set as starting points of the text particle’s movements, and “leftEar” keypoints are set as destinations. Line 32~47 in the attached js code is how “leftEar” and “right” ear keypoints are processed respectively.
At every frame, a new Particle is created, initially place at a “rightEar”. Starting from Line 71 in the code is the Particle class. Some key properties of a Particle are its start and end coordinations, x & y speed, text content, size and color. The update() method dynamically updates the Particle’s destination coordination, so that when the target “leftEar” moves, the Particles on their way follows along.
Issues
-
- Feedback from poseNet on poses are not kept in a set order in the “poses” array. This made it troublesome for a Particle to keep a fixed reference to its destination keypoint. In this current version
-
- In the current version, I used an an array (“destinations”) to store destination “leftEar”s. This array is off fixed length, every element corresponding to a pose (one person).
- This solution is still raw and problematic, for it is subject to the order in which poseNet stores all poses.
- A possible solution might be creating a “Pose” class to have a “Pose” object corresponding to each pose detected. (necessity?)
-
- When a “rightEar” is detected before any “leftEar” is detected, an TypeError might occur, because the destination keypoint sent to this “rightEar”‘s Particles might be “undefined”. I initialized the “destination” array with a artificial keypoint element (with only position data), but it doesn’t seem to be reliable. Reasons and solutions needs further investigation. Effective error protections are needed
- Building on top of this current version, I’d like to
- give Particle’s text contents more possibilities/randomness, so that every whisper would be different;
- add audio elements to the system, so that whispers received would be read out, in a voice that’s customized for each pose (person) detected. This would most likely make it necessary to build Pose as objects.
- Feedback from poseNet on poses are not kept in a set order in the “poses” array. This made it troublesome for a Particle to keep a fixed reference to its destination keypoint. In this current version
Code