Jannie Zhou | Traces: Reimagining movement in 3D webspace.

 
My project is an interactive 3D movement visualization webpage. Developed using Three.js, p5.js, and the machine learning 3D motion detection model written in Tensorflow.js, the project runs in web browsers and still provides a good synchronization of movement prediction and visual rendering. It aims to change the context of our everyday movement, visualize its traces in the virtual space, and disclose the information our movement carries.
The Tensorflow.js machine learning model, developed by Google, predicts the users’ joint coordinates in three dimensions (x,y,z). I deployed the model in my project and used the output data to create visuals in the web space. The visuals are abstract and geometrical, including lines, icosahedrons, butterflies, and particles, representing the traces of the users’ movement. The aesthetics of the visuals are deliberate, contributing to the overall abstract experience. The butterflies, as geometrical as they look, are the only things that associate the visual world with the real-life experience. They are programmed to fly toward the users’ wrists, creating lifelike experiences. Every visual is closely tied together to tell a coherent story about the movement. The users are able to observe how much information their movement has: the angels, trails, speed, and dynamics, and play with them.
A backend server was also implemented for the webpage to allow interactions among users in different physical locations. Using Socket.io, the server stores the data from the clients and renders the visuals on each client’s device in real-time. Since it is developed using solely JavaScript and I have published the project on Glitch, users can join the server using their own devices and see each other’s traces in real time with the link. The server is designed to promote exchange and communications between users. While substituting verbal communication with movement interaction, I hope to remind the users that movement remains an expressive and direct method of communication, although many of us have neglected this ability in today’s society.