INSPIRATION
Trying to build upon my concept development on digital preservation, I feel like the example Body, Movement, Language: AI Sketches With Bill T. Jones from Google is something I want to explore. In this project, the words move with the movement of your body, so a sentence follows the flow of your movements. I think it would be interesting to use the text from people’s tweets or Instagram captions as something that the user can interact with.
Another one that could work as well is Touch Type, which is also an experiment with Google. However with this one, there is one word on the screen and each letter is made up of points. The interaction is that when your hand is near the word, the particles are repelled and move away from you. This is another interaction that could be interesting to play with, for instance the word could be what’s trending on Twitter recently.
PROCESS
I first started by implementing the ml5.js code that I wanted to use, which was bodypose movenet. To start of simple and make sure the interactions work, I pulled the example code from the github. For some reason, I kept getting an error on “X-Content-Type-Options”, so I tried not using the main.js file and just running the p5, which worked. So instead of working with ml5 first, I decided to set the scene first and then add ml5.
From the inspiration, I decided to play around with text in three.js. It took a while to find references and help from online, but after finding this video, adding text felt relatively straightforward. I managed to get this on the screen:
I then went back to the ml5 example code, and after talking to Professor Moon the issue was with the naming of my file. After fixing that, everything was working.
Then I tried to get the text to move with the body. At first, I didn’t realize there was an update body pose function, so I thought the cube on the hand was there by coincidence. I added the text without updating, so it was just floating there. After seeing there was an update function, I tried adding the text mesh to it and got this result:
While this works, for some reason the p5 canvas disappeared and I got multiple errors about “mesh” being “undefined” in updateWithBodypose();. I wanted to change the parameters as well, such as the text size, but like the last assignment, it doesn’t work.
FUTURE IMPROVEMENTS
I think this is a good starting point for my final project. I’m still not sure if I want to add ml5 interactions, since I want to focus on simpler interactions first such as using the mouse. It is also not at all like the inspiration, so later on I would like to create an effect like Touch Type.