My plan
My plan is to create an interactive painting. The user is a butterfly in the painting, and the coordinates of the butterfly is the location of the user’s left eye. I want to add some surprising part to let users explore. For example, there is a tree in the picture. If the butterfly gets close enough to it or even touches it, the leaves will fall down to the ground. Also, the user can do some specific poses to add the object to the picture. For example, if the user puts his or her hand in the air, there will be a cloud in the sky. To achieve this effect, I used poseNet as the core model since it can get the coordinates of the human’s body parts.
My process
First, I searched for some beautiful paintings about natural scenes and set the following picture as the background. And then I observed the painting to find the surprising part.
I found that there were plenty of trees. So maybe I could add a tree in the painting and when the butterfly approached, the leaves would fall down to the ground. Then I searched online for the GIF about trees, and I’ve found this gif. And I used screenshot to get a still PNG file.
First, I put the still image into the draw function and added an “if statement” so that when the butterfly approaches the still tree, the dynamic tree will show up.
I also applied Same method to this group:
To achieve the other idea, I used two body parts and used the distance between them to trigger things, since the model couldn’t recognize poses directly. I used left eye and left wrist. For example, if the hand is a certain amount above the eye, a cloud will come out. In this way, the user needs to lift his/her hand to trigger the cloud.
These are about visual part. I also noticed that there were deer in the painting. So I wanted to add audio output. If the butterfly get close to the deer, the user will hear the cry of the deer.
Difficulties
At first I used the coordinates of left wrist to control the position of butterfly but it was too tired to keep my hand in the air for such a long time. I also tried to use nose position, but it was not good as well. At last I decided to use the position of left eye to move the butterfly.
Also, the butterfly couldn’t fly smoothly. The professor suggested me I can use lerp () to make it smooth.
Here, the number is 0.05, so the butterfly will cover 5% of the distance from (bfX, bfY) to (x, y) and then cover 5% of the rest part. Finally, it will get really close to the destination but never reach it.
Another difficulty for me is to differentiate the coordinates of left wrist and left eye. At first I used x1=x, y1=y in the section of left wrist and x2=x, y2=y in the section of left eye. I didn’t declare x1 and x2, but it worked. But that was not a right way to get it. Professor told me I can declare x1 or other name and give the variable a value first. And then use the distance function to calculate the result.
Reflection
I have received so many excellent suggestions in the presentation. I am really interested in Professor Moon’s idea. And my thought is that I can set different thesis and give users the corresponding filters. For example, in the map of Christmas thesis, if the user finds the “present” in the painting and collects it, there will be a red hat and white beard on his or her face (since I don’t want to hide the camera, the filter can appear in the camera canvas). Also, I like the suggestion provided by Tristan. I can make the interaction more smooth by making it continuous and gradual. For example, as he said, the user can hear a slight sound of running water when he gets close to the waterfall and the sound will become louder and louder as he gets closer to the waterfall. Also if the user leaves the waterfall, the sound will gradually fade away. Besides, Shenshen’s project inspired me a lot. Since I want to enrich my paintings into various pictures with some connections (storytelling part). And Shenshen’s project provides a way for the users explore or observe my project. They can use their eyes to “walk” in the painting. I can add these thought to my final project. And I really appreciate anyone who helped me promote my project or test it, especially Professor Moon!