Inspiration
The inspiration of this project is quite ridiculous. At first, I didn’t have any ideas about my final project, so I just browsed some random things on the Internet. Then I found this picture:
And after I clicked it to learn more about it, I found that there were so many interesting and weird spoof paintings on the Internet. These paintings reminded me that I could build a project to let people add their creative ideas and their characteristics to the well known paintings. Besides, I noticed that people tend to imitate the signature moves of the characters in the painting, such as Mona Lisa and The Scream. The connection between the motion or poses and visual communication reminded me a project called Move Mirror. Here is the demonstration of it.
I really liked the connection and I wanted to apply such an idea to my project. In my case, if the users imitate the poses of the figures in the painting, there will be a corresponding painting beside the canvas to tell them which painting the machine think they are imitating.
The last one is the artistic filters of Beauty Camera that can transfer the original camera style to the oil painting style. I wanted to use style transfer model to achieve such effect.
My Project
Firstly, there are six paintings on the top of the web page working as the reference. The user can click the “Load Dataset” button and then click the “Start Predicting” button to play the project. If the users imitate the poses correctly, they can see the painting on the right side of the canvas. Also, if they press the spacebar, they can transfer the style to the painting style. You can see a brief demo video through this link:
https://drive.google.com/file/d/120leTwPVZvnTRMPI0Y-0_3AK9wFFSrUC/view?usp=sharing
Methodology & Difficulties
I use KNN & PoseNet and Style transfer to build my final project. I have already used KNN&PoseNet in my midterm project so the logic is similar. I just need to define the classification result and the corresponding output. For this case, the output is the picture of the painting and the style of it. For the style transferring, I referred to Aven’s example about multi-style transferring and used the array to decide which kind of style it would display.
One if the difficulties is the logic of my project. At first, my plan is to let the user take a screenshot and the result of style transfer will show on that image. However, I failed to take a screenshot directly. The only thing I could do was using saveCanvas() to download the screenshot and the outcome was not that good. Thus, I gave it up and tried to show the change through the live video rather than photos. And it worked well.
The other problem is the outcome of style transfer models. I chose a lot of famous paintings as follows:
However, the result of training models was lower than my expectations. Some of them were quite similar, so I just abandoned them and found different one. And these are some results of the models.
According to Professor Aven, it was because some of my images didn’t have vivid shape and color. Thus, the model couldn’t give style with strong features. I accepted his suggestions to choose the part with bright color and strong sense of geometry instead of the whole image. And the result was better than before. For example, I only used a part of the portrait of Van Gogh to train the model.
What’s more, I learned how to train several models at the same time. Before I only changed the file but didn’t create new checkpoint and model folders. In this way, I can only get one model because the latest one could cover the previous models. Now I know how to create the checkpoints and models and download multiple models.
The last step is to beautify the UI. I chose a simple but classical image as the background to fit the characteristic of a gallery. Then I only kept “Start Predicting” and “Load Dataset” buttons and changed their original style to an artistic one. Next I made some cards with the basic information of the paintings and the paintings themselves. These cards were put on the top of the web page as a reference for the users.
Significance & Further development
I have noticed that building this project helped me learn more about the paintings, especially their name and creators. Therefore, the significance of my project could be teaching the users some basic information about the famous paintings. For the further development, I wanted to enhance the educational function of my project. If the reference card contains more information such as the background or style of the paintings, the users will learn more about the paintings in a relatively interesting way. However, it’s essential to figure out how to draw the user’s attention to the reference. My idea is to add some text bubbles on the painting so that it seems like the figure in the painting is telling the information to the users. Also, I can add some sound files in my project to enrich the form of interaction. In addition, I plan to increase the types and the number of the paintings to enrich both input and output.