Background
When I was young, I was fascinated by the magic world created by J.K. Rowling in Harry Potter. She has created so many bizarre objects in that world of magic that I still find very remarkable today. “The Daily Prophet”, a form of newspaper in the Harry Potter world, is the main inspiration of my final project. “The Daily Prophet” is a series of printed newspaper that contains magic which allows the image on the printed paper to appear as if it’s moving. It inspires me to create an interactive newspaper with an “AI editor” where not only the images on the newspaper will update every second according to the video captured by the webcam, but also the passage on it will change according to the image. In my final project, I will use Style Transfer to make the users’ face appear on the newspaper and utilize im2txt to change the words of the passages on the newspaper according to what the user is doing. I will build an interactive newspaper that constantly reports the users’ action.
Motivation
Even with the development of social medias which allow new information to be spread almost every moment and every second, it still requires human people behind the screen to type, collect and then post those news. However, if there is an AI editor that could document, write and edit the news on the newspaper for us, the real-time capability of spreading information of the newspaper would be even better. Thus, I want to create an interactive self-edited newspaper that asks an AI to write the news about the action of the people it sees by generating sentences on their own.
Reference
I’ll refer to the im2txt model on github https://github.com/runwayml/p5js/tree/master/im2txt here to create the video caption. This model will generate sentences according to the object and action the webcam video caught. I will run this model on the runway and then it will sent the result of the caption to html so that I can manipulate the outcome. Since some of the captions aren’t that accurate, I still need to find some ways to improve on that.