My final project is mainly inspired by the re-created famous paintings, especially the portraits. Some people replace the Mona Lisa’s face with the Mr. Bean’s face and the painting is really weird but interesting.
Also, I found that some people tend to motivate the poses of the characters in the painting, such as The Scream:
Therefore, I want to build a project to let the users add their creativity to the famous painting and personalize the paintings to recreate these paintings. It reminds me my previous assignment for style-transfer. For that assignment I use a painting from Picasso to train the model, so that everyone or everything showing in the video can be changed into Picasso’s style. Even though the result is not that good, it still shows a way to personalize the painting or to let the users create their own version of paintings.
My idea is that the user can trigger a famous painting by imitating the pose of the characters in that painting. For example, if the user wants to trigger The Scream, he or she needs to make the pose like this: 😱. After the painting showing up, the user can choose to transfer the style of the live camera to the style of The Scream. If the users want to change to another painting, they just need to do the corresponding pose to trigger the expected painting.
Reference
My reference is the project called moving mirror. The basic idea is that when the user makes a certain pose, there will be lots of images with people making the same or similar pose.
What attracts me most is the connection between images and human poses. It displays a new way of interaction between human and computer or machine. Users can use certain poses to trigger things they want, and in my project it is the painting.
The second one is style-transfer. It reminds me some artistic filters in Meituxiuxiu, a popular Chinese photo beautification application. These filters can change the style of the picture to sketch style, watercolor style or crayon style.
But the filter is only for still picture. I want to use style-transfer model to add this filter to the dynamic video so that the user can see their style-changed motions in a real time.
To differentiate between the terms of VJing, Live Cinema and Live Audiovisual Performances, we must understand the relation between them and their supposed hierarchal standings. Live Audiovisual, as depicted by Ana Carvalho, works as an “umbrella that extends to all manner of audiovisual performative expressions” (134). This artistic umbrella harnesses under its wing, expressions that include VJing, live cinema, expanded cinema and visual music. The term itself is generic and vast for it fails to identify a single style, technique or medium, rendering it complex at the same time. Its “live”, “audiovisual” and “performative” features are grounded in a nature of improvisation. This sense of improvisation has developed alongside the increase in immediacy and ‘liveness’ in technology (cameras, mixers, software) for image and sound manipulation that now permit the capturing and presenting of performance simultaneously while an action is happening (134). The category is often commended for the opportunity it provides audience members across the global to understand the numerous innovative expressions that it entails.
Though similar, the practices of VJing and Live Cinema are crucially distinct under the wing of Live Audiovisual Performance. VJing, as we had learned in the past, runs parallel to the responsibilities of a disk jockey (DJ). VJs are rooted in the manipulations of live visuals as DJs are rooted in the manipulation of audio. It is however, unlike Live Cinema, much more interactive based in terms of its relationship with the performer and the audience. The acts of VJing, can be relatively restrictive when compared to the likes of Live Cinema. VJs may have the artistic freedom of improvisation and a lower demand for narration, but for the most part, VJs lack the upper hand in a performance. They have less control; having to rely on their fellow collaborators (lighting engineers, DJs, set producers) as well as the response of the audience members. Since the VJing aspect is so heavily interaction-based, it is here we see that the performative aspects of VJs are often times more than not, restricted to the monotonous setting of a nightclub, where they are treated like wallpaper and supporting acts.
Live Cinema on the other hand is a much more hands on and demanding genre. In essence, the goals of Live Cinema are described to be much more personal and artistic to the eyes of the creator as well as the audience member on the receiving end. This is where “many live cinema creators feel the need to separate themselves from the VJ scene altogether” (93), for goals of live cinema are relatively difficult to achieve in the club environment. The creator is given a much “larger degree of creative control over the performance” (95), there is much more leeway for the artist to create what they want, given that they don’t need to follow trends and situational norms. Furthermore, compared to VJing, Live Cinema houses a much larger importance on narration and communication, where story telling then becomes a needed skill in articulating meaningful representations to the audience.
Examples
Live Audiovisual Performance
Ryoichi Kurokawa is a household name in the genre of Live Audiovisual Performance. The performance’s usage of synthesized and impactful sounds that play in collaboration with distinct visuals do not have strong narrative sense, but nonetheless work as a personal and artistic piece. His usage of human depictions, animal species and other meaningful representations further convey a sense of artistic expression to the audience member.
Live Cinema
This performance by Ge-Suk Yeo can be categorized underneath the Live Cinema wing of live audiovisual performances for its usage of concrete narrative aspects to form visual art. The theme of aquatic life and narration of the light down under within the dark seas is prominent in this performance.
VJing
This example of VJing is a standard performance that does not necessarily harness any narrative components, but makes use of live manipulation of audio and its visual storage to create an atmosphere that allows members present to become harmonious with the performance. The performance brings people closer to the performance they are seeing.
Sources:
Carvalho, Ana. “Live Audiovisual Performance.” The Audiovisual Breakthrough, 2015, pp. 131–143.
Menotti, Gabriel. “Live Cinema.” The Audiovisual Breakthrough, Edited by Cornelia Lund, 2015, pp. 83–108.
In this project, my partner and I are aiming to educate young children on the dying languages and dialects in different provinces around China. Unlike most Western countries, China has multiple languages and variations of language spoken throughout the country. Even though Mandarin is the most common one, some cities like Shanghai also have their own dialect called “Shanghainese” (上海话) (Asia Society). In order to preserve cultural dialects such as Shanghainese and others, my partner and I have devised a plan to educate younger children on the many dialects throughout China. In many articles, it is evident that languages other than Mandarin in China are being phased out because the Government is requiring people to use Mandarin in official scenarios (GaoKao exam and government forms for example) (The Atlantic). It is important to maintain the vast array of cultures in China through spoken language . We will appeal to the younger generation through a puzzle game which displays a map of China. The child will take one of the provinces (a singular puzzle piece) out and trigger the screen and audio to begin playing a clip from a native speaker of that region. Each province that is available to source from will have a native speaker from NYU Shanghai introducing themselves. This way children can listen, practice, and most importantly, be aware of the diversity among languages in China. Hopefully through this project, my partner and I can inspire classmates, teachers, mentors, and children to keep dialects alive in China.
C. Project Plan
The main goal of this project is to preserve the cultural traditions of different languages and dialects around the provinces of China. In order to appeal to younger kids through education, we’ve created a puzzle game which correlates with a screen, joystick, and audio speakers. The first component is creating a map of China, which is split into different puzzle pieces based on provincial boundaries. Each province puzzle piece will be linked through wires to the screen so that when it is taken out of the puzzle, a video of the native speaker from that selected province introducing themselves plays. The joystick will be linked to the screen and control a cursor which can move over a virtual map of China. As the cursor mouses over a province, the voice of that dialect begins to whisper, creating intrigue in the participant. My partner and I plan to film and record different students from NYU Shanghai who come from separate provinces such as Guangdong, Shanghai, Beijing, Hebei, Sichuan and more. We realize that not every province is represented at our school, so we will try and accrue as many as possible. Although, the search process for finding students that speak dialects other than Mandarin has been difficult because of the Government mandated policy of Mandarin only learning in recent years. We have designed the base of the puzzle and plan to laser cut it while we will 3-D print the model of China and its pieces to fit within the base. We also need to coordinate the sensor wires with the puzzle pieces once they are fully printed. After, we will link the joystick with the on screen graphics through the Arduino values we’ve learned and processing skills to create a virtual map.
D. Context and Significance
In my preparatory research for this project, I stumbled upon a similar project named Phonemica which also aims to preserve Chinese cultural dialects by physically traveling to those places and recording the people there. I found it very inspiring that there were other people who are also passionate about preserving the local culture of China. Our project differs from Phonemica though in that we are only in one location and are recording people in our school. We are also trying to appeal to a younger generation by making it a game, rather than a website. Another project that inspired me was the musical bikes, just a block away from our school. The premise of this project was interactivity through music and exercise. When all four people were riding the bikes, the whole song would play. I think that our puzzle and joystick feature creates a similar incentive because at first the voice whispers when they mouse over it with the joystick and only once the child takes the puzzle piece out will it fully play the phrase. The joystick creates an incentive to take out a puzzle piece just as the songs created an incentive for all four people to be exercising on the bikes. I think our project is unique in that we are enlisting the help of the whole school to guide our educational process. It’s not only my partner and I working to preserve Chinese dialects, but those from that region too. To me, it aligns with my definition of interaction because it involves a child playing with a puzzle which translates a result to the screen and helps that child learn. It’s a conversation through the physicality of the puzzle, and the encouragement to repeat the phrase after they are finished watching the video. Just as Chris Crawford explains in what exactly is interactivity? , interactivity has to do with conversations moving back and forth between the user and the machine. It’s not just once, but multiple times in a fluid back and forth process (Crawford). I agree with Crawford and think that my project on interaction is a conversational learning process geared towards children. Hopefully, this project can inspire others in the future to build similar models such as Phonemica’s to help maintain cultural identities in China. These educational tools such as speech repetition games, apps, and computer games could possibly be used in schools around China and specifically in each province too. Seeing as the future is digital, I hope Chinese dialects can survive the new wave of technology.
When I was young, I was fascinated by the magic world created by J.K. Rowling in Harry Potter. She has created so many bizarre objects in that world of magic that I still find very remarkable today. “The Daily Prophet”, a form of newspaper in the Harry Potter world, is the main inspiration of my final project. “The Daily Prophet” is a series of printed newspaper that contains magic which allows the image on the printed paper to appear as if it’s moving. It inspires me to create an interactive newspaper with an “AI editor” where not only the images on the newspaper will update every second according to the video captured by the webcam, but also the passage on it will change according to the image. In my final project, I will use Style Transfer to make the users’ face appear on the newspaper and utilize im2txt to change the words of the passages on the newspaper according to what the user is doing. I will build an interactive newspaper that constantly reports the users’ action.
Motivation
Even with the development of social medias which allow new information to be spread almost every moment and every second, it still requires human people behind the screen to type, collect and then post those news. However, if there is an AI editor that could document, write and edit the news on the newspaper for us, the real-time capability of spreading information of the newspaper would be even better. Thus, I want to create an interactive self-edited newspaper that asks an AI to write the news about the action of the people it sees by generating sentences on their own.
Reference
I’ll refer to the im2txt model on github https://github.com/runwayml/p5js/tree/master/im2txt here to create the video caption. This model will generate sentences according to the object and action the webcam video caught. I will run this model on the runway and then it will sent the result of the caption to html so that I can manipulate the outcome. Since some of the captions aren’t that accurate, I still need to find some ways to improve on that.
Dancing With Strangers is a development of my midterm project Dancing With a Stranger. In my midterm project, I used Posenet model to mirror human movements on the screen and exchange the controls of their legs. With my final project Dancing With Strangers, I’m hoping to create a communal dancing platform that enables every user to use their own terminal to log onto the platform and mirror all the movements on the same platform. In terms of the figure that will be displayed on the screen, I’m planning to build an abstract figure based on the coordinates provided by Posenet, The figure will illustrate the movements of the human body but will not look like the skeleton or the contour.
Motivation
My motivation for this final project is similar to my midterm project: interaction with electronic devices can pull us closer, but can also drift us apart. Using them to strengthen the connections between people become necessary. Dancing, in every culture, is the best way to pull together different people, a communal dancing platform can achieve this goal. Compared with my midterm project, the stick figure I create was too specific, in a way, being specific means assimilation. Yet people are very different, they move different, Therefore, I don’t want to use the common-sense stick figure to illustrate body movement. Abstract provides us with diversity, without the boundary of the human torso, people can express themselves freer.
Reference
For building the communal dancing platform, I’m using the Firebase server as a data collector that records live data sent by different users from different terminals.
For the inspiration of my abstract figure, I’m deeply inspired by the artist and designer Zach Lieberman. One of his series depicts the human body movement in a very abstract way, it tracks the speed of the movements and the patterns illustrate this change by changing its size. With simple lines, bezier and patterns, he creates various dancing shapes that are aesthetically pleasing. I plan to achieve similar results in my final project.
