Category: capstone2022

 
Path is an interactive virtual experience connecting audiences’ presence and attention with Chinese paper cutting, which allows them to navigate through a 3D web space and generate a paper cut art. Interactive experience is not only about user actions, but also involves their subsequent cognitive processes, such as the perception of the interactive system, the reflection on their presence and the environment, and the understanding of the behaviors as well as subjects in the experience. Following research on the impact of interactive experience on human cognition, this project is investigating how interaction design can promote people’s understanding of the marginalized traditional culture on a different level than before, specifically in this case of Chinese paper cutting, how the sense of presence and multiple layers of feedback can help audiences understand the subject better.

Due to the pandemic situation, the experience is shifted from a physical installation to a web-based virtual exhibition. The contents to deliver are arranged within a fan shape and have four categories, including patterns, techniques, contexts and spirits. Each piece is attached to a symbol extracted from the repetitive and representative units from the corresponding paper-cuts. This repetition is also one characteristic of Chinese paper cutting. The whole design is centered around the core spirit of Chinese paper cutting, where it uses its special language of cutting to hide infinite possibilities between square inches of a small piece of paper. The user can move in the 3D web space to interact with them. Meanwhile, the project will transform the path of user movement to an animated pattern related to paper-art in order to highlight the users’ presence and attention. It is because, for one thing, the path can reveal their stream of thought in face of Chinese paper cutting contents; for another, when it changes size based on users’ dwell time, the path can also indicate how much the user pays attention to the content of a certain spot. The path transformation is based on the most widely used form in Chinese paper cutting, Tuanhua(团花), where paper is always folded, cut flat and unfolded with separate but linking graphic units. The path leaves negative space at first, and will shift between positive and negative every time the user has stopped for a while. The fan shape area will be replicated to form symmetrical circles, just like how we unfold paper after cutting, which relates to the Yin & Yang philosophy as well.

By connecting users’ virtual presence and their underlying thoughts and providing real-time feedback with an animated pattern, this project intends to give contemporary audiences an interactive and reflective experience about Chinese paper cutting, which is one of the most important forms of Chinese folk art but is marginalized as a result of globalization and capitalism. The interaction paradigm in this project seeks to make a positive impact on audiences’ cognition in the given cultural context, which can also be extended to other traditional cultures.

 
Lockdown Ties is a project that documents neighbors’ collect glimpses into residents’ lives under lockdown. It is a web-based video diary project and creative documentation of one neighborhood’s experience during the covid19 pandemic in Shanghai in 2022. Although face-to-face interaction is prohibited, residents develop new ties that exist uniquely during the Covid-19. Until this time of the writing, impressions shared usually zoom in on the mishaps and the suffering. However, besides the sadness, individuals also shared more happiness concerning their neighbors on the social platform. The lock-down forces residents to enhance the interaction between neighbors since living in groups becomes a new way to survive. By presenting a collection of videos from my neighbors, this project aims not only to document how people’s lives have changed because of the lock-down but also to present the enhanced group interaction inside a community.

My community in Pudong Laoshan Road, with 400 people arranged in 4 buildings is not allowed to leave the building, and the compound of food comes from both group purchases and government support. This website is a re-creation of the actual space, people are absent but viewers can encounter their traces when they explore the webpage. The recorded videos will be displayed on the website once the viewers get closer to the layout object. I constructed separate homes based on residents’ room numbers. The project consists of a website, constructed as a simulation of the housing complex with the residents’ individual rooms, and numerous audiovisual materials recorded by the inhabitants themselves, responding to my prompts about lockdown routines. The display of furniture is based on the content of the record. Viewers are put into the role of a fellow resident to walk around, seeing the actual lock-down lives and hearing each resident narrating their stories. When the character gets close to the furniture, videos will automatically be displayed. The lock-down atmosphere is also demonstrated through the medical workers displayed at the border.

Within this space, online ‘rooms’ reflect the residents’ actual spaces, while the outside environment suggests an ongoing lockdown – outside the border of the flats, medical workers in hazmat suits can be seen. The visitors can explore the complex, approaching different furniture items in the rooms to watch the video diaries. By inviting residents to narrate and film their stories, this project serves as a documentary to store memories during the special period. Compared with the impression of modern, high-tech, and busy streets, this project reveals another living style that Shanghai people are creative and energetic even being lock-down at home. The culture of Shanghai is also revealed in the project as viewers can listen to the Shanghai language. This project also intends to direct people’s focus to the enhancement of community brought by the Covid-19. Though documentaries traditionally utilize filmmaking, web-based documentaries expand the definition. The ‘subjects’ of the documentary can participate in various ways, recording the video material themselves. This project serves as a reflection on the boundary between public and private, intimate and isolated.

 
WaveNav is a ML powered wearable accessibility system that allows users to utilize customizable gestures to complement KBM (Keyboard and Mouse) navigation of PC navigational interfaces. It is powered by TinyML, which includes edge impulse model trainings and an Arduino Nano 33 BLE Sense to provide fast and accurate gesture classification, and AutoHotKey (AHK), a Windows scripting language. To be more specific, the Edge Impulse model is run through an arduino sketch with extra code that set certain outputs to respective simulated keypresses, which is then picked up as an input by AHK and AHK then performs the task associated with that key combination. For example, tilting your head to the left could trigger the browser’s “back” button, and the opposite for “forward”. Another example would be kicking your left foot out three times to open Google Chrome and a specific webpage. The small form factor allows users to wear the device on any dextrous parts of the body, and everything from the gestures themselves to the AHK outputs can be customized to individuals’ specific needs. While the system was inspired by the concept of easing computing for individuals with physical disabilities, anyone can take advantage of navigational gestures. Academic research has shown that while utilizing gestures for very specific inputs (similar to KBM) is not efficient or useful enough, gesture based navigational control, which has more tolerance for different movements, has shown great user feedback. Overall, this project is intended to be a proof-of-concept that TinyML along with open source scripting languages like AHK can be a powerful tool for accessibility, or even music or performance arts.

 
Contrary to most language scripts, Chinese is a pictographic script whose structure and composition consist not of a simple linear arrangement of letters, but of interwoven combinations of strokes in different shapes. Therefore, there has been a phenomenon among current Chinese people called “character amnesia” (提笔忘字), which refers to the situation of forgetting how to write Chinese characters under the prevalence of the Smart Pinyin Input Method. To avoid the worsening situation of character amnesia, some people have started to use the handwriting input method instead of the Smart Pinyin Input Method to practice writing in their daily lives.

The handwriting input method refers to using a finger to handwrite Chinese on the phone, which will “translate” the handwritten scripts into standard typefaces to display on the screen. What Do (AI) See is a project based on a collection of people’s handwritten Chinese, the AI predictions of what the characters were intended to be, as well as the extraction of new appearances for existing Chinese characters based on the handwritten script. These new forms of written Chinese are derived from the moment of every recognition made by the handwriting input method during the writing process. For the AI to decipher the Chinese handwriting input method, they need to be trained by enormous sample datasets. However, this process is largely invisible to the user. This project shed light on the transformation of handwritten strokes into digital language. It asks what makes handwritten strokes a language and how is it that artificial intelligence sees and understands Chinese calligraphy.

People write characters by memory emphasizing the character’s most important strokes. As people write on their phones, this digitized handwritten character is interpreted by the AI which presents us with its best guesses. By recording the whole process of these interactions, this project monitors AI’s guesses as individual strokes of a character are drawn one by one by the human. Despite the obvious differences in the actual stroke combinations, it detects stroke compositions that AI recognizes as specific Chinese characters. Upon the detection, the stroke composition, as well as the AI’s best guesses, are captured and collected. What emerges from programming and assembling these collections is a database of new appearances of existing Chinese characters.

AI learning for Chinese characters has become sophisticated. But as the widespread use of Chinese begins to incorporate AI learning, we can take a deeper look and gain more insight into the characters by observing the learning of AI. This project not only serves as a lens to peek into AI’s mindset but also reveals the beauty of the strokes and structure of Chinese characters.

 
Gao Su Du De Qian Jin! is a shadow art installation that investigates and calls attention to the construction and demolition (C&D) waste problem in China and its relationship with China’s modernization. With the fast urbanization throughout China, the C&D waste has become a major contributor to China’s waste production. However, this connection between the two is rarely mentioned, instead the attention of the public is steered towards the economic growth and the construction achievements of urbanization with propaganda. This project attempts to uncover this rarely-recognized relationship between China’s urbanization and C&D waste, calling for attention to this problem, by combining and comparing imageries that represent both sides with a sculpture, and presenting the irony that a seemingly prosperous and praised process is rather messy when looked at from another perspective.
This project consists of a rotating sculpture made from recycled concrete, a series of images of abandoned/torn buildings, as well as a light source that projects red lights on to the sculpture and casts its shadow onto the wall. Modeled from two distinct imageries that represent two sides of China’s C&D waste problem — an “Open and Reform” propaganda poster and a half-torn building — the sculpture rotates slowly while its shadow constantly morphs between those two imageries. A Pinyin transcription of the poster’s title “高速度地前进!” is attached to the bottom of the sculpture and rotates with it, appearing along with the human figure and disappearing when the building is shown. Supporting images of more broken/abandoned buildings are also placed surrounding the main piece.
Through the comparing of the two shadows of the same object, I wish to convey that the seemingly triumphant achievement of China’s urbanization and the problem of C&D waste is two faces of the same issue. The title is also purposefully presented in Pinyin so that its sound is more directly accessible to audiences speaking any language, creating the feeling of chanting empty propaganda languages. While the shadow rotates to the broken building and the line disappears, the audience is invited to think about their place between these two sides of the story and wonder what is the result of such “development at a great speed”.