Yu Yan | What Do (AI) See: How AI Perceives Chinese Characters That Uses Handwriting Input Methods

 
Contrary to most language scripts, Chinese is a pictographic script whose structure and composition consist not of a simple linear arrangement of letters, but of interwoven combinations of strokes in different shapes. Therefore, there has been a phenomenon among current Chinese people called “character amnesia” (提笔忘字), which refers to the situation of forgetting how to write Chinese characters under the prevalence of the Smart Pinyin Input Method. To avoid the worsening situation of character amnesia, some people have started to use the handwriting input method instead of the Smart Pinyin Input Method to practice writing in their daily lives.

The handwriting input method refers to using a finger to handwrite Chinese on the phone, which will “translate” the handwritten scripts into standard typefaces to display on the screen. What Do (AI) See is a project based on a collection of people’s handwritten Chinese, the AI predictions of what the characters were intended to be, as well as the extraction of new appearances for existing Chinese characters based on the handwritten script. These new forms of written Chinese are derived from the moment of every recognition made by the handwriting input method during the writing process. For the AI to decipher the Chinese handwriting input method, they need to be trained by enormous sample datasets. However, this process is largely invisible to the user. This project shed light on the transformation of handwritten strokes into digital language. It asks what makes handwritten strokes a language and how is it that artificial intelligence sees and understands Chinese calligraphy.

People write characters by memory emphasizing the character’s most important strokes. As people write on their phones, this digitized handwritten character is interpreted by the AI which presents us with its best guesses. By recording the whole process of these interactions, this project monitors AI’s guesses as individual strokes of a character are drawn one by one by the human. Despite the obvious differences in the actual stroke combinations, it detects stroke compositions that AI recognizes as specific Chinese characters. Upon the detection, the stroke composition, as well as the AI’s best guesses, are captured and collected. What emerges from programming and assembling these collections is a database of new appearances of existing Chinese characters.

AI learning for Chinese characters has become sophisticated. But as the widespread use of Chinese begins to incorporate AI learning, we can take a deeper look and gain more insight into the characters by observing the learning of AI. This project not only serves as a lens to peek into AI’s mindset but also reveals the beauty of the strokes and structure of Chinese characters.