Project name: WHAT? I cannot SEE you talking!
Professor: Eric Parren
Documented by: Skye Gao
Date: March 11th, 2019
Project Description
My project is a glasses-like device which exams the interference between vision and hearing. The idea was initially developed upon my personal experience: as a very short-sighted person, I often run into the situation where without my glasses or clear vision, I cannot hear people talking clearly. While I conducted researches, I found out this is a common phenomena among people with poor visions. According to Lawrence Rosenblum, professor of psychology at the University of California, Riverside, the phenomena – though not highly researched – is plausible. “What we understand about the perceptual brain is that our senses integrate, and though we don’t think we use sight in hearing, we lip-read all the time,” and we also get clues from teeth and tongue movement. The statement is also in a way supported by the McGurk effect, a perceptual phenomenon that demonstrates an interaction between hearing and vision in speech perception. ( A link for illustration of McGurk effect)
Based on these findings, I built my project to simulate a blurred visual experience which will have effect on hearing. The project is intended to project the interference vision has on hearing, and discover possible by-effects on an individual level.
Perspective and Context
As Maurice Merleau-Ponty states in the book The World of Perception: “rather than being absolutely separate, each of these qualities has an affective meaning which establishes a correspondence between it and the qualities associated with the other senses.”(60). There are connections between each senses, however, people with full sense seldom realize such connections. How do we actually perceive the world? How do our senses connect with each other? What is the balance between them and what will happen if we break the balance?
My project hence picks the specific sense of vision and hearing, and experiment with the interaction between them. People name our senses with specific names, but the truth is they are always corresponding and coexisting with each other. By weakening one sense, I tend to break the harmony and reveal the indivisibility among them.
Development & Technical Implementation
There are two essential parts for my project: visual experience and acoustical experience. I first started with the visual part and tried two ways to implement the idea: one is to make some videos with high/poor qualities; the other is to built physical devices that make people see unclearly.
As for the first approach: I first found some random speech videos and covered the screen with frosted plastic board for temporary effect. However, the result for user-testing was not ideal. The majority of the users were aware that they were under test and thus would consciously pay special/no attention to the content of the video with hearing. As I discussed with professor, I also got recommendation that conversation may be a better way to bring out the interference between hearing and vision.
Therefore, I started to go with the second approach. In terms of the physical device that makes people’s vision blur, my idea was to built a glass to achieve the effect. Considering that though a pair of glasses with lens can certainly have effect on normal people, the lens can also cause discomfort or even harm on peoples’s eyes. Thus I decided to use some simple material to simulate the effect. One kind of ideal material is frosted plastic. After trying out with several kinds of plastic, I found the one used for plastic gloves works the best, and I got several pair of these gloves from delivery food. 
With these plastics, I borrowed a pair of protective glasses from Lily. I simply attached the plastic to the glasses (the image of glass one and the vision through it are shown as below):
As I proceeded to some user tests, I realized that the outcome was really differed from person to person as well as the environment: with the glasses on, people tend to identify the words more clearly when talking to a familiar person or/and in a more quiet environment, and more unclearly the opposite. Also, people having chronic experience with glasses were more likely to understand situation than others. Therefore, I learned that the test should be conducted with settled variables. Considering the final test will be in a certain room which the environment is settled, I focused more on the objects of conversation (person and content).
To ensure the randomness of the conversation objects, I searched for some “common misheard sentences” online and paired up random people to do the test. Compared to previous tests, such arrangements has led to much more obvious outcomes as I expected.
As I played around with the material, I found out that different layers of frosted plastic can also create different blurred visions. Therefore, I further made another pair of glasses with double layers of the same plastic which made the vision extremely blurred(glasses2 and the vision through it are shown as below). The outcome of user testing are very interesting: with two layers (a very blurred vision), users’ hearing are actually better than wearing glasses of less blurred vision. According to some users’ feedbacks, when their visions fail more, users tended to rely more on hearing and hence adjusted to the situation quickly.
Presentation
The presentation went quite well in class. My classmate Rosie participated in the tests as a volunteer, and Luis helped me read out the sentences to her. While Luis read the same sentence three time, Rosie wore glass two, glass one and did not wear glasses for each time. The result quite matched my assumption: with blurred vision, Rosie could not hear clearly than usual; Moreover, with glass two, Rosie said that she heard even more clearly than wearing glass one. While other people played around with glasses during conversation, the effect that blurred vision can lead to poor hearing was confirmed.
I did not manage to record the presentation in class, but here is a test I run after class with students outside our class:
I also got several feedbacks and suggestions during the class presentation:
- Since each test the testee would hear the same sentence for three times, they may become familiar with the sentences and thus spare the role vision plays in the process. Therefore, I got suggestions that I should read different sentences each time to the testees.
- As for the person who reads the sentence, the testee can also get familiar with his/her voices. Thus I could try to use audio recording or let different people read sentences to overcome the influencing factors.
- To better simulate the vision of a poor-sighted person, I could explore other materials to create more realistic effect. For example, instead of using plastic to create only blurred vision, I can also use some materials to make double images effect.
Conclusion:
This project is a exploration of daily experience with sense interference between vision and hearing. During the whole duration of building my project, I learnt the following things:
- Always left enough time for project implement and adjustment. While we had a week for project preparation, I still spent more than three day trying to figure out how to implement my ideas. After I settled down with my final plan, I only got about two days to built everything. Such intensive duration of working left me with limited time and choices on materials and the scale of my project. At the night before the presentation, I got feedback that the blurry vision was not like shorted-sighted vision very much. Though this was only one feedback out of twenty, when I tried to improve this point, I did not have enough time for perfection. So I could only make full use of what I had and take this as a random sample.
- People’s perception of sense really differs from individual to individual. Such outcome should not be considered as failure, instead, it is exactly the by-effetc I wanted from such experiment. Just as Irwin states “conception is the faculty of unique individual to appropriate and mold reality as presented in perception into a systematic experience of cognizance” (25), people’s experience with sense and perception is not only demonstration of some definitions but parts of a larger picture of conception.
Future improvement: Considering all the feedback and suggestions I got from the precess of building my project, I am thinking about 1) designing my test more cautiously and let the outcome more accurate; 2) trying different materials to experiment with various visual experience, and exploring the more connections between vision and other senses.