A brief introduction of our project:
We designed and implemented an interactive device that aims at extending human’s perception on colors, that is, electromagnetic waves. When a user scans a point on an object, our interactive device will play a note on piano corresponding to the specific color, with the distance between the user and the object corresponding to the loudness, in a high quality of sound. A more advanced use of this project is to scan through a color board and play some music.
How we come up with the idea.
In the previous research we were amazed by the interactive musical instrument that one can control the pitch by the distance from his hand to the device. Then we consider an interactive device that relates to music can be a newly designed musical instrument, that is, a human send some signal to a machine and it gives out a sound.
We thought about making a musical instrument at first, then we realized whatever we make could neither have the potential to excel the performance of piano, violin … etc. which has been tested in hundreds of years, nor have the potential to add more artistic or practical value into it in a musical instrument (for example, we can make a musical device that will also light an LED on press, but what’s the point of it?).
Yet in the following research we discovered a MIT neuro science project that shows: we can create a map in our brain from specific sound to specific pattern, for example, if in blind people’s mind we create the “beep” sound to the pattern of black-white strips, the blind people would be able to “see” the pattern in their brain. The team is building such a device to test the discovery.
link: https://youtu.be/2w3bfmL0RXg
This triggers the my thought as I just encountered the idea of extended perception at that time when I was randomly watching cool design videos online. Then the idea started to form in my mind:
From electromagnetic waves to mechanic waves: an interactive perception extending device
Since the existence of human, no matter what fancy forms human presents visual art: in different style of paintings, in calligraphy, for modern times in videos, the way of percept the basic element of visual art: colors, never changed. It seems like a cliché but human can only see colors, with eyes. In some ways, our understanding and aesthetics are determined by our sensors: the three type of cells in our eyes. (For example, for an “ugly” painting, is the painting itself ugly, or is our perception limited us to understand the beauty of it?)
The MIT project inspired us. It covert some color patterns to some simple sound patterns. But we can also map each color to a musical note, so that people can percept visual art using the prat of brain for audio analyzation.
After presentation Tristian recommended some articles that are relevant to our subject, after some search I find some online and planned to read them later (perhaps it will help with the final)
for example:
https://pdfs.semanticscholar.org/a1bd/629ba961a9635ec7ff186ed7b8171b31fbb6.pdf
/*Something else we want to mention
1. Trasty mentioned we could also use the device to specially target at the blind people, either to help them understand color or to help the in daily life (such as sensing red light and green light). We considered it as a very useful advice, and it is a practical idea (while our perception idea is perhaps a bit philosophical?). So we considered it also a very important purpose for our device.
2. Actually before that we thought of implement a music game, but Rudi told us there is already a very similar game. It taught me a important lesson: always google your “original” idea before you “invent something new”.
*/
Interactive Model
After studied the theory of interactivity and extended perception and recalled what we read and learned in class before, we the finalized an interactive model for such a device:
1. a agent (user) send a signal to an interactive device with a coordination in 3D space
2. the device convert the electromagnetic waves (color) emitted (or reflected) from the position in to mechanic waves (sound), then send the sound back to the user
3. the user process the sound and the color itself with neuro cells that analyzes visual art and audio art.
4. The agent decided another color it wants to sense and gives signal to the machine.
Device and UX Design
Based on our model, what we need to make is a handy device with a set of sensors in the front, when the user use it to scan different objects, it will play a piano note that corresponding the color of the object. Some key elements are: color sensor (to sense color for sure), a ultrasonic sensor to sense distance covert to loudness (to have a full sense of music, and perhaps to pay some attribution to our first inspiration, the music instrument). MP3 module, Speaker/headphone/SD card and related interface These are all functional necessary parts.
We also pay special attention to the user experience design. The idea of UX came to me in class for the first time, and I immediately realized its importance when someone mentioned the doors at our university. Indeed they are either slow or heavy. So we have the following UX guidelines:
1. hide all the wires and technical parts and gives out a clear Interface
2. gives the user a basic idea on how to use it on the first look.
To achieve 1. we laser cut a box to contain the Arduino, all the modules and all the wires, with only part of the sensors showing out.
Achieving 2 is more tricky, we finally settled down several ways: we have a earphone interface to suggest the user to plug in earphone and listen, or plug in a speaker; the name suggests the device can let you “hear” the colors; and we made the shape of our device into a gun, or rather, a handy scanner we commonly see at supermarket to scan barcode or Wechat pay code, as the intuition of seeing such a stuff is to use it scan something, or at least to point at something closely. (We also have an alternative to make it the shape of a torch, but we finally gave up the idea because the intuition of seeing a torch is to use it to light up something, with rather large distance, while the color sensor has best performance when it is close an object.)
Implementation Highlights
The UX design took 3 iterations and 1 more in the future:
The first time we just connect all the units and all the wires and boards are showing outside;
The second time we laser cut a small box to contain the Arduino, the mp3 module and most of the wires, but there is no explicit hint of the function of this device as well as how to use it.
So in the third time we developed the current model
The technology part is also very challenging, as we have never encountered most of the units we used so we have to learn from scratch. I tried 3 different mp3 implementation and SD card implementation, tested on the color sensor, and tried different speaker implementation while Steve researched on color theory and computer color theory to create a RGB-HSV-Frequency mapping that can be more easily understood my human, and calibrated the color sensor by hand. The 3D-printing caused an overnight stay at the school.
Modification
After the presentation, we realized there are still many things we can improve on.
1. setting
Rather then designing a universal application of our device, we could focus on a specific scene where it is used. For example, it is used in a museum where people can use it to scan some paintings. This way of designing enables us to ignore the complex interference in different environment and focus on the details and improving the performance in a specific setting.
2. shape
Although the reason of our device’s shape is explained, it is, a bit weird as I now realized the subtle difference between a gun and a gun like scanner. It is strange to point a gun at Mona Lisa. A very simple modification yet can generate good result is to make the box horizontal so it is more like a scanner.
3. duration
So far every note has the same duration, which limits the music it can play. Right now I can think of two ways of modification: add a control interface so that the user can control the duration, it gives best performance but also make the device a bit more difficult to use as user needs to master a new function; or we can let the chip decided: if it keeps sensing the same color, it will play a long note until the color changed instead of a note every 500ms
Conclusion & Reflection
Our goal is to build a interactive perception extending device. We intended to let the user actively experience visual art — colors in the form of audio arts. The outcome is quite satisfying. We conduct several test among our friends and received good results. The interaction model we proposed succeeds.
To be honest I underestimated the difficulty in the project a bit, as we always operated on things we don’t know the theory and we have to learn everything in two weeks by our self and from scratch — color theory, inter-hardware communication, theory of speakers/earphones, user experience design … sometimes I was very worried, because we sometimes don’t know what we are doing: the SD card cannot load on serial 115200 and we don’t know why, we tried whatever we came up with, is it a bug in coding, a conflict among pins? etc. and finally when I changed the serial to 9600 it worked! It happens everyday and finally we solve the problems one by one either knowing why, or perhaps not knowing why at first. Then I thought of what Rudi said in class (or on ppt?): it is not an exploration if you now what you are doing.
It is true.
Crisis broke out in presentation. We force ourselves too calm down and solve the problem: the extended board somehow got detached from Arduino. We have to disassemble all the boxes to fix it. But for us, it is never too late.
Tristian, who knows the concept and theory of extend perception well, praised our project and pointed out the future research in the field. Interactivity is a complex systematic behavior, and so is human brain and perception. The combination of these to does not only serves for entertainment, or help the blind, but it might triggers more interesting study and ideas on the frontier science: the cognition and interaction in human (the awesome function of the brain is the “interaction” of cells!) and inter-human, as well as what is art and aesthetics.