Project B Reflection
Kiri Tian, Project CONVERT: interactive converting between the audio and the visual
ELEVATOR PITCH
By interacting with the web, users can create their own art pieces by either converting the audio input into something visual or converting their drawing on the web into sounds.
Abstract
Synesthesia is a neuropsychological trait in which the stimulation of one sense causes the automatic experience of another sense. It is a genetically linked trait estimated to affect from 2 to 5 percent of the general population.
The project CONVERT is inspired by the idea of synesthesia – this atypical trait provides people with the opportunity to feel the world with a more stereoscopic, layered, and rich texture.
Making use of minimalism by adopting simplicity and black&white background, the focus is shed on the area of interaction – connecting the two senses.
Front Page:
From the audio to the visual:
From the visual to the audio:
Practical Reflections:
1) Process: Design and Composition
I’ve always kept in my heart this idea of creating something that connects the senses of hearing and seeing, so the idea-generating process was not hard for me. I write the javascript sketch first, and then designed the visual appearance of the website. At first, I tried to make the page look fancy by adding an abundance of elements; but once I added the java sketch, it was hard to tell which part of the website was the core and which was just peripheral. So I decided to make the page as simple as possible both in terms of the operations and the aesthetics, to make contrast with and highlight the part I want to express the most: the interactive art generation.
There is a small trick I put on the front page: after bouncing for several rounds, the little semi-transparent black circles remain at the bottom of the page. But as time progresses, they will gradually swell and eventually inundate the whole page (usually after 4 minutes if staying on the front page).
2) Process: Technical
One of the most tricky parts was to draw the huge circle in the middle of the front page. It is a CSS shape, not a javascript one, although you can actually interact with it. I tried to use Java to create it, but that wouldn’t allow the smaller bouncing circles to show behind the huge circle unless I put two individual Java sketches with two canvases. Instead, I came up with a more effective solution to draw curves twice to create a circle, and set the color to gradually change from white to black when the mouse is hovering. It now looks natural and harmonious with the whole page.
To create the Java sketch, I set visual and audio parameters that connect to each other:
For the “from the audio to the visual”, the sound frequency is linked to the brightness of the color, and the sound volume is linked to the stroke weight; for the “from the visual to the audio”, the speed of the mouse movement is linked to the sound frequency, the mouse wheel movement is related with stroke weight while stroke weight is linked to the sound volume. I achieved these by:
mic = new p5.AudioIn();
mic.start();
fft = new p5.FFT();
fft.setInput(mic);
mic.getLevel()
fft.analyze()
map()
3) Reflection and Future Development
Overall, I would say my project is more concept-driven instead of being more narrative/storytelling or presentable. I have this mechanism of converting between the audio and the visual first, then developing anything else based on that. It’s more about providing a digital space for interactively experimenting with the two human senses. And I’ve executed the tasks mentioned in my initial proposal pretty well.
I would like to implement further these in the future:
– Hyperlink at the bottom -> change the default color to black&white to fit in with the whole website better
– Structure: Linear is not the best way -> put the “from the audio to the visual” and the “from the visual to the audio” on the same layer
– Color: Instead of purely random, plan the color combo
– Instructions -> Independent pages instead of buttons and pop-ups