Seeing the different world-Jiayi-Eric
A.CONCEPTION AND DESIGN:
This project is designed to allow people to experience the life of individuals with visual impairments. By simulating both farsightedness and nearsightedness, participants can grasp the challenges faced by those with visual impairments in daily life and learning, particularly by using word retrieval as a means of understanding the difficulties. The simulation includes scenarios where participants control the mouse by shaking their heads, aiming to replicate the experience of searching for objects. The use of vocabulary as the grabbing object aims to simulate the difficulties students with visual impairments encounter when trying to read content on a blackboard.
Since my definition of interaction is “A way to create connection with other subjects using different methods”, my project aims to align with this definition by creating a connection between audience and machine using different sensors and methods. Also I tried to add more stages to make the project more abundant as well as integrated.
During the User Testing Session, professors, fellows and my classmates provided me with plenty of interesting and helpful information. They recommended that I add an ending and sound effects during the game, make the processing scene more connected with vision and set my computer higher to create a better effect. I found the advice truly inspiring so I did them accordingly. First I added sound effects and an ending page for celebration, which make the game more engaging and mature. Then I changed the grabbing objects from triangles to words to simulate knowledge on the blackboard. Last I stick 2 boxes together to set my computer higher. All these changes contribute to my final version of the project and make it more integrity.
B.FABRICATION AND PRODUCTION:
Regarding my production progress, I think the test of the accelerometer and the codes of blur are the most significant steps. I tested the accelerometer at the very beginning to ensure this crucial sensor can work well. First I tried analog and digital accelerometers but neither of them worked well. Then Professor Andy gave me a Leonardo Arduino board and a new accelerometer and this time it works!Moreover, since my goal is to simulate visual impairments, the blurry mode is also definitely a core part. Professor Eric helped me with this part of the code(twice! Because the first version vanished like a ghost 0.0).
I would say the criteria for me to select electronics is whether it can work constantly and stably. Sometimes it may work because of luck but this is not a permanent solution. An example would be the use of the distance sensor. At first I put the distance sensor on the Leonardo board, same to the accelerometer sensor. It works depending on luck and is often stuck. As a result I split 2 sensors into 2 Arduino boards so they wouldn’t mess up with each other. Now I will record the process in detail.
- Arduino
For the Arduino part, I need to read the value of the sensor. Since I split them into 2 boards I need 2 separate Arduino programs. Professor Andy helped me with the codes of the accelerometer by providing me with a sample. As for the distance sensor I followed the instructions on the slides and finished the coding.
- Processing
The code of Processing is much more complex than Arduino.
Before user testing, I set up a basic structure by editing the code from a recitation program at first. The program consists of floating triangles. If you catch the triangle it will stop and when you catch all the triangles you win. After that, I added the transfer from Arduino to Processing. Next comes to the first difficult part—adding the blurry. Professor Eric added a new graph to control the blurriness in my code. To make it work, every drawing needs to add “src.” at the beginning.
After user testing, I downloaded a sound effect and added it to the code so when you click the vocabularies it will sound something. What’s more, I added an ending page to the code. Considering changing the grabbing objects to another that can make more connection of the topic “visual impairments”, I decided to adapt an advice from a tester to change the triangles into words to simulate the “class environment”. While I was stuck at what vocabularies I should use, Andy inspired me to add a new stage that includes both farsighted and nearsighted. Then I thought since there are 2 sighted stages, I want the audience to recognize which stage they are in by themselves so I need to offer them some tips. As a result, I built sentences including “ farsighted” and “nearsighted” and broke them into vocabularies to substitute the floating triangles. After that I encountered The most difficult part, adding the stages. The problem was mainly due to “delay()”. Kevin helped me to solve this problem by changing “delay” into an if sentence including “millis()”. Finally, all the codes worked well!!!😭
- Fabrication
I used a 3D printer to print my main installation—-the glasses. I downloaded the model from Printables and printed it out.
After user testing I decided to make a board to hint to the audience that they should move their heads back and forward to see the floating words clearly.
Another thing I would like to mention is soldering. Since my project needs a quite large range for the audience to move around, it requires longer wires. At first I soldered the superfine wire to make the wire more flexible but sadly they can’t transfer data well. Then I changed to a thicker wire but they can not be plugged into Leonardo board. Fine, so I soldered extra 4 MM wires on them. At last they worked. To make the soldered part stronger I also used heat shrink tubings to reinforce them. Finally, based on incomplete statistics, I soldered at least 16 wires. I feel like I could become a novice welder.😜
C.CONCLUSIONS:
My goal of my project is to allow people to experience the life of individuals with visual impairments. I think it achieved the goal. My audience’s interaction met my expectations. They usually first put on glasses, move their head around and find out they are actually controlling the mouse. Then I may need to remind them they can also move back and forward to see where they can see the words most clearly. I believe the project aligned with my definition of interaction. There are 2 ways you can interact with the computer, one is catching the vocabulary and another is moving back and forward to experience visual impairments. There are many considerable improvements that can be made if there is more time, I will make the scene more like a classroom to create a better atmosphere. Moreover I am willing to add new stages that simulate more visual impairments, such as glaucoma and cataracts. Adding more hints is also a practical idea which enables the audience to know how the project functioned in a more explicit way. All in all, I am proud of the project I made.I also attended the IMA show where I gained a lot of compliments, a tester even provided me with a practical application of a product similar to my project called “glassouse”.
Due to numerous setbacks I’ve encountered, I’ve learned to be cautious—just because code runs doesn’t mean it runs correctly, and having code that seemingly works doesn’t guarantee it’s flawless. Additionally, I’ve acquired patience as problems arise one after another, requiring calm and patient analysis for resolution. The Interaction Lab has truly nurtured resilience within me; faced with countless challenges and issues, the initial worry has transformed into a proactive mindset of how to address each problem.💪
Initial wires⬆️
Most importantly, I want to express my sincere gratitude to all the professors, fellows, and LAs at the Interaction Lab. Without them, my project wouldn’t have been possible. Thanks to Professor Andy for providing me with the accelerometer and offering advice on adding stages. Special thanks to Professor Gohai for assisting me with multiple debugging sessions, and to Professor Eric for patiently rewriting my blurry code twice. Kevin dedicated a whole two hours to help me troubleshoot issues with my stage code, while Dalin provided invaluable assistance with 3D printing and laser cutting, teaching me how to polish and sharing useful soldering experiences and heat shrink tubing.I’m also grateful to Anya for helping me transform triangles into words, among other things.
Additionally, I appreciate the camaraderie of my fellow students who fought alongside me in the studio, offering encouragement, support, debugging assistance, and serving as each other’s testers with constructive feedback. I’ve witnessed many creative, powerful, and complex designs, constantly amazed by the extraordinary creativity and execution skills of my classmates. We’ve built deep friendships, and all of this has made me feel the warm embrace of IMA. I believe this journey will be a lasting memory, and as we cheer and applaud for each other’s works, it represents our sincerest blessings from the depths of our hearts.
Love comes from IMA❤️🔥.
D. DISASSEMBLY:
E.APPENDIX:
- Full video: https://drive.google.com/file/d/1EULyJEDRA16fs5fHewuG5PGK6z6UhdBX/view?usp=drive_link
- Libraries:
<Wire.h><I2Cdev.h><MPU6050.h><Mouse.h>
- 3D model: Printables
- Codes credits
- Blurry codes from Prof.Eric
- Accelerometer codes from Prof.Andy
- Arduino code(Mouse):
- Arduino code(Blurry):
- Processing code:
import processing.serial.*;
import processing.sound.*;
// declare a SoundFile object
SoundFile sound;
Serial serialPort;
//String[] words = {"This", "is", "how", "the", "world", "looks", "like", "if", "you", "are", "long", "sight", "!"};
int NUM_OF_VALUES_FROM_ARDUINO = 1; /* CHANGE THIS ACCORDING TO YOUR PROJECT */
/* This array stores values from Arduino */
int arduino_values[] = new int[NUM_OF_VALUES_FROM_ARDUINO];
boolean gameFinished = false;
boolean stage0R = true;
int a = 30;
float[] xs = new float[a];
float[] ys = new float[a];
String[] textArray = new String[a];
color[] colors = new color[a];
float[] xspeed = new float[a];
float[] yspeed = new float[a];
int blackTriangles = 0;
float stage = 0;
int timetostop;
int b;
float s;
PShader blur;
PGraphics src, src2;
PGraphics pass1, pass2;
void setup() {
fullScreen(P2D);
size(720, 480, P2D);
printArray(Serial.list());
serialPort = new Serial(this, "/dev/cu.usbmodem101", 9600);
for (int i=0; i<a; i=i+1) {
xs[i] = random(width);
ys[i] = random(height);
colors[i] = color(random(255), random(255), random(255));
xspeed[i] = random(-5, 5);
yspeed[i] = random(-5, 5);
}
sound = new SoundFile(this, "mixkit-retro-game-notification-212.wav");
blur = loadShader("blur.glsl");
blur.set("blurSize", 9);
blur.set("sigma", 5.0f);
src = createGraphics(width, height, P2D);
src2 = createGraphics(width, height, P2D);
pass1 = createGraphics(width, height, P2D);
pass1.noSmooth();
pass2 = createGraphics(width, height, P2D);
pass2.noSmooth();
}
void draw() {
getSerialData();
//src.beginDraw();
if (stage == 0f) {
String[] words = {"This", "is", "how", "the", "world", "looks", "like", "if", "you", "are", "far", "sighted", "!"};
src.beginDraw();
src.background(255);
src.fill(0);
src.text("catch the floating words to let them stop!",500,50);
src.text("Move back and forward to see words clearly!", 500,800);
src.noStroke();
for (int i=0; i<words.length; i=i+1) {
if(stage0R){
xs[i] = random(width);
ys[i] = random(height);
colors[i] = color(random(255), random(255), random(255));
xspeed[i] = random(-5, 5);
yspeed[i] = random(-5, 5);
stage0R=false;
}
src.fill(colors[i]);
src.textSize(50);
src.textAlign(CENTER);
src.text(words[i], xs[i], ys[i]);
//triangle(xs[i], ys[i], xs[i]-20, ys[i]+20, xs[i]+20, ys[i]+20);
xs[i] = xs[i] + xspeed[i];
ys[i] = ys[i] + yspeed[i];
if (xs[i] > width) {
xspeed[i]=-xspeed[i];
}
if (xs[i] < 0) {
xspeed[i]=-xspeed[i];
}
if (ys[i] > height ) {
yspeed[i]=-yspeed[i];
}
if (ys[i] < 0) {
yspeed[i]=-yspeed[i];
}
if (dist(mouseX, mouseY, xs[i], ys[i]) <20) {
if (colors[i] != color(0)) {
colors[i]=color(0);
blackTriangles = blackTriangles+1;
xspeed[i] = 0;
yspeed[i] = 0;
// play the sound on loop
sound.play();
}
}
}
b = int(map(arduino_values[0], 15, 60, 50, 0));
s = map(arduino_values[0], 15, 60, 50, 0);
if (blackTriangles >= words.length) {
stage = 0.5;
timetostop = millis();
src.clear();
src.background(src.color(255,0));
src.endDraw();
} else{
src.endDraw();
blur.set("blurSize", b);
blur.set("sigma", s);
// Applying the blur shader along the vertical direction
blur.set("horizontalPass", 0);
pass1.beginDraw();
pass1.shader(blur);
pass1.image(src, 0, 0);
pass1.endDraw();
// Applying the blur shader along the horizontal direction
blur.set("horizontalPass", 1);
pass2.beginDraw();
pass2.shader(blur);
pass2.image(pass1, 0, 0);
pass2.endDraw();
image(pass2, 0, 0);}
}
if (stage == 0.5f) {
if (millis() <= timetostop+3000) {
pushMatrix();
background(255);
fill(0);
noStroke();
textSize(50);
textAlign(CENTER);
text("This is how the world looks like if you are far sighted!", width/2, height/2);
popMatrix();
} else {
stage = 1;
stage0R = true;
blackTriangles = 0;
}
}
if (stage == 1f) {
String[] words = {"This", "is", "how", "the", "world", "looks", "like", "if", "you", "are", "short", "sighted", "!"};
src.beginDraw();
src.background(255);
src.noStroke();
if (stage0R) {
for (int i=0; i<words.length; i=i+1) {
xs[i] = random(width);
ys[i] = random(height);
colors[i] = color(random(255), random(255), random(255));
xspeed[i] = random(-5, 5);
yspeed[i] = random(-5, 5);
}
stage0R = false;
}
for (int i=0; i<words.length; i=i+1) {
src.fill(colors[i]);
src.textSize(50);
src.textAlign(CENTER);
//println(words[i] + " " + xs[i] + " " + ys[i]);
src.text(words[i], xs[i], ys[i]);
//triangle(xs[i], ys[i], xs[i]-20, ys[i]+20, xs[i]+20, ys[i]+20);
println(xspeed[i] + " " + yspeed[i]);
xs[i] = xs[i] + xspeed[i];
ys[i] = ys[i] + yspeed[i];
if (xs[i] > width) {
xspeed[i]=-xspeed[i];
}
if (xs[i] < 0) {
xspeed[i]=-xspeed[i];
}
if (ys[i] > height ) {
yspeed[i]=-yspeed[i];
}
if (ys[i] < 0) {
yspeed[i]=-yspeed[i];
}
if (dist(mouseX, mouseY, xs[i], ys[i]) <20) {
if (colors[i] != color(0)) {
colors[i]=color(0);
blackTriangles = blackTriangles+1;
//println("trigg");
xspeed[i] = 0;
yspeed[i] = 0;
// play the sound on loop
sound.play();
}
}
}
b = int(map(arduino_values[0], 15, 60, 0, 50));
s = map(arduino_values[0], 15, 60, 0, 50);
if (blackTriangles == words.length) {
//for (int i=0; i<a; i=i+1) {
// colors[i] = color(random(255), random(255), random(255));
//}
//gameFinished = true;
stage = 1.5;
timetostop = millis();
src.clear();
src.background(src.color(255,0));
src.endDraw();
} else{
src.endDraw();
blur.set("blurSize", b);
blur.set("sigma", s);
// Applying the blur shader along the vertical direction
blur.set("horizontalPass", 0);
pass1.beginDraw();
pass1.shader(blur);
pass1.image(src, 0, 0);
pass1.endDraw();
// Applying the blur shader along the horizontal direction
blur.set("horizontalPass", 1);
pass2.beginDraw();
pass2.shader(blur);
pass2.image(pass1, 0, 0);
pass2.endDraw();
image(pass2, 0, 0);}
}
if (stage == 1.5) {
println("stage1.5");
if (millis() <= timetostop+3000) {
pushMatrix();
background(255);
fill(0);
noStroke();
textSize(50);
textAlign(CENTER);
text("This is how the world looks like if you are short sighted!", width/2, height/2);
popMatrix();
} else {
stage = 2;
stage0R = true;
blackTriangles = 0;
}
}
if (stage == 2) {
timetostop = millis();
if(millis() <= timetostop+3000){
sound = new SoundFile(this, "T9RZL9V-fab-game-celebrate.mp3");
sound.play();
pushMatrix();
background(255);
textSize(50);
fill(0);
textAlign(CENTER);
text("Congratulation! You have finished the experience!", width/2, height/2);
textSize(30);
text("Normal action will be harder for people with visual impairments", width/2, 300);
popMatrix();
} else{
stage = 0;
stage0R = true;
}
//if (dist(mouseX, mouseY, width/2, height/2) < 20) {
gameFinished = false;
blackTriangles = 0;
for (int j=0;j<a; j=j+1) {
colors[j] = color(random(255), random(255), random(255));
xspeed[j] = random(-5, 5);
yspeed[j] = random(-5, 5);
// textArray[j] = words[j];
}
}
src.endDraw();
}
/*src.endDraw();
blur.set("blurSize", b);
blur.set("sigma", s);
// Applying the blur shader along the vertical direction
blur.set("horizontalPass", 0);
pass1.beginDraw();
pass1.shader(blur);
pass1.image(src, 0, 0);
pass1.endDraw();
// Applying the blur shader along the horizontal direction
blur.set("horizontalPass", 1);
pass2.beginDraw();
pass2.shader(blur);
pass2.image(pass1, 0, 0);
pass2.endDraw();
image(pass2, 0, 0);*/
/*void keyPressed() {
for (int i=0; i<a; i=i+1) {
if (key == 's') {
xspeed[i]=0;
yspeed[i]=0;
} else if (key == 'r') {
xspeed[i] = random(-5, 5);
yspeed[i] = random(-5, 5);
}
}
}*/
void getSerialData() {
if (serialPort == null) {
return;
}
while (serialPort.available() > 0) {
String in = serialPort.readStringUntil( 10 ); // 10 = '\n' Linefeed in ASCII
if (in != null) {
//print("From Arduino: " + in);
String[] serialInArray = split(trim(in), ",");
if (serialInArray.length == NUM_OF_VALUES_FROM_ARDUINO) {
for (int i=0; i<serialInArray.length; i++) {
arduino_values[i] = int(serialInArray[i]);
}
}
}
}
}