A. A Robot Psychiatrist-Rachel Li-Margaret Minsky
B. Conception And Design:
The concept of my design was simply to literally create a back-and-forth dialogue between humans and computers but also add the role of patient and psychiatrist as the background of the conversation to send my central message. In terms of preparatory research, I would like to mention the research I did for the Group Project Garden of Birds by Fei Jun. As I described it in my blog post-Group Project: Research, “It is an interactive installation, a piece of artwork in Beijing Daxing International Airport, displaying a digital and live landscape of nature. And as passengers slowly walk by, the birds will consciously follow them and excitedly “talk” to them” (Rachel). Moreover, it will show information like the weather and the arrival and departure time of the flights as passengers approach, which was something I thought wasn’t that interactive at that time. And as I look back on the research I did, I found that it may have inspired me subconsciously in the way to merge information perfectly into the background. And that’s why I merged real-life anxieties into the background story. And as for the preparatory research I did specifically for the final project, which was: A Simple Interactive Project with Processing on SBC by Haoming Weng, and Interactive Gloves by emcnany, they can both fall into the “game” category, and before I proposed the three ideas, I thought that a game is exactly the most original and the best form of “interaction” because there are always immediate feedback and can engage the users to the largest extent. However, when I came to the three proposals, I found that it’s not actually a game, but I can include these good sides of a game into my project instead. So, in my understanding, the dialogue should be smooth without deliberate human intervention, with a little bit of uncertainty to keep the users closely engaged. There should absolutely be adequate feedback to show them when to speak and when to listen. That’s why I decided to use the original wavy line that came with google TTS to show the “robot” speaking while using FFT, another type of visualization, combined with the clear instructions “Speak:” to indicate users to speak. Also, in other to make the dialogue go smoother, I wanted the “robot” to detect the amplitude of users’ voices and based on that to decide when to go to the next step, instead of clicking the mouse or hitting the keyboard. However, in the User Testing Session, I had to go back to the keyboard because the background noise was too loud and the code would think that the user still hasn’t finished talking, so Sylvia suggested that I can probably use a pair of earphones or a headphone instead so that it won’t be as sensitive as the embedded microphone in the computer. And I think it somehow solves the problem because when I was testing it myself, I could see that the waves of the FFT were lower than using the embedded mic, but we will probably never know as the IMA end-of-year show is now online 🙁 . Lydia also suggested that I can put a small cushion on the floor to create an intimate little space as what it usually is in a psychiatry room. I did buy one but I don’t think I can be able to test how effective it is in the future. She also suggested that I can paint a little on the plain box and that was when I thought I could use the painting to show the sense of satire more clearly. I think that didn’t work very well because other students observing and commenting obviously didn’t get to know that until I explain a little and I found it so hard to create that sense of satire. Another feedback I got from Nicole was to fix the drawer and the touch sensor, which was probably not working properly because I added a delay to the stepper motor at that time and other things could be sleeping while I was giving them all the orders. At last, I did fix that and the proper popping out of the stepper motor accounted a lot for the success of the whole.
C. Fabrication And Production:
Step 0: Tracking down Google TTS (text-to-speech)
Many many thanks to Professor Rudi for wasting a lot of his precious time and tracking down google TTS for me online!!! And we (he) found the original code HERE. However, the url to google translate in the original code wasn’t working, so Professor Rudi changed the url into another effective one instead, and the full code for google TTS looks like this:
/**
* This was adapted from a code Professor Rudi found on this website: https://amnonp5.wordpress.com/2011/11/26/text-to-speech/
void googleTTS(String txt, String language) {
//String u = "http://translate.google.com/translate_tts?tl=";
//u = u + language + "&q=" + txt;
//u = u.replace(" ", "%20");
String u = "https://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&tl=en&";
u = u + language + "&q=" + txt;
u = u.replace(" ", "+");
//https://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&tl=en&q=Hello+World
try {
URL url = new URL(u);
try {
URLConnection connection = url.openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 1.2.30703)");
connection.connect();
InputStream is = connection.getInputStream();
File f = new File(sketchPath() + "/1.mp3");
OutputStream out = new FileOutputStream(f);
byte buf[] = new byte[1024];
int len;
while ((len = is.read(buf)) > 0) {
out.write(buf, 0, len);
}
out.close();
is.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
catch (MalformedURLException e) {
e.printStackTrace();
}
}
*/
Step 1: Experimenting with touch sensors and fruits
As soon as Professor Misky suggested that I can use fruits as sensors to make choices and I thought that was a great idea to show the sense of irrelevance and peculiarity, I started to experiment with that. I wanted to buy a set of Makey Makey online, but then as we consulted with Professor Rudi, we found out that we can simply use a touch sensor instead if what I want to achieve was just to detect and send the signal. And then Professor Rudi kindly soldered a wire to the touch sensor so that I could plug the other end of the wire into the fruit that I will choose. Professor Minsky also bought me an apple pie so that I could test it at once, and it worked! The video below is recorded by Professor Minsky:
It was the same week that weekend, I came to AB and started with the code and tried to solder the other two touch sensors as I was planning to use three fruits as three choices. And when I started to actually get on hand to it, I found that it was just impossible to solder the wire onto the sensor, and whenever I was about to succeed, the soldering tin starts to fall off. And below are pictures I took that show how poorly I soldered when compared with Professor Rudi’s.
The shiny one is Rudi’s. Major Failure It fell off (of course!)
And so I gave up keeping on soldering that day and I happened to have a short chat with Professor Minsky on Tuesday morning before class. I told her about my difficulties in soldering and also I started to think about asking whether it was a good idea to use stepper motors as actuators or if she had other good suggestions for alternatives. And for the first question on soldering, when I showed her the soldering piece I made, she quickly pointed out that I just didn’t remove the coating. Although I thought I had already scraped the sensor hard enough to remove the coating, it was when she showed me the correct extent, which was to the extent that I could clearly see the metal part inside that I realized that “OH, so that was all the coating and that was why I couldn’t solder the wire onto it!” So, after I scraped all the coating off from the sensor, I successfully and smoothly soldered the wire firmly onto the surface of the sensor. I also plugged the other end of the wire into a lemon and an apple to experiment with the sensor to check out at what distance the electricity in the wires will affect each other, and below is the video showing my experiment:
Other than that, I also ran this code below to confirm that the connection actually works:
/**
* This was adapted from an example embedded in Processing.
import processing.serial.*;
import osteele.processing.SerialRecord.*;
Serial serialPort;
SerialRecord serialRecord;
String s = "";
void setup() {
size(1280, 720);
background(0);
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 9600);
serialRecord = new SerialRecord(this, serialPort, 2);
}
void draw() {
serialRecord.read();
int value = serialRecord.values[0];
int Distance = serialRecord.values[1];
println(Distance);
if (Distance == 10) { //< 20 && Distance >0
welcome();
}
if (value == 1) {
conversation1();
}
if (value == 2) {
conversation2();
}
if (value == 3) {
conversation3();
}
}
void welcome() {
textSize(50);
stroke(255);
textAlign(CENTER, CENTER);
s = "Psychiatrist: Are things going well these days?";
text(s, 0, 0, width, height);
//waitAnswer();
//launch("/Users/rachel/1.command");
}
void waitAnswer() {
background(0);
strokeWeight(0.5);
textSize(25);
fill(255);
text(":", 20, 30);
}
void conversation1() {
background(0);
textSize(50);
stroke(255);
textAlign(CENTER, CENTER);
s = "Okay, it's obvious that you're suffering from sleep disorders. Tell me what it is that is preventing you from sleeping these days.";
text(s, 0, 0, width, height);
}
void conversation2() {
background(0);
textSize(50);
stroke(255);
textAlign(CENTER, CENTER);
s = "Apparently, you are struggling with relationships recently.";
text(s, 0, 0, width, height);
}
void conversation3() {
background(0);
textSize(50);
stroke(255);
textAlign(CENTER, CENTER);
s = "I can see that you are stressed with your finals. Would you like to share more about your feelings and what particular subjects you are having trouble with?";
text(s, 0, 0, width, height);
}
*/
The code above was working all well, although you can see that I couldn’t realize the effect to show the effect of colon blinking to show it’s time to speak and the speech is being analyzed, just as the cursor is blinking when we are typing words to show that the words are “waiting” to be typed in. And that was partly why I simply eliminated the part that shows the “fake” or programmed answers from the patients.
Step 2: Experimenting with stepper motors as actuators:
I was questioning whether it was a good choice to use stepper motors as actuators because according to my experience of using stepper motors in Recitation#4, I know that the knobs that are attached to them are pretty small and the parameter probably cannot realize the effect to push the drawers out as I had imagined (but now I am thinking that the knobs are 3D printed out anyway, so if I really wanted a knob with a larger parameter, I could simply print one out! Anyway, the ones already printed out for us worked at last, but if given more time, I may want to try printing out them by myself to make the fruits more like a surprise coming out from nowhere.) At that time, however, Professor Minsky introduced more types of actuators to me, like worm gears that can support heavier things, which is one of the goals that I want to realize but may be a little too slow as I wanted kind of a surprising effect, and another choice is to use a solenoid, but then we had to think about how to make it go back. I was not sure, so I turned to Professor Rudi again, and he suggested that I can probably use stepper motor linear sliders instead as they have slides that can push the drawer out. And he was pointing to a piece of work hanging on the top of the roof by other students:
And so I searched for it on Taobao and got these results:
But there was also the problem of the size isn’t right and I was also not sure if there is going to be a large slider, or how big would my whole box be. Moreover, it was already Dec 1st that day, only 8 days to go before the user testing, so I also have to consider how long it takes to arrive, and what I shall do if the slider didn’t work well… I was really stressed and then I made an appointment with Christine to discuss how to draw the box and what kind of actuator I should use. She suggested that I can probably use the solenoid with the combination with a slope so that the drawer can fall from the top to the bottom, but in that way, the box is going to look completely different, and after discussing it with my dad, we thought it was too much a risk and stayed to the original idea of stepper motors.
I have to thank Christine for making the box design with me, as it was the first time for me to do something like this and I often get panicked at things I’m not familiar with. But it was with her help that I had more confidence in what I was doing and I also learned how to start designing, although it was such a pity that she promised me that she would come to see my project at IMA end-of-year show but no one could have known that it was going to be online 🙁 🙁 🙁 .
the sketch Christine taught me to draw the idea of using a slope and solenoid
I thought that I could simplify the stepper motor structure we did in Recitation#4, and I did try to make it happen for a long time. First, I made a prototype with cardboard:
At that time, because I was struggling with the box design and the way to actuate at the same time, so I made an appointment with Evelyn to seek help. And she gave me great suggestions on how to make the drawer move with the stepper motor; otherwise, I would have to stick to my two weird ideas I came up with: one is connecting the rotation link to the upper part of the drawer as a lid, but that may seem a little bit meaningless and that probably was not what I had imagined; another is sticking the rotation link to the bottom of the drawer, but then I will have to think whether the stepper motor is powerful enough to hold the three fruits. However, she suggested that I can connect the rotation link to a bent board that is connected to the back of the drawer so that the stepper can truly be able to “push” the drawer out literally.
And then I thought I can laser cut the drawer part out even though I haven’t figured out if it can work properly, so then I started to experiment with the actuator. And it turned out that I was oversimplifying the complicated structure. Although I tried to add a rectangular frame to restrict the rotation link from being all wobbly, it didn’t work well.
So after a long struggle, I finally realized that I have to stick to the original though the complex way of making the actuator system. And I simply copied what I did in Recitation#4 two times and it succeeded and worked much much better than what I “creatively” came up with:
As you can see, it did work much better, it’s now the stepper motor’s turn to be all wobbly. So, I put some hot glue onto the bottom of the stepper motor, and then the problem was solved! When I put it on the top of the box to it, it looks all good now:
Step 3: Experimenting with the code:
I was trying to do it step-by-step at first and based on the structure I tried and succeeded, I managed to get to this step only by using keyPressed (video shown below). But then I also wanted to replace keyPressed with a certain amplitude of human voice, so that the users won’t bother to press the key every time they want to continue. But when I found that I have many many more problems and so I wrote an email to my dad:
What it was like before dad’s help:
And the code was like this at that time:
/**
* This code merges google TTS, voice visualization fft and keyPressed control. The fft part comes from the Processing examples.
Amplitude amp;
AudioIn in;
import ddf.minim.*;
AudioPlayer player;
Minim minim;
import java.net.*;
import java.io.FileOutputStream;
// Declare the sound source and FFT analyzer variables
FFT fft;
AudioIn mic;
// Define how many FFT bands to use (this needs to be a power of two)
int bands = 1024;
// Define the bands wanted for our visualization.
// Above a certain threshold, high frequencies are rarely attained and stay flat.
int bandsWanted = 128;
// Create an array to store the bands wanted
float[] spectrum = new float[bandsWanted];
//void keyPressed() {
float startTime;
import processing.sound.*;
String s = "Welcome to my psychiatry room!";
String y = "";
int status = 0;
void setup() {
amp = new Amplitude(this);
in = new AudioIn(this, 0);
in.start();
amp.input(in);
//fullScreen();
size(1280, 720);
minim = new Minim(this);
textAlign(CENTER, CENTER);
textSize(50);
stroke(255);
// Create the AudioIn object and select the mic as the input stream
mic = new AudioIn(this, 0);
// Start the input stream without routing it to the audio output.
mic.start();
// Create the FFT analyzer and connect it to the sound input
fft = new FFT(this, bands);
fft.input(mic);
}
void draw() {
background(0);
textSize(50);
textAlign(CENTER, CENTER);
text(s, 0, 0, width, height);
textSize(20);
y = "Press 'enter' to continue.";
text(y, 631, 590);
if (player != null) {
translate(960, 550);
for (int i = 0; i < player.left.size()-1; i++) {
line(i, 100 + player.left.get(i)*50, i+1, 100 + player.left.get(i+1)*50);
//line(i, 150 + player.right.get(i)*50, i+1, 150 + player.right.get(i+1)*50);
}
}
if (keyCode == ENTER) {
y = ""; //
text(y, 631, 590); //bug
textAlign(CENTsit downTER);
y = "Press 's' to sitdown."; //
text(y, -100, 40);
textAlign(CENTER, CENTER);
textSize(50);
fill(255);
s = "Are things going well these days?";
text(s, 0, 0, width, height);
}
if (key == 's') { //bug
textAlign(CENTER, CENTER);
textSize(40);
text("Speak:", -100, 3);
// Perform the analysis because //does not WORK :(((((( I think it is becasue this key event
fft.analyze(spectrum)tore we are reversing the spectrum array in order to get the higher frequencies on the right edge of canvas.
spectrum = reverse(spectrum);
pushMatrix();
//Translating the position to start drawing in the middle
translate(0, 42);
//Drawing a line for every frequency of the spectrum
for (int i=0; i<bandsWanted; i++) {
stroke(255);
line( i*2, 0, i*2, -spectrum[i]*1000 );
}
popMatrix();
}
if (amp.analyze()>0.2) {
status = 1;
}
println(status);
if (status == 1) {
s = "Okay, let's deal with that, but first let me know: Which one of them do you think best describes the problem that has recently been disturbing you? Touch it to continue.";
text(s, 0, 0, width, height);
speech();
println(status);
mic.stop();
}
if (keyPressed) {
if (key == 'f') {
status = 2;
}
}
if (status == 2) {
s = "Apparently, you are struggling with relationships recently.";
textSize(50);
textAlign(CENTER, CENTER);
text(s, 0, height/3, width, height/3*2);
//text(s, 0, 0, mouseX, mouseY);
speech(); //working
mic. sttofft.analyze(spectrum);
//Here we are rethe versing the spectrum array in order to get the higher frequencies on the right edge of canvas.
spectrum = reverse(spectrum);
pushMatrix();
//Translating the position to start drawing in the middle
translate(0, 42);
//Drawing a line for every frequency of the spectrum
for (int i=0; i<bandsWanted; i++) {
stroke(255);
line( i*2, 0, i*2, -spectrum[i]*1000 );
}
popMatrix();
}
}
void speech() {
if (amp.analyze()>0.3) {
googleTTS(s, "en");
if (player != null) {
player.close();
}
player = minim.loadFile(1 + ".mp3", 2048);
player.play();
s = "";
}
}
void speechWithoutReply() {
googleTTS(s, "en");
if (player != null) {
player.close();
}
player = minim.loadFile(1 + ".mp3", 2048);
player.play();
s = "";
}
void keyPressed() {
if (key == 's') { //bug: doesn't work
y = "Press 's' to sitdown.";
text(y, 631, 590);
}
if (keyCode == ENTER) {
googleTTS(s, "en");
if (player != null) {
player.close();
} // comment this line to layer sounds
// player = minim.loadFile(s + ".mp3", 2048);
player = minim.loadFile(1 + ".mp3", 2048);
player.play();
s = "";
} else if (keyCode != SHIFT && keyCode != CONTROL && keyCode != ALT && key != 's' && key != 'f' && s.length() < 100) {
s += key;
}
}
void googleTTS(String txt, String language) {
String u = "https://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&tl=en&";
u = u + language + "&q=" + txt;
u = u.replace(" ", "+");
//https://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&tl=en&q=Hello+World
try {
URL url = new URL(u);
try {
URLConnection connection = url.openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 1.2.30703)");
connection.connect();
InputStream is = connection.getInputStream();
// File f = new File(sketchPath() + "/" + txt + ".mp3");
File f = new File(sketchPath() + "/" + 1 + ".mp3");
OutputStream out = new FileOutputStream(f);
byte buf[] = new byte[1024];
int len;
while ((len = is.read(buf)) > 0) {
out.write(buf, 0, len);
}
out.close();
is.close();
// println("File created: " + txt + ".mp3");
}
catch (IOException e) {
e.printStackTrace();
}
}
catch (MalformedURLException e) {
e.printStackTrace();
}
}
void stop() {
player.close();
minim.stop();
super.stop();
}
*/
And he replied to me with this code below:
/**
* written by my dad Glen Li
import ddf.minim.*;
import java.net.*;
import java.io.FileOutputStream;
import processing.sound.*;
import processing.serial.*;
import osteele.processing.SerialRecord.*;
// Declare the sound source and FFT analyzer variables
FFT fft;
AudioIn mic;
Minim minim;
AudioPlayer player;
AudioIn in;
Serial serialPort;
SerialRecord serialRecord;
Amplitude amp;
// Define how many FFT bands to use (this needs to be a power of two)
int bands = 1024;
// Define the bands wanted for our visualization.
// Above a certain threshold, high frequencies are rarely attained and stay flat.
int bandsWanted = 128;
// Create an array to store the bands wanted
float[] spectrum = new float[bandsWanted];
// voice buffer
FloatList voice = new FloatList();
int state = -1; // idle
int q = 0; // question index
boolean ready = true; // ready for new message
boolean pendingReady = false; // pending to set ready
int distance = 1000;
void setup() {
//fullScreen();
size(1280, 720);
minim = new Minim(this);
// Create the AudioIn object and select the mic as the input stream
mic = new AudioIn(this, 0);
// Start the input stream without routing it to the audio output.
mic.start();
amp = new Amplitude(this);
amp.input(mic);
// Create the FFT analyzer and connect it to the sound input
fft = new FFT(this, bands);
fft.input(mic);
// Serial control
String serialPortName = SerialUtils.findArduinoPort();
serialPort = new Serial(this, serialPortName, 9600);
serialRecord = new SerialRecord(this, serialPort, 2);
}
void draw() {
background(0);
switch(state) {
case -1:
idle();
break;
case 0:
welcome(qset0());
break;
case 1:
conversation(qset1());
break;
case 2:
conversation(qset2());
break;
case 3:
conversation(qset3());
break;
default:
changeState(0);
break;
}
}
void stop() {
player.close();
minim.stop();
super.stop();
}
//////////////////////////////////////////////////////////////////////////////////
/// Conversation control
//////////////////////////////////////////////////////////////////////////////////
void idle() {
String s = "Welcome to my psychiatry room!";
message(s);
serialRecord.read();
int Distance = serialRecord.values[1];
println(Distance);
if (Distance < 30 && Distance > 0) {
changeState(0);
}
}
void welcome(StringList qs) {
//fft();
if (q < qs.size()) {
String s = qs.get(q);
message(s);
talk(s);
checkVoice();
}
// last question
if (q == qs.size() - 1) {
serialRecord.read();
int s = serialRecord.values[0];
if (s != 0) {
changeState(s);
}
}
}
void conversation(StringList qs) {
fft();
if (q < qs.size()) {
String s = qs.get(q);
message(s);
talk(s);
checkVoice();
} else {
// No more questions, return to welcome state
changeState(-1);
}
}
/////////////////////////////////////////////
void changeState(int s) {
state = s;
q = 0;
ready = true;
pendingReady = false;
voice.clear();
println(s);
}
/////////////////////////////////////////////
void checkVoice() {
if (player != null) {
if (player.isPlaying()) {
// we need human voice instead of computer voice
translate(0, 96);
for (int i = 0; i < player.left.size()-1; i++) { stroke(255); line(i, 100 + player.left.get(i)*50, i+1, 100 + player.left.get(i+1)*50); } voice.clear(); return; } speakSign(); } float v = amp.analyze(); voice.append(v); float sum = 0; int duration = 60; // voice checking duration if (voice.size() > duration) {
voice.remove(0);
for (int i = 0; i < voice.size(); i++) { sum += voice.get(i); } } float average = sum/voice.size(); if (average > 0.05) {
println("got voice!");
pendingReady = true;
} else {
// silence
if (pendingReady) {
// patient finished talking
ready = true;
q++;
pendingReady = false;
}
}
}
void speakSign() {
background(0);
fft();
textAlign(CENTER, CENTER);
textSize(30);
fill(255);
text("Speak:", 899, 625);
}
void message(String s) {
textAlign(CENTER, CENTER);
textSize(50);
fill(255);
text(s, 0, 0, width, height);
}
void talk(String s) {
if (ready) {
speak(s);
ready = false;
}
}
/////////////////////////////////////////////////////////////
// for test only
void keyPressed() {
if (key == 's') {
distance = 10;
} else if (key == 'l') {
distance = 30;
changeState(-1);
} else if (key == '0') {
changeState(0);
} else if (key == '1') {
changeState(1);
} else if (key == '2') {
changeState(2);
} else if (key == '3') {
changeState(3);
} else if (key == 'r') {
ready = true;
q++;
}
}
////////////////////////////to/////////////////////////////
void fft() {
the fft.analyze(spectrum);
//Here we are reversing the spectrum array in order to get the higher frequencies on the right edge of canvas.
spectrum = reverse(spectrum);
pushMatrix();
//Translating the position to start drawing in the middle
translate(960, 650);
//Drawing a line for every frequency of the spectrum
for (int i=0; i<bandsWanted; i++) {
stroke(255);
line( i*2, 0, i*2, -spectrum[i]*1000 );
}
popMatrix();
}
////////////////////////////////////////////////////////////////////////////////////////////////////// Questions ////////////////////////////////////////////////////////////////////
StringList qset0() {
StringList qs = new StringList();
qs.append("Are things going well these days?");
qs.append("Okay, let's deal with that, but first let me know: Which one of them do you think best describes the problem that has recently been disturbing you? Touch it to continue.");
return qs;
}
StringList qset1() {
StringList qs = new StringList();
qs.append("Okay, it's obvious that you're suffering from sleep disorders. Tell me what it is that is preventing you from sleeping these days.");
return qs;
}
StringList qset2() {
StringList qs = new StringList();
qs.append("Apparently, you are struggling with relationships recently. Would you like to share more about that?");
return qs;
}
StringList qset3() {
StringList qs = new StringList();
qs.append("I can see that you are stressed with your finals. Would you like to share more about your feelings and what particular subjects you are having trouble with?");
return qs;
}
*/
The code written by my dad worked perfectly well using keyPressed for testing and the voice check achieved the effect that I wanted, using the average of a period of voice and fixing the problem that I had when I was trying to do so but only could make the google TTS in draw loop and thus repeating the speech for many many times at one second.
Step 4: Decoration and better sending the message:
In the user testing session, I think many users failed to get the idea of why it was created and the sense of satire. I know it’s still hard to sense that feeling even though I have already painted the picture on the box now, but I think it somehow works by giving the viewers a sense of weirdness and making them think what this project is actually created for. And for the painting, I searched for “absurdity in art” and I decided to imitate THIS painting:
And I also wanted to make the users feel a sense of closeness and isolation from the outside world just as what I usually imagine the atmosphere in a psychiatry room is like, so I decided to 3D print some furniture I got from THIS website and then laser cut a box with acrylic material to cover the pieces of furniture to show a sense of surveillance, and this is what I got:
Failures:
- Repeated Serial: It was in the email I sent my dad that I explained what I was having trouble with, and although I copied the error message and searched it online, there weren’t any useful results. And my dad explained to me that there might be repeated “serials” in two libraries, and that
import xxx.xxx.*is an abbreviation of the whole library, even though the code may not need the whole. So I followed his suggestion and commented each library in turns to see whether there is still an error. In that way, I managed to find out that it was java.io that also has Serial inside. So, I tracked down what google TTS was using inside that library and I simply specified that import line intoimport java.io.FileOutputStream;and there was no error message showing that error anymore.
Screenshot of the website I searched for confirmation - Distance sensor out of range: When I was experimenting with the distance sensor in Recitation#10 in preparation, everything was just fine. However, when I wanted to use Serial communication between Arduino and Processing, there came a problem: when I was confirming if everything was going okay by printing the distance sent to Processing by Arduino, I found that most of the time, the numbers printed were more than ten thousand. And after a lot of hard thinking, I searched for the datasheet for the Sharp infrared distance sensor (as I learned from Professor Minsky when I was asking her about how the relay works) and then I saw the graph below from THIS website. And then it suddenly came to me: It’s probably because if an object is out of range from the distance sensor, which is more than 40cm, in this case, it will generate very large numbers. So, I experimented with the distance and then wrote these codes in Arduino and Processing accordingly: Arduino:
if (Distance > 40) { Distance = 0;}Processing:if (distance < 30 && distance > 0) {And it worked! Yay!
- 3D printing failure: missing support: It wasn’t until I saw something like this that I knew support is extremely important during 3D printing:
And it was simply because, without the necessary support, the materials will fall due to gravity. And the following video, we can clearly see the support and its significance.
Photo of Cissy helping me remove the support
D. Conclusions:
The goals of my project were to tell the users if they have ever met a psychiatrist who could not give them any emotional support and simply make them question themselves, then they should realize it was not their fault but rather the psychiatrist’s fault, through playing with my project and making fun of the unsupportive psychiatrists. In this sense, I think it’s not really achieving that goal until I clarify what message I actually want to send. But in the sense of interaction, I think my classmate Joy who tested my project in the final presentation did great and even amplified the whole effect. Honestly, I did not expect that there would be such a great laugh when the psychiatrist’s answer came out, and that was kind of an unexpected buff. And I think the voice check function checking for an average amplitude of 0.1 actually played a good role in the sense sometimes it was the laugh from the audience that reached that condition, and while the audience wait for the next answer, they somehow started to expect what will happen next and that is how they are all engaged. And as my definition of my interaction goes, there should be a back-and-forth dialogue between the user and the computer, and most importantly, there should be immediate feedback to show what the users should do next, and the users should always stay focused and engaged, I think my project realized that sense of interaction. If I have more time, I’ll probably try to:
1. figure out how to send the message of satirizing useless psychiatrists more clearly;
2. ask people to fill out surveys about how they think of psychiatrists and psychiatry rooms;
3. experiment with self-print larger knobs so that the tissue box can stay inside before it pops out and that would probably also strengthen the central message;
4. ask more people to test my project to adjust the right amplitude to move on to the next step.
Although this project still has room for improvement, I love and cherish it just as everyone else does, not just because I have spent days and nights on this project, but also because I have learned precious lessons from going through all the difficulties both from all the people I have sought help and also from myself. And the key lesson I learned from the whole process is that anything can go wrong. That’s why Professor Minsky told us that the whole process of making the project should be a constant iteration of idea and implementation, and although that should be a lesson already learned from my midterm project, I didn’t get that until I had so many setbacks in my Finals that I learned this lesson. And I think the reason behind it is that I tried to stick to my original idea as much as possible in my Finals while we were constantly changing between ideas and then abandoning ideas. And as for my first experience of basically trying everything, including box design, laser cutting, and 3D printing, I think the lesson I learned from those first experiences is that practice makes perfect. And it’s natural to panic the first time we get our hands on something, but with some help, we should have the confidence to make it work, and with more experiences gathered, we definitely will become better at the task we’re focusing on. And the third lesson I learned from the whole process is that the effort we paid onto our project is always worth paying. From the final presentation and also from the user testing session, we can see that the projects that surprise us or attract us all need hard work and we can see their hard work based on the organization of ideas, the combination of the physical work, and the computer screen, and the whole concept of their ideas. I believe I’ll benefit from those lessons in the future, and I really appreciate all of those who helped me in any way.
The final presentation:
E. Annex
Material list:
touch sensors 
SHARP GP2Y0A41SKOF Infrared Distance Sensor (4-30cm) 
42STH33-0404AC stepper motor + shaft adapters
an apple 
a mandarin 
a lemon 
L293D chip (H-bridge) 
12 VDC power supply 
USB protector 
Power jack
Wiring diagram:
I couldn’t find stepper motors or touch sensors in TinkerCad Circuits, so I didn’t put touch sensors on it, and used DC motors instead as a substitution of the stepper motors I used.
Laser cut projects in cuttle:
the whole box 
the three drawers for fruits and another one for the tissue box 
the covering box using acrylic 3mm 
the welcome sign
3D prints:
The original dialogue I imagined:
Psychiatrist: Are things going well these days? Answer: I’m not sure. I feel like I’m pressured but I’m okay. Psychiatrist: Okay, let’s deal with that, but first let me know: Which one of them do you think best describes the problem that has recently been disturbing you? Touch it to continue. When the tissue box pops out: That’s all right. It’s always good to outlet your emotions. And I used to cry a lot when I feel pressured, so I encourage you to cry as a way to relieve all your stress. Conversation1: Sleeping disorder P: Okay, it’s obvious that you’re suffering from sleep disorders. Tell me what it is that is preventing you from sleeping these days. A: (My roommate keeps talking with her friends till late at night.) P: Have you talked about this with your roommate? A: Nah, I feel like it’s not that big a problem and I can just fit my sleeping schedules with theirs and that wouldn’t be a problem then. P: I can see that you have a strong inclination to avoid confronting the problem and I bet that has something to do with your childhood. Can you relate that to any of your unhappy childhood memories? A: Not really… P: From my experience of counseling, I know many patients who are neglected and lack attention on them at a young age show this same kind of inclination. I think you have a similar type of problem, right? A: Hmmm. Compared with other children of the same age, I’d say I’d lived a happy childhood with my family and friends. Although it was my grandparents who had taken care of me most time when I was young, it was a complete childhood overall. P: I notice that you are missing from the love given by your parents and that is probably the root reason. A: Okay, but how can I solve my sleeping problems anyway????? P: Don’t worry. No one is perfect and you don’t have to be. I know you’re good enough and you should also keep in mind that you’re great. And I’m always here for you. Conversation2: Struggling with relationships P: Apparently, you are struggling with relationships recently. A: I mean, I do feel like I can improve my relationship with some of my friends…\ P: So, what kind of problems are you having with your friends currently? A: Perhaps I should contact my high school friends more because I haven’t contacted them for a long time since I entered college. P: Is there any particular reason why you have not contacted each other for a long time? And what is preventing you from contacting them? A: Sometimes we just feel like we shouldn’t be disturbing each other and over time, we don’t get as close as before. P: I see. What about the new friends you made in NYUSH? Do they somehow fulfill your need for friendship? A: Yeah I mean, but I have only known them for less than a semester and I still miss my friends from high school… P: I think you are too nostalgic and you have to move on. (pop out tissue box) A: Really? Maybe I’m not that close with my high school friends and I should just move on? P: Yeah definitely. Don’t worry. You’ll be fine. Conversation3: Concerned with studies P: I can see that you are stressed with your finals. Would you like to share more about your feelings and what particular subjects you are having trouble with? A: Definitely. I hate GPS more and more as I get to the finals because it’s such torture that I have to find all the connections and the relationship between individuals and society!!! P: I see I see. Have you looked for ARC for help? A: Yesss. I did try to log in WCOnline, but apparently, there are no remaining availabilities now:). P: I feel so bad for you. (pop out tissue box) A: You know what? I’ll be fine. I can handle this. And it’s only the first semester. P: Apparently, what you just said was a typical way to hide feelings, which can cause a possible major mental collapse if you keep on burying your negative emotions. A: Really? I feel like I’ve already calmed myself and I think I’m good now. P: No. You have to let out all your emotions so that you can truly feel relieved. Just cry out if you want. I’m here for you. (pop out tissue box) A: … P: Don’t worry. You’ll be fine.
The full code:
Arduino:
/**
* This code is adapted from the Arduino example SendReceiveMultipleValues.
#include "SerialRecord.h"
#include <Stepper.h>
SerialRecord writer(2);
SerialRecord reader(1);
const int stepsPerRevolution = 200;
Stepper myStepper1(stepsPerRevolution, 8, 9, 10, 11);
Stepper myStepper2(stepsPerRevolution, 5, 6, 7, 12);
void setup() {
myStepper1.setSpeed(30);
myStepper2.setSpeed(30);
Serial.begin(9600);
}
void loop() {
uint16_t distance = analogRead(A1);
double Distance = get_IR(distance);
int sensorValue1 = digitalRead(2);
int sensorValue2 = digitalRead(3);
int sensorValue3 = digitalRead(4);
if (Distance > 40) {
Distance = 0;
}
int value = 0;
if (sensorValue1 == 1) {
value = 1;
} else if (sensorValue2 == 1) {
value = 2;
} else if (sensorValue3 == 1) {
value = 3;
}
writer[0] = value;
writer[1] = Distance;
writer.send();
delay(20);
int stepperState = 0;
if (reader.read()) {
int stepperState = reader[0];
if (stepperState == 11) {
myStepper1.step(stepsPerRevolution / 2);
}
else if (stepperState == 12) {
myStepper1.step(-stepsPerRevolution / 2);
}
else if (stepperState == 21) {
myStepper2.step(stepsPerRevolution / 2);
}
else if (stepperState == 22) {
myStepper2.step(-stepsPerRevolution / 2);
}
Serial.println(stepperState);
}
}
double get_IR(uint16_t value) {
if (value < 16) value = 16;
return 2076.0 / (value - 11.0);
}
*/
Processing:
/** * This code is adapted from the code wrote by my dad Glen Li. import ddf.minim.*; import java.net.*; import java.io.FileOutputStream; import processing.sound.*; import processing.serial.*; import osteele.processing.SerialRecord.*; // Declare the sound source and FFT analyzer variables FFT fft; AudioIn mic; Minim minim; AudioPlayer player; AudioIn in; Serial serialPort; SerialRecord sender; SerialRecord receiver; Amplitude amp; // Define how many FFT bands to use (this needs to be a power of two) int bands = 1024; // Define the bands wanted for our visualization. // Above a certain threshold, high frequencies are rarely attained and stay flat. int bandsWanted = 128; // Create an array to store the bands wanted float[] spectrum = new float[bandsWanted]; // voice buffer FloatList voice = new FloatList(); int state = -1; // idle int q = 0; // question index boolean ready = true; // ready for new message boolean pendingReady = false; // pending to set ready boolean stepper1Out = false; boolean stepper2Out = false; int distance = 1000; void setup() { //fullScreen(); size(1280, 720); minim = new Minim(this); // Create the AudioIn object and select the mic as the input stream mic = new AudioIn(this, 0); // Start the input stream without routing it to the audio output. mic.start(); amp = new Amplitude(this); amp.input(mic); // Create the FFT analyzer and connect it to the sound input fft = new FFT(this, bands); fft.input(mic); String serialPortName = SerialUtils.findArduinoPort(); serialPort = new Serial(this, serialPortName, 9600); // send stepper motor control commands sender = new SerialRecord(this, serialPort, 1); // read touch sensor choice and distance sensor receiver = new SerialRecord(this, serialPort, 2); } void draw() { background(0); switch(state) { case -1: idle(); break; case 0: welcome(qset0()); break; case 1: conversation(qset1()); break; case 2: conversation(qset2()); break; case 3: conversation(qset3()); break; default: changeState(0); break; } } void stop() { player.close(); minim.stop(); super.stop(); } /////////////////////////////////////////////////////////////////////// ////////Conversation control /////////////////////////////////////////////////////////////////// void idle() { String s = "Welcome to my psychiatry room!"; message(s); receiver.receiveIfAvailable(); distance = receiver.values[1]; println(distance); if (distance < 40 && distance > 0) { changeState(0); } } void welcome(StringList qs) { if (q < qs.size()) { String s = qs.get(q); message(s); talk(s); println(s); checkVoice(); } // last question if (q >= qs.size() - 1) { if (!stepper1Out) { sender.values[0] = 11; sender.send(); stepper1Out = true; } else { // only receive after stepper1 is out receiver.receiveIfAvailable(); int s = receiver.values[0]; println(s); if (s == 1 || s == 2 || s == 3) { println("state:"+ s); changeState(s); } } } } void conversation(StringList qs) { if (q < qs.size()) { String s = qs.get(q); message(s); talk(s); checkVoice(); if (q == 3) { if (!stepper2Out) { sender.values[0] = 21; sender.send(); stepper2Out = true; } } } else { // No more questions, return to welcome state changeState(-1); } } ///////////////////////////////////////////// void changeState(int s) { state = s; q = 0; ready = true; pendingReady = false; voice.clear(); println(s); if (stepper1Out && (s == 1 || s == 2 || s == 3)) { sender.values[0] = 12; sender.send(); stepper1Out = false; } if (stepper2Out) { sender.values[0] = 22; sender.send(); stepper2Out = false; } } ///////////////////////////////////////////// void checkVoice() { if (player != null) { if (player.isPlaying()) { // we need human voice instead of computer voice translate(0, 96); for (int i = 0; i < player.left.size()-1; i++) { stroke(255); line(i, 100 + player.left.get(i)*50, i+1, 100 + player.left.get(i+1)*50); } voice.clear(); return; } speakSign(); } float v = amp.analyze(); voice.append(v); float sum = 0; int duration = 80; // voice checking duration if (voice.size() > duration) { voice.remove(0); for (int i = 0; i < voice.size(); i++) { sum += voice.get(i); } } float average = sum/voice.size(); if (average > 0.1) { println("got voice!"); pendingReady = true; } else { // silence if (pendingReady) { // patient finished talking ready = true; q++; pendingReady = false; } } } void speakSign() { background(0); fft(); textAlign(CENTER, CENTER); textSize(30); fill(255); text("Speak:", 899, 625); } void message(String s) { textAlign(CENTER, CENTER); textSize(50); fill(255); text(s, 0, 0, width, height); } void talk(String s) { if (ready) { speak(s); ready = false; } } ///////////////////////////////////////////////////////////// // for test only void keyPressed() { if (key == '0') { changeState(0); } else if (key == '1') { changeState(1); } else if (key == '2') { changeState(2); } else if (key == '3') { changeState(3); } else if (key == 'r') { ready = true; q++; } else if (key == 's') { distance = 10; } else if (key == 'l') { distance = 0; changeState(-1); } } void fft() { fft.analyze(spectrum); //Here we are reversing the spectrum array in order to get the higher frequencies on the right edge of canvas. spectrum = reverse(spectrum); pushMatrix(); //Translating the position to start drawing in the middle translate(960, 650); //Drawing a line for every frequency of the spectrum for (int i=0; i<bandsWanted; i++) { stroke(255); line( i*2, 0, i*2, -spectrum[i]*1000 ); } popMatrix(); } StringList qset0() { StringList qs = new StringList(); qs.append("Are things going well these days?"); qs.append("Okay, let's deal with that, but first: Which fruit do you think best describes the problem disturbing you? Touch it to continue."); return qs; } StringList qset1() { StringList qs = new StringList(); qs.append("Okay, it's obvious that you're suffering from sleep disorders. Tell me what it is that is preventing you from sleeping these days."); qs.append("Have you talked about this with your roommate?"); qs.append("I can see that you have a strong inclination to avoid confronting the problem and I bet that has something to do with your childhood. Can you relate that to any of your unhappy childhood memories?"); qs.append("From my experience, many patients who are neglected at a young age show this same kind of inclination. I think you have a similar type of problem, right?"); qs.append("I notice that you are missing from the love given by your parents and that is probably the root reason. But don't worry. No one is perfect and you don't have to be. And I'm always here for you."); return qs; } StringList qset2() { StringList qs = new StringList(); qs.append("Apparently, you are struggling with relationships recently. Would you like to share more about that?"); qs.append("What kind of trouble are you having with your relationships?"); qs.append("Is there any particular reason why you have not contacted each other for a long time? And what is preventing you from contacting them?"); qs.append("I see. What about the new friends you made in NYUSH? Do they somehow fulfill your need for friendship?"); qs.append("I think you are too nostalgic and you have to move on."); qs.append("Yeah definitely. Don't worry. You'll be fine."); return qs; } StringList qset3() { StringList qs = new StringList(); qs.append("I can see that you are stressed with your finals. Would you like to share more about your feelings and what particular subjects you are having trouble with?"); qs.append("I see I see. Have you looked for ARC for help?"); qs.append("I feel so bad for you."); qs.append("Apparently, what you just said was a typical way to hide feelings, which can cause a possible major mental collapse if you keep on burying your negative emotions."); qs.append("No. You have to let out all your emotions so that you can truly feel relieved. Just cry out if you want. I'm here for you. Don't worry. You'll be fine. "); return qs; } void speak(String s) { googleTTS(s, "en"); if (player != null) { player.close(); } player = minim.loadFile(sketchPath() + "/1.mp3", 2048); player.play(); } void googleTTS(String txt, String language) { "https://translate.google.com/translate_tts?ie=UTF-8&client=tw-ob&tl=en&"; u = u + language + "&q=" + txt; u = u.replace(" ", "+"); ob&tl=en&q=Hello+World try { URL url = new URL(u); try { URLConnection connection = url.openConnection(); connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 1.2.30703)"); connection.connect(); InputStream is = connection.getInputStream(); File f = new File(sketchPath() + "/1.mp3"); OutputStream out = new FileOutputStream(f); byte buf[] = new byte[1024]; int len; while ((len = is.read(buf)) > 0) { out.write(buf, 0, len); } out.close(); is.close(); } catch (IOException e) { e.printStackTrace(); } } catch (MalformedURLException e) { e.printStackTrace(); } } */
