Category: interaction lab

CONCEPTION AND DESIGN: 

This time, I still choose to collaborate with Arial. Since both of us are interested in traditional Chinese medicine and have past experiences of visiting Chinese medicine clinics, and we both want to promote its principles, we have decided to work on a project related to Chinese herbal medicine. In my three idea documents, the video I referenced served as my inspiration. At that time, we came up with the idea of creating a Chinese medicine simulator, inspired by the concept of a Chinese medicine cabinet. The goal was to allow people to experience the charm of combining different herbs.

Considering time constraints and cultural differences, we designed our presentation to include two modes: Easy Mode and Hard Mode. When participants click “Start,” they are introduced to the role they will play as a Chinese medicine practitioner, and they need to select herbs based on the patient’s symptoms. The next page displays a conversation between the patient and the doctor, providing the name of the herbal prescription.

In Easy Mode, the screen directly displays the herbs required in the prescription along with their respective quantities. Participants only need to open the corresponding drawers and take out the labeled bags of herbs. However, each drawer may contain one to three bags, each containing different weights of medicinal ingredients. Participants need to use a scale to find the appropriate weight. We purchased a traditional Chinese medicine scale online, which is not an electronic scale and can be a bit challenging to use. We will provide instructions to the participants before they start the experience. As participants open a drawer, the corresponding images and information will be displayed on the screen. They can drag and rearrange the images, and when they close the drawer, the images disappear. This way, they can also acquire relevant knowledge during the process.

In Hard Mode, we only provide the effects of the herbal prescription, the number of different herb types required, and the total weight. Participants need to open each drawer and match the herbs based on the descriptions and prompts provided in the images. After participants take out a bag, I will assist them in using an RFID detector in a medicine pot to identify the RFID sticker attached to the bag. Finally, the screen will display a success or failure message and play corresponding music.

During user testing, many people provided feedback that they were unsure about the first steps when they saw the screen, and they felt confused. They also didn’t know how many bags to take out from each drawer. Therefore, in subsequent presentations, we added verbal prompts, informing them to select only one bag from each drawer and guiding them to open the drawers.

Additionally, during user testing, we did not label the herbs on each bag, which caused confusion when participants put the bags together. So, before the presentation, we labeled each bag with the names of the herbs in both Chinese and English.

Furthermore, during user testing, we found that the current flow detection between the drawers was affected. Initially, we followed a homemade switch design from the first recitation, which I will explain in the fabrication process. In the end, we switched to using reed switches.

During the presentation, the computer randomly selected the most challenging herbal prescription, so the participant spent a long time searching for the corresponding drawer in Easy Mode. She also provided feedback that the scale was too difficult to use. As our project is suitable for 1-2 people to experience, other observers also found it confusing.

The entire process of opening the drawers was relatively successful, and since we also placed herbs inside the drawers, the participant expressed curiosity about these herbs, which inspired us to create an Ima Show version of the project. However, unfortunately, during the detection phase, the participant felt a disconnect because it was conducted on another computer, separate from the medicine cabinet. Additionally, I didn’t anticipate that the RFID stickers on the bags would not make flat contact with the detector due to their placement, and the metal weights inside the bags would interfere with the detection.

Based on the feedback received during the presentation and the advice from the professor, we ultimately decided to abandon the gaming aspect and shift towards an educational direction for the Ima Show version. In the Ima Show, participants wear headphones, and for each opened drawer, the screen displays informative images and plays videos related to the production or introduction of the corresponding herbs. The drawers contain real herbs, allowing participants to observe, touch, and smell the herbs, providing them with a multi-sensory experience. Based on the participants’ reactions, they found this approach to be more direct and engaging.

FABRICATION AND PRODUCTION:

At the beginning, we planned to divide the tasks, with Arial handling the laser cutting part and me handling the 3D printing of the medicine pot. When searching for a 3D model, I couldn’t find one specifically for a Chinese medicine pot, so I settled for a Japanese teapot model that closely resembled its shape. After downloading the file, I initially intended to modify it using the Tinkercad website. However, since I was not familiar with advanced operations in Tinkercad, I ended up using Blender software to make the necessary modifications. However, during user testing, many people commented that the 3D printed medicine pot looked too modern and didn’t blend well with the overall design. As a result, we ultimately decided to use wood as the material for the medicine pot. Arial took on the task of handcrafting a wooden medicine pot.

In terms of production, my responsibility was to ensure the feasibility and difficulty of the game. I organized a spreadsheet with the required weights of different herbs and simplified them. I removed any distracting text from the herb descriptions. I also created fictional herbs representing different forms of water (dew, spring, and snow) to reduce the difficulty for non-Chinese participants. Additionally, based on real-world information, I provided recommended herb combinations in the images, hoping to make the game less challenging. However, unfortunately, I still underestimated the issue of cultural differences, and it proved to be too difficult for many non-Chinese participants.

Next, I will explain the issue with the switches. Initially, we wanted to use Arduino’s digitalRead pin to detect the current and output a value of 0 or 1. We would then send this value to Processing to control the display and disappearance of images. We initially followed the homemade switch design from the first recitation, where we attached conductive tape to the back panel of each drawer and drilled holes in the back panel of the medicine cabinet to insert the wires. This setup created a closed circuit when the drawer was closed and opened when the drawer was pulled out.

However, during user testing, we found this setup to be unstable. There were still gaps between the back panel of the drawer and the wooden panel of the cabinet, causing vibrations and interfering with the current detection of nearby drawers when opening and closing a drawer. To address this, we initially planned to use wood glue to tightly connect the dividers with the other wooden panels. However, when the glue dried, we realized that due to angle issues, some drawers couldn’t fit properly, so we had to use force and a hammer to loosen some of the wooden panels.

In addition to the switches, there were also issues with the detector. Initially, we wanted to combine the detection process with the previous herb selection process into a single Processing program. However, we didn’t realize that there are additional digitalRead pins on the Arduino Mega board in other area. While driving the previous program, it has already caused confusion in Arduino detection and frequent computer crashes. In this situation, we urgently abandoned the original plan and switched to using a second computer before the final presentation. To simplify the visualization, we set it up so that when a sticker was detected, the corresponding herb name would be displayed on the screen. Due to time constraints, the effect was quite rough. Additionally, I wanted to achieve the effect where the detector would display success only when it detected the correct combination of RFID stickers, and it would display failure only after all the stickers were detected. However, the issue was that it would immediately display failure when it detected an incorrect sticker during the process. I sought help from Kevin, but he also expressed his inability to solve the problem. Finally, I decided to “fake it to make it” by manually checking the stickers based on their numbers while assisting with the detection, and then displaying failure through a keypress at the end.

On the night before the “IMA Show,” based on suggestions, we searched the internet for introduction videos of various herbs and cut out the key parts. However, we couldn’t find suitable videos on YouTube, so we had to rely on Chinese websites, which could pose some understanding difficulties for non-Chinese participants. Another unexpected issue we encountered was frame lag when playing the videos in Processing, resulting in an unsmooth video playback. We sought help from various individuals, but ultimately, we couldn’t resolve this problem.

CONCLUSIONS:

Regardless, I believe our IMA Show version achieved my initial goal, which was to let people experience the charm of traditional Chinese medicine through interaction. They were able to directly sense the herbs through visual observation, touch, and smell, aligning with my definition of interaction that I wrote at the beginning of the semester. Regarding the presentation version, I think we also achieved interaction, but the overall process was too lengthy, and the use of two computers made the experience less immersive. My inspiration actually came from games, where the key point is to be engaging. However, I overlooked the differences in cultural backgrounds and people’s ability to comprehend and memorize text within a short period of time, resulting in the game being too challenging and affecting participants’ experience. If we had more time, I would have been more meticulous in finding various sources, such as animations, to diversify the content being presented and include bilingual subtitles. My pursuit of interaction has also undergone some changes. It doesn’t necessarily have to be a highly entertaining game; projects with educational and promotional significance can also be valuable. In ima show, some people even asked us that if we tend to coorporate with the museum, which makes me feel so moved. And I am glad to see that in the ima show, many parents encouraged their children to experience and used our project to tell some knowledge to their children.We shouldn’t strive for overly complex elements; a more intuitive approach would facilitate users’ experience and provide them with a more direct sensory experience.

During the process of organizing various materials, I also realized the challenges of game design, which involves ensuring logical game mechanics and setting appropriate difficulty levels, requiring extensive user testing. In the process of modifying the detector code, I learned a lot. Since it was unfamiliar territory for me, I had to go online or seek assistance from people around me to learn coding, which greatly enhanced my patience and coding skills. Additionally, I understood the importance of promptly contacting relevant departments at the school when encountering issues. I want to express my gratitude to my professor, Rudi, and the school’s facilities for their assistance. In the future, when undertaking activities that may impact others, I will carefully consider and review the relevant regulations.

DISASSEMBLY:

APPENDIX

Here is the code：

Arduino RFID detector

From What is RFID? How It Works? Interface RC522 RFID Module with Arduino

*
* --------------------------------------------------------------------------------------------------------------------
* Example sketch/program showing how to read data from a PICC to serial.
* --------------------------------------------------------------------------------------------------------------------
* This is a MFRC522 library example; for further details and other examples see: https://github.com/miguelbalboa/rfid
*
* Example sketch/program showing how to read data from a PICC (that is: a RFID Tag or Card) using a MFRC522 based RFID
* Reader on the Arduino SPI interface.
*
* When the Arduino and the MFRC522 module are connected (see the pin layout below), load this sketch into Arduino IDE
* then verify/compile and upload it. To see the output: use Tools, Serial Monitor of the IDE (hit Ctrl+Shft+M). When
* you present a PICC (that is: a RFID Tag or Card) at reading distance of the MFRC522 Reader/PCD, the serial output
* will show the ID/UID, type and any data blocks it can read. Note: you may see "Timeout in communication" messages
* when removing the PICC from reading distance too early.
*
* If your reader supports it, this sketch/program will read all the PICCs presented (that is: multiple tag reading).
* So if you stack two or more PICCs on top of each other and present them to the reader, it will first output all
* details of the first and then the next PICC. Note that this may take some time as all data blocks are dumped, so
* keep the PICCs at reading distance until complete.
*
* @license Released into the public domain.
*
* Typical pin layout used:
* -----------------------------------------------------------------------------------------
* MFRC522 Arduino Arduino Arduino Arduino Arduino
* Reader/PCD Uno/101 Mega Nano v3 Leonardo/Micro Pro Micro
* Signal Pin Pin Pin Pin Pin Pin
* -----------------------------------------------------------------------------------------
* RST/Reset RST 9 5 D9 RESET/ICSP-5 RST
* SPI SS SDA(SS) 10 53 D10 10 10
* SPI MOSI MOSI 11 / ICSP-4 51 D11 ICSP-4 16
* SPI MISO MISO 12 / ICSP-1 50 D12 ICSP-1 14
* SPI SCK SCK 13 / ICSP-3 52 D13 ICSP-3 15
*
* More pin layouts for other boards can be found here: https://github.com/miguelbalboa/rfid#pin-layout
*/
#include 
#include 
#define RST_PIN 5 // Configurable, see typical pin layout above
#define SS_PIN 10 // Configurable, see typical pin layout above
MFRC522 mfrc522(SS_PIN, RST_PIN); // Create MFRC522 instance
void setup() {
Serial.begin(9600); // Initialize serial communications with the PC
while(!Serial); // Do nothing if no serial port is opened (added for Arduinos based on ATMEGA32U4)
SPI.begin(); // Init SPI bus
mfrc522.PCD_Init(); // Init MFRC522
delay(4); // Optional delay. Some board do need more time after init to be ready, see Readme
mfrc522.PCD_DumpVersionToSerial(); // Show details of PCD - MFRC522 Card Reader details
Serial.println(F("Scan PICC to see UID, SAK, type, and data blocks..."));
}
void loop() {
// Reset the loop if no new card present on the sensor/reader. This saves the entire process when idle.
if( ! mfrc522.PICC_IsNewCardPresent()){
return;
}
// Select one of the cards
if( ! mfrc522.PICC_ReadCardSerial()){
return;
}
// Dump debug info about the card; PICC_HaltA() is automatically called
mfrc522.PICC_DumpToSerial(&(mfrc522.uid));
} 

import processing.serial.*;
import processing.sound.*;
SoundFile file1;
SoundFile file2;
//PFont myFont;

Serial myPort;
String val;
boolean card1Detected = false;
boolean card2Detected = false;
boolean card3Detected = false;
boolean card4Detected = false;
boolean card5Detected = false;
boolean card6Detected = false;
boolean card7Detected = false;
boolean card8Detected = false;
boolean card9Detected = false;
boolean card10Detected = false;
boolean card11Detected = false;
boolean card12Detected = false;
boolean card13Detected = false;
boolean card14Detected = false;
boolean card15Detected = false;
boolean card16Detected = false;
boolean card17Detected = false;
boolean card18Detected = false;
boolean card19Detected = false;
boolean card20Detected = false;
boolean card21Detected = false;
boolean card22Detected = false;
boolean card23Detected = false;
boolean card24Detected = false;
boolean card25Detected = false;
boolean card26Detected = false;
boolean card27Detected = false;
boolean card28Detected = false;

boolean card29Detected = false;
boolean card30Detected = false;

//boolean card30Detected = false;

//boolean card31Detected = false;

long lastDetectionTime = 0; // 记录最后一次检测到卡的时间
int timeout = 10000; // 设置超时时间为20秒
boolean timeoutActive = false; // 超时计时器是否激活

int isDetectedValue1 = 0;
int isDetectedValue2 = 0;
int isDetectedValue3 = 0;
int isDetectedValue4 = 0;
int isDetectedValue5 = 0;


PImage image1, image2; // 图像对象

void setup() {
  //size(640, 360);
  fullScreen();
  println("Available serial ports: " + Serial.list());
  myPort = new Serial(this, Serial.list()[5], 9600);
  myPort.bufferUntil('\n');
  background(0);

  file1 = new SoundFile(this, "success-1-6297.mp3");
  file2 = new SoundFile(this, "wah-wah-sad-trombone-6347.mp3");
  // 加载图像，确保图像文件在项目的根目录下或者指定了正确的路径
  image1 = loadImage("success.jpg");
  image2 = loadImage("fail.jpg");

  if (image1 == null) {
    println("Error loading image1");
  } else {
    println("Image1 loaded successfully");
  }

  if (image2 == null) {
    println("Error loading image2");
  } else {
    println("Image2 loaded successfully");
  }
  //resetDetection();
}

void draw() {
  //if (timeoutActive && millis() - lastDetectionTime > timeout && isDetectedValue == 2) {
  //  println("Timeout reached, displaying image2");
  //  image(image2, 0, 0, width, height);
  //  //resetDetection();
  //}
  if (card1Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Peony Root 30g ", 50, 100 );
    println("card1Detected");
  }
  if (card2Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Licorice 20g ", 500, 100);
    println("card2Detected");
  }
  if (card3Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Firewood 30g ", 950, 100 );
    println("card3Detected");
  }
  if (card4Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Angelica 30g ", 1400, 100);
    println("card4Detected");
  }
  if (card5Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Wolfiporia 30g ", 50, 250 );
    println("card1Detected");
  }
  if (card6Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Atractylodes 30g ", 500, 250);
    println("card2Detected");
  }
  if (card7Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Spring 100g ", 950, 250 );
    println("card3Detected");
  }
  if (card8Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Milk Vetch root 30g ", 1300, 250);
    println("card4Detected");
  }
  if (card9Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Licorice 10g ", 50, 400 );
    println("card1Detected");
  }
  if (card10Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Firewood 5g ", 500, 400);
    println("card2Detected");
  }
  if (card11Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Angelica 5g ", 950, 400 );
    println("card3Detected");
  }
  if (card12Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Atractylodes 10g ", 1300, 400);
    println("card4Detected");
  }
if (card13Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Dried Orange Peel 5g ", 50, 400 );
    println("card1Detected");
  }
  if (card14Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Jute 5g ", 500, 400);
    println("card2Detected");
  }
  if (card15Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Ginseng 5g ", 950, 400 );
    println("card3Detected");
  }
  if (card16Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Snow 100g ", 1300, 400);
    println("card4Detected");
  }
  if (card17Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Milk Vetch Root 50g ", 50, 550 );
    println("card1Detected");
  }
  if (card18Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Atractylodes 50g ", 500, 550);
    println("card2Detected");
  }
  if (card19Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Wind-Proof Grass 30g ", 950, 550 );
    println("card3Detected");
  }
  if (card20Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Jujube 5g ", 1300, 550);
    println("card4Detected");
  }
  if (card21Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Cinnamon 10g ", 50, 700 );
    println("card1Detected");
  }
  if (card22Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Ginger 10g ", 500, 700);
    println("card2Detected");
  }
  if (card23Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Peony Root 10g ", 950, 700 );
    println("card3Detected");
  }
  if (card24Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Licorice 5g ", 1300, 700);
    println("card4Detected");
  }
  if (card25Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Dew 100g ", 50, 850 );
    println("card1Detected");
  }
  if (card26Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Mid-Summer 10g ", 500, 850);
    println("card2Detected");
  }
  if (card27Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Bitterness 5g ", 950, 850 );
    println("card3Detected");
  }
  if (card28Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Ginseng 10g ", 1300, 850);
    println("card4Detected");
  }
  if (card29Detected) {
    stroke(10);
    fill(255);
    textSize(50);
    text("Scrutellaria 10g ", 50, 1000 );
    println("card1Detected");
  }
  
  checkCardsAndDisplay();
}

void serialEvent(Serial myPort) {
  val = myPort.readStringUntil('\n');
  if (val != null) {
    val = trim(val);
    println("Received UID: " + val);
    if (!timeoutActive && (card1Detected || card2Detected)) {
      lastDetectionTime = millis(); // 在检测到第一张卡后开始计时
      timeoutActive = true; // 激活超时计时器
    }

    if (val.equals("0   0D D3 1E DE  1E 08 04 00  03 D4 57 FA  7B CE 73 90  [ 0 0 0 ]")) {
      card1Detected = true;
      println("Card 1 detected");
      isDetectedValue1 = 1;
    } else if (val.equals("0   1D D3 1E DE  0E 08 04 00  03 94 55 01  EB 4D 46 90  [ 0 0 0 ]")) {
      card2Detected = true;
      println("Card 2 detected");
      isDetectedValue1 = 2;
    } else if (val.equals("0   2D D3 1E DE  3E 08 04 00  03 12 B5 84  9C AB 4B 90  [ 0 0 0 ]")) {
      card3Detected = true;
      println("Card 3 detected");
      isDetectedValue1 = 3;
    } else if (val.equals("0   3D D3 1E DE  2E 08 04 00  03 CC D4 33  12 5A 9E 90  [ 0 0 0 ]")) {
      card4Detected = true;
      println("Card 4 detected");
      isDetectedValue1 = 4;
    } else if (val.equals("0   4D D3 1E DE  5E 08 04 00  03 85 46 73  50 65 E8 90  [ 0 0 0 ]")) {
      card5Detected = true;
      println("Card 5 detected");
      isDetectedValue1 = 5;
    } else if (val.equals("0   5D D3 1E DE  4E 08 04 00  03 0F C2 A6  88 18 14 90  [ 0 0 0 ]")) {
      card6Detected = true;
      println("Card 6 detected");
      isDetectedValue1 = 6;
    } else if (val.equals("0   6D D3 1E DE  7E 08 04 00  03 74 0F F8  7E A5 CB 90  [ 0 0 0 ]")) {
      card7Detected = true;
      println("Card 7 detected");
      isDetectedValue1 = 7;
    }
    //else {
    //  println("Received unknown card UID");
    //}
    //} else {
    //  println("No data received or incomplete data");
  }



  if (val.equals("0   7D D3 1E DE  6E 08 04 00  03 A5 8E F4  33 2E DE 90  [ 0 0 0 ]")) {
    card8Detected = true;
    println("Card 8 detected");
    isDetectedValue2 = 1;
  } else if (val.equals("0   8D D3 1E DE  9E 08 04 00  03 15 9E BA  3C 20 6C 90  [ 0 0 0 ]")) {
    card9Detected = true;
    println("Card 9 detected");
    isDetectedValue2 = 2;
  } else if (val.equals("0   9D D3 1E DE  8E 08 04 00  03 CA 79 7C  E9 BB 3B 90  [ 0 0 0 ]")) {
    card10Detected = true;
    println("Card 10 detected");
    isDetectedValue2 = 3;
  } else if (val.equals("0   AD D3 1E DE  BE 08 04 00  03 A0 6E 95  17 DA C1 90  [ 0 0 0 ]")) {
    card11Detected = true;
    println("Card 11 detected");
    isDetectedValue2 = 4;
  } else if (val.equals("0   BD D3 1E DE  AE 08 04 00  03 1D E7 6D  86 71 4C 90  [ 0 0 0 ]")) {
    card12Detected = true;
    println("Card 12 detected");
    isDetectedValue2 = 5;
  } else if (val.equals("0   CD D3 1E DE  DE 08 04 00  03 22 93 D0  EA FF AF 90  [ 0 0 0 ]")) {
    card13Detected = true;
    println("Card 13 detected");
    isDetectedValue2 = 6;
  } else if (val.equals("0   DD D3 1E DE  CE 08 04 00  03 02 AA F4  48 4A 4C 90  [ 0 0 0 ]")) {
    card14Detected = true;
    println("Card 14 detected");
    isDetectedValue2 = 7;
  } else if (val.equals("0   ED D3 1E DE  FE 08 04 00  03 25 07 05  F0 45 86 90  [ 0 0 0 ]")) {
    card15Detected = true;
    println("Card 15 detected");
    isDetectedValue2 = 8;
  } else if (val.equals("0   FD D3 1E DE  EE 08 04 00  03 4C BB B8  5C 0D 7A 90  [ 0 0 0 ]")) {
    card16Detected = true;
    println("Card 16 detected");
    isDetectedValue2 = 9;
  } else {
    // println("Received unknown card UID");
  }
  //else {
  //  println("No data received or incomplete data");
  //}
  if (val.equals("0   0D D4 1E DE  19 08 04 00  03 3A 5D 4A  0F 28 1E 90  [ 0 0 0 ]")) {
    card17Detected = true;
    println("Card 17 detected");
    isDetectedValue3 = 1;
  } else if (val.equals("0   1D D4 1E DE  09 08 04 00  03 CF 75 A4  E9 22 96 90  [ 0 0 0 ]")) {
    card18Detected = true;
    println("Card 18 detected");
    isDetectedValue3 = 2;
  } else if (val.equals("0   2D D4 1E DE  39 08 04 00  03 73 E1 09  02 EE 63 90  [ 0 0 0 ]")) {
    card19Detected = true;
    println("Card 19 detected");
    isDetectedValue3 = 3;
  } else if (val.equals("0   6D D3 1E DE  7E 08 04 00  03 74 0F F8  7E A5 CB 90  [ 0 0 0 ]")) {
    card7Detected = true;
    println("Card 7 detected");
    isDetectedValue3 = 4;
  } else {
    // println("Received unknown card UID");
  }


  if (val.equals("0   3D D4 1E DE  29 08 04 00  03 77 78 BC  B0 0A 3C 90  [ 0 0 0 ]")) {
    card20Detected = true;
    println("Card 20 detected");
    isDetectedValue4 = 1;
  } else if (val.equals("0   4D D4 1E DE  59 08 04 00  03 E2 4E 7C  D3 A6 5B 90  [ 0 0 0 ]")) {
    card21Detected = true;
    println("Card 21 detected");
    isDetectedValue4 = 2;
  } else if (val.equals("0   6D CA 1E DE  67 08 04 00  03 A8 B2 9F  D9 FE 27 90  [ 0 0 0 ]")) {
    card22Detected = true;
    println("Card 22 detected");
    isDetectedValue4 = 3;
  } else if (val.equals("0   8D CA 1E DE  87 08 04 00  03 E4 08 1B  42 C4 3D 90  [ 0 0 0 ]")) {
    card23Detected = true;
    println("Card 23 detected");
    isDetectedValue4 = 4;
  } else if (val.equals("0   9D CA 1E DE  97 08 04 00  03 94 17 AA  D4 BF F7 90  [ 0 0 0 ]")) {
    card24Detected = true;
    println("Card 24 detected");
    isDetectedValue4 = 5;
  } else if (val.equals("0   AD CA 1E DE  A7 08 04 00  03 2A 22 CB  E7 82 0D 90  [ 0 0 0 ]")) {
    card25Detected = true;
    println("Card 25 detected");
    isDetectedValue4 = 6;
  } else {
    // println("Received unknown card UID");
  }

  if (val.equals("0   3D D4 1E DE  29 08 04 00  03 77 78 BC  B0 0A 3C 90  [ 0 0 0 ]")) {
    card20Detected = true;
    println("Card 20 detected");
    isDetectedValue5 = 1;
  } else if (val.equals("0   6D CA 1E DE  67 08 04 00  03 A8 B2 9F  D9 FE 27 90  [ 0 0 0 ]")) {
    card22Detected = true;
    println("Card 22 detected");
    isDetectedValue5 = 2;
  } else if (val.equals("0   BD CA 1E DE  B7 08 04 00  03 DD 6E C4  09 C8 D9 90  [ 0 0 0 ]")) {
    card26Detected = true;
    println("Card 26 detected");
    isDetectedValue5 = 3;
  } else if (val.equals("0   8D D3 1E DE  9E 08 04 00  03 15 9E BA  3C 20 6C 90  [ 0 0 0 ]")) {
    card9Detected = true;
    println("Card 9 detected");
    isDetectedValue5 = 4;
  } else if (val.equals("0   CD CA 1E DE  C7 08 04 00  03 77 AE 93  10 C6 AB 90  [ 0 0 0 ]")) {
    card27Detected = true;
    println("Card 27 detected");
    isDetectedValue5 = 5;
  } else if (val.equals("0   DD CA 1E DE  D7 08 04 00  03 68 87 32  94 6A 7A 90  [ 0 0 0 ]")) {
    card28Detected = true;
    println("Card 28 detected");
    isDetectedValue5 = 6;
  } else if (val.equals("0   AD CA 1E DE  A7 08 04 00  03 2A 22 CB  E7 82 0D 90  [ 0 0 0 ]")) {
    card25Detected = true;
    println("Card 25 detected");
    isDetectedValue5 = 7;
  } else if (val.equals("0   ED CA 1E DE  E7 08 04 00  03 98 92 4A  7F EB 99 90  [ 0 0 0 ]")) {
    card29Detected = true;
    println("Card 29 detected");
    isDetectedValue5 = 8;
  }
}

void checkCardsAndDisplay() {
  if (card1Detected && card2Detected && card3Detected && card4Detected && card5Detected && card6Detected && card7Detected) {
    //println("Both cards detected, displaying image1");
    image(image1, 0, 0, width, height);

    if (file1.isPlaying() == false) {
      file1.play();
    }

    resetDetection();
    //} else if (card1Detected ^ card2Detected) {
    //  //println("Only one card detected, displaying image2");
    //  image(image2, 0, 0, width, height);
    //  resetDetection();
  } else if (card30Detected) {
    image(image2, 0, 0, width, height);

    if (file2.isPlaying() == false) {
      file2.play();
    }
    //file2.rate(2);
  }

  if (card8Detected && card9Detected && card10Detected && card11Detected && card12Detected && card13Detected && card14Detected && card15Detected && card16Detected) {
    //println("Both cards detected, displaying image1");
    image(image1, 0, 0, width, height);
    if (file1.isPlaying() == false) {
      file1.play();
    }
    resetDetection();
    //} else if (card1Detected ^ card2Detected) {
    //  //println("Only one card detected, displaying image2");
    //  image(image2, 0, 0, width, height);
    //  resetDetection();
  } else if (card30Detected) {
    image(image2, 0, 0, width, height);
    if (file2.isPlaying() == false) {
      file2.play();
    }
  }

  if (card17Detected && card18Detected && card19Detected && card7Detected ) {
    //println("Both cards detected, displaying image1");
    image(image1, 0, 0, width, height);
    if (file1.isPlaying() == false) {
      file1.play();
    }
    resetDetection();
    //} else if (card1Detected ^ card2Detected) {
    //  //println("Only one card detected, displaying image2");
    //  image(image2, 0, 0, width, height);
    //  resetDetection();
  } else if (card30Detected) {
    image(image2, 0, 0, width, height);
  }

  if (card20Detected && card21Detected && card22Detected && card23Detected && card24Detected && card25Detected) {
    //println("Both cards detected, displaying image1");
    image(image1, 0, 0, width, height);
    if (file1.isPlaying() == false) {
      file1.play();
    }
    resetDetection();
    //} else if (card1Detected ^ card2Detected) {
    //  //println("Only one card detected, displaying image2");
    //  image(image2, 0, 0, width, height);
    //  resetDetection();
  } else if (card30Detected) {
    image(image2, 0, 0, width, height);
    if (file2.isPlaying() == false) {
      file2.play();
    }
  }

  if (card20Detected && card22Detected && card26Detected && card9Detected && card27Detected && card28Detected && card25Detected && card29Detected) {
    //println("Both cards detected, displaying image1");
    image(image1, 0, 0, width, height);
    if (file1.isPlaying() == false) {
      file1.play();
    }
    resetDetection();
    //} else if (card1Detected ^ card2Detected) {
    //  //println("Only one card detected, displaying image2");
    //  image(image2, 0, 0, width, height);
    //  resetDetection();
  } else if (card30Detected) {
    image(image2, 0, 0, width, height);
    if (file2.isPlaying() == false) {
      file2.play();
    }
  }
}

void resetDetection() {
  timeoutActive = false;
}

void keyPressed() {
  card30Detected = true;
}

int box1 = 21;
int box2 = 2;
int box3 = 3;
int box4 = 4;
int box5 = 5;
int box6 = 6;
int box7 = 7;
int box8 = 8;
int box9 = 9;
int box10 = 10;
int box11 = 11;
int box12 = 12;
int box13 = 13;
int box14 = 14;
int box15 = 15;
int box16 = 16;
int box17 = 17;
int box18 = 18;
int box19 = 19;
int box20 = 20;

void setup() {
Serial.begin(9600);
// pu.t your setup code here, to run once:
pinMode(box1, INPUT_PULLUP);
pinMode(box2, INPUT_PULLUP);
pinMode(box3, INPUT_PULLUP);
pinMode(box4, INPUT_PULLUP);
pinMode(box5, INPUT_PULLUP);
pinMode(box6, INPUT_PULLUP);
pinMode(box7, INPUT_PULLUP);
pinMode(box8, INPUT_PULLUP);
pinMode(box9, INPUT_PULLUP);
pinMode(box10, INPUT_PULLUP);
pinMode(box11, INPUT_PULLUP);
pinMode(box12, INPUT_PULLUP);
pinMode(box13, INPUT_PULLUP);
pinMode(box14, INPUT_PULLUP);
pinMode(box15, INPUT_PULLUP);
pinMode(box16, INPUT_PULLUP);
pinMode(box17, INPUT_PULLUP);
pinMode(box18, INPUT_PULLUP);
pinMode(box19, INPUT_PULLUP);
pinMode(box20, INPUT_PULLUP);
}

void loop() {
// put your main code here, to run repeatedly:
int state1 = digitalRead(box1);
int state2 = digitalRead(box2);
int state3 = digitalRead(box3);
int state4 = digitalRead(box4);
int state5 = digitalRead(box5);
int state6 = digitalRead(box6);
int state7 = digitalRead(box7);
int state8 = digitalRead(box8);
int state9 = digitalRead(box9);
int state10 = digitalRead(box10);
int state11 = digitalRead(box11);
int state12 = digitalRead(box12);
int state13 = digitalRead(box13);
int state14 = digitalRead(box14);
int state15 = digitalRead(box15);
int state16 = digitalRead(box16);
int state17 = digitalRead(box17);
int state18 = digitalRead(box18);
int state19 = digitalRead(box19);
int state20 = digitalRead(box20);

Serial.print(state1);
Serial.print(“,”);
Serial.print(state2);
Serial.print(“,”);
Serial.print(state3);
Serial.print(“,”);
Serial.print(state4);
Serial.print(“,”);
Serial.print(state5);
Serial.print(“,”);
Serial.print(state6);
Serial.print(“,”);
Serial.print(state7);
Serial.print(“,”);
Serial.print(state8);
Serial.print(“,”);
Serial.print(state9);
Serial.print(“,”);
Serial.print(state10);
Serial.print(“,”);
Serial.print(state11);
Serial.print(“,”);
Serial.print(state12);
Serial.print(“,”);
Serial.print(state13);
Serial.print(“,”);
Serial.print(state14);
Serial.print(“,”);
Serial.print(state15);
Serial.print(“,”);
Serial.print(state16);
Serial.print(“,”);
Serial.print(state17);
Serial.print(“,”);
Serial.print(state18);
Serial.print(“,”);
Serial.print(state19);
Serial.print(“,”);
Serial.print(state20);

Serial.println();

delay(10);
}
 

import processing.serial.*;
Serial serialPort;

int NUM_OF_VALUES_FROM_ARDUINO = 20;
int NUM_OF_IMG = 20;
int NUM_OF_PATIENT = 5;
int NUM_OF_DOC = 5;
int NUM_OF_PRE = 5;
int NUM_OF_PROMPT=5;


int numImages = 20;
float spacing = 150;
float imageWidth = 1181/2;
float imageHeight = 944/2;
float scroll = 0;

PFont myFont;

int m_pre;

/* This array stores values from Arduino */
int arduino_values[] = new int[NUM_OF_VALUES_FROM_ARDUINO];
//int arduino_values1[] = new int[NUM_OF_BUTTON];

PImage[] images = new PImage[NUM_OF_IMG];

//images of the traits
PImage angelica1;
PImage angelica2;
PImage atractylodes1;
PImage atractylodes2;
PImage atractylodes3;
PImage bitterness;
PImage cinnamon;
PImage dew;
PImage dried_orange_peel;
PImage firewood1;
PImage firewood2;
PImage ginger;
PImage ginseng1;
PImage ginseng2;
PImage jujube;
PImage jute;
PImage licorice1;
PImage licorice2;
PImage licorice3;
PImage licorice4;
PImage midsummer;
PImage milkvetch1;
PImage milkvetch2;
PImage peony1;
PImage peony2;
PImage scutellaria;
PImage snow;
PImage spring;
PImage wolf;
PImage wpg;

//images of patients with 5 different symptoms
PImage[] images_patient = new PImage[NUM_OF_PATIENT];
PImage img21;
PImage img22;
PImage img23;
PImage img24;
PImage img25;

//images of doctors with 5 different solutions
PImage[] images_doc = new PImage[NUM_OF_DOC];
PImage img26;
PImage img27;
PImage img28;
PImage img29;
PImage img30;

//images of 5 different prescriptions for easy mode
PImage[] images_pre = new PImage[NUM_OF_PRE];
PImage img31;
PImage img32;
PImage img33;
PImage img34;
PImage img35;

//user testing upload
PImage patient;
PImage easymode_guizhi;
PImage prompt;
PImage start;
PImage sym;

//images of prompt for hard mode
PImage[] images_prompt = new PImage[NUM_OF_PROMPT];
PImage promptXY;
PImage promptBZYQ;
PImage promptGZ;
PImage promptYPFS;
PImage promptXX;

int mode = 0;

//positions for the pop up
float imgX[]= new float[NUM_OF_IMG];
float imgY[] = new float[NUM_OF_IMG];

void setup() {
  fullScreen();
  rectMode(CENTER);
  textAlign(CENTER, CENTER);

  printArray(Serial.list());
  serialPort = new Serial(this, "COM5", 9600);

  imageMode(CENTER);
  for (int i=0; i < 20; i++) {
    imgX[i] = random(1754/4.8, width-1754/2.4);
    imgY[i] = random(1240/4.8, height-1240/2.4);
  }

  //load images of the herbs
  angelica1 = loadImage("Angelica_BZYQ.png");
  angelica2 = loadImage("Angelica_xiaoyao.png");

  atractylodes1 = loadImage("Atrac_BZYQ.png");
  atractylodes2 = loadImage("Atrac_xiaoyao.png");
  atractylodes3 = loadImage("Atrac_YPFS.png");

  bitterness = loadImage("bitter.png");
  cinnamon = loadImage("cinnamon.png");
  dew = loadImage("dew.png");
  dried_orange_peel = loadImage("DOP.png");

  firewood1 = loadImage("FW_BZYQ.png");
  firewood2 = loadImage("FW_xiaoyao.png");

  ginger = loadImage("Ginger.png");

  ginseng1 = loadImage("Ginseng_BZYQ.png");
  ginseng2 = loadImage("Ginseng_xiexin.png");

  jujube = loadImage("jujube.png");
  jute = loadImage("Jute.png");

  licorice1 = loadImage("Licorice_BZYQ.png");
  licorice2 = loadImage("Licorice_guizhi.png");
  licorice3 = loadImage("Licorice_xiaoyao.png");
  licorice4 = loadImage("Licorice_xiexin.png");

  midsummer = loadImage("Mid_sum.png");

  milkvetch1 = loadImage("MVR_BZYQ.png");
  milkvetch2 = loadImage("MVR_YPFS.png");

  peony1 = loadImage("PR_guizhi.png");
  peony2 = loadImage("PR_xiaoyao.png");

  scutellaria = loadImage("Scute.png");
  snow = loadImage("snow.png");
  spring = loadImage("spring.png");
  wolf = loadImage("Wolf.png");
  wpg = loadImage("WPG.png");
  //load images of the patients and doctors
  img21 = loadImage("patient_1.png");
  img22 = loadImage("doc_1.png");
  img23 = loadImage("patient_2.png");
  img24 = loadImage("doc_2.png");
  img25 = loadImage("patient_3.png");
  img26 = loadImage("doc_3.png");
  img27 = loadImage("patient_4.png");
  img28 = loadImage("doc_4.png");
  img29 = loadImage("patient_5.png");
  img30 = loadImage("doc_5.png");
  
  //load images of the prescriptions
  img31 = loadImage("Xiao_Y_S.png");
  img32 = loadImage("BZYQ_T.png");
  img33 = loadImage("Cinna_T.png");
  img34 = loadImage("YPFS.png");
  img35 = loadImage("Mid_Sum_T.png");

  //user testing upload
  patient = loadImage("patient.png");
  easymode_guizhi = loadImage("easymode_guizhi.png");
  prompt = loadImage("prompt.png");
  start = loadImage("start_back.jpg");
  sym = loadImage("sym.jpg");

  //load images of the prompts
  promptXY = loadImage("xiaoyao_prompt.png");
  promptBZYQ= loadImage("BZYQ_prompt.png");
  promptGZ= loadImage("guizhi_prompt.png");
  promptYPFS= loadImage("YPFS_prompt.png");
  promptXX= loadImage("xiexin_prompt.png");

  images[0]= angelica1;
  images[1]= atractylodes1;
  images[2]= bitterness;
  images[3] = cinnamon;
  images[4]= dew;
  images[5]= dried_orange_peel;
  images[6]= firewood1;
  images[7]= ginger;
  images[8]= ginseng1;
  images[9]= jujube;
  images[10]= jute;
  images[11]= licorice1;
  images[12]= midsummer;
  images[13]= milkvetch1;
  images[14]= peony1;
  images[15]= scutellaria;
  images[16]= snow;
  images[17]= spring;
  images[18]= wolf;
  images[19]= wpg;

  images_patient[0] = img21;
  images_patient[1] = img23;
  images_patient[2] = img25;
  images_patient[3] = img27;
  images_patient[4] = img29;

  images_doc[0] = img22;
  images_doc[1] = img24;
  images_doc[2] = img26;
  images_doc[3] = img28;
  images_doc[4] = img30;

  images_pre[0] = img31;
  images_pre[1] = img32;
  images_pre[2] = img33;
  images_pre[3] = img34;
  images_pre[4] = img35;

  images_prompt[0]=promptXY;
  images_prompt[1]=promptBZYQ;
  images_prompt[2]=promptGZ;
  images_prompt[3]=promptYPFS;
  images_prompt[4]=promptXX;

  m_pre= int(random(0, 5));
}

void draw() {
  background(start);

  getSerialData();

  //start page
  if (mode == 0) {
    drawButton(0.5*width, 0.5*height, 0.2*width, 0.15*height, 1, "START!!!");
  }

  //page changer
  if (mode == 1) {
    drawDescription();
  } else if (mode == 2) {
    drawSymptom();
  } else if (mode == 3) {
    drawEasymode();
  } else if (mode == 4) {
    drawHardmode();
  } else if (mode==5) {
    drawSuccess();
  }
}

void drawDescription() {
  myFont = createFont("隶书", 60);
  textFont(myFont);
  String textDisplay = "You are a CTM(Chinese Traditional Medicine) doctor \n who is working to prescribe medicine to your patients. \n Based on their symptoms and the corresponding remedy, \n please pick the right herbs for the prescription.\n 你是一个中医药大夫，\n需要根据病人的症状或者所给出的药方配出一副药";
  text(textDisplay, 0.5*width, 0.5*height);
  drawButton(0.5*width, 0.7*height, 0.2*width, 0.1*height, 2, "Continue");
}

void drawSymptom() {
  image(patient, 0.5*width, 0.5*height, 620, 570);

  //pictures of patients and docs
  image(images_patient[m_pre], 0.2*width, 0.35*height, 1181/1.2, 944/1.2);
  image(images_doc[m_pre], 0.8*width, 0.5*height, 1181/1.2, 944/1.2);

  //buttons easy and hard
  drawButton(0.4*width, 0.85*height, 0.15*width, 0.1*height, 3, "EASY");
  drawButton(0.6*width, 0.85*height, 0.15*width, 0.1*height, 4, "HARD");
}


void drawEasymode() {
  //prescription
  image(images_pre[m_pre], 0.8*width, 0.5*height, 1417/2, 2125/2);

  //pop up
  for (int i = 0; i<20; i++) {
    if (arduino_values[i] ==1 ) {
      if (m_pre == 0) {
        images[0] = angelica2;
        images[1] =atractylodes2;
        images[6] =firewood2;
        images[11] =licorice3;
        images[14] =peony2;
      } else if (m_pre == 3) {
        images[1]= atractylodes3;
        images[13]=milkvetch2;
      } else if (m_pre == 4) {
        images[8]= ginseng2;
        images[11]= licorice4;
      } else if (m_pre==2) {
        images[11]= licorice2;
      }
      image(images[i], imgX[i], imgY[i], 1754/2.4, 1240/2.4 );

  //drag
      if (mouseX>imgX[i]-0.05*width && mouseX<imgX[i]+0.05*width && mouseY<imgY[i]+0.05*height&& mouseY>imgY[i]-0.05*height) {
        if (mousePressed) {
          imgX[i] = mouseX;
          imgY[i] = mouseY;
        }
      }
    }
  }
  textSize(40);
  text("You can drag the pictures anywhere you want!", width/2, 24*height/25);
}


void drawHardmode() {
  //prompt
  image(images_prompt[m_pre], 0.8*width, 0.5*height, 1417/1.5, 2125/1.5);

  //pop up
  for (int i = 0; i<20; i++) {
    if (arduino_values[i] ==1) {
      if (m_pre == 0) {
        images[0] = angelica2;
        images[1] =atractylodes2;
        images[6] =firewood2;
        images[11] =licorice3;
        images[14] =peony2;
      } else if (m_pre == 3) {
        images[1]= atractylodes3;
        images[13]=milkvetch2;
      } else if (m_pre == 4) {
        images[8]= ginseng2;
        images[11]= licorice4;
      } else if (m_pre==2) {
        images[11]= licorice2;
      }
      image(images[i], imgX[i], imgY[i], 1754/2.4, 1240/2.4 );

  //drag
      if (mouseX>imgX[i]-0.05*width && mouseX<imgX[i]+0.05*width && mouseY<imgY[i]+0.05*height&& mouseY>imgY[i]-0.05*height) {
        if (mousePressed) {
          imgX[i] = mouseX;
          imgY[i] = mouseY;
        }
      }
    }
  }

  textSize(40);
  text("You can drag the pictures anywhere you want!", width/2, 24*height/25);
}



void getSerialData() {
  while (serialPort.available() > 0) {
    String in = serialPort.readStringUntil( 10 );  // 10 = '\n'  Linefeed in ASCII
    if (in != null) {
      print("From Arduino: " + in);
      String[] serialInArray = split(trim(in), ",");
      if (serialInArray.length == NUM_OF_VALUES_FROM_ARDUINO) {
        for (int i=0; i<serialInArray.length; i++) {
          arduino_values[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

void drawButton(float xPosition, float yPosition, float buttonWidth, float bottonHeight, int mode_num, String text) {
  if (mouseX>= xPosition - 0.5*buttonWidth && mouseX<= xPosition+ 0.5*buttonWidth && mouseY>=yPosition-0.5*bottonHeight && mouseY<= yPosition+0.5*bottonHeight) {
    strokeWeight(5);
    fill(#CEAC66);
    rect(xPosition, yPosition, buttonWidth, bottonHeight);
    textSize(100);
    fill(0);
    text(text, xPosition, yPosition);
    if (mousePressed) {
      mode = mode_num;
    }
  } else {
    strokeWeight(5);
    fill(#CEAC66);
    rect(xPosition, yPosition, 2*buttonWidth/3, 2*bottonHeight/3, 50);
    textSize(80);
    fill(0);
    text(text, xPosition, yPosition);
  }
} 

Processing ima show version

import processing.video.*;
import processing.serial.*;
Serial serialPort;
int NUM_OF_VALUES_FROM_ARDUINO = 20;
int arduino_values[] = new int[NUM_OF_VALUES_FROM_ARDUINO];

int NUM_OF_VIDEO = 20;
Movie[] video = new Movie[NUM_OF_VIDEO];

int NUM_OF_IMG = 20;
PImage[] images = new PImage[NUM_OF_IMG];

Movie myMovieAngelica;
Movie myMovieAtract;
Movie myMovieBitterness;
Movie myMovieCinnamon;
Movie myMovieDew;
Movie myMovieDOP;
Movie myMovieFirewood;
Movie myMovieGinger;
Movie myMovieGinseng;
Movie myMovieJujube;
Movie myMovieJute;
Movie myMovieLicorice;
Movie myMovieMidS;
Movie myMovieMVR;
Movie myMoviePR;
Movie myMovieScutellaria;
Movie myMovieSnow;
Movie myMovieSpring;
Movie myMovieWolf;
Movie myMovieWPG;

PImage angelica1;
PImage atractylodes1;
PImage bitterness;
PImage cinnamon;
PImage dew;
PImage dried_orange_peel;
PImage firewood1;
PImage ginger;
PImage ginseng1;
PImage jujube;
PImage jute;
PImage licorice1;
PImage midsummer;
PImage milkvetch1;
PImage peony1;
PImage scutellaria;
PImage snow;
PImage spring;
PImage wolf;
PImage wpg;

PImage start;

void setup() {
 fullScreen();
 // size(2560, 1440, P2D);
  myMovieAngelica = new Movie(this, “Angelicafinal.mp4”);
  myMovieAtract = new Movie(this, “Atractylodes1.mov”);
  myMovieBitterness = new Movie(this, “Bitternessfinal.mp4”);
  myMovieCinnamon = new Movie(this, “Cinnamonfinal.mp4”);
  myMovieDew = new Movie(this, “dew.mp4”);
  myMovieDOP = new Movie(this, “DOP1.mov”);
  myMovieFirewood = new Movie(this, “Firewoodfinal.mp4”);
  myMovieGinger = new Movie(this, “Gingerfinal.mp4”);
  myMovieGinseng = new Movie(this, “Ginsengfinal.mp4”);
  myMovieJujube = new Movie(this, “Jujubefinal.mp4”);
  myMovieJute = new Movie(this, “Jutefinal.mp4”);
  myMovieLicorice = new Movie(this, “Licoricefinal.mp4”);
  myMovieMidS = new Movie(this, “MidS1.mov”);
  myMovieMVR = new Movie(this, “MVRfinal.mp4”);
  myMoviePR = new Movie(this, “PeonyRootfinal.mp4”);
  myMovieScutellaria = new Movie(this, “Scutellaria1.mov”);
  myMovieSnow = new Movie(this, “snow.mp4”);
  myMovieSpring = new Movie(this, “spring.mp4”);
  myMovieWolf = new Movie(this, “Wolf1.mov”);
  myMovieWPG = new Movie(this, “WPG1.mov”);

  video[0] = myMovieAngelica;
  video[1] = myMovieAtract;
  video[2] = myMovieBitterness;
  video[3] = myMovieCinnamon;
  video[4] = myMovieDew;
  video[5] = myMovieDOP;
  video[6] = myMovieFirewood;
  video[7] = myMovieGinger;
  video[8] = myMovieGinseng;
  video[9] = myMovieJujube;
  video[10] = myMovieJute;
  video[11] = myMovieLicorice;
  video[12] = myMovieMidS;
  video[13] = myMovieMVR;
  video[14] = myMoviePR;
  video[15] = myMovieScutellaria;
  video[16] = myMovieSnow;
  video[17] = myMovieSpring;
  video[18] = myMovieWolf;
  video[19] = myMovieWPG;

  angelica1 = loadImage(“Angelica_BZYQ.png”);
  atractylodes1 = loadImage(“Atrac_BZYQ.png”);
  bitterness = loadImage(“bitter.png”);
  cinnamon = loadImage(“cinnamon.png”);
  dew = loadImage(“dew.png”);
  dried_orange_peel = loadImage(“DOP.png”);
  firewood1 = loadImage(“FW_BZYQ.png”);
  ginger = loadImage(“Ginger.png”);
  ginseng1 = loadImage(“Ginseng_BZYQ.png”);
  jujube = loadImage(“jujube.png”);
  jute = loadImage(“Jute.png”);
  licorice1 = loadImage(“Licorice_BZYQ.png”);
  midsummer = loadImage(“Mid_sum.png”);
  milkvetch1 = loadImage(“MVR_BZYQ.png”);
  peony1 = loadImage(“PR_guizhi.png”);
  scutellaria = loadImage(“Scute.png”);
  snow = loadImage(“snow.png”);
  spring = loadImage(“spring.png”);
  wolf = loadImage(“Wolf.png”);
  wpg = loadImage(“WPG.png”);

 start = loadImage(“start_back.jpg”);
 
  images[0]= angelica1;
  images[1]= atractylodes1;
  images[2]= bitterness;
  images[3] = cinnamon;
  images[4]= dew;
  images[5]= dried_orange_peel;
  images[6]= firewood1;
  images[7]= ginger;
  images[8]= ginseng1;
  images[9]= jujube;
  images[10]= jute;
  images[11]= licorice1;
  images[12]= midsummer;
  images[13]= milkvetch1;
  images[14]= peony1;
  images[15]= scutellaria;
  images[16]= snow;
  images[17]= spring;
  images[18]= wolf;
  images[19]= wpg;
  
  imageMode(CENTER);

  printArray(Serial.list());
  serialPort = new Serial(this, “COM5”, 9600);
}

void draw() {
  background(start);

  getSerialData();

  for (int i = 0; i<20; i++) {
    if (arduino_values[i] ==1 ) {
      if (video[i].available()) {
      video[i].read();
    }
      video[i].play();
      image(video[i], width/2, height/2);
      image(images[i], 1754/4.8, 1240/4.8, 1754/2.4, 1240/2.4 );
    }
    else {
      video[i].pause();
    }
  }
}

void getSerialData() {
  while (serialPort.available() > 0) {
    String in = serialPort.readStringUntil( 10 ); // 10 = ‘\n’ Linefeed in ASCII
    if (in != null) {
      print(“From Arduino: “ + in);
      String[] serialInArray = split(trim(in), “,”);
      if (serialInArray.length == NUM_OF_VALUES_FROM_ARDUINO) {
        for (int i=0; i<serialInArray.length; i++) {
          arduino_values[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

CONTEXT AND SIGNIFICANCE

Before the midterm, we made a mini project, a sleeve sword, which awakened my interest in organ-based facilities, and inspired my midterm IDEA. An example I gave in analysis 1 is Alias, a smart product by Bjorn Karmann and Tore Knudsen that adapts appliances based on user feedback. It works only if the user gives it a command, and only if the user gives it feedback so that it can interact with it. The user needs to think when giving commands, so this example inspired me: if I want to enhance interactivity, I need to enhance the user’s thinking so that they can actively join the process. And I think that setting up multiple organs is good for this purpose. In addition, as an escape room player, I enjoy the process of deciphering and the joy of solving the mystery, and I especially like the interlocking of the various links, which allows me to think more coherently, so we finally decided to change from Mini Theater to a combination of Mini Theater and Escape Room. In my perception, most of the audience in a theater is sitting in the audience offstage, and there is no strong interaction with the story, I wanted to improve that with the escape room format.

CONCEPTION AND DESIGN

As I mentioned earlier, I believe that to achieve interactivity, participants need to engage in critical thinking and take action in response to the design, not just through facial expressions but also through physical engagement. I believe that incorporating puzzles and mechanisms within an escape room setting can effectively achieve this goal. Participants can physically explore the mechanisms, decrypt clues, and receive feedback through audio recordings. To make the escape room experience more immersive and logical, I decided to incorporate a story background.

In the first escape room, we used several components, including a button, light sensor, electromagnet, magnet switch, and servo motors. In our design, when the participant opens the bed board, the light sensor detects the light to satisfy the if-condition. Initially, we envisioned having the painting fall down, and before learning about the electromagnet, we thought of using a servo motor to pull a line and bring the painting down. However, we found this approach challenging to implement and realized that the servo motor could be easily noticed. Therefore, we decided to use an electromagnet instead, as it would provide a more natural effect.

We also wanted to open the first escape room from both sides. However, during our repeated testing, we found that the servo motor inside the box was not strong enough to move the entire room. To address this, we obtained two larger 270-degree servo motors. We created space within the paper wall and placed the servo motors inside. After testing, we were able to achieve our desired effect.

In the second room, we required the participants to classify and arrange objects. To facilitate concealment, we decided to use a pressure sensor. The sensing area of this sensor is just a transparent thin film, making it easier to hide. Initially, we planned to use three sensors, but we found that if they were covered with tape, their sensitivity decreased. As a result, only the weight of the mini laptop could be detected by the pressure sensor. The small size of the laptop’s placement made it convenient to restrict the arrangement of objects, ensuring the pressure sensor’s detection range was more precise.

In the final escape room, we placed a large mirror on the wall. This was done to provide a more direct and impactful visual experience for the participants compared to using a printed electronic display screen, which would not provide real-time changes. Additionally, the mirror symbolizes the display screen that monitors the participants.

FABRICATION AND PRODUCTION

Actually, I personally believe that in the overall team collaboration, my role was that of an idea provider and working together with Arial to create the entire project.

We conducted several user tests, and the most common feedback we received was that our story wasn’t clear enough, and it didn’t provide a compelling reason for them to follow the steps. Initially, our plan was to integrate each step into a diary written at different times, so they would follow the chronological order. However, the participants provided feedback that some people might not understand the hints in the diary or might not have the patience to read its contents carefully. As a result, we changed our approach and used step-by-step voice prompts, which allowed them to understand each step without forgetting what they needed to do. Additionally, they also mentioned that the connections between our puzzle rooms were not tightly linked, which led them to unintentionally skip steps. The step-by-step prompts effectively addressed this issue because they didn’t know what to do until they received the next prompt.

 We encountered a few other problems as well. Initially, we attempted to connect a reed switch and a servo in series, so that when they threw the book into the fireplace, the switch would close, activating the servo. However, during the process, we discovered that the servo would rotate even without closing the switch. Later, LA pointed out our mistake – we shouldn’t have connected the switch in series with the servo because it required reading data from a pin to control the servo.

Furthermore, we had issues with our rolling shutter door. Initially, we followed a picture we found online, where one end of the paper was fixed to a wire, and rotating the wire would lift the paper up. However, we found it difficult to stick the paper to the circular cross-section of the wire. So, we decided to roll up one end of the paper, stick it 5cm away, create a channel, and then insert the wire into it. We also discovered that the cardboard paper we initially used had too much friction, causing it to get stuck in the groove. Therefore, we switched to using smooth white paper. However, we noticed that it would deviate from its intended position during the reset. To address this, I suggested installing a guide board to allow the paper of the rolling shutter door to slide back to its original position. We wanted to use a stepper motor to control it, but even with hot glue, it was challenging to connect them. So, we decided to “fake it to make it.” We manually rotated the wire after the participants completed the task in the second room.

The same solution was applied to the issue with the buzzer. We had initially allocated a day to make the speaker play audio, but the professor reminded us that it would require a significant amount of time, which we didn’t have. Following his suggestion, we ultimately chose to manually play the recorded sound.

I believe the most successful part of the project was actually the first room splitting into two halves and rotating to the sides. When we realized the blue servo wasn’t sufficient, we immediately replaced it with a larger servo. We also created space at the corner to solve the servo’s positioning issue. To make the traces less noticeable, we covered the cracks on the floor with a carpet, so the participants wouldn’t notice them at the beginning.

Working with Arial throughout the project was very enjoyable because we usually worked together, brainstorming solutions when problems arose, and ensuring we didn’t miss out on the project’s progress. Arial was proficient in coding, so she generally handled the coding issues, while I focused on refining the overall project’s logic and storytelling, as well as providing new ideas when solving equipment operation problems.

CONCLUSIONS

In general, my goal was to engage participants in thinking and responding, making them firsthand participants in the story rather than observers. Based on my observations, I believe I achieved that during the presentation. They read the diary, listened attentively to the recordings, understood the story’s development, and made decisions based on the audio prompts for their next actions. This aligns with my definition of interactivity. However, what didn’t align was that in my definition, I mentioned the use of multi-sensory stimulation, but in this project, I only incorporated visual and auditory elements. In the future, I would consider more diverse combinations, including olfactory and tactile experiences, to enhance realism. Given more time, I would also expand the project and make the rooms more detailed, such as adding furniture. This would make the scenes more realistic, enhance immersion, and accommodate more interesting mechanisms to increase playability.

Throughout the project, my biggest takeaway was the concept of “fake it to make it.” This approach allowed me to save time by not getting stuck on problems that couldn’t be solved at my current skill level, while still achieving my goals flexibly. After all, the ultimate aim of the project is to create interactivity rather than merely showcasing technical prowess. Otherwise, I could have easily lost focus by fixating too much on technical issues and forgetting how to enhance engagement. I also realized the importance of effective team communication. I believe Arial and I played significant roles in problem-solving through our communication, as different individuals offer different solutions. Trying out multiple ideas increases the chances of successful problem-solving, and actively listening to each other’s perspectives is highly valuable. Lastly, inviting different people for user testing was crucial. Since both Arial and I are avid escape room enthusiasts, our thinking naturally gravitates toward puzzle-solving when we assess room designs. As project designers, we may subconsciously assume that others can easily understand our designs. However, for individuals who are not familiar with escape rooms, their thought processes may differ. Therefore, we invited friends who are not escape room enthusiasts to test our project and provide their perspectives.

Finally, I would like to express my gratitude. Firstly, I want to thank the Professor for providing us with valuable suggestions and inspiring ideas for improving our project. I am also grateful to LA and the other Fellows for their assistance in identifying project issues and helping us acquire various materials. A special thank you goes to my partner, Arial. Her resourcefulness and craftsmanship played a significant role in shaping the entire project, allowing us to complete it smoothly and efficiently. I would also like to thank my friends Han, Isabel, and Benjamin for their encouragement and support throughout the process. Their presence and motivation were truly invaluable to me. Thank you all for being part of this journey and contributing to the success of the project.

DISASSEMBLY

When we returned the items, we did not keep a record. However, I can provide a list of the items I returned: 2 large servos, 1 small servo, 1 reed switch, 1 electromagnet, 2 power supplies, and 1 diode.

APPENDIX

The inspiration for my design comes from “The Truman Show.” Before experiencing this project, participants will be informed that the protagonist of the story has lived their entire life inside this room, but now they have discovered something is amiss, and they need help to escape. As participants solve puzzles and uncover hidden cameras in the room, audio recordings will play to provide background information on the story. When they follow the voice prompts and open the closet door, pressing the button inside, the first escape room is successfully unlocked, and they proceed to the second scene.

The second scene is a photography studio, symbolizing the constant surveillance the protagonist has been under. Inside the studio, there is a diary left behind by a kind-hearted employee who states that if the protagonist takes over their tasks, they will help the protagonist escape. Participants can deduce the proper arrangement of items based on clues on the wall. To increase the game’s difficulty, we haven’t directly indicated where each item should be placed. Instead, we used black tape to highlight three different areas, and participants had to infer the correct placement based on the shape and size of the items. Once they successfully categorize the items, the rolling shutter door will automatically open, leading to the climax and resolution of the story. A recorded message will play, congratulating them on completing the challenge, and revealing that the participants themselves were the ones being observed.

At this point, they enter the third room, a real surveillance control room. Facing the participants is a large mirror on the wall, allowing them to see their faces the moment the rolling shutter door fully ascends. This mirror symbolizes a large screen specifically designed to monitor the participants. Photos of the surveillance control room’s computers are also displayed on the sides of the room, representing surveillance from multiple angles. Through this story, I aim to convey the message that when you think others are being observed, there may be someone observing you from behind. By being indifferent bystanders to the victim, you may end up getting hurt, as you never know who the real target is.

#include 


const int servo2Pin = 6;  
const int servo3Pin = 5;  

const int buttonPin = 2;  


Servo servo2;  
Servo servo3;  


bool buttonPressed = false;  

void setup() {
  Serial.begin(9600);

  pinMode(buttonPin, INPUT_PULLUP);  

  servo2.attach(servo2Pin);  
  servo3.attach(servo3Pin);  
  servo2.write(0);           
  servo3.write(90);          
  delay(1000);
  servo2.detach();
  servo3.detach();
}

void loop() {
  //Serial.println(digitalRead(buttonPin));
  if (digitalRead(buttonPin) == LOW) {

    servo2.attach(servo2Pin);  
    servo3.attach(servo3Pin);  
    servo2.write(90);          
    servo3.write(00);         
    delay(1000);
    servo2.detach();
    servo3.detach();
  }


  

  delay(100);  
}
 

#include 

const int lightSensorPin = A0; 
const int electromagnetPin = 9；
const int pressurePin = A1;
const int servo1Pin = 5; 
const int reedPin = 13;

Servo servo1;

// bool electromagnetClosed = false;  
// bool buttonPressed = false;        

void setup() {
Serial.begin(9600);
pinMode(electromagnetPin, OUTPUT); 
pinMode(reedPin, INPUT);
servo1.attach(servo1Pin); 
}

void loop() {
int lightSensorValue = analogRead(lightSensorPin)

digitalWrite(electromagnetPin, HIGH);
if (lightSensorValue > 50) {

digitalWrite(electromagnetPin, LOW);
}

int reedValue = digitalRead(reedPin);
Serial.println(reedValue);
if (reedValue == 1){
servo1.write(70);
}


Serial.print("Light Sensor Value: ");
Serial.println(lightSensorValue);

delay(100);

int pressureValue = analogRead(pressurePin);
 Serial.print("Pressure: ");
 Serial.println(pressureValue);
// if (pressureValue > 0){

// }
}