Project II. Interactive Experience
Eggy and Cube
Description & Concept
This project is based on Eggy and Cube, a story I wrote. I want to introduce the setting of my story and the main character’s personality through interactive storytelling. The project contains 5 scenes and users will follow the instructions and hints to interact with each scene and enter the next scene. Through effectively utilizing Object-Oriented Programming, p5 libraries as well as other software, I developed diverse interactions (through the keyboard, sounds, and mouse) and composed the graphic following a unified aesthetic. I still need to simplify and organize the diverse interactive ways, so that users can follow the narrative more easily.
Demo
Conceptual Development
Inspirations:
I was inspired by These memories won’t last, a story told by dragging one’s mouse. I decided to interactively tell my story through this project. I then researched online for more interactive storytelling ideas and found a scrolling interactive story: The Boat as well as Judgement Interactions: Digital Deadly Sins. I decided to embed sound interactions into my project and form different scenes.
Initial Design:
Originally, I want to develop 3 scenes that will showcase where Eggy lives. Users will first enter the forest where the invisible world lies and interact with the visual elements through clicking their mouse:
When they click the special physalis hidden behind some grass, they will be navigated to the invisible world scene. They will then interact with the physalis through making sounds:
When users click on the door (the circle in the middle of the physalis) of the middle physalis where Eggy lives, they will go inside the Physalis and see Eggy’s living room:
And when they click on the ellipse on the floor, they will enter Eggy’s bedroom.
However, when I started to create the forest scene, I found that constructing 12 classes and arranging them properly in one scene is too overwhelming. I may sacrifice the quality of visuals as well as the interaction design due to the limited time. Therefore, I chose to use only one class (Dandelion developed in week6 recitation) in the forest scene and focus on creating diverse interaction experiences in multiple scenes.
List of techniques and equipment
- Main technique: Object-oriented programming
- Programming: p5.js; p5 sound libraries; ml5 pitch detection model
- Drawing the storyboard and scenes’ background: Procreate
- Music and sound effects: Garageband
Coding
Fabrication
Importing the image:
When I first import my storyboard to p5 Web Editor, I got an error message that was really confusing:
I checked the file name and the variable name but found no mistakes. Finally, I realized that I should write “.JPG” instead of “.jpg” :
After importing the image, I had some trouble with defining the y position. I couldn’t move the image when I defined h in the draw function / IntroScene function:
I found that it was because every time the program runs h will be defined again as “height + 780”. The correct codes:
Using console.log I found the position (the min y = -1360) where the storyboard reaches its end. I then defined the transition as:
In this way, users will be led to the forest scene automatically when they finished reading the storyboard intro. Here’s a problem I met when creating the transition:
I forgot why it occurred, but I managed to solve it in the end.
Animating dandelion seeds’ transparency:
I utilized the dandelions created in week6 recitation and tried to animate the transparency of the seeds:
It took some experiments to make the seeds fade at the right speed:
The animation didn’t work after I rearranged the codes and finalized the scene, which might be caused by using for loop to generate the dandelions. I felt that it was not a necessary effect, so I didn’t animate the transparency in the final version.
Generating random positions of the dandelions using for loop and array:
In the week6 recitation, I manually defined the parameters of the seeds and dandelions:
This was very redundant when I had to create more dandelions, and it was frustrating to scale the dandelions with the scale function:
. 
I found that the x, y positions of the stem and the seeds are the same. So I decided to use two arrays to store the x, y positions of the dandelions and generate the positions through the random function.
But the seeds and stems parted and the circle at the end of each seed disappeared, and there was no error message:
After a long time of debugging, I found that I missed the parameter “dirty”.
Another problem was, the dandelions appeared randomly across the canvas, and Eggy couldn’t appear at the position where the dandelion crowd is the densest. I struggled a lot the associate the Eggy’s position with the density of the dandelions. After researching online, I found p5 had a “randomGaussion” function which corresponds to the knowledge of Gaussian distribution. By experimenting with the standard deviations and means, I found the most satisfying visual effect while making sure that Eggy was hidden well behind the dandelions:
Constructing the physalis:
At first, I drew a physalis and import it PNG file with transparent background. I intended to make it look darker (by adjusting the tint value) when no sound is captured and become lighter when the designated note is heard. The following is the original picture & the tinted version.
. 
However, the picture lost colors when imported to onlineProcessing and looked really weird. Also, I couldn’t adjust the properties of the physalises and duplicate them on the editor, unless I draw other physalises and import them again. So I used ellipse and circles to construct the class physalis instead.
Playing the background music when Eggy is clicked:
At first, I tried to assign each lantern elf a specific note so that when I clicked on one elf, it would sing his note. I tried to achieve this by adding a “checkMouse” method in the class “Eggy”, using this.x and this .y as the first two parameters of the “dist” function. However, I could only trigger the sound when clicked on the left upper corner of the window. I tried to replace “this.x” and “this.y” with “width / 2” and “height / 2”. And it successfully located the clicking region to the elf in the middle (which is very weird because I’ve already translated the elf to the middle position).
I failed to solve this problem. So I tried an alternative: using the keyboard to trigger the individual notes.
To make the physalis’s door glow its associated note is detected, I used an array to store the scale and utilize the ml5 pitch detection model to detect the pitch. Different from the music visualization in my Project One, I only used the notes in C major scale because I created the lantern elf’s song in C major:
Here is a test of the lantern elf’s song and the vocal parts:
Play audio files using mouse interaction:
I imported several pieces of music from YouTube as well as my compositions and used “.play()” & “.loop()” to play the audio file. At first, I just put the play method in the draw function, and the music is started and repeated by every frame which caused a horrible sound. I solved this problem by using mouseIsPressed as a signal of starting the sound:
This works well in the first two scenes where only one mouse click interaction is needed (It might still run into trouble if users happen to press twice or press the button for a long time). But in the last two scenes, users have to press Eggy for a long time to read the message. And Eggy’s sound effect is played every frame during the press, which sounded as if Eggy was screaming. So I used “.isPlaying” to avoid the sound being played too many times:
Background of the living room and bedroom scenes:
At first, I used the sketch I drew when designing the scene and I used a testing ellipse to determine the mouse interaction region:
However, the visual was not satisfying and the image’s width and height didn’t fit into the full window in presentation mode. So I redrew images using different brushes and size:
The full window mode for presentation:
I created this project with preview mode in p5 Web editor, so the size of the window differed greatly from that in the presentation mode:
I had to rearrange the size and positions of visual elements in the scenes to make sure that they fit the window size. Next time I may open the presentation mode before creating the project to save time.
A failed attempt of using OOP to create a recursive tree:
Somehow I was only able to create one branch. Concerned with time, I decided to leave this to future work.
Feedback from Critiques
In the first and second peer review, I got a lot of feedback that I should add some instructions to the scene so that users wouldn’t be confused. I asked Mo if I should create instruction scenes before each scene so that users could fully understand what to do next. She told me that it would be better to just provide the instructions on each scene because creating additional scenes might break the cohesive feeling and users might forget the instructions when they enter the actual scene.
Combining the suggestions from my peers and the concerns that both the interactions and the narrative are complicated, I decided to draw a storyboard intro to better introduce this project at the beginning (and I really love the little poem I wrote hhh). I also added text instructions in each scene. To make it fun, I provide hints to the users instead of telling them what to do directly.
In the final presentation, professor Moon found that the sounds seem to be distorted because of improper use of the p5 sound library. He also suggested me to use more intuitive ways to instruct the users (e.g. using graphic such as arrows to point out the place where users should click or press). He also recommended me to organize the diverse interactions and make them more user-friendly and cohesive (the current choice of keyboard interactions may cause confusion). Linda suggested that I should simplify the way of interaction to make users focus on experiencing the scene and narrative. Ruby suggested that I could make the sound fade out when transiting to the next scene, as the sound stopped abruptly.
Attributes
- Music Credit:
- Forest bgm (Forest Music Instrumental – Raindrop Woods)
- Storyboard bgm (Edward Scissorhands – Ice Dance Music Box)
- Living room bgm (crossing field_LiSA [Music Box])
- Eggy’s Sound Effect
- The lantern elf’s song and vocals in the 3rd scene, as well as the bedroom bgm is generated by me using Garageband
- Font Credit: Indie Flower
Reflection
In this project, I explored diverse interactions such as mouse-clicking, keyboard-pressing, and sound-making. I gained a better understanding of utilizing OOP to create objects and manipulate their properties effectively. I was able to combine imported images drew by myself and visuals created using p5.js to form a coherent aesthetic. I also learned sounds importation as well as p5 libraries such as ml5 (pitch detection model) and sound library. However, the interactions are too diverse and complicated, which makes users confused and stuck in some scenes. I should improve the user experience by designing the interaction ways more carefully.
Future Plan
- Further study the ml5 pitch detection model so that I can figure out why some notes cannot be detected (e.g. the physalis didn’t react to the A and B notes);
- Debug the mouse click region of checkMouse method in the Eggy class to make every lantern elf clickable and react with sounds;
- Some music cannot be triggered (e.g. the bedroom bgm). I will debug this problem and make sure that all music could be played properly;
- The keyboard interactions in the invisible world scene are messy. I will redesign the keys for triggering the sounds so that users may not accidentally press the “b” key and return to the storyboard intro scene;
- The transitions will be improved by a) making the music fade out when transiting and b) providing navigation among all scenes (currently user can only go back to the first scene by pressing “b”);
- Debug the animation of seeds’ transparency;
- Learn how to use OOP to generate recursive trees.