Refrigerator Door
Refrigerator Door is an interactive installation that features movable magnetic letters attached to an acrylic panel. Viewers can move around letters on the board, and when letters come into a certain distance with another, letters are projected onto the panel that complete the word. Attempting to recreate the nostalgia of playing with letters on a refrigerator door as a child while also creating an engaging experience for today’s children, the piece combines a physical interface through the alphabet magnets along with a digital interface through the projected letters.
Ideation
After seeing a video by OpenFrameworks that featured an interactive tabletop installation that augmented the movement of physical objects through changes in a tabletop projection, I wanted my project to be very similar, using objects to dictate the state of the sketch. My initial idea was to create a tabletop installation as well, making a project that relied on light, shapes and sound to create an experience that was similar to the video I saw. I am really interested in using sound and light to create an installation, and thought that using a tabletop and object tracking would be the easiest way to do it.
I thought that I would use a Kinect or webcam to track the location of the objects from above while they would be on the tabletop, along with a projector projecting the sketch from above.
I initially, with the help of Aaron, thought of three ways to technically implement this:
- Color tracking: I thought that this might work provided the colors would be distinct from the hands and possibly clothes of the people interacting with it.
- Shape detection: Might have been the most reliable technique if I used it, but after some digging around I was not able to turn it into a working prototype.
- Marker Detection: This is what I ended up doing. Aaron suggested that I look at the Reactivision library and how it uses marker tracking with an infrared camera to view unique markers’ locations.
The Process:
As stated earlier I wanted to use light and shapes to create this installation. So I started off by finding an addon that allowed me to do this. I found Light2D to be an interesting addon to accomplish this, with its ability to render shadows and different shapes easily. I started off using color tracking and found a working example with the ofxCv addon. I then combined the two, rendering the shapes using the Light2D addon to specific colors found using the color tracking example. This proved to work but was extremely unstable, changing with different light conditions. I think I made some progress with it as seen below, but I think it would not have worked very well because of how unreliable it would be.
Prototype using color tracking and the Light2D addon.
I then tried to use background subtraction with the Kinect in order to filter out the noise that the background might cause for the stability of the sketch. However, I was not able to get this fully working as well. I also experimented with shape detection, and Aaron was able to get a working example of it for me because the old version was extremely buggy and would not run. This also did not work too well with the sketch.
I also ran into issues with the idea behind the project and after talking to Aaron, realized that there was not much substance to it. I was still interested in moving objects and having the sketch update accordingly and the idea that me and Aaron came up with was based on my interest in alphabet magnets on a refrigerator door. The new project would be something geared towards children, perhaps as something educational or entertaining. The letters could be moved around a panel and when two letters would come within a certain proximity, letters to create real words would be projected onto the panel. This required me to change the physical structure of the project to fit the premise of the refrigerator door, so I changed it to a vertical panel with people interacting with it from the front and the projection as well as the camera detecting positions from behind the panel.
This actually made things simpler for me, since I did not have to worry about people interfering with the marker tracking since it would all be done from behind.
Initial tests using the ofxAruco library
I originally wanted it to be on the semi-opaque windows of 153, but was not able to find magnets strong enough for both sides of the panels. So I decided to use a clear acrylic panel and fine sandpaper to create the same effect of a semi-opaque board that could be projected onto from behind.
After Aaron referred me to Reactivision, we were able to find ofxArcuo, an addon that allowed me to use marker tracking. I combined this with the PS3Eye infrared camera, allowing me to make the detection of the markers more stable regardless of lighting conditions, along with two infrared lights to see the markers and use the ofxAruco addon.
Final Stages:
With all this in place, I was pretty sure that the installation would work with the infrared tracking after I tested this, it was just designing it and coding the features of the letters rendering between the physical letters on the panel. With some help from Aaron with vectors to detect distances between markers, I changed it so that it would only detect letters of a certain direction as well. The markers would correspond to certain letters with their ids, and would then loop through the array with all the words to see if the first letter and the last letter are the same for a word. If it is found to be the same, the middle letters will be taken from the entry in the array, and will be rendered at the correct distance between the two letters’ positions. I then created two arrays with four-letter and three-letter words. If two letters were extremely close, the three letter words would be rendered, with just one letter displayed between the two. If they were slightly farther apart, two letters would be rendered to make a four-letter word.
The setup
Magnets attached to markers
To physically create the installation, I used a cart from the IM lab along with two clamps to hold the acrylic panel vertically on the top of the cart and moved this in position outside 153. I placed the projector and camera (which was clamped down using arms) onto a shelf which was positioned behind the panel and cart. I then cut out the markers that I printed out and attached them to cardboard squares using hot glue. On the other side of this I would attach a magnet. I bought cheap foam alphabet pieces from Daiso and also attached magnets on the back of them. The foam magnets would go on the front side where the users would move them, while the cardboard with the corresponding marker and magnet would go on the opposite side.
For the design of it, I was able to find a font (https://www.dafont.com/alpha-fridge-magnets.font) for something similar to alphabet magnets. I also added moving, rotating flowers to show that it is a dynamic sketch. I was worried that people might not be interested in seeing or interacting with the piece if they did not see that there was some movement on the acrylic panel. This also created an experience that might be aimed at children, resembling iPad apps that the children of today might use to learn the alphabet.
Finally, I used the projection mapping example to ensure that the letters detected would be properly corresponding to the letters rendered on the screen. Although it was not perfect, it still allowed me to get the positioning correct which was a huge challenge when I was prototyping it in the final location.
Since the presentation of the projects two Wednesdays ago, I have added a debug mode, slightly changed the color of the letters to be more visible and increased the size of the rendered letters.
Final Reflection:
This was a very challenging project and I felt overwhelmed a lot of the time. However, it was extremely rewarding and I felt that I learned a lot. Even though using Openframeworks is still challenging, I am definitely a lot more comfortable using the addons and all of the functions that Openframeworks has. I do wish that my code was more clean though, and felt that it could have been a lot more efficient. It was a challenge to constantly detect the magnets in real-time, so I felt like I needed to record the previous positions of every marker, and only update this if the marker is detected once again.
The code to the github repo is here: https://github.com/slw515/Refrigerator-Door