This week we presented our final project, Remake, to the class. Here are some process documentation and challenges we encountered.

p5.js

Our project relies heavily on programming in p5.js, including a color tracker and display images. The most significant progress we had this week was solving the jitteriness of the display by setting layers of thresholds. 

Challenge

However, the biggest challenge was also the color tracking. The reading value of each color varies a lot under different lighting settings, which also became a huge problem in our presentation, for the light was a lot warmer in the classroom than on the floor. Also, the color of each object was too similar to computer vision; for example, it often mixed orange and yellow together. On the other hand, I’m not sure if it’s because we’re overdoing things in p5.js or there are some errors in sketches that are yet to be solved; we could not manage to show every image completely when we let them move on both X and Z-axis. It was a disappointment for me. I feel like the impact of the projection was much weaker without different scales of the elements. Therefore, this would be the first thing I would try my best to solve before the winter show.

Fabrication

Thanks to Sarah’s access to Tandon’s Maker Space, we printed out several 3D blocks for users to interact with. Also, the platform was laser cut from acrylic plates. It was our first time using those machines for both of us, and we’re both happy that the interface turned out as we planned.

User Feedback

We got some interesting feedback regarding the interface during our presentation. First, since there are four states of displays in our project, Olivia suggested changing the input into something more related to the output. So instead of turning the potentiometer, maybe we could use different kinds of pens to determine what state to show – for example, a pencil could indicate the sketch mode. Also, since we only presented with the X-axis movement, Keer mentioned that we could make tracks for the objects, pointing to the users that these objects should only be moved horizontally. 

All in all, I feel like this project has developed a lot since we first got this concept. I enjoyed the process, and working with Sarah was also a delight! We’ll try to fix more things and hope to enjoy this project with the audience during the winter show!

After the playtest last week, we looked into computer vision and decided to use color tracking to monitor the objects’ location. 

Computer Vision: tracking.js

For color tracking, we used the library tracking.js, and by entering the color we want the computer to monitor, it can return the x,y position of the color objects. With the x,y position, we can then send them to control the location of the projected element. 

Also, for the projection, instead of going from low to high fidelity, we decided to make them into 4 different states controlled by a potentiometer, showing 4 styles of the landscape – photorealistic, children drawing,  wireframe, and one more that we’re not yet sure about. 

Mapping to Projection

And then, for my part, I tried to map the x and y positions of the objects to the x-axis and z-axis of the projected elements. There were several challenges – extracting data from the tracking results was harder than I thought, there tended to be delays and glitches, which we would try to solve. Second, in order to move along the z-axis (so that it scales up and down as the user moves the objects forward and back on the platform), we need to use WebGL in p5.js to create a 3D space. However, the tracking works only on a 2D canvas. Therefore, in the end, I tried to write the projected elements on another 3D graphic plane by using the function createGraphic(). There are still lots of details that should be revised and considered, we will work more along the way.

 

Fabrication

We planned to 3D print out our objects for the users and would start testing them with the tracking model. For the objects, we were thinking about abstract, colorful polygon shapes.

Ideally, we will also hide the camera (maybe decorate it as a lamp?) to create a lean look for the whole piece. The platform that the users would be playing on would be a white acrylic plate, cut with a simple user interface, attaching the potentiometer to control the states.

 

For the last Pcomp project of this semester, Sarah and I partnered up again. We decided to make an interactive installation, which will consist of a platform with small objects and a projection of a landscape.

As a participant moves abstract physical objects on a surface, the projection of a  high fidelity landscape is manipulated. E.g. as the participant changes the location of a grey block, the mountains in the landscape that is being projected also move. For our playtest, we use legos to represent our small models and cut some cardboard to mimic the elements that are going to be projected.

What we are yet to figure out is what sensor to use beneath the object platform so that we can monitor the objects’ movements and make the landscape change accordingly. We might also add a potentiometer or another element to let the audience control the fidelity of the landscape. E.g. the low fidelity might look more like wireframes and the high fidelity would be represented by photorealistic landscapes.