Style Filter

Style Filter is an interactive installation powered by Arduino, p5.js and ml5.js. It incorporates a physical interface for users to apply neural style transfer filters to images, in a way that one would apply a square filter on a camera lens into a holder. As multiple filter holders are provided, users are free to explore applying filters on top of each other in various orders.

Model Training

The model involved is fast neural style transfer model only. My training process, as to be implemented with ml5, follows this guide from ml5.js documentation. All models were trained on a linux workstation with a single GTX1080 with 200 epochs, taking about 8 hours each.

Initial ideas and changes

In the previous proposal, the idea was to transfer a selfie photo to the style of the an existing self-portrait piece with the most similar facial expression, using PoseNet, KNN Classifier and Style Transfer. It is also an educational interaction that provides reference of art pieces, targeted at art classes in high school. However, after some testing on the neural style transfer model, the aesthetic result turned out far from expected. It seems that the model had a hard time relating the facial features in the self-portrait to an actual face in reality.

After showing the prototype results to an experienced Chinese high school art teacher, Hu Lihua from Jianping High School, she commented that it is hard to tell the strokes, textures and styles from these transferred selfies, which makes less sense as an educational tool. Therefore, I discontinued this idea and decided to focus on the little spotlight from my midterm project – stacked style transfer. The models used in the final project are the same with the midterm project, so check them out in the previous documentation if you want.

Concept

Since midterm, I have tried to highlight the possibility of applying multiple style transfer in various orders. However, other than technical difficulties, an intuitive interface is the key to the whole interaction. The original idea was to apply a node-based interface on web to implement freedom in connections.

The problem was, although it allows precise expression on the orders and clear representation of the style images, the node-based interface was found rather oriented to professional users than average users.

The final answer is a physical interface, inspired by square filter system from photography. I believe a filter holder system for image style transfer filters would intuitive for users to apply and a fun experience.

Implementation

Physical Computation

The ideal physical interface comes in two parts – filters and filter holders. The holders should be able to identify different filters in some way. My approach is using resistors of different ohms as the identifier.

The bottom of the filter holders are wrapped with conductive tapes, and a resister is attached at the bottom of each filter. The fabrication technique is laser cutting on transparent acrylic boards to imitate filters. The holders are put together with stainless steel bars with m-8 screw threads, screws and nuts.

The detection mechanism is implemented by making Arduino am ohm meter. The resistor on different filters would divide the voltage differently so that Arduino can recognize it from an analog read port.

Web development

The Arduino talks to the computer with serial communication, which is not quite common for usage on web. The project p5.serialport  provides a reasonably simple way for implementing serial on p5. It requires a node.js server opening to serve the serial data to the frontend, which is nicely done with a GUI in this library.

The frontend is implemented using p5.js and ml5.js. It utilizes the style transfer model in ml5 and a nested callback for transfer iterations. Here is a demo video for a minimum working prototype, which is also the one presented in class. (Notes: #1 stands for Mondrian and #2 stands for Pollock).

Future & Further Implementation

There are two major drawbacks for the current prototype – the freedom of image source to be transferred and the visual icon reminding users which style each filter stands for. The next version will replace the built-in image with a webcam capture interface, so that the user can take pictures and style transfer on the go. Also, the style image will be etched onto the acrylic filter using laser cutter to indicate which one it is.

Additionally, the processing time is rather long and also results in lack of responsiveness during processing, due to the ml5 implementation. Adding a python Flask backend for inferencing models is a great idea, but meanwhile meaning discarding most of the existing codes and models (takes too long to train) and a more complicated structure.