Style Transfer Interface for Diffuse Texture

Intro

The aim of the project is to provide of an interactive interface for 3D model texture style transfer, powered by three.js and ml5.js. Through this project, I want to explore the possibilities of machine learning in the 3D world. The user will be able to upload obj and texture files to try the pre-trained styles on their own models. A style transfer model is one of the most performance consuming ones to train. Through this project, I want to get familiar the training setup and process. 

Demo

A working prototype can accessed here which does not offer model uploading functionality.

Approach

Before the implementation of the interface, there was also model training of the style transfer functionality. I grabbed the photo I am interested in and trained them following the ml5 guide. The training process for each epoch of a style is roughly one hour on my GTX1080. I configured my desktop to run on Ubuntu with proper commands for replicate the guide. (Dealing with GPU driver, CUDA and cuDNN was also a painful process.) In the end, I trained 4 of the styles and used the Fuchun model trained by Yining Shi on the example code. More could be added for sure as a certain selection of theme.

The interface is implemented with three.js and ml5.js on the web platform. The model is loaded through a obj file and the original texure images are loaded as static resources. There were a lot out troubleshooting concerning the WebGL context lost when using three.js and ml5.js at the same time. Some unknown problems still exist occasionally when running on Linux and Windows, not on Mac.

Highlight

The style transfer for texture on 3D model is just an extension of the 2D. However, adding styles on top of each other is a really interesting operation, which is a side product of this project. (Adding several lines of code, why not?)

These are the outcomes with only one style applied.

These are the outcome of stacked styles:

Next Step:

The next step would be a dedicated interface for stacked style transfer. I have been thinking about the proper user experience design for a while. The experience would be great with an AR interface with rectangle image recognition or webcam interaction on web. The history of every step of style transfer will be gathered as a collection presenting to the user.