Category: aiarts

Overview

This week, I trained a ml5js style transfer model with devCloud services. I chose the engraving artwork “Thangka” from artist Wenna as the target style for this model.

Continue reading “Training Style Transfer with ml5js | aiarts.week10”

For this week’s assignment, I trained a style transfer model using a cyberpunk style image as my base. I personally fell in love with this genre thanks to movies such as Tron, Beyond the Aquila Rift and others. I wanted to create a model that would translate normal landscapes into something out of one of these movies.

Beginning the project, I didn’t realize that it would take me such a long time to complete. Being on a Windows computer didn’t help as well. I followed along with the slides provided to us however some of the lines of code would not transfer well in the command terminal of the windows system. Therefore, I resorted to transferring my work onto a mac.

From there, I downloaded the finished model and uploaded into the folder that was provided to us. Running the localhost server, I then proceeded to test the finished work out. 

Here are some of my finished work:

Personally, I’m very satisfied with the results. It manages to change normal everyday landscapes into actual Tron images. However, I think that the result created a very digitized version of the landscapes. Understandably, there is a lot of loss and the accuracy might not be quite high, but it is safe to say that the model was able to perform better than expected.

The effect doesn’t really work with portraits as seen in this example. If given pictures of landscapes it works very well. I think that this model has potential and if given enough time to develop, it can potentially create even better examples of said images. 

Overall, I think that I was able to recreate the cyberpunk aesthetic with the model. It went really well however there is a lot of room to improve. A larger sample size would give me a better result and translate the landscapes into a more accurate depiction of cyberpunk/Tron-esque landscapes. This idea can become a reality and is a very feasible idea in my opinion. This is definitely a good idea to look into.  

TRAINING

In this week’s assignment, I am going to train a styleTransfer model for ml5 on devCloud. I select an image from the Insider. Using Skittles to draw the portraits of people. This work of art binds people’s mentality with materials and soul, which creates huge visual and social impact.

In this training assignment, I am going to train a style transfer network using one of the portraits to let it transfer camera rolls to the art expression. So I follow the instructions in the slides and set up the environment on the devCloud:

$ # Create SSH connection with devCloud
$ ssh devCloud

$ # Clone the training code from GitHub
$ git clone https://github.com/aaaven/train_style_transfer_devCloud.git
$ cd train_style_transfer_devCloud

$ # Set up virtual environment and training image
$ conda env create -f environment.yml
$ sed -i 's/u24235/u31325/g' train.sh
$ sed -i 's/styletransferml5/styleTransfer/g' train.sh
$ sed -i 's/zaha_hadid.jpg/train.jpeg/g' train.sh
$ curl -L https://image.insider.com/5cb7888866ae8f04915dccc8?width=2500&format=jpeg&auto=webp -o images/train.jpeg

$ # Start training
$ qsub -l walltime=24:00:00 -k oe train.sh

The instructions above are the essential ones that extracted from the original slides to simplify the set-up process. After submitting the job, we can run qstat to check out the training process. It usually shows like that:

Job ID Name User Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
410873.v-qsvr-1 train.sh u31325 07:16:30 R batch

Indicating the training is still on the go. If the result of the command shows nothing after hours of wait, the training process is done, and the model can be extracted from devCloud:

$ # On my own laptop
$ scp -r devcloud:/home/u31325/train_style_transfer_devCloud/models myModels

And the model is saved to the local path under myModels folder.

TESTING

The next step is to load the model in ml5js and let if transfer the style of images. I followed the instruction here. 

Indicating the training is still on the go. If the result of the command shows nothing after hours of wait, the training process is done, and the model can be extracted from devCloud:

const style = ml5.styleTransfer("myModels/", modelLoaded);
function modelLoaded() {
  style.transfer(document.getElementById("img"), function(err, resultImg) {
    img.src = resultImg.src;
  });
}

The JavaScript code will load models from the previously copied folder and run styleTransfer after the model is completely loaded.

RESULTS

I transferred some pictures and get the following results: