Author: Keyin Wu

This week, I chose to explore the Deep Dream because I think its visual effect is mysterious and cool. Firstly, I used the default parameters and tries it on my painting and a photo took before. The results turned out in this way:

• ‘mixed2’: 0.2,
• ‘mixed3’: 0.5,
• ‘mixed4’: 2.,
• ‘mixed5’: 1.5,

• step = 0.01         # Gradient ascent step size
• num_octave = 3      # Number of scales at which to run gradient ascent
• octave_scale = 1.4  # Size ratio between scales
• iterations = 20     # Number of ascent steps per scale
• max_loss = 10.

Then I changed the parameters and wanted them to be more sharp. The former one is the result in the first case and the second one is the result in this case.

• ‘mixed2’: 2,
• ‘mixed3’: 0.5,
• ‘mixed4’: 2.,
• ‘mixed5’: 1.5,

• step = 0.05         # Gradient ascent step size
• num_octave = 3      # Number of scales at which to run gradient ascent
• octave_scale = 1.4  # Size ratio between scales
• iterations = 20     # Number of ascent steps per scale
• max_loss = 5.

I notice that when mixed2 gets dominant and the step value goes up, the picture will have more lines and patterns, which I think is a good addition to picture of landscapes or scenery. Actually I also tried Deep Dream on some people’s face, but it turned out that the picture would get messy and the face’s outline would be stretched. I personally prefer its effect on landscapes so I remained these pictures. However, I feel like the use of Deep Dream is very random and sometimes to get better visual effect, what we can do is only keep trying. But without doubt it is a cool model which I enjoy a lot!

For this week’s assignment, we are asked to train the style transfer by ourselves and do some inference.

I chose a picture with colorful geometries so I named it as “colorfulgeo”.

Training

As for the training part, I simply followed the instructions. The obstacle I met is about the setup step because I originally operated in the login node. And there were also some problems with unzip which caused results like these.

Inference

After figuring out those problems, I successfully ran the task and finally got the training result. Firstly, I tried the style transfer in webcam and got screenshots of myself like these:

Later, I replaced the video input with some images and got these results:

I think when the input has a certain main object, it works better than when there is only some similar colors or patterns. Maybe it is also because my original image is not so large and has less information. However, at least I have a basic understanding of how style transfer works and there still remain a lot of aspects that we can explore to improve the visual effect at last.

THE WEIGHT OF STROKE

Introduction

Basically, it is a drawing game where two players can draw by round and see whose drawing has higher “value”. In this case, I would prefer to refer to “value” as the complexity of the drawing, considering different people have various perspectives of the value of an artwork. And here I just pick the world-famous paintings as the highest value class. With machine learning technique, the drawings can be classified by machine based on the data set we provide in advance, so I think it is also an interesting try in terms of exploring how machine treats human’s artwork.

Process

1. KNN Classifier with Feature Extractor

Technically, I mostly made use of KNN Classifier with Feature Extractor to define my own classes. Originally, KNN Classifier with Feature Extractor connects with the webcam, so I broke its connection and rebuilt it with the canvas by createGraphics.

Here are my tests about how KNN works with the canvas input. I define three classes based on the number of the circles on the canvas, where the first class has few circles, the second one has some and the third one has a lot. The result shows that around ten to fifteen examples are enough for KNN to work. Besides, in the first three screenshots, examples I added to each class are partly repeated, and I found the confidence value would be easily stick to 100%. Therefore, I later made examples in one class also different from each other, and I then got the result of 66.666% as the fourth screenshot shows.

Then I replace three original labels with new examples drawn by myself.

• First Label – Childish Painting

• Second Label – Sketch

• Third Label – Famous Painting

2. Front End by P5

Firstly, I created a balance that moves like this with sin value and vectors.

Then I created user interface with two parts. One part is the balance with the mass showing on its two sides. The other part is the drawing part where the two players can draw on the canvas by round.

Actually, I spent quite a lot time on this part, because it involves many details like different buttons and needs a lot of coding. And as the player is done with his drawing and clicks “Done”, there will be colorful balls falling down to the balance on their side. The ball’s size and mass are based on what the player drew on the canvas. It is because each label is given a range of value, where the first childish painting label has the lowest values and the third famous paintings label has the highest values. Then the confidence of each label will multiply the according value of label and generate the final result about the ball’s size.

How it Works

Here is an accelerated video demo.

When the mass on one side is 100 greater than the other or there have been 10 rounds, the game ends.

Further Development

• Collect more data painted by different people and different styles;
• Enrich the way of drawing;
• Improve the user interface;