Tag: aiarts10

For this week’s assignment, I chose “Taenghwa” as the style input. Taenghwa, paintings of Korean Buddhism that are mostly for indoor display of temples, is characterized by its rich symbols and intense colors. Below is the image I used:

Using Intel DevCloud, The entire training process took roughly 20 hours. Since I tried several times but eventually failed to utilize DevCloud for my midterm project, I could correctly learn this time how to submit my shell file and export trained data. After the training process was completed, it generated a folder with a json file and a number of files that are labelled “variable_#.” I integrated this data-set with Aven’s Inference Style Transfer model. 

Despite running slowly, it stylized my webcam input in real time. We can clearly see how the color scheme got changed into red and green, which are main colors of the Taenghwa. Most lines seem, as expected, somewhat vague and lumpy. And yet, I was excited to spot some characteristics of Taenghwa from the output. So I tried it with other images, using Aven’s another code: Style Transfer for Images.

First image I tried was The Birth of Venus by Sandro Botticelli. Its output seems rather surreal than giving the impression of Taenghwa. Another remarkable thing I found is that most of the stylized images have glitched parts. As you can see from the image above, the clam shell and Venus’ leg are partially glitched like rgb subpixels of an analog TV screen. They can also be found on the image in which I have my hands together; the curtain behind me also has a glitchy part. I’m wondering what caused this effect because the original Taenghwa input doesn’t have this sort of characteristics. 

The other image I tried was the 12th century mosaic of Christ Pantocrator. Since it has clear distinctions between the figure and the background, the generated image seems clearer than the past results. And this image definitely displays more characteristics of Taenghwa, considering its composition and solemn atmosphere.

Overall, the whole process went smoothly and I could learn a lot from training the algorithm with the data I chose. I’m excited to learn more about the GANs in upcoming weeks, so I can realize the Hanafuda GAN for my final project.

Model training

I choose this painting from Picasso as the target style and use cd image_path and qsub image_name colfax to upload this image to my devcloud. 

But when I submitted my training task, I couldn’t see the expected result. Professor checked and found that I didn’t have the local train.sh. So I created one and upload it to the devcloud. This time it worked and then I downloaded the model by using scp -r colfax:train_style_transfer_devCloud/models 111(my name). At first, I did this on the devcloud platform, so I failed to download the model. Then I realized that I should download the model on the local terminal and got my model successfully.

Application

I changed the original code to apply my own model and got this:

This image is abstract and is quite different from my original image, but it looks funny. Also, the canvas isn’t smooth when the style is changed. I think  the reason might be the huge calculation like my midterm project. I thought this was simple because I could only change the style by pressing the space. Thus, I added poseNet model to use the pose to replace space. When the coordinates of the user’s nose reached a certain range, the style will change. However, every time I changed my position, the style would change from the original one to Picasso style. Therefore, it looked so weird and it blinked. So I had to give up this method and still applied the keyPressed function. 

But it inspired me a lot for my final project. Maybe I can use the pitch or tone of sound to trigger the change of the style so that it can enrich the visual output. For example, if the tone is high enough, the style will transfer to another one. However, the precondition is to solve the problem of stuck live video.

For this week’s assignment, I trained a style transfer model with the painting below.

The biggest difficulty I met when training this model is that since the internet is extremely unstable, I failed again and again when downloading the model for training. I tried at least 10 times and finally manage to download that at 2 am in the morning…Other than that the procedures are smooth and I finally have a general understanding of what those commands do with the help of Aven. The output for the trained model is as follows.

（The style transfer can be triggered by a loud sound)

It’s only after I saw the output did I realize that I probably chose the wrong picture to train. Since the image is black and white, so is the output, which makes it hard for me to identify the similar pattern. Originally I want the output image to have the similar drawing line pattern as the input. However, I think  such detailed imitation requires more training. I should have chosen an image with an obvious color pattern that is easier to observe in output image…Still, I guess the pattern of black, white and gray lines shown in the output is somewhat noticeable, even though it’s not as  obvious as I hope.

Overall, it’s a very interesting experiment. I think it helps me a lot to understand how the style transfer process works and allows me to get my hand on to train a model. I also tried using different signals to trigger the style of the web to change, such as using p5’s pitch detection. The style of the web cam image will change when the mic reaches certain volumn. I also hope I can apply this process of training style transfer model in my final project. The style transfer model can be used to generate different characters or battle scene with same style and theme for my battle game.