Tag: aiarts11

For my deepdream exploration, I created a zoom video that’s a compilation of two deepdream sessions. I created one deepdream with setting mixed4c, then another with the last image picked up from the first video and zoomed in further, also on mixed4c.

deepdream zoom low quality

I had a lot of challenges with this, as at first I didn’t realize I need to change the video file name to save the video (leading to some confusion) and I also had some random errors where img0.jpg wasn’t defined as I didn’t realize that I had to run all the cells together every time for everything to work. I also had trouble with the original photos I tried to use as they were very high quality and I didn’t realize that the session would close down after 30 minutes so the videos kept failing to generate and getting stuck loading at one or more of the cells.

After getting some help from Aven I was able to realize these errors and generate the videos. I plan on using this technique for my final video project, so I’m excited to explore it more!

I also put together the class video for the bigGAN collaboration, available here:

https://www.youtube.com/watch?v=xxrblvC-VOM 

For this final project, we had two ideas in our minds. I will introduce them one by one:

Refine the CartoonGAN project

This idea is a continuation of our midterm project by adding two functionalities: gif converting and foreground/background only converting.

In detail, the first one requires work on unpacking gif into frames and pipe all frames as a batch to the model. Then the model gives the output value and our app will pack those frames into a new gif.  Through this process, we need to put extra attention to avoid running out of memory. 

As for the second idea, I am planning to combine the BodyPix model with our CartoonGAN model. However, to maximize the efficiency of the computation.  I will apply some WebGL tricks with the BodyPix and CartoonGAN output. So that we can deliver the best user experience.

Generative ART with Sentiment Analysis on both audio and text

The idea is that the text could contain certain emotions so does the audio/speech. What’s more interesting is that even the same content, the text, and the audio could have different emotions. We can use this kind of conflictions to make generative arts.

However, there are some problems remaining here.  First, there is no real-time sentiment analysis during these years. We can find an alternative that uses face expression.

The other thing is how to visualize or say generate the emotions that will attract the audience?  This seems to be a harder and more critical question than the previous one.

The assignment for this week is to play around DeepDream, a GAN network to transfer the style (or pattern) of an image, for example, this one:

WORK

There are 5 parameters that are customizable in the step of generation in total. These are:

octave_n = 2
octave_scale = 1.4
iter_n = 15
strength = 688
layer = "mixed4a"

octave_n

– Test Range: [1, 2, 3, 4, 5, 6]

– Test Outcome:

Form the test we can see the parameter determines the depth of deep dream. The larger octave_n becomes, the deeper the render/transfer process will be. When it is set to 1, the picture is only slightly changed, the color of the sheep remains almost the same as its original source. However, when the parameter becomes larger, the contrast colors become heavier and the picture loses more features.

octave_scale

– Test Range: [0.5, 1, 1.5, 2, 2.5, 3]

– Test Outcome:

This parameter controls the scale of the deep dream. Although the contrast colors are not as heavier as the first parameter octave_n, the size of each transfer point scales and affect a larger area. So, we can see from the last picture, the intersections of several transfers are highlighted.

iter_n

– Test Range: [10, 15, 20, 25, 30, 35]

– Test Outcome:

This parameter controls the number of iteration of the deep dream. In other words, it determines the times of image processing. When the number is smaller, the output woulld be more similar to its original input. When the number becomes larger, the output would be more ‘deepdreamed’.

strength

– Test Range: [300, 400, 500, 600, 700, 800]

– Test Outcome:

The strength determines the scalar condition of each deep dream process. As we may see from the pictures above, the 6 transforms of the original picture are almost the same while only differ in the strength of colors (patterns). The higher strength outputs the sharper result.

layer

– Test Range: [“mixed3a”, “mixed3b”, “mixed4a”, “mixed4c”, “mixed5a”]

– Test Outcome:

The layer gives different patterns of the deep dream. It is also the pattern GAN used to train. So, each pattern would render different shape of DeepDream.