Background

In recent years, sneaker culture is becoming more and more popular. We can see people wearing sneakers not only on a court but also in an office, on a street or even on a formal event. The group of people who love sneakers is expanding as well. Compared to the sneaker in the last century, which was mainly designed for sports, sneaker design experience a huge change: it becomes more casual and has a better design. They are unique and they have their own design language. In the era when people hope to pursue their own personality and style, many sports brands provide customizing service to their customers. For example, Nike ID and Nike Bespoke. However, that services are still limited, we cannot design our own sneaker. There fore, I hope to utilize AI to help the people who want to design their own sneakers but have no design skills to achieve dreams.

Model

The model I chose is cycleGAN, which can achieve style translation from categoryA to categoryB or vice versa. For my project purpose, I hope to use cycleGAN to translate the sneaker sketch(drawn by users) into a good sneaker design. The model was provided by Juyan Zhu

Github link: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

In order to train this model for my purpose, I prepare a dataset that contains approximately 1500 sneakers images from Nike and Adidas. For sketch data, I use the image2sketch sample code provided by Aven Zhou(AI Arts instructor) to translate my sneaker data into sketch data.

Sketch sample:

Sneaker sample:

Training

All the training process took place on Google collab, an online platform for people to train the AI model. This is the most difficult part of my project. I encountered many obstacles and failed so many times. Here are my training reports:

1. My own laptop: stopped at 97/200 epoch (testing)
2. Studio PC: stopped at 24/200 epoch (testing)
3. Studio PC: stopped at 27/200 epoch (testing)
4. My own laptop: stopped at 79/200 epoch (with edited dataset)
5. Ed’s (peer) laptop: stopped at 79/200 epoch (with edited dataset)
6. Studio PC: stopped at 50/200 epoch (final training version)
7. Studio PC: stopped at 50.200 epoch (final training version)

For an unknown reason, Google won’t allow me to finish my training even if I activate the pro version. But I still got a training model.

Training process

Model testing

As we can see, this model can basically fill sneaker sketches with black, white, and grey colors. Also, it will show the lightness and greyness. But it is still my expected output, which can translate simple sneaker sketch (profile or layout) into fine design. The potential reason for this output might be:

1. There is no common design language for sneaker color design.
2. The dataset is not the perfect one for training.
3. Training process didn’t completely finish

Supplementation in Processing and RunwayML

Back to my project purpose, I am about to make a project that can allow users to draw and receive a sneaker design output from the model. To achieve this, I decide to use Processing as a sketchbook, and RunwayML as the host for my model. 

• Processing

This is the code screenshot. Basically, users can draw a sneaker sketch on Processing canvas. Press c, the programming will automatically save the sketch into the Processing folder, and the user needs to choose that image file and send it to RunwayML. After receiving input from Processing, RunwayML will process the input in the model and send output back to Processing.

Resolution testing

Suggested by Aven, I adjusted the resolution of input images in order to experiments whether the size of input images will affect the output quality. 

The offset setting:

 

256×256:

128×128

As we can see, the smaller the resolution is, the darker the output image is.

Future development

• Dataset
  • Sneaker side images’ color varies too much, it should be simple for the AI model to extract features
  • Part of sneaker sketch data are not full sketch
• Training
  • Train with a larger dataset
  • Go through all training process(200 epochs)

Reflection

Regarding the training, I should start training as soon as possible since there will be a bunch of issues I may encounter.

Prepare a perfect dataset according the goal I hope to achieve is really really important.

Background:

Inspired by image-to-image translation, I was thinking about exploring the possibility of translating sneaker design language among different brands. Hence I keep researching on image translation model on the Internet and find a useful tool–the Pix2Pix model.

Reference link: https://ml4a.github.io/guides/Pix2Pix/

Pix2Pix is shorthand for an implementation of a generic image-to-image translation using conditional adversarial networks, originally introduced by Phillip Isola et al. Given a training set which contains pairs of related images (“A” and “B”), a pix2pix model learns how to convert an image of type “A” into an image of type “B”, or vice-versa. 

Why it is powerful: “The nice thing about pix2pix is that it is generic; it does not require pre-defining the relationship between the two types of images. It makes no assumptions about the relationship and instead learns the objective during training, by comparing the defined inputs and outputs during training and inferring the objective. This makes pix2pix highly flexible and adaptable to a wide variety of situations, including ones where it is not easy to verbally or explicitly define the task we want to model.”

In general, we don’t need to define the relation between input images and output images. It is really flexible when we are in real production.

Research Project

On the same website, there is a project called “Invisible cities”. In Invisible Cities, a collection of map tiles and their corresponding satellite images from multiple cities were downloaded from the MapBox API. A pix2pix model was trained to convert the map tiles into the satellite images.

Example:

Below are the examples from training dataset:

After the training process, we can translate maps into AI-generated satellite image.

Also, we can translate human-created input image:

So taking this idea, I think it is possible for me to do sneaker style transfer, and also hand-sketch translation proejcts.

My plan

Dataset

So the first thing is to prepare the dataset. Since we need a huge dataset, we need a bunch of sneaker images. So I decided to just focus on the brand design language translation: Adidas & Nike design language translation. Sounds interesting!

So my plan is to collect all the Nike and Adidas sneakers side pictures to train a model. Here are some examples:

Model

The model should be possible to be trained with two datasets: Nike dataset and Adidas dataset. After the training process, the users should be able to upload a sneaker image, and then choose a design language(it could be just Nike or Adidas, and also it could be mixed mode). The output should be a sneaker image with a relatively clear design. I am supposed to train my own model and run it in runwayML. But for now, I need to test it the model. I choose to test the pre-trained model in runwayML, Pix2Pix model.

Pix2Pix model in runwayML

In runwayML, the Pix2Pix model only provides two styles. For a testing purpose, it is ideal to get a clear output image. I upload a sneaker picture, which is

And here are what I got from the model:

So I think they are not the ideal ones. But I think if we change the model, it will work better. But they look amazing though.

Final Project: iSneaker

Background：

In recent years, sneaker culture is spreading throughout the whole world. More and more people wear sneakers for their daily outfit. Inspired by Nike ID service(you can design your shoe colorway by yourself on NIKE.com), I am going to build a project to design a sneaker, not just colorway, but the whole design language. iSneaker project is supposed to make sneaker design possible to the sneaker lovers or public people who have no design background. Everyone can design their shoes by drawing sketches or combining different existing sneakers.

Potential AI Models

For sketches design, I am considering to use the same plan as I mentioned above, which will use Pix2Pix to achieve this function. For combining different sneakers, I will consider training my own model(something similar to style transfer). The available model now is styleGAN.

Datasets

For datasets, I will try to collect sides picture of sneakers as many as possible. Nike, Adidas, Puma, Converse, Under Armor, etc. 

 

My work

This week, we are asked to create GAN generative content and explore more about GAN. I follow the collab instruction and create my first bigGAN video.

Firstly, I encountered this error. I searched on stack overflow and realized that the problem is the version of tf and tf hub is not correct. 

I changed the setup code into:

I choose strawberry first!

frame number and noise seed setting:

Interpolation: For category A and category B, I chose violin and jeep.

Finally, I got the result!

My research

The first thing comes into my mind about GAN generative content is AI human face project. So I decided to research on GAN human face project

This is the AI-generated faces video that I found on YouTube. It looks really similar to what I got from collab. But to be honest, I did not fully understand how this algorithm works.

This is the Medium articles I found that explains GAN human faces generation.:

https://towardsdatascience.com/generate-realistic-human-face-using-gan-e6136876e67c

My work

This week we are asked to create our neural style transferred image or video. The model I used is a fast-style transfer model in runwayML. 

The style I choose is Hokusai painting. That reminds me of one direction of neural style transfer–comics style transfer.

My research

I found out a paper that discussion about comics style transfer.

Link: https://www.groundai.com/project/neural-comic-style-transfer-case-study/1

The above is the result they put in the paper. Comics style transfer indicates that neural style transfer is a good tool for comics or animation industry to animate/transfer video content into comics style in their creative work. 

For example:

Through the style transfer on each frame, content creators can transfer video into any style. Compared to hand-drawn/computer making comics or animation, style transfer video is a more convenient way to create such content. Compared to my work, I think this 3d style transfer for video is the advanced version of image style transfer. 

 My work

This week, we are asked to create our own deep dream images or videos. I have three try for deep dreams.

The basic setting is:

Firstly, I try to use my own profile to create a deep dream image.

However, this is a PNG file, which is not available for the deep dream. I did not find out this rule so I tried another PNG image.

The second try is my wallpaper. At first, I concluded that deep dream cannot dream the white background(it is true), and it does not matter with the format of images. Therefore, I try another PNG image without white background.

Even if I got deep dream image, I could not animate it. That is the problem.

Therefore, I shifted my attention to JPG image. I uploaded a JPG image to collab.

I tried with different setting actually. But I forgot to change the output name, resulting in that the output video was overwritten times by times.

 

My research

My research is divided into three steps.

Step1:

Since my output video is only two seconds long, so I am really curious about what a long deep dream video looks like. I searched on YouTube and reached to a video.

This is a really long deep dream video. Compared to my video, this video moves more smoothly. At the same time, since the time is long enough, the deep dream style is keeping changing while our vision is keeping diving. That is really amazing. I was told by Aven that the author has a really strong hardware computer to processing this video. That is why they can generate this long-time, high-quality deep dream video. It is impossible for my computer to run this deep dream video–it might take a week or more.

Step2

The first video is a 2d version deep dream video. I am also curious about how deep dream looks like in 3d space. Hence, I found a VR deep dream video.

Actually it is really scary to watch this video. It reminds me of multidimensional spaces and nightmare. Compared to my work, this VR work, I guess, is nothing special actually. How the author made it is to edit the original deep dream video, setting it into a 360 file. Then the author edit this 360 file and put it into the VR headset.

Step3

This video is more like a video with a deep dream filter. The author apply deep dream programming into each frame of this video, then put all frame together to form this weird woods video. It is quite interesting to see how it makes a deep dream. Different from a normal deep dream video, it changes the texture of the video gradually, it is a process. Look closely, we can tell.

Pixels-Reality

Abstract

We are living in an era that the digital world(the virtual world) intersects with the real world. AR/VR technologies and Smartphones allow us to create and interact with virtual stuff based on our real world. For example, we can see virtual Pokemon in Pokemon Go game, we can use Microsoft Hololense to interact with a virtual object in the real world, VR even brings us to an entirely new world which is completely different from where we really are. This evokes me to have critical thinking on the virtual creature and the real world: how about virtualize people in the real world? We are virtual creatures like Pokemon in Pokemon Go?

Concept

Everything on a screen is shown as a collection of pixels set. So the idea of my project is really simple: pixelate human in video/image. We are also a set of pixels in video/image. In order to show the pixel style, we need to manipulate the pixels that consist of our body, changing the size.

Development

Model

DenseDepth is a machine learning model that can detect the depth of an image/video. By using this, I can get a depth image that I can operate on. The image show depth by showing brightness.

As you can see, my head is a little bit darker than the surrounding. The RGB value of my body pixels is smaller than the background pixels. This mechanism allows me to pixelate my body and keep the initial background.

Combine with Processing

After getting the depth video from the DenseDepth model, I export the model result in Processing. I used the sample code provided by Processing libraries.

This is the key code block of my project. This is also my hand-written code. The basic idea of my project is: Keep the original background, only pixelate body. To achieve this, I separate my work into four steps:

1. Copy background image from camera
2. Create a new image, draw out the body contour and set the rest part of the image transparent so that we can see the background image I copy in the step1
3. Pixelate body contour
4. Draw background and pixelated image

Copy background image from camera

This is the easiest step for my project. image(camera,640,0) can simply achieve this.

Create a new image, draw out the body contour and set the rest part of the image transparent so that we can see the background image I copy in the step1

This is a little bit tricky. I utilize the depth image I got from the model in this step. Firstly, I need to create a new image, I called it a threshold image. Then by going through each pixel of the original image, I can set the threshold image’s pixel transparent if the brightness is larger than a threshold value(103 in my testing). This means that the pixel is the pixel that consists of the background. Else, I will draw the pixel out if the brightness is smaller than the threshold value. 

Here is what I got. For testing purposes, I set the body contour pixels’ RGB value to be random.

Pixelate body contour

To pixelate the body contour, I did a pixel manipulation:

for (int x=0; x <thresholdImage.width; x += 6) {
  for (int y=0; y < thresholdImage.height; y += 6) {
    int index = y*thresholdImage.width+x;

    noStroke();
    fill(thresholdImage.pixels[index]);
    rect(640+x, y, 6, 6);
 }
}

This code block indicates the computer to go through all the pixels of the threshold image, and when the computer passes by 6 pixels, it will draw a rectangle(size 6*6) and fill in according to the current RGB value. After the pixel manipulation, I got:

This is the final version of the threshold image. The body contour is pixelated, the background is transparent. So the final step is to draw the background image and the threshold image.

Draw background and pixelated image

This is the final step, and it is also quite easy. Firstly, I draw the background image first, meaning that the background image will at the bottom. Secondly, I draw the threshold image beyond the background image so that the threshold image won’t be covered. Since the threshold image is transparent in the part outside the body, we will see the background image I draw first. The final result is:

Even though it is a blur, we can tell that the head part is pixelated.

Conclusion&Future Development

This is a simple try with the DenseDepth model and pixel manipulation skill. The idea is simple but the process makes me struggling. I learned a lot from this try and thank Aven for his patient help! From the perspective of the quality of the project, there is one thing I think is worth refining: if the resolution size is too large, the people in front of the camera will look like a suspected criminal. It is funny but I am not supposed to achieve this. So in the future, it will be a good idea to add interaction. The audience of this project can control the size of the rectangle by themselves so that they can decide their own experience.

Presentation slide

Midterm Project Plan

For this week’s assignment, I defined my first machine learning model and train it with my computer based on the previous setup and the tutorial videos/coding.

Setup

Firstly, I used the foundational setting from the fashion_mnist coding to setup my cifar10 model setting. I changed several places: 

1. Import cifar10 in the first code block
2. import other optimizers from Keras
3. load cifar10 data in the second code block
4. change the class names of dataset 

Then I spent 681 seconds on downloading cifar10 data.

I encountered some problems in the whole process. For example:

Since it is different from the fashion mnist project, the input of cifat10 is not the same as fashion mnist(actually I didn’t know what it is.) I follow the error message and change the code: class_names[int(train_labels[index])]

start training:

Finish training.:

Optimizer: RMSprop

Loss function: Cross-Entropy

Total time: 136s

Loss: 1.5224

Accuracy: 0.4642

Change optimizer and loss function

I changed optimizer and loss function to SGD and Hinge. For SGD, I did not know that much. But for Hinge, I learned from loss function cheat sheet

 https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#cross-entropy

Hinge is a loss function for classification, which fits the cifar1o project. 

However, I got an error and I could not solve it. Aven told me that the problem is the input shape is not exactly what the next layer expects to receive.

After that, I tried several loss functions, but all failed.

Only change optimizer

SGD+Cross-entropy

Time: 120s

Loss: 2.2906

Accuracy: 0.1748

Conclusion: compared to RMSprop, SGD is faster but worse in loss and accuracy.

Adam + Cross-entropy

Time: 125s

Loss: 1.5115

Accuracy: 0.4684

\

Conclusion: Adam as an optimizer, is almost as fast as SGD, as accurate as RMSprop. 

Data augmentation method

I tried to use a data augmentation method to enrich the dataset. But I failed. I used the code below

(from https://machinelearningmastery.com/how-to-configure-image-data-augmentation-when-training-deep-learning-neural-networks/ )

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

# example of vertical shift image augmentation from numpy import expand_dims from keras.preprocessing.image import load_img from keras.preprocessing.image import img_to_array from keras.preprocessing.image import ImageDataGenerator from matplotlib import pyplot # load the image img = load_img(‘bird.jpg’) # convert to numpy array data = img_to_array(img) # expand dimension to one sample samples = expand_dims(data, 0) # create image data augmentation generator datagen = ImageDataGenerator(height_shift_range=0.5) # prepare iterator it = datagen.flow(samples, batch_size=1) # generate samples and plot for i in range(9): # define subplot pyplot.subplot(330 + 1 + i) # generate batch of images batch = it.next() # convert to unsigned integers for viewing image = batch[0].astype(‘uint8’) # plot raw pixel data pyplot.imshow(image) # show the figure pyplot.show()

I adjusted the code to fit in my code.

But I got one error:

After googling, I found out I need a “pillow” package to run this code. And then

I downloaded it. However, I still got the same error.

 

I think neuron networks are inspired by the biological neuron structure. As we all know, information is transmitted in the form of an electric signal through each neuron. The outside stimuli will be transferred to a neurotransmitter that will be accepted by the brain eventually. Human beings will do reaction according to the influence they receive from the neurotransmitter. Neuron networks, similarly, will receive the input data and process calculation. Since the function of activation function, the input will be transferred to the “neurotransmitter” of neuron networks—the result of the activation function(0 or 1, 0 or the sum of input, etc. Depends on the activation function). These “neurotransmitters” will be pass from the hidden layer to the next neuron of other layers. After going through all the layers of the neuron network structure, the programming will reach to the final output, and make reaction according to the output. Hence, we can conclude that neuron networks mimic the operating principle of the human body’s neuron structure and brains.

However, there are also differences between neuron networks and neuron structure of the human body. Since programming makes prediction instead of responding accordingly as the neuron system of humans do, we need to use abundant data to train the programming model so that the accuracy of neuron networks will be improved. What’s more, the direction of the correcting process is from the output layer to the input layer, which is a mechanism that the human neuron system doesn’t have. From the output layer to the input layer, each layer will correct the mistakes it makes. But in the human neuron system, the neuron signal’s conduction has its directivity. There is no way that neurons can reverse the direction of the signal’s transmit and pass the signal back to the last neuron.

Deep neural networks have something in common with the human neuron system, but we cannot say something like “the deep neural networks are completely simulating”. The way both two solve problems is the same.

Computer Vision art project: Two-Stream Convolutional Networks for Dynamic Texture Synthesis

This week I cased study on “Two-Stream Convolutional Networks for Dynamic Texture Synthesis”

Link: https://ryersonvisionlab.github.io/two-stream-projpage/

This project mainly utilizes pre-trained convolutional networks (ConvNets) to recognize dynamics texture and synthesize optical flow prediction. Through the recognition and analysis of dynamic texture, the dynamic texture of each frame of dynamic texture is encapsulated separately, and the model will initialize and optimize to form many different sub-sequences. Typically a sequence will have 12 frames. This is the basic process of generating a dynamics texture.

As we can see in this example, the water flow in painting style is the result of the recognition and synthesis of real water flow texture in the middle of the top.

Data source: The author applied their dynamic texture synthesis process to a wide range of textures that were selected from the DynTex database as well as others collected in-the-wild. The total amount of synthesized results they provided is nearly 60. 

In order to avoid the consistency between the synthesized dynamic texture and the original texture, the subsequence is optimized and initialized. For example, the first frame of a subsequence becomes the last frame of the preceding subsequence. Therefore, the differences between the original texture and synthesized texture can be told by our eyes in the endless loop of playing.

Besides, this AI model can transfer one dynamics texture into another one. As the documentation page says, “The underlying assumption of our model is that the appearance and dynamics of a dynamic texture can be factorized. As such, it should allow for the transfer of the dynamics of one texture onto the appearance of another“.

General feeling:

This project is a little bit different from the previous CV cases I studied. Firstly, it is not based on the camera but based on the inputs of texture(images/videos). It is such a genius project that we can use this model to make something static(like paintings) dynamic by analyzing the existence of texture from nature.