AI Arts Final Project: SneakerGAN-Sketch2Sneaker translation

Background

In recent years, sneaker culture is becoming more and more popular. We can see people wearing sneakers not only on a court but also in an office, on a street or even on a formal event. The group of people who love sneakers is expanding as well. Compared to the sneaker in the last century, which was mainly designed for sports, sneaker design experience a huge change: it becomes more casual and has a better design. They are unique and they have their own design language. In the era when people hope to pursue their own personality and style, many sports brands provide customizing service to their customers. For example, Nike ID and Nike Bespoke. However, that services are still limited, we cannot design our own sneaker. There fore, I hope to utilize AI to help the people who want to design their own sneakers but have no design skills to achieve dreams.

Model

The model I chose is cycleGAN, which can achieve style translation from categoryA to categoryB or vice versa. For my project purpose, I hope to use cycleGAN to translate the sneaker sketch(drawn by users) into a good sneaker design. The model was provided by Juyan Zhu

Github link: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

In order to train this model for my purpose, I prepare a dataset that contains approximately 1500 sneakers images from Nike and Adidas. For sketch data, I use the image2sketch sample code provided by Aven Zhou(AI Arts instructor) to translate my sneaker data into sketch data.

Sketch sample:

Sneaker sample:

Training

All the training process took place on Google collab, an online platform for people to train the AI model. This is the most difficult part of my project. I encountered many obstacles and failed so many times. Here are my training reports:

1. My own laptop: stopped at 97/200 epoch (testing)
2. Studio PC: stopped at 24/200 epoch (testing)
3. Studio PC: stopped at 27/200 epoch (testing)
4. My own laptop: stopped at 79/200 epoch (with edited dataset)
5. Ed’s (peer) laptop: stopped at 79/200 epoch (with edited dataset)
6. Studio PC: stopped at 50/200 epoch (final training version)
7. Studio PC: stopped at 50.200 epoch (final training version)

For an unknown reason, Google won’t allow me to finish my training even if I activate the pro version. But I still got a training model.

Training process

Model testing

As we can see, this model can basically fill sneaker sketches with black, white, and grey colors. Also, it will show the lightness and greyness. But it is still my expected output, which can translate simple sneaker sketch (profile or layout) into fine design. The potential reason for this output might be:

1. There is no common design language for sneaker color design.
2. The dataset is not the perfect one for training.
3. Training process didn’t completely finish

Supplementation in Processing and RunwayML

Back to my project purpose, I am about to make a project that can allow users to draw and receive a sneaker design output from the model. To achieve this, I decide to use Processing as a sketchbook, and RunwayML as the host for my model. 

• Processing

This is the code screenshot. Basically, users can draw a sneaker sketch on Processing canvas. Press c, the programming will automatically save the sketch into the Processing folder, and the user needs to choose that image file and send it to RunwayML. After receiving input from Processing, RunwayML will process the input in the model and send output back to Processing.

Resolution testing

Suggested by Aven, I adjusted the resolution of input images in order to experiments whether the size of input images will affect the output quality. 

The offset setting:

 

256×256:

128×128

As we can see, the smaller the resolution is, the darker the output image is.

Future development

• Dataset
  • Sneaker side images’ color varies too much, it should be simple for the AI model to extract features
  • Part of sneaker sketch data are not full sketch
• Training
  • Train with a larger dataset
  • Go through all training process(200 epochs)

Reflection

Regarding the training, I should start training as soon as possible since there will be a bunch of issues I may encounter.

Prepare a perfect dataset according the goal I hope to achieve is really really important.