Author: Jiayan Hou

The idea

I really hope to combine what I have learned in class with my interest area in daily lives and as I said at the first class of this semester, I am interest in using photography or video in art forms to express myself. At the same time, I want to make this interactive process fun and easier for people to use. It could be better if my project could have some real-life usage. 

Then I landed on the idea to create an outfit generator. I would like to create an either real-time or non real-time platform where people can select the favorite  style they want to have. The detailed form could be as follows. The first way is letting the user simply type in the type of tops and bottoms they want to have and then the system will generate a predicted outfit image. The second way is letting the user input an image and then generate the predicted outfit image onto the user’s body. More detailed plan could be generated later on.

I think in everyone’s daily life, there is a need to create some visualized fitting system to help us select outfits. In reality, some online shopping website like has already implemented this kind of function. In the future, I hope everyone can have this kind of tools by empowering AI technology.

Action Plan

First of all, I planned to search similar work online including their models, their data set and any special techniques they used for reference. Then based on my initial research, I am going to do some deep dive into the areas I am interested in. 

I scanned the major fashion-related interactive machine learning project and find one research paper highly related to my initial idea. I requested for the dataset and find out that the model they used is called fashionGAN. There may have other similar models that could probably generate similar results and that’s the part I am going to research further.

Initial Findings

After taking a closer look at the models I found, I found out it is based on Torch and PyTorch, which I did not have any experience before. That could be hard for me to implement this project and I would like to find more related sources or models and also rethink of the whole workflow for this project.

Reference

https://github.com/zhusz/ICCV17-fashionGAN#complete-demo

http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/FashionSynthesis.html

Project Name: Outfit Generator

Concept & Introduction

The initial concept of this project is to do something about photo or video synthesis with a special focus on fashion topic. Then I found an interesting project called Deep Fashion made my the multimedia lab of the Chinese University of Hong Kong. They created a large-scale clothes database including different categories and attributes and they they builded up a model called fashionGAN to classify this database into different clothing images as well as retrieve the edge of clothes in each image. 

After that, when you input different characteristics of the clothes you would like to generate, the model will select all the labeled images and synthesize them together to generate a resulted output.  It will work as follows.

Research & Development

1）

After the concept presentation, I started to reach out to the former research team and ask for the original dataset. I would like to see whether I could possibly train the deep fashion dataset with fashionGAN on my own computer. But unfortunately, the dataset takes 53G which may be hard to store and run on my computer. Also, the fashionGAN model is designed with Torch which I am not quite familiar with. Then after consulting with professor Moon, he suggested me to run this dataset with fashion_mnist which we played in class. But since fashion_mnist can only process data with a scale of 28*28 and the dataset of fashion_mnist is actually open source online, I found out there it is not quite realistic to reshape all the raw image data into 28*28 size and use fashion_mnist. 

2）

Then I did some research on some interesting projects or models about image or video synthesis to see whether I can integrate some of their techniques into my own projects. I found an image synthesis model called CNNMRF and a video-to-video translation model called vid2vid. They either classify the edge of a facial image or classify the edge of people’s faces in a video. I think the techniques behind is really similar to the clothes retrieval benchmark function in Deep Fashion project. However, at this time I found that CNNMRF is built on Torch and vid2vid is built on ByTorch. And they are not using python but using lua.

/CNNMRF/

/vid2vid/

3）

Then I consulted with professor Moon again to see if there is any other options I can try. We finally decided to integrate two or three different models we have on ml5js to imitate similar classification and image synthesis effect. We chose to use BodyPix , which is newly released based on PoseNet, Pix2Pix and Style Transfer (this is still tentative) as the simplified model I could possibly use for this project. I am still working on the demo of BodyPix on my own computer and hopefully it will work this or at the beginning of next week.

The mechanism behind is that the model will first classify the edge of a figure and then classify the body parts of a figure.

/BodyPix/

//Drawing the Person Segmentation

//Drawing the Person Segmentation

After that, I also need to try the other two models and then integrate them all together for my final. More detailed workflow breakdown will be demonstrated in the next section. 

Next Step

1）

Finish the demo of BodyPix and label the body part for toppings and the body part for bottoms. 

2）

Try to train Pix2Pix on COCO Dataset or with part of the deep fashion dataset I get access to. 

3）

Integrate these two models in one combined model and return value in BodyPix to Pix2Pix. Integrate image-to-image translation to different body segmentation by using this combined model.

4）

Design the interactive user interfaces on my personal host. 

In order to find the Cifar-10 well, I tried to adjust two parameters, the batch size and epoch. The batch size means how many data are we going to select and train each time. Epoch means how many times are we going to train the whole data set in total.

Generally if you increase the epoch, the accuracy will definitely increase. I tried to adjust epoch number to 50 but I failed to run so much time. It seemed that I need to take three or four hours to finish the whole training process so I just changed the epoch number to 5. 

Under this condition, I simply adjust the batch size to 32, 64, 1280 and 2560. When the batch size is 32, test loss is 1.0839 and test accuracy is 0.6203.

When the batch size is 64, test loss is 1.2065 and test accuracy is 0.5686.

When the batch size is 1280, test loss is 1.7189 and test accuracy is 0.3912.

When the batch size is 2560, test loss is 1.8184 and test accuracy is 0.3663.

When the epoch is 5, it seems that as batch size increases, the test loss will increase and the test accuracy will decrease. In this case, when batch size is 32, we get the best test accuracy which is 0.6203. 

But here we need to notice that under different number of epoch, the optimal number of batch size might be different as well. In order to better train Cifar-10 model (get the highest possible test accuracy), we need to increase the number of epoch to 50 or even 100 and try to find the optimal batch size under that condition. We should notice that the total number of data set = batch size * #of iteration / epoch. Both the #of iteration and the #of epoch will have some effect on the test accuracy.

For this assignment, I first tried all the models available on ml5js.org and then select StyleTransfer as my target for this assignment. In order to add more fun to this model, I searched online and find this interesting YouTube video. It shows how to use Style Transfer model and further apply it to real time video importing from computer webcam.

Style Transfer Part 2: Real-Time Style Transfer with ml5.js with Yining Shi

I really love this idea and I tried to develop my own project based on this demo and the ml5 demo on their website. First of all, I started from transfer an image into the style of another image. I used inputImg and resultImg to draw the input image and output image on my canvas. Also, I added a key function in my script so that I can control when to start and when to stop this image transfer process.

Then secondly, I started to transform my code from image based to real-time video based. I added video = createCapture(VIDEO) and changed the resultImg to video in the draw function. And in order to better control this video transferring process, I added a variable called isTransferring. This variable will help me use Style Transfer function only if I press my key. So after these adjustments, I can get a real-time style transfer from my computer webcam. You can see my project demo below.

In the demo I found online, Yining also created a function which you can import your own image and apply it as a image style to the real-time video. I thought that function was really cool as well and I think after I learn how to train my own model, I can add this function into my own project as well.

Source: Style Transfer Demo from ml5

                 Real-Time Style Transfer demo with Yining Shi

My Code:  https://drive.google.com/drive/folders/11Nnwe2ZvMiPxrwZz9vWqmmJpKo6iSeJl