TrackLog Generator
Introduction
As I developed my final project, I made some adjustment based on my original proposal. I used to plan to make a simple way for people to do daily logs in which they can spend less effort achieving pretty visual effects. But later I decided to narrow the concept to target a more specific group of people who want to record the places they have been. Thus, TrackLog Generator can be a useful tool for them to create some artful and unique visuals for their tracks. As for the machine learning part, I mainly used Style Transfer and CycleGAN and trained them separately.
Process
1. StackGAN 🙁
At the beginning, I would like to use StackGAN to use text-to-image conversion. However, I came across a lot of bugs when I set the environment and modules. I will try to figure it out later, but for this final project, I would like to go back to CycleGAN to realize image-to-image conversion.
2.CycleGAN
As for the CycleGAN, I trained it with facades dataset. With different epochs, the results are different as follows.
I think Epoch_114_400of400.ckpt performs best so that I chose it to generate the model. Also, I applied it to other pictures of buildings, or say facades, where it also performs well.
3.Style Transfer
As for Style Transfer, I trained six different styles with the six following pictures. All of them are related to a kind of emotion expression so that they can be labels for users to choose.
And the training results are as follow:
Also, I applied the style transfer model to the building to see its performance, which turned out not bad.
4. Style Transfer + CycleGAN
Next, I tried to use style transfer and cycleGAN together. Firstly, I tried to make the output of style transfer as the input of cycleGAN. The results are a little too abstract and similar in which case it can hardly express effective meanings. So, I tried the other sequence.
5. CycleGAN + Style Transfer
When I used the output of cycleGAN as the input of style transfer, the results are quite satisfying to me, because there are multiple shapes and colors. As a result, I decided to make my TrackLog generator apply cycleGAN first and style transfer next.
6. CycleGAN + Style Combo
Moreover, cycleGAN plus a simple style is not interesting enough and the results can be very limited. So, I made the style combo transfer where the user can pick three labels for style transfer. And the output of each style will be the input of the next style.
- Each label has a representative color, and by clicking the according label, a square with the according color will pop up besides the Transfer button.
- When clicking the Transfer button, three styles will be applied in one step.
- The three styles can be the same, and the order does matter.
I think the benefit of doing so is that the user does not need to refresh the webpage again and again whenever they want to change the style, and they can compare different results in one page and pick his or her favorite more easily.
Here are some screenshots of the pictures generated by style combo transfer. 
7. Optional Notes
Furthermore, to add more flexibility to my project, I used getContext(“2d”) so that the user can also user the mouse to create his or her own patterns/words on the picture generated by cycleGAN. And they are free to add more details on the visual.
How it Works
1. Three Steps
My project has three steps in total to generate TrackLog visuals.
- Step 1: DIY by Dragging Your Mouse (Optional)
- Step 2: Pick your three keywords for the day (The order matters!)
- Step 3: Pick Your Favorite, or Try it Again!
2. A Sample–My Stay in Florence
To illustrate the further application of these generated pictures, I created a simple sample about Florence to show how dynamic and flexible it can be.
First, there are three groups of pictures. A is the original ones, B is the ones with style combo and C is the cycleGAN plus DIY.
With the three groups of pictures in three layers, we can create a lot of different visuals by simply modifying the opacity of layers. Here are some experimental results I get by setting different opacity to each layer. This sample was made in Photoshop, and it only took me a few minutes to get multiple results.
As we can see, these results have a more consistent style and can be more artful and creative.
Further Development
- Firstly, due to the time limit, I did not make a mature model for users to generate something like the sample above. But I think it can definitely be based on website and make everything easier again.
- Secondly, I think there can be some login and logout where the user can save or upload the picture more conveniently. Now I save the pictures mostly by screenshots.
- Lastly, the front-end can be further improved and the window should be resizable. Now, the mouse position is not so stable relative to the canvas position.
- Besides, as for the artistic part, as Moon mentioned, it can be extended to some collaborative creating website and do some animation because it is accumulative. 🙂