The visual reference I choose is one of the interactive installations of the teamLab Borderless project.
My current work looks similar to this project at the very beginning.
This is a huge physical installation. I am thinking of make this fantastic interactive installation into digital form by utilizing what we have learned so far.
Here is the link for my further plan:
This project is inspired by teamLab project.
In the performance, all the light could be rotated and point in a certain direction. Also, the performing pattern will keep changing. For example, the light at the sides will stop projecting and there will be only the lights in the center shooting lines towards one direction. By using the flow field, I can move the “lights” into the center by utilizing seekVec(centerVector). However, I still feel unfamiliar with the flow field technique. I think I still do not know where should I adjust when I try to achieve a specific visual effect.
Inspired by image-to-image translation, I was thinking about exploring the possibility of translating sneaker design language among different brands. Hence I keep researching on image translation model on the Internet and find a useful tool–the Pix2Pix model.
Reference link: https://ml4a.github.io/guides/Pix2Pix/
Pix2Pix is shorthand for an implementation of a generic image-to-image translation using conditional adversarial networks, originally introduced by Phillip Isola et al. Given a training set which contains pairs of related images (“A” and “B”), a pix2pix model learns how to convert an image of type “A” into an image of type “B”, or vice-versa.
Why it is powerful: “The nice thing about pix2pix is that it is generic; it does not require pre-defining the relationship between the two types of images. It makes no assumptions about the relationship and instead learns the objective during training, by comparing the defined inputs and outputs during training and inferring the objective. This makes pix2pix highly flexible and adaptable to a wide variety of situations, including ones where it is not easy to verbally or explicitly define the task we want to model.”
In general, we don’t need to define the relation between input images and output images. It is really flexible when we are in real production.
On the same website, there is a project called “Invisible cities”. In Invisible Cities, a collection of map tiles and their corresponding satellite images from multiple cities were downloaded from the MapBox API. A pix2pix model was trained to convert the map tiles into the satellite images.
Example:
Below are the examples from training dataset:
After the training process, we can translate maps into AI-generated satellite image.
Also, we can translate human-created input image:
So taking this idea, I think it is possible for me to do sneaker style transfer, and also hand-sketch translation proejcts.
So the first thing is to prepare the dataset. Since we need a huge dataset, we need a bunch of sneaker images. So I decided to just focus on the brand design language translation: Adidas & Nike design language translation. Sounds interesting!
So my plan is to collect all the Nike and Adidas sneakers side pictures to train a model. Here are some examples:
The model should be possible to be trained with two datasets: Nike dataset and Adidas dataset. After the training process, the users should be able to upload a sneaker image, and then choose a design language(it could be just Nike or Adidas, and also it could be mixed mode). The output should be a sneaker image with a relatively clear design. I am supposed to train my own model and run it in runwayML. But for now, I need to test it the model. I choose to test the pre-trained model in runwayML, Pix2Pix model.
In runwayML, the Pix2Pix model only provides two styles. For a testing purpose, it is ideal to get a clear output image. I upload a sneaker picture, which is
And here are what I got from the model:
So I think they are not the ideal ones. But I think if we change the model, it will work better. But they look amazing though.
In recent years, sneaker culture is spreading throughout the whole world. More and more people wear sneakers for their daily outfit. Inspired by Nike ID service(you can design your shoe colorway by yourself on NIKE.com), I am going to build a project to design a sneaker, not just colorway, but the whole design language. iSneaker project is supposed to make sneaker design possible to the sneaker lovers or public people who have no design background. Everyone can design their shoes by drawing sketches or combining different existing sneakers.
For sketches design, I am considering to use the same plan as I mentioned above, which will use Pix2Pix to achieve this function. For combining different sneakers, I will consider training my own model(something similar to style transfer). The available model now is styleGAN.
For datasets, I will try to collect sides picture of sneakers as many as possible. Nike, Adidas, Puma, Converse, Under Armor, etc.
This week, we are asked to create GAN generative content and explore more about GAN. I follow the collab instruction and create my first bigGAN video.
Firstly, I encountered this error. I searched on stack overflow and realized that the problem is the version of tf and tf hub is not correct.
I changed the setup code into:
I choose strawberry first!
frame number and noise seed setting:
Interpolation: For category A and category B, I chose violin and jeep.
Finally, I got the result!
The first thing comes into my mind about GAN generative content is AI human face project. So I decided to research on GAN human face project
This is the AI-generated faces video that I found on YouTube. It looks really similar to what I got from collab. But to be honest, I did not fully understand how this algorithm works.
This is the Medium articles I found that explains GAN human faces generation.:
https://towardsdatascience.com/generate-realistic-human-face-using-gan-e6136876e67c
This week we are asked to create our neural style transferred image or video. The model I used is a fast-style transfer model in runwayML.
The style I choose is Hokusai painting. That reminds me of one direction of neural style transfer–comics style transfer.
I found out a paper that discussion about comics style transfer.
Link: https://www.groundai.com/project/neural-comic-style-transfer-case-study/1
The above is the result they put in the paper. Comics style transfer indicates that neural style transfer is a good tool for comics or animation industry to animate/transfer video content into comics style in their creative work.
For example:
Through the style transfer on each frame, content creators can transfer video into any style. Compared to hand-drawn/computer making comics or animation, style transfer video is a more convenient way to create such content. Compared to my work, I think this 3d style transfer for video is the advanced version of image style transfer.
This week, we are asked to create our own deep dream images or videos. I have three try for deep dreams.
The basic setting is:
Firstly, I try to use my own profile to create a deep dream image.
However, this is a PNG file, which is not available for the deep dream. I did not find out this rule so I tried another PNG image.
The second try is my wallpaper. At first, I concluded that deep dream cannot dream the white background(it is true), and it does not matter with the format of images. Therefore, I try another PNG image without white background.
Even if I got deep dream image, I could not animate it. That is the problem.
Therefore, I shifted my attention to JPG image. I uploaded a JPG image to collab.
I tried with different setting actually. But I forgot to change the output name, resulting in that the output video was overwritten times by times.
My research is divided into three steps.
Step1:
Since my output video is only two seconds long, so I am really curious about what a long deep dream video looks like. I searched on YouTube and reached to a video.
This is a really long deep dream video. Compared to my video, this video moves more smoothly. At the same time, since the time is long enough, the deep dream style is keeping changing while our vision is keeping diving. That is really amazing. I was told by Aven that the author has a really strong hardware computer to processing this video. That is why they can generate this long-time, high-quality deep dream video. It is impossible for my computer to run this deep dream video–it might take a week or more.
Step2
The first video is a 2d version deep dream video. I am also curious about how deep dream looks like in 3d space. Hence, I found a VR deep dream video.
Actually it is really scary to watch this video. It reminds me of multidimensional spaces and nightmare. Compared to my work, this VR work, I guess, is nothing special actually. How the author made it is to edit the original deep dream video, setting it into a 360 file. Then the author edit this 360 file and put it into the VR headset.
Step3
This video is more like a video with a deep dream filter. The author apply deep dream programming into each frame of this video, then put all frame together to form this weird woods video. It is quite interesting to see how it makes a deep dream. Different from a normal deep dream video, it changes the texture of the video gradually, it is a process. Look closely, we can tell.
After learning attraction, I was obsessed with the magic of various kinds of forces in the coding world. But in the course coverage, we only learn the macro physics force. So I decided to explore and test whether the microphysics law fits in the coding world or not. The topic that I chose is Coulomb forces. Also, inspired by the icon of Atom(text editor), I want to build an atom structure with the P5.js.
I have mentioned my inspiration and visual reference in my last post.
Link: NOC Midterm Project Concept
Pastel Themed Natural Plants Mood Board
This is the mood board of my project. The images on it are my most straightforward visual director.
General Plan Guidelines:
I started my project by testing the Coulomb Forces in the P5 sketch.
The formula is: F = kq1q2/r2
Look closely, it is similar to what we learn for gravitational attraction
After finding out this, my project shift to another direction: trying to build an atom structure with gravitational attraction.
However, after testing different value setting, I realize that it is impossible to achieve my expected visual effects with only one gravitational setting. The reason is:
Above all, I think for my current ability, I cannot build an atom system with gravitational attraction. In order to finish the project, I decided to achieve the visual effect with simple attraction.
This is the basic structure by applying a simple attraction function to particles.
For proton and neutron, I design them to be sparkle particles. To achieve this, I use for loop to draw many circles, which no fill in body but fill in stroke. When looping through, the radius of a circle increases, but the opacity of filled color in stroke decrease. As a result, proton and neutron look shining. Besides, by applying frameCount and sin/cos value, I make P&N animated.
For electron, I try to keep it simple because it will be hard for watching if the visual effect is too fancy.
For the particles sphere, it is the most tricky one. At first, I want to only make many particles distributed randomly within a circle. I did not know how to do it. So I turned to Moon. Moon provided me a solution, which keeps all the particles outside an invisible circle. When the particles reach to a specific position, we change the direction to opposites. Then we adjust the value of forces so that particles won’t go away but keep bouncing since they are always attracted by the core. Hence visually, it looks randomly distributed on the sketch.
I did not put a lot of efforts in the interaction design. So I would say interaction design is one of the potential improvements in my project. I want to keep exploring the how can I combine add electron interaction with music playing. It can be a music interaction project actually.
To be honest, my visual design ability is limited and my imagination is restricted by what I have learned so far. I am not surprised that I cannot do a good visual design. So if I have more time, I will definitely dig deeper to visual experiments.
We are living in an era that the digital world(the virtual world) intersects with the real world. AR/VR technologies and Smartphones allow us to create and interact with virtual stuff based on our real world. For example, we can see virtual Pokemon in Pokemon Go game, we can use Microsoft Hololense to interact with a virtual object in the real world, VR even brings us to an entirely new world which is completely different from where we really are. This evokes me to have critical thinking on the virtual creature and the real world: how about virtualize people in the real world? We are virtual creatures like Pokemon in Pokemon Go?
Everything on a screen is shown as a collection of pixels set. So the idea of my project is really simple: pixelate human in video/image. We are also a set of pixels in video/image. In order to show the pixel style, we need to manipulate the pixels that consist of our body, changing the size.
DenseDepth is a machine learning model that can detect the depth of an image/video. By using this, I can get a depth image that I can operate on. The image show depth by showing brightness.
As you can see, my head is a little bit darker than the surrounding. The RGB value of my body pixels is smaller than the background pixels. This mechanism allows me to pixelate my body and keep the initial background.
After getting the depth video from the DenseDepth model, I export the model result in Processing. I used the sample code provided by Processing libraries. 
This is the key code block of my project. This is also my hand-written code. The basic idea of my project is: Keep the original background, only pixelate body. To achieve this, I separate my work into four steps:
This is the easiest step for my project. image(camera,640,0) can simply achieve this.
This is a little bit tricky. I utilize the depth image I got from the model in this step. Firstly, I need to create a new image, I called it a threshold image. Then by going through each pixel of the original image, I can set the threshold image’s pixel transparent if the brightness is larger than a threshold value(103 in my testing). This means that the pixel is the pixel that consists of the background. Else, I will draw the pixel out if the brightness is smaller than the threshold value.
Here is what I got. For testing purposes, I set the body contour pixels’ RGB value to be random.
To pixelate the body contour, I did a pixel manipulation:
for (int x=0; x <thresholdImage.width; x += 6) {
for (int y=0; y < thresholdImage.height; y += 6) {
int index = y*thresholdImage.width+x;
noStroke();
fill(thresholdImage.pixels[index]);
rect(640+x, y, 6, 6);
}
}
This code block indicates the computer to go through all the pixels of the threshold image, and when the computer passes by 6 pixels, it will draw a rectangle(size 6*6) and fill in according to the current RGB value. After the pixel manipulation, I got:
This is the final version of the threshold image. The body contour is pixelated, the background is transparent. So the final step is to draw the background image and the threshold image.
This is the final step, and it is also quite easy. Firstly, I draw the background image first, meaning that the background image will at the bottom. Secondly, I draw the threshold image beyond the background image so that the threshold image won’t be covered. Since the threshold image is transparent in the part outside the body, we will see the background image I draw first. The final result is:
Even though it is a blur, we can tell that the head part is pixelated.
This is a simple try with the DenseDepth model and pixel manipulation skill. The idea is simple but the process makes me struggling. I learned a lot from this try and thank Aven for his patient help! From the perspective of the quality of the project, there is one thing I think is worth refining: if the resolution size is too large, the people in front of the camera will look like a suspected criminal. It is funny but I am not supposed to achieve this. So in the future, it will be a good idea to add interaction. The audience of this project can control the size of the rectangle by themselves so that they can decide their own experience.