Author: Jonghyun Jee

Posted on

VR/AR Final Project Documentation – Jonghyun Jee

Title: D.R.E.A.M. 

Subtitle: Data Rules Everything Around Me

Description: D.R.E.A.M. brings you into the middle of Shanghai, where neck-craning skyscrapers and bustling passers-by are surrounding you. Everything seems just another ordinary day in Shanghai, and you might wonder what’s so special about this cityscape. As time goes by, you’re going to notice there is something strange about your surroundings. When you look closely, you can spot a number of buildings (including Oriental Pearl Tower) that are gradually disintegrating. Background noise sounds a bit different now, as footsteps and voices get slightly distorted. The sky is peeling off and the black screen beneath reveals itself; all the people around begin to disintegrate too. Everything what you see now is, quite literally, data. 

Location: Lujiazui Skywalk, Shanghai

Goal: Our team tried to make the maximum use of glitch effect in VR experience in terms of not only visual but audio part, too. As its title indicates, our video touched upon an idea that we are surrounded by data—or anything that can be reduced to data. We were partly influenced by The Matrix Trilogy, and a thought experiment known as “Brain in a Vat,” in which I might actually be a brain hooked up to a complex computer system that can flawlessly simulate experiences of the outside world. We hope our audience to experience an unreality that is so realistic that they feel as though they are in a simulation within the simulation. The title is also a subtle, intended pun because what rules everything around me is, at least when I am wearing a VR headset, nothing but data.

Filming: Amy, Ryan, and I went to the skywalk twice to shoot 360 degree video. For the first shot, weather was fine but a little shiny so there was a slight light flare in the footage. We took video in two spots; one in a less crowded, park-like place and the other in the middle of the skywalk. Since we picked the late morning time in Tuesday, there were less people than usual. The whole process was quite smooth. Insta 360 Pro 2 camera was equipped with a straightforward user interface. For some reason, the footage we took in the middle of the skywalk was later found to be corrupted, and that was a little bit of a downer. The other footage seemed nice but on the very top of it was the tip of a peeping antenna. For the second shot, we did not make any mistake and could get the full footage of the both spots. The weather was pretty cloudy so more suitable, and luckily nobody disturbed our shooting. A few number of people showed an interest, but most passers-by just continued their way.

Post-Production: The first thing we did after the shoots was to stitch these video files together. We watched our videos on the Oculus Go and checked how they looked in 3-D display. The videos were highly immersive without no additional edit; they were already quite powerful. We decided to use the second video we took on the skywalk, as it had more a variety of elements that can be manipulated. Since we only used Adobe Premiere Pro for the visual part, effects and tools we could utilize were somewhat limited. If not labelled “VR,” most of the effects were inapplicable to our files since those effects only worked for flat surface. At first, we added a glitch effect on the entire video just to see how it looks like. When the effect is applied for the whole part, the features of 3-D stereo image were hardly visible. So we traced a mask layer from each building and specified the area in which the effect got kicked in—a bit of manual labor but totally worth it. We modified parameters separately for each building so that their glitch effects look distinct from each other, to avoid seeming a way too identical. Later in the video, the glitch effects get contagious so the sky and people also become distorted, gradually disintegrated into particles. The room for improvement, I think, is to make more use of a man who looks the audience straight in the face. In the later part of the video, there is a guy who stands in front of the camera and takes a video of it. We tried to add more effects on this guy, but realized that we needed After Effect to visualize what we wanted to try (to make him continuously back flip or so). Many people, after watching our demo product, gave us a feedback that our video would become more interesting if we put  spatial audio. I tried to add spatial audio multiple times, but could not reach a satisfactory result; sci-fi sound effects were positioned on each building as an audio indicator that hints where to look at, however, the result had no significant differences between the spatial audio one and the stereo one. If given more time, we would definitely work more on the spatial audio part so we can possibly direct our audience to see where we want them to see. Overall, we could successfully visualize what we envisioned—but could not maximize the possibilities of audio. 

Reflection:  Throughout the semester, I definitely enjoyed learning both theoretical and practical knowledge of immersive media. The new concepts and terms we learned were at first a little confusing; and yet, they got more and more clear as we began to work on our own project. During the show, it felt great to be noticed for our efforts when we could see a lot of “wow” faces from our testers. What I learned the most during the course is to think in a 3-D way; all of my filming and video editing skills were limited to 2-D flat screen and so was my visual imagination. This course helped me to add a new dimension in the canvas of my mind. Now I have more understanding on how VR/AR actually works—now I can feel such a deep consideration and hard efforts behind the scenes of virtual reality. 

Posted on

Week 14: Final Project Documentation by Jonghyun Jee

Slides for the finals can be viewed here.

Project Title: Object-Oriented Art

Background

I’m recently interested in the nascent philosophical movement known as Object-Oriented Ontology (usually called by the acronym OOO), which has attracted a lot of attention from the arts and humanities scene. In short, OOO rejects the idea of human specialness: we should not place the privilege of human existence over the existence of nonhuman objects. Working on my AI Arts final project, I convinced myself that my works, in part, reflect the idea of OOO, so I named this series “Object-Oriented Art.” In contrast to the mainstream phenomenology that presumes things are only real insofar as they are sensible to human conception, OOO claims that things do exist beyond the realm of human cognition. In his article, Dylan Kerr lists some examples of questions that are posed by OOO artists: “what does your toaster want? How about your dog? Or the bacteria in your gut? What about the pixels on the screen you’re reading off now—how is their day going? In other words, do things, animals, and other non-human entities experience their existence in a way that lies outside our own species-centric definition of consciousness?” One of the main criticisms against OOO is that it is simply impossible for us to withdraw from human perception. Wondering about how the day of pixels on my screen might go is fascinatingly imaginative; and yet, that idea itself is still too human-centered—it is nothing more than applying human-only notions to other nonhuman objects. An interesting parallel can be found from an emerging AI arts: a controversy about whether the credit of AI-generated artworks goes to AI. As of now, I think, all the algorithmic artworks in the world are still human’s brainchild. Humans programmed the code, collected data, and fed algorithms these data to create a piece of art. Unless AI does the same process by itself without any single human involvement, I think AI is no more than an art tool like a brush. For my project, I put emphasis on the possibilities of AI as an art tool—algorithms as my brush, data as my paint.

Motivation

To put in a nutshell, my project is to feed a number of algorithms with my own sketches in order to visualize my ideas and impressions. Below are my original drawings:

Chameleon is the living example of “style transfer,” which I’m using primarily to color and morph my drawings.  

This is my self-portrait. 

And this is a skeleton of frog. When I was very young, probably 3 or 4 years old, I was floating around my house and found this tiny frog skeleton. I stared at it for quite a while, knew right away that it is not alive anymore. It was my first encounter with the idea of death.

Let’s see how AI spices these drawings up!

Methodology

If I fed algorithms my raw sketches, the result would be disappointing. As their background is just plain white, AI will fill up this blank with dull, redundant patterns. So I had to do a sort of “biscuit firing” by adding some colors and patterns. I used the tool called “Painnt” to apply thin styles on my drawings.

The next step was to choose data. For the chameleon, I intended to visualize a future chameleon that is surrounded by human-caused environmental pollution. 

I combined my drawing with an image of plastic waste using “DeepStyle” powered by Google. DeepStyle allows its user to easily apply the style transfer effect on one’s image; it usually takes five minutes to train and yield a result. The generated result is already pretty interesting, but it the distinction between the object and background is not highly evident. 

So I repeated the same process with a different image of plastic waste. You can see how the sky and shadow of the right image are partially shown in the generated result. 

Using Photoshop, I combined the two results together and got this final image. However, I needed AI’s help once more.

The resolution of this image is 775×775, which is not very favorable to print out. When zoomed into the arm of the chameleon, the image became visibly pixelized.  I used AI image upscale to enhance the resolution of my image by 3100×3100. 

I repeated the same process for my other sketches as well.

The other chameleon  combined with images of forest fire.

The skeleton of frog combined with a picture of frog eggs. In so doing, I tried to blur the line between life and death, drawing and photography. 

My self-portrait combined with an image of Dancheong, a Korean decorative patterns for traditional wooden buildings, especially for temples. I chose Dancheong as my style input because it is a symbolic representation of my cultural background (Korea & Buddhism). 

Conclusion

I intended to focus on the effectiveness of AI as an art tool, especially in terms of creating a piece of fine arts. Using traditional art mediums such as paint and ink is not only time-consuming but mostly irreversible. We cannot simply press CTRL+Z in a canvas. When I create an artwork, the biggest obstacle has always been the lack of my techniques; my enthusiasm cooled off when I could not visualize my thoughts, ideas, and impressions in a way I had envisioned. The AI tools I have learned during the class, in this sense, could fill in the technical gap of my art experiments. After using AI to color and morph my drawings, I printed out the generated results and juxtaposed my original sketches with AI-modified versions of them in order to show the process of how AI spiced up my raw ideas. One remarkable thing I noticed during my project is that, AI arts also requires a sort of “technique.” I had to choose tools and data that are appropriate to visualize my ideas, modify the parameters, manipulate my data (sketches and photos) to yield more satisfying results. Some may think AI artwork is just a click away, but in fact I think it requires inspiration and consideration as much as traditional art mediums do. I would like to continue my art experiments with the tools I learned during this course, and explore more about the possibilities of artificial intelligence and computer vision in terms of creating artworks. Huge thanks to Aven who spared no pains to help us learn the inside scoop of AI Arts, and all other classmates who gave me extremely valuable feedback.

Posted on

Week 13 Best and Worst VR/AR News – Jonghyun Jee

Assignment: VR/AR News of the Week has closed for the season with 92 entries, all of which we’ve seen at least briefly in class. Some of the stories will be looked back in 5 years and be considered accurate, prophetic, powerful. Others will be looked back in 5 years and be considered off-track, clueless, and ridiculous. PLEASE SELECT YOUR TOP 4 OF EACH.

Promising

1. New wearable skin lets you touch things in VR and and be touched, too

Most of the VR contents have been confined to the audiovisual experience. Only a very few number of them include sense of touch; and yet, tactile sensation in VR environment has been no more than electronic vibrations. According to the article, this second skin can “actually make you feel are touching something real.” I am not sure if this wearable skin will be introduced to the market in 5 years; if so, it will definitely create no little excitement in the VR scene. I agree to the journalist’s opinion that “touch is the last piece needed for virtual reality’s true killer application.” The emergence of “touch” in VR will undoubtedly make our experience more rich and immersive.

2. Create An Entire Home Gym With Oculus Quest

There are a couple of problems when you try to set up home gym. It takes a lot of space, equipment not cheap, and once you get bored with your treadmill or rowing machine–it will give you a headache. Workouts in VR will remedy these disadvantages; they not only require less space, but also are cheaper and far more versatile. The writer of this article notes that “virtual reality headsets deserve serious consideration as an exercise tool.” I also believe this VR fitness will become more and more widespread as the price of standalone headsets (i.e. Oculus Quest) becomes cheaper.

3. Phone-based VR is officially over

After its first introduction to market in 2016, Daydream, Google’s phone-based VR cardboard headset contributed to lowering the entry barriers of VR experience. This article explains why companies decided to discontinue their phone-powered VR services. “Mobile VR was good for short experience,” said the spokesman of Google, “but it had clear limitations.” I think their decision to give up the mobile VR was right, because the course of development is definitely heading to a wireless, standalone equipment. Mobile VR did everything on its part, and is now making way for more powerful successors.

4. Sifting Reality From Hype: What 5G Does (and Doesn’t) Mean for VR & AR

This article gives its reader rich information and context about what 5G is and how it can or cannot impact the development of VR/AR experience. 5G, in short, provides greater bandwidth and lower latency than current mobile networks does. What this means for VR/AR is that, according to the writer, it will enable immersive video streaming and cloud-rendered VR/AR gaming, which are only theoretically possible as of now. And yet, this article points out that “possibility doesn’t always mean viability.” Even if 5G can revolutionize the immersive media, whether people are willing to open their wallet is a whole different question. I think this article has a keen-eyed insight because it is not only showing the rosy future of VR but also considering realistic issues about its market value. 

Not Likely

1. Sony’s Shuhei Yoshida: ‘The Human Brain Is Getting Used To’ VR

Sony’s Shuhei Yoshida said in an interview that people are beginning to overcome the dreaded VR simulation sickness. Simulation sickness is a syndrome similar to motion sickness, often experienced during VR exposure. Although a number of researches proved that there is a kind of “adaptation effect,” their result varies strongly between individual studies. I think the recent upgrades of equipment quality may account for a larger portion of this effect, instead of people’s brain actually getting accustomed with immersive media.

2. Facebook’s Latest Purchase Gets Inside Users’ Heads—Literally

Facebook purchased a start-up called “CTRL-Labs” that “uses a mix of machine learning and neuroscience to allow people to manipulate computer interfaces simply by brainpower.” It is remarkable that Facebook, which is often accused of getting inside of its users’ head, actually turns its eyes to this technology. Even though the entire process is supposed to be exclusive to a user, I do not think people will be willing to give someone a direct access to their brain, regardless of how “helpful” it can be. Customers will not let technology intrude into the last bastion of their privacy. 

3. Is the world ready for virtual graffiti?

The developers of Mark AR describes their app “the world’s first augmented reality social platform.” It might be the first one, but I think it has some severe limits: this type of platform heavily depends on the size of users. I am skeptical about whether this app can attract enough users in the first place to utilize its full potentials. This article writes that “the idea of wandering around a city, finding the random tags people have left behind, is fascinating.” It might sound fascinating, but in its early stage, people will not find it very interesting as there is nothing much to see.

4. Russian dairy farmers gave cows VR goggles with hopes they would be happier and make better milk

Regardless of whether cows with VR headsets are going to make better milk, this idea is just entirely impractical in many ways. First, I hardly believe someone designed a headset specifically for cow-eyes. If these farmers are just showing their cows VR video set for human-eyes, it will just make these cows confused and even more stressed. Second, let’s say cows somehow produce better milk. Will its benefit outweigh the price of giving VR goggles for every cow? There are many more reasons I can give, including ethical issues too. In short, I hope this news is just another fake news.

Posted on

Week 12: Final Project Concept by Jonghyun Jee

Presentation slides can be viewed here.

Background

There is hardly any question about the fact that humans are the only species that create art. Some might bring up examples to refute this; the painting elephants of Thailand, the male bowerbirds that build a collage-display with sticks and glasses to impress the females, bees that build structurally perfect honeycombs, and so on. Yes, they are clearly showing kinds of artistry; and yet, I cannot put them on the same level as artists. They have techniques but not the thoughts—the essential core that makes art, art. What did these animal artists mean by their artworks? Marcel Duchamp displayed a toilet to question the traditional values of craftsmanship; Damien Hirst put a tiger shark in a vitrine filled with formaldehyde to visualize the physical impossibility of death. Many modern artists, including these two, present pieces that seemingly lack artistic techniques in a traditional sense, but their philosophy underneath makes their work “artwork.”

In this sense, it is no wonder that the emergence of AI in the field of art has triggered such a myriad of controversies. Some people even envisioned the dystopian future of the art world in which most of the human artists are replaced with AI artists. This apprehension climaxed when an AI-generated portrait “Edmond de Bellamy” was sold for $432,500 in a Christie’s auction last year. A year later, however, the hype seems to have faded. In November 15th, the Obvious Art—the creator of “Edmond de Bellamy”—put another AI-generated painting for a Sotheby’s auction; the result turned out disappointing for them. Their new Ukiyo-e artwork was sold for $13,000, barely above the presale high estimate. This price crash is indicative of how skeptical the art world is of electronically created artworks. The staggering price of “Edmond de Bellamy” was, in my opinion, mainly because it was the first AI-generated artwork that came under the auctioneer’s hammer. Their second Ukiyo-e was not that special anymore and it was exactly reflected in its price. The artworks of the Obvious art team, strictly speaking, are not “created” by artificial intelligence. It was human who fed the algorithm lots of data. I would not say the AI is an artist here. Humans who collected the data and wrote the code are rather closer to the definition of an artist; AI was just a tool. No one will say the brush in a painter’s hand is an artist, even though it is what actually draws a painting.

Motivation

I intend to focus on the effectiveness of AI as an art tool, especially in terms of creating a piece of fine arts. Using traditional art mediums such as paint and ink is not only time-consuming but mostly irreversible. We cannot simply press CTRL+Z in a canvas. When I create an artwork, the biggest obstacle has always been the lack of my techniques; my enthusiasm cooled off when I could not visualize my thoughts, ideas, and impressions in a way I had envisioned.

The AI tools I have learned during the class, in this sense, can fill in the technical gap of my art experiments. For my final project, I will use AI to color and morph my rough sketches and print out the generated outcomes.  Juxtaposing my original sketches and AI-modified versions of them, I want to show the process of how AI spices up my raw ideas.  

Reference

Among the models we have covered in the class, I will mostly use the Deep Dream to explore the possibilities of AI as an art tool, and Style Transfer as an inspiration. To break down the whole process, the first step is to draw a sketch and take a photo of it; next, I will briefly color the drawing with Photoshop so the background will not remain totally blank (if there is nothing on the background, AI might just fill it up with dull, repetitive patterns); Last, I will feed algorithms my drawings and repeat the retouching processes. I found that Deep Style tool of this website is particularly powerful. 

Below are the articles that gave me some insights:

AI Is Blurring the Definition of Artist

Has the AI-Generated Art Bubble Already Burst? Buyers Greeted Two Newly Offered Works at Sotheby’s With Lackluster Demand

With AI Art, Process Is More Important Than the Product

Posted on

Week 11 – DeepDream Experiment by Jonghyun Jee

DeepDream, created by Google engineer Alexander Mordvintsev, is a computer vision program that chews up the reality and renders it into trippy, somewhat nightmarish image. With a help from CNN (Convolutional Neural Networks), the effect of deepDream is a result of how the algorithm views images; that’s why this pattern recognition is called algorithmic pareidolia. For this week’s assignment, I tried a number of experiments with varying parameters to see what sort of results it would yield.

Instead of photographs, I drew a self-portrait and took a picture of it. I colored my drawing with Photoshop and Painnt:

Then I uploaded my drawing on this site, which allows its users to easily apply DeepDream effects on their images–without knowing much of how this DeepDream actually works. 

We can see from the generated image above that it warped the original image with mostly animal-related features. We can spot the dog-like and parrot-like visuals, but still the original portrait looks like a human face. To control more parameters of this effect, I used the notebook called “DeepDreaming with Tensorflow” provided by Alex Mordvintsev. I tried different layers to see which one yields the most interesting output.

Those layers are characterized by edges (layer conv2d0), textures (layer mixed3a), patterns (layer mixed4a), parts (layers mixed4b & mixed4c), objects (layers mixed4d & mixed4e).

Mixed 4b created spirals in the background.

And Mixed 4c showed the floral patterns. The way how it transformed the background elements was pretty cool; and yet, my face didn’t change much. I could see there was something interesting going in terms of computer vision. I moved on to the next step: video!

This notebook powered by Google Colaboratory provides a simple yet powerful user environment to generate a DeepDream video. To break it down with several steps, the first thing I had to do was mounting my Google Drive. It allows users to import their own Google Drive and upload an input image and download the output (generated video, to be specific). The next step is to load the model graph–the pre-trained inception network–to the colab kernel. After loading the starting image, we can customize our own neural style by adjusting the sliders (the strength of the deep dream and the number of scales it is applied over). Then we can finally begin generating the video by iteratively zooming into the picture.

Layer: mixed4d_3x3_bottleneck_pre_relu Dreaming steps: 12 Zooming steps: 20 From its thumbnail, we can see some interesting architectural images and dogs. And yet, 32 frames were too small to enjoy a full DeepDream experience.

Layer: mixed4c Dreaming steps: 60 Zooming steps: 20 Dreaming steps were a bit too high compared with zooming steps. At the point where it begins to zoom, the image doesn’t even look like the original portrait. It rather seems a way too deep-fried.

Layer: mixed4c Dreaming steps: 16 Zooming steps: 80 When I added more zooming steps, it goes far deep but the images look a bit too redundant. It would have been better if I tried different layers.

Overall, it was a very exciting tool to play around with. The whole rendering process didn’t take a long time thanks to the pre-trained model. I still don’t have clear idea for my upcoming finals, but DeepDream will be a definitely interesting option.