Category: VR / AR Fundamentals

The VR production project is a great way to actually get involved with hands on production techniques and put the theories we learned in class into practice, I had a great time working with VR production and also learned a lot of VR related knowledge along the way. Since I have a certain level of proficiency with video editing in premiere, I thought this project will be fairly simple in the post production phase. However, I ended up spending the whole weekend to accomplish what I had in mind. I learned that while working with new forms of media such as VR, there is not as much resources at our disposal since it is still a developing area. Not only is there a lack of support from different video applications, VR specific technologies such as the Insta 360 camera we used in class can have many problems since the algorithms for VR is still in development for perfection. The production process itself was also much more complicated than I thought it would be, I have to do many effects by hand just so that the left and right eye can have the same effect. While it is very simple to fool the eyes in a traditional video, having post production effects align is another story, I ended up manually masking the sky on both eyes to achieve a successful special effect, while also creating a separate mask for the ground to make the colors look natural as opposed to just having filters over the lens. If I get the chance to do this project again, I will definitely film the day and night footage in one shooting session, without moving the camera

When we first got access to the 2k files from previous classes, I experimented with premiere of what I can and cannot do: 

adding greenscreen effects and avatars

Another problem I had during production is the alignment of day and night footage, filmed at the same spot but with slightly different angle. I had a feeling that this would not work well according to the degrees of freedom talk we had in class, how it is extremely difficult to find the exact same pan/tilt angle as the previous shot. Yet through some experiments with post production it seems like I can manipulate the footage to some degree to make it better.

One attempt I made was to adjust the footages in premiere so that they can line up, however changing one section of the VR video will throw the rest of the alignment off, making it impossible to get a perfect alignment without having the original footage be correct, or have heavy photoshop editing like the “911” call of Avatar, which is expensive/time-consuming and not ideal.

alignment attempt

On second thought I decided to only photoshop one frame of the night footage to lay over the day footage, Instead of using a whole video which is extremely hard to manipulate frame by frame by hand.

showing day and night footage

Aside from all the technical difficulties with video editing skill, the biggest problem I had with this project is the handling of huge data files. The 8k 30fps footage from the 360 pro is a completely different thing from the 2k files I have been experimenting with. The IMA studio has decent computers(32 ram, 1080 GPU), yet the 8k footage is too for the computer to even function normally. However, the production was a lot smoother after the downgrading from 8k to 4k, though the process of downgrading took a few hours as well. I believe that if the hardware catches up, we can have a much easier time with VR production because it is a very processing power intensive form of media. 

video encoding constantly freezing

As for demoing, I was not able to do so during the IMA show because I had other projects I had to demo during the show. But I did demo the video to many of my friends during the production phase just to get suggestions and feedback. It seemed like most people are interested in VR and its possibilities, and I believe that the Hyperreal theme that I decided to go with is a perfect fit for the VR platform. Since no other forms of media can have the same amount of immersiveness and impact to humans, VR combined with spatialized audio is the best way to convince the human body to really engage with what they are watching.

creating a hyperrealistic world

glitch effect and audio

I am very grateful that I had the chance to formally learn the theories of VR and AR in an academic setting, since I learned most of my audio and video knowledge on my own. I think this class is a great way to get me started in VR production, but most importantly get me thinking in a VR mindset. Before formally learning about the VR AR terminologies and theories I had a rough idea of how the technology works by aligning left and right eye with their corresponding difference in position depending on their distance from you, but now I can use the correct terms such as disparity and parallax. One exchange I had with Professor Naimark changed the way I think about VR, when I proposed the idea of using machine learning to replace humans with solid silhouettes, I thought that the VR will look correct as long as the left and right eye positions are correctly matched. However, the response I got really surprised me, I was told that even if they look spatially correct, it will still be a flat figure in the correct spot. Very obvious observation if you think about it, yet so abstract in concept. Which is why actually getting to try VR production in this class is a great way for us to explore and learn how things work in the world of virtual reality. I will definitely continue to think about human perceptions with the concept I learned in this class, I am sure that these theories will be beneficial to the projects that I will work on in the future. 

Project Name: DREAM (Data Rules Everything Around Me)

Description: DREAM is one of the four “On Shanghai” VR projects in a NYU Shanghai Fall 2019 IMA course named VR/AR Fundamentals, completed by Ryan, John and Amy. It is a 360 video experience located at the center of Shanghai near the Pearl Tower in Lujiazui. What we wanted to express in this one minute experience is that everything in VR headset, or any technical equipments like computer, mobile or whatever nowadays, is composed of data and may vanish and collapse. We wanted to audience to get some futuristic and cyberpunk feelings from it.

Below is the 360 video:

Production: Firstly we started with getting used to a very “high-end” camera, Intsa 360 Pro 2. It was quite different with normal cameras but actually also very easy to use under the clear and in-order instructions. Then about the post-production, we went through the stitching them together, adding visual glitch effects pixel by pixel within certain areas and attempting to modify spatial audio. Stitching six files into a full 360 video was also easier than expected with present mature application, for the reason we have discussed in class how hard manual stitching was before the new camera like Insta 360 and application came out several years ago. About the effects, greatest thanks to my teammates’ hard working. Though I didn’t participate in this part much, I also explored a lot in post-production’s effects on a normal video by myself and got a taste about how powerful Premier Pro is to modify even just from pixel level. The VR toolset in Premier Pro can do more than we expected and gave us a great experience to try “movie-like” effects though ended with requiring more workloads, totally in expectation. The spatial audio in the end was not achieved because of time limits either. Anyway, the outcome is quite satisfying, especially with the demoing reactions in IMA Show. 

Demoing: The hardest part while demoing our project was in fact to introduce a zero-VR-experience user how to use a VR headset. They need to immediately know how to use the controller and how to chose and click in their own views. Fortunately all of them got used to it very quickly. It was a huge paid-off when you saw their exciting and “wow” reactions while watching the one-minute video. Some users even had questions after the experience, like “What course this project was made for?”, “How did you make this”, “Do you have more to show” and so on. We all believe VR experience will get more and more popular and if this short one-minute experience can trigger more people’s interests, we have succeeded.

Below is the documentation video intercut with the project and demoing reactions:

This semester I am very fortunate to work with wonderful teachers and classmates on a VR video project which our team named “Century in a Minute”. As an one-minute VR rephotography of the street scene on Century Avenue, the project is far from perfect, however, it was indeed a great learning experience that opens a door to the VR production and experiment world. I was able to learn how to shoot 360 with Insta 360 Pro 2, our latest 8K VR camera, edit it with Premiere Pro tools that are tailored for VR videos (including the amazing “Immersive Video” effects). I also tried the Facebook 360 Spatial Audio Workstation using Reaper, which enables the spatialization of our background music. Last but not least, it was great to have IMA show participants trying it on with a portable Oculus Go device and observe people’s reactions. Most of these experiences were unexpected beforehand but definitely rewarding looking backward.

What works well in our production is the application of Premiere Pro VR toolset. We applied the plane effect to titles so as to avoid distortion, and glitches and blocks to transit between scenes. The automation completed by the software. such as matching content on both eyes’ views yet allowing for discrepancy to create sense of depth, is really a boost to productivity. In the meantime, crop and color hue filters were used in our project and after working with the over-under footage we have for a few weeks, I started to view VR video editing not as mysterious and intimidating, but just like any other videos with exciting new opportunities to explore.

What didn’t work as well as I would expect is the shooting. Taking into account weather and hardware malfunctioning, we had four shoots excluding preparatory shoots without VR camera. Especially when we want to do rephotography, the requirements on exact location and angle posted a huge challenge. On the last two shoots, we figured out the xyz position of the camera correctly, however the orientation of the camera is 180 degrees different from the previous shoot. The preview from phone app wasn’t showing exactly what will come out after stitching and given enough time, a new shoot taking into account the full 6 dofs might produce better day and night matching than we currently achieve. But the process of panning and restitching the video post-production also taught us good skills, so it was a rewarding experience after all.