VR Production and Demo Summary

This semester I am very fortunate to work with wonderful teachers and classmates on a VR video project which our team named “Century in a Minute”. As an one-minute VR rephotography of the street scene on Century Avenue, the project is far from perfect, however, it was indeed a great learning experience that opens a door to the VR production and experiment world. I was able to learn how to shoot 360 with Insta 360 Pro 2, our latest 8K VR camera, edit it with Premiere Pro tools that are tailored for VR videos (including the amazing “Immersive Video” effects). I also tried the Facebook 360 Spatial Audio Workstation using Reaper, which enables the spatialization of our background music. Last but not least, it was great to have IMA show participants trying it on with a portable Oculus Go device and observe people’s reactions. Most of these experiences were unexpected beforehand but definitely rewarding looking backward.

What works well in our production is the application of Premiere Pro VR toolset. We applied the plane effect to titles so as to avoid distortion, and glitches and blocks to transit between scenes. The automation completed by the software. such as matching content on both eyes’ views yet allowing for discrepancy to create sense of depth, is really a boost to productivity. In the meantime, crop and color hue filters were used in our project and after working with the over-under footage we have for a few weeks, I started to view VR video editing not as mysterious and intimidating, but just like any other videos with exciting new opportunities to explore.

What didn’t work as well as I would expect is the shooting. Taking into account weather and hardware malfunctioning, we had four shoots excluding preparatory shoots without VR camera. Especially when we want to do rephotography, the requirements on exact location and angle posted a huge challenge. On the last two shoots, we figured out the xyz position of the camera correctly, however the orientation of the camera is 180 degrees different from the previous shoot. The preview from phone app wasn’t showing exactly what will come out after stitching and given enough time, a new shoot taking into account the full 6 dofs might produce better day and night matching than we currently achieve. But the process of panning and restitching the video post-production also taught us good skills, so it was a rewarding experience after all.

VR/AR News Collection

Promising trends:   

1. Facebook’s DeepFovea AI promises power-efficient VR foveated rendering.

This semester we witnessed Google announce the end of its phone-based VR grand trial, which essentially discard its efforts with the cardboard and Daydream projects. Two of the major reasons behind the low adoption rate, I believe, are the relatively low resolution and high power consumption. Thus, the research on foveated rendering becomes the focus of many VR firms, since it promises to provide high resolution for areas where the user pays attention to and achieve that with lower energy consumption and probably less latency. This technology, along with other power-saving improvements, is critical in building wearable VR/AR devices. 

Source: DeepFovea

2. Apple Working on AR Headphones

While most of the AR companies out there compete in fooling the eyes, the world of spatial audio augmentation receives less attention. The patent Apple recently filed shows the firm’s ambition in building AR headphones that could spatialize sound in a video/voice conference setting.  This use case could have great potential for business customers but also for public spaces like museums and schools. Also, compared to the complex optical requirements to create AR visuals, AR audio seems a low-hanging fruit that would hit the market in the coming five years. 

Source: USPTO

3. Microsoft’s DreamWalker lets you see VR cities during real-world walks

This invention might seem weird at first glance: having the user see a VR city rendered by detecting the real-world city environment looks like a redundant step, especially considering the rough CGI models and blurry textures. However, I think this signals a potential use case which has a huge customer pool: commuter recreation. As cities expand, an increasing number of people need to take a long commute to work every day. People might read and watch videos on their smartphones, but a VR experience during the commute would definitely make the previously boring commute something to look forward to. A modified train cabin that allows travelers to safely experience VR during their commute would be very interesting.

Source: Microsoft Research

4. Create An Entire Home Gym With Oculus Quest

Another piece of news that I believe shows a promising commercial trend. It demonstrates how in less than $ 600 one can build his/her own VR gym at home. The vested weight and other accessories could be deliberately designed in the gym VR titles so that the experiences are better. Considering there is a growing demand in China for gym and fitness training,  and that many physical gyms are run inefficiently and shut down soon after they open, people would probably show more interest in home gym options.

Not so promising trends:

1. Facebook’s Latest Purchase Gets Inside Users’ Heads—Literally

I used to believe that the brain-machine interface is an important step towards intelligence augmentation, IA (an idea that focuses on strengthening human intelligence instead of replacing it with machine intelligence). Facebook’s acquisition of CTRL-Labs gives me mixed feelings. Even though the tech giant claims the interface only enables users to give instructions to applications and collect no more information, the idea that one can share some photos or information by just “intending to” is unsettling. The ease of sharing encourages leakage of private information that users might not want to publish given a second chance to think about it. This brain-machine interface could be useful in many cases, but it should be heavily regulated in a social network application.

2. Russian dairy farmers gave cows VR goggles with hopes they would be happier and make better milk

This piece of news sounds like a joke but I think it suggests a  possibility that VR headsets and related technology could be cheap enough one day to become a tool to make tedious work more tolerable. However, the value it brings to its users (cows in this case) might not be “real” and substantial. Though the cows see an ideal living environment in VR, their real living conditions are not any better. Calmed by the virtual reality, they may be confined to even smaller space physically. Such kind of virtual experience is more like a means of control, which should not be encouraged, no matter its audience is animals or humans.

Source: Ministry of Agriculture of the Moscow region

3. MediView XR raises $4.5 million to give surgeons X-ray vision with AR

The startup company is working towards a mixed reality solution for tumor removal surgeries. Despite the good intention and help it provides, I am not sure if it is reliable enough for real surgeries. Medical use cases are high stake scenarios and inaccurate AR projection could lead to legal mistakes. Even if we assume the AR imaging is correct, surgeons who get used to having AR assistance might find it hard to perform the operation when AR is not available due to technical reasons. A better use case might be medical personnel training, meanwhile, the technology should go through rounds of checks to confirm its accuracy.

Source: Mediview

4. Virtual Reality is the latest dinner party trick

Appealing as the news title sound, the dining experience described in the news piece is far from a casual and fun dinner party. The VR environment was somewhat avant-garde, featuring an abstract and artistic environment that don’t arouse my appetite. The unusual container of food and the bite-size limitations also makes the food more of a tool in an exotic VR experience, rather than a food itself. The news is reporting a trend that food served in such experiences is going high-end, with top chefs joining the team. However, I think a more popular use case might be creating VR representations of snacks for VR gamers or movie views who want to enjoy snacks while experiencing VR.  It will be a natural extension of the VR environment instead of transporting the user to a random place and alienizing the food itself.

Thoughts on Mantra, Cara, Mocha VR

Mantra VR

liquify

Highlights:

  1. A comprehensive set of 16 filters to stylize your 360° footage in After Effects and Premiere Pro.
  2. Drag-and-Drop effects for transforming into surreal and fantastic animations. Compatible with the Immersive tools in After Effects and Premiere Pro.
  3. Keyframe animations from 0-100% effect applied.
  4. Node-based Panel lets you tweak, merge, and chain the effects, with automatic scripting.

CARA VR

Camera rig solving instantly with presets in Cara VR
 

Highlights:

  1. Solve and refine camera position, rotation, focal length and lens distortion parameters inside Nuke.
  2. GPU-accelerated stitching without line artifacts or ghosting, even on moving scenes.
  3. Automatically correct exposure and white balance differences between cameras during the stitching process.
  4. Feature-point tracker optimized to handle distortion and continuity at the edges of lat/long images.
  5. Ray trace rendering engine
  6. Preview project with  HTC Vive and Oculus Rift headset

MOCHA PRO 2020

Highlights:

  1. Optimized workflow for mono and stereo 360° to achieve fewer pre-composing, nesting or rendering. (After Effects)
  2. Simplified Undistort/Distort workflow makes mono or stereo clean-ups on equirectangular footage easy.
  3. User-selected planar tracking analysis on “both eye views” which is better for set extensions, 3D text, and particle tracking, masking, and object removal.
  4. Assist other 3D tracking applications on difficult shots with low detail or significant foreground occlusions.

Common trends:

  • Much emphasis on optimizing workflow for 360 video post-production. Many use node-based interfaces to allow user chain different effects together. 
  • Stitching and rendering mostly powered by GPU.
  • Two types of tracking methods: feature-point tracking and planar tracking. 
  • Some compatible with many platforms and others designed to fit certain tool.

Specialized features:

  • Mantra: Filters for stylized effects, animations.
  • Cara: Exposure and color adjustment, continuity of stitching.
  • Mocha: Masking, object tracking and removal.

Rephotography of Shanghai

This is an interesting video exploring the changes in many famous Shanghai landmarks. Thanks to deliberate cultural conservation, many historical buildings still exist, while other places (like the Lujiazui area) underwent great development and completely changed its look. 

It could be interesting to shot 360 videos from certain spots around the upper Bund area because as shown in this video, many historical buildings (especially those from the 19th and early 20th-century cluster there. The downside, however, is that these spaces are mostly occupied by buildings and man-made artifacts, we might lose the natural aspect of Shanghai.

The spot is marked on Google Maps and it can potentially cover three views, making it very interesting for the 360 project.