Author: Guangyu WU

Posted on

Reflection on Image-based VR and Photosynth

The TED Talk by Blaise Aguera y Arcas is an inspiring video showing that the image-based VR techniques were taking form as early as in 2006. However, it also suggests that the field has encountered some challenges so we are not experiencing it on a large scale now.  From my perspective, the main challenges that might have held technologies like Photosynth back are the computational techniques, data privacy issues, and user demands. 

The speaker Blaise Aguera y Arcas focuses on zooming interface and how to seamlessly transit from one picture to another, using a system like Seadragon. And his latest patents and research shows that he is exploring viewing interface that is stronger in performance and requires less computational power. And an interesting fact is that he filed a patent to help phone users protect their data, probably include images, from massive mining, which is the kind of work required to create a project like Photosynth. That seems a sign that privacy concerns will hold back the development of this semantic-rich network of photos.

After looking through the research by other researchers mentioned in the talks,  some other interesting trends emerged. Richard Szeliski, for example, starts to explore more of video-related topics and space construction technology. Noah Snavely continues to research depth and scene reconstruction. Steve Seitz is researching texture-related topics. What surprises me is that the majority of the research explicitly mentions “virtual reality” in their title. Only very recently one of their research points out that the outcome of the paper is aimed at VR consumption. So I think one of the reasons that we are not seeing a lot of examples of Photosynth is that we don’t have the ideal viewing interface for it. With VR developing at a high speed and the increasing ownership of VR devices, technology like Photosynth might have more role to play.

Posted on

Comment on the OC 6 Developer’s conference Keynotes

After watching the entire video of the opening keynote of Oculus Connect 6 Developer’s conference, I was really impressed by the vision the Oculus and Facebook team have for VR: many of the endeavors they are taking on now are vivid examples of what a “Mirror World” can be in the future. The three keywords that many of the speakers used are “proximity”, “diversity”, and “presence”. And the goal for VR, they claimed, is to allow people from anywhere (proximity) to do many things (diversity) together (presence). 

From the presentations of different departments at Facebook and Oculus, these goals are partially achieved with the current versions of Oculus devices. My impression is that “proximity” is most readily achieved and “diversity” of experiences and content is growing fast, while “presence” requires further development on both hardware and software.  

The new releases that caught my attention were “Oculus Link” that links Quest to PC so that Rift content can be seen on Quest. Many Go apps are also being transferred to Quest, so it seems Quest, especially the next version with hand-tracking, will likely be the flagship model for a long time.

Another interesting new release is Horizon, which sounds like a VR version of Facebook, with gaming and design experiences built-in similar to some social games. It sounds like a virtual part of the Mirror World depicted by Kevin Kelly. But considering one speaker talks about developing the “real world index” at the Reality Lab of Facebook. I think the team certainly has the ambition to create a true AR mirror world of our reality.

However, such rich-detail mirror world also comes with security and privacy concerns. While VR and AR allow people to interact with each other in an easier way than ever, this platform is also more prone to Internet attacks and manipulations. When people rely on information that automatically overlays on reality and stop to check for themselves, security and trust will have a new meaning.

Two more comments on what I think the conference really wants to convey: one is that VR experiences need to be “natural”. The addition of hand tracking to Quest and Facebook’s acquisition of CTRL-Labs both imply a future of VR with fewer pieces of hardware, and more human-natural interface. Another point is “more entry points”, Facebook is using its own social network, its workplace subsidiary, as well as fan community like Star War to promote the VR experiences and utilize its network effect to attract more users to try Oculus.

Posted on

VR Titles Review – Guangyu Wu

BBC Home Spacewalk 

Type: modeling (maybe some photography); Visual: 180, 3DOF; Sound: spatial effect not obvious;

Interaction: through gamepads;

Movement: move by grabbing “handles” in the game and pulling the virtual body forward. The player movement seems to follow a pre-determined trajectory most of the time.

Actions: Grab “handles”, occasionally take other “tools”.

Plot: A new astronaut joining ISS who went on a training session and inspect a damaged part of the space station. 

What I liked: 

1. Moving my virtual body by pulling the handle feels real. The relative movement of the virtual environment gives me a feeling that I moved up a bit in the real world.

2. The learning curve for interaction (at least in easy mode) is very friendly, I quickly learned how to grab things and navigate.

3. The plot is well-designed so that player can see what the ISS looks like, view Earth from space, witness a sunset, and experience an accident, all within a few minutes.

What I disliked:

1. The view is limited to about 180 degrees and no 6DOF. It is understandable that an astronaut’s view is limited but it could be better if the player has the agency to decide which direction to look. 

2. Since the setting is in the space, we can’t experience strong spatial sound effect, which could make VR more real.

3. During the scene where the astronaut spins away from ISS, I got really dizzy and wanted to take off the headset to rest. Without any feeling of movements from the body, this kind of visual makes me uncomfortable. 

Meet the miner WDR

Type: modeling; Visual: 360, 6DOF; Sound: strong spatial effect.

Interaction: through real body movements(walking and bending), gamepads.

Movement: can walk around and tilt head to see more. 

Actions: includes pick up, smash, shovel. Gamepads have vibrating feedback when the tool held by the player hits a hard surface.

What I liked:

1. Player can move around, bend body or tilt head to see more details of the environment and NPCs. Sound is spatial and light condition is rendered real-time according to the position of the lantern that the player holds.

2. Items that can be picked up or acted upon are highlighted, forming clear instructions about what to do next.

3. The vibration feedback is well synchronized with hand movement. When I used the pick to hit the rock, the vibration, sound, and visuals are all happening together.

What I disliked:

1. Player doesn’t have much agency in the game and follows instruction most of the time. 

2. The space that a player can move around is relatively small. Player is tele-transported to the next position in a black-out transition. 

Virtual Tour of Dunhuang Cave

Type: photography (static picture); Visual: 360, 6DOF; Sound: no spatial sound effect, plain background music.

Interaction: through real body movements(walking and bending), gamepads.

Movement: use gamepad to point at the place to tele-transport to.

Actions: focus for five seconds to watch visual-audio explanation of part of the frescos, use the gamepad as a flashlight to lighten up part of the wall, point the gamepad and click to tele-transport.

What I liked:

1. The scene of Dunhuang Cave, though static, is 360 and 6DOF. As we discussed in last class, a 6DOF picture is really hard to capture. Yet, one can look from any angle in this VR tour.

2. The VR flashlight achieves something not feasible in real-life frescos visit – light will damage the delicate artwork. This VR tour is beyond the limits of a real tour.

3. The tele-transport function allows users to see the room from more angles (even from the ceiling) and observe the patterns on the wall close-up (closer than 10 centimeters).

What I disliked:

1. There is no sound effect that is spatial and simulates the experience in a cave. The plain background music is okay but not as immersive.

2. The room in the cave is well-lit, which is not realistic since the light should only come from the entrance in reality. This makes the room more like a model than a real picture.

I also viewed many other titles including Accounting (which I think is immersive and have the cool concept of VR in VR), Medical deck (model of human body and organs with both hands as tool to shape and edit like in tilt brush), and Primitive (which visualizes a complex program), and more. But the three ones above gave me most thoughts regarding its VR techniques.

Posted on

VR/AR Week 2 Post – Guangyu Wu

1.  Response to  16 Lessons for a VR-First Future From ‘Ready Player One’

AGREE: 12. VR platforms should put in safe guards for managing physical health into future systems

VR can be addictive and immersive: Addiction to VR and subsequent long-time usage might cause stress on users’ eyes. Immersion means that users cannot see their environment and their body movement is likely to be confined to a small, room-size space. Thus, adding health management is important.

DISAGREE: 5. Virtual Schools will democratize high quality education to the world 

Internet-based learning such as MOOCs have proven to be moderately successful in democratizing high quality education, but current VR doesn’t seem to solve the “lack of interaction” problem in Internet-based learning and it’s more costly. Thus, I think the benefits doesn’t justify the costs of going from 2D to 3D.

2. Three VR experiences:

Lucid Space Dreams(Oculus Go), Jurassic Park (Oculus Go), Boxing Game (Vive).  Along with other youtube experiences.

3. Short review of Kevin Kelly’s article AR Will Spark the Next Big Tech Platform—Call It Mirrorworld

In this encyclopedic article, the author compares AR to the web and social media and reveals AR’s potential to create a dynamic platform that further connects the digital world with the physical world. The author also discusses the “reflexive recoil ” that many people have towards new and unknown technology such as AR, and I believe the author is right in pointing out that the only way to understand the impact of AR is by immersing ourselves in it and embrace the “mirrorworld”.