Vision3 creates a photorealistic 360VR with 6DOF movement

Vision3 creates a photorealistic 360VR with 6DOF movement thumbnail

Source: Vision3

It is generally accepted that for VR to continue developing, immersive content needs to be about more than gaming. Filmic content is almost entirely limited to passive VR360 content. The new generation of headsets which give inside out tracking, mean that movement within an environment is now possible. This is typically seated VR where the viewer isn't going to walk around a scene, but needs to be able to look-around the principal subject. 6DOF movement also addresses the nausea of VR360 films and creates significant increased perceived value.

There needs to be a solution to creating video experiences that one can move around within. 

Vision3, in collaboration with Sony, has successfully created a methodology that gives the viewer a degree of movement within a film. The process uses stereoscopically captured imagery projected onto a 2D card within the scene, which gives realistic and photographic quality results. The card needs to be deformed to avoid ground contact inconsistencies between the character and the ground plane caused by placing a 2D object into a 3D scene. In the proof of concept, this was manageable through distorting the 2D card because we filmed a stationary character. The quality of the photographically captured subjects is a significant improvement on anything so far created in VR. However, the proof of concept did not allow for characters to move around within their environment due to geometry of the subjects being 2D and that of the environment being 3D.

The resulting ground contact issue is addressed with the Stereo Environment Contact Automation Tool or SECAT.

SECAT works by capturing depth information from the scene at the time of shooting, using witness cameras. The depth information can be used to produce a depth map. Geometry is created from this, onto which we project the stereo image. However, projecting a full stereo image onto full geometry creates distorted depth. Instead, we reduce the depth of the geometry over most of the image to instead get the most stereoscopic detail. Conversely, at the point where the geometry meets the environment (e.g. the ground plane where a characters is walking across a floor) we need the full geometry to correctly position the feet in depth, correctly sitting the subject within its environment. The key development of SECAT is the algorithm that optimises the depth map and stereo image resulting in a tool that allows us to straightforwardly place the stereoscopically captured subjects into a 3D environment. This will turn a high quality but limited process into one we can use to create content across a significantly wider range of interactive experiences. As it is automated, it will increase efficiency and enable character movement.

As a byproduct of the witness cameras it is expected that additional tools will be created to help the process including separating characters from backgrounds so reducing roto. The use of geometry should also enable an increase in look-around and consequently the area within which the viewer can move.

SECAT makes it possible to capture truly lifelike human performances and place them within a 3D scene, within which the viewer has the ability to move.

The development of SECAT enables the process to be commercially viable for a wide range of applications, kick starting the creation of experiences featuring characters and bringing a level of intimacy, empathy and connection that has not so far been possible in VR environments.

SECAT will also be available as a hardware and plugin package, enabling third party content creators to benefit. This will help the industry to thrive, increasing audiences and helping to create the market.

Vision3 will release SECAT to market at the end of 2019. SECAT will consist of a witness camera hardware package and software plugin. These effectively distil the results of the research, into a package that together with industry standard processes such as rotoscoping and paint, will give third party content creators the tools to create high quality 6DOF video content. Vision3's work in VR is concentrated in interactive photographically captured experiences. The development of this technique and the tools to achieve it are fundamental to achieving growth and filling the market gap of photographic 6DOF content for the estimated 5.7 Million next-gen headsets in 2019 (SuperData) and growing VR/AR market, estimated to reach $182B by 2025 (Goldman Sachs).