Acquiring detail down to the finest granular is paramount. I’d liken the habit to an unhealthy obsession, a self proclaimed pinnacle of mild madness. However, the dividends do pay off, as the level of immersion and realism, detail to form, truly achieves trickery on a scope never experienced with any other VR scenario of today. We are tantalizingly close to achieving a level beyond photo-realism, a sense of actual realism.
Digital cultural preservation of indigenous communities and heritage sites, nostalgia and captivating VR storytelling of times past. This is what we aim for when developing our processes and one must stress developing, as we are always striving to make our own pipeline of in itself obsolete. You must keep ahead of yourself as you’re only as good as your last incarnation. This manic drive and obsessive behaviour for perfection came from a moment of pure desperation of circumstances, as you don’t know what you’ve truly got until it’s gone.
Almost four years to the day. January 20th 2013, the morning after my 31st Birthday. Mine and my daughters home of eight and a half years was arsoned. There’s this great footage of a shirtless Albino running across a burning two story inner city warehouse apartment rooftop. This one scenario would lead to a cascade of howling personal events that would bring me to the absolute brink. Post traumatic stress disorder, the recurring dreams. I knew I needed to get busy living, to find a means of dealing with the grief, we needed to find a way home. I don’t think it was so much the house itself, but rather what it represented. A sense of security, a peace of mind.
So here's a wacky idea, maybe, just maybe, we could utilize our old family photographs to rebuild an experience not to different from the place in question. And there you go, this is how one was lead to the magic of photogrammetry, virtual reality, game engines and all that is VFX goodness. Prior to all this, I was just drummer / singer in a band. Necessity truly is the mother of all invention.
2016 started not too different. Turning through pipelines and processes, crash coursing every avenue out of my bedroom, as one had been doing for quite some time already. What was to come after posting the first of a series of photogrammetry to VR scans on YouTube was just bonkers. That moment when you’re getting calls from Chinese VCs like the Shanda Group and you're just sitting there thinking, holy hell I don’t even have a lawyer, an accountant, office, or a job!
From approx 300m pointcloud resolution in February to 2.75Bn by October. By December we were demonstrating preliminary results from our 3.5D human capture techniques, concluding that such performances can be captured at much higher compression ratios than are currently available on market. This is a huge deal, as being able to record an experience similar to 4D volumetric video, however only using 50-100mb per minute vs 1-2GB per minute, while retaining great texture, silorette and geo resolution, is a game changer. Now for true human performance cinematography integrated into our realistic VR environments.
2017, we will witness the birth of VR cinematography, not to dissimilar to the talking pictures era of a century ago. As the process we follow is far more akin to the movie industry than that of a traditional game / VR company, as storyboarding, scouting, finding actors & talent and even our pricing models are based on this methodology.
For VR to truly shine, a sturdy efficient and automated pipeline / toolset must be delivered in conjunction with a strong refined language and understanding of effective narrative for a new industry of storytellers. The technology defines what is plausible and we must abide and respect these rules. I believe, most crucially though that rapid succession of content is required, we must talk about wrapping up projects in weeks rather than years. This is a must for true VR.
Markets such as China are craving for content and everyone has came to their sensors that this is not all about hardware. If we want to be developing seasons and episode of stand up comedy, theatre, experiences of the human condition, what I like to refer to as macro experiences, we have to automate, streamline the tools required, package them and they must scale. We are catering for a growing audience with a growing appetite and 360 just doesn't cut it, it never did. At the end of the day, content is king.
Our strategy for 2017. First we demonstrate ‘proof of concept’, then we lead to short sweet experiences with little narrative, then we shall demonstrate something not to dissimilar to a short film. We must take baby-steps. Anyone who tells you that we are ready for a 6 degrees of freedom feature film type experiences in 2017 is simply full of shit, or is literally selling snake oil. We will finally be releasing, to the joy of our fans, many of these experiences via the steam VR marketplace over 2017.
These demos will demonstrate three holy pinnacles of our technological prowess. We will deliver captivating, immersive spaces, with the faces to match. Extremely highly detailed & accurate asset & environment acquisition, 3.5D volumetric human performance capture and stunning HRTF based 3D audio.
Photogrammetry partnered with Deep Learning is another avenue I am strongly investing resources into, they make for good bed fellows as they both require vast amounts of GPU processing power. One might ask what possible avenues would deep learning and all of its CNNs, autoencoders, reinforced learning and visual hallucinations would entice. Simple, it's all signal processing, AI enhanced super resolution, smart filler techniques for troublesome areas and what I currently refer to as procedural photogrammetry. This is going to get exciting.
Like any good production house, scouting is a necessity. Such scenarios like ‘Maori Chieftain on the Beach’ were probably not the safest scouting sessions for a discoordinated, visually impaired, combustible individual under the harsh New Zealand sun. In conjunction with the tide being a continued threat, the rocks were insanely slippery, long cliff drops to your side, sunscreen irritating your eyes while lugging $10k worth of UAVs and DSLRs around. Had many falls, one of which actually fractured my finger, pulled a ligament which to this day is still mending. The passion we place and the things we do for an unpaid POC.
The scene consists of 2.75Bn points. 2 Billion of the beach scene itself and 750m for the Maori Chieftain statue. The scanning process was actually remarkably quick, as we had very little time due to the tide. Approximately 45 minutes, 2000 photos with a Nikon D810 and 12/24mm wide angle lens. We had to return with the UAV a few days later due to poor weather, however acquiring UAV shots in addition to ground-level is absolutely vital for what we refer to as ‘the filler’. The Phantom 4 with its 12mp sensor only needed a few hundred photo’s but certainly good enough capture for the job.
One must have good spacial awareness, as with scenarios like this, we didn’t have time to align the photos on location. Always remember where you have been. It’s an art and you just get very systematic and efficient over time. Hilarious though, because I still routinely screw it up. You see, the experience was meant to paint a picture of precolonial days of Aotearoa, New Zealand, yet you can clearly see the markings of my Vans in the sand,, though an impressive feat of detail, definitely time period inaccurate. If you explore long enough, you’ll find a plastic bag or two, a bottle lid and a melted powerade bottle. Obviously we will remove these in post, something we couldn’t have done easily even months ago.
The Chieftain himself is actually located in Auckland CBD on Queen St and is strongly associated with local Iwi. We used a delighting approach via cross-polarization for this asset. Our techniques push out enough flash lighting power to delight objects even in the middle of a sunny day. The Chieftain is incredibly detailed, as stated above 750m points is a hell of a lot for an object of this size and you are far more likely to go cross-eyed before seeing any pixilation of texture.
Pre-production planning is a must, this critical dialogue will dictate the final desired outcome. You are constantly weighing up between realism vs interactivity. For instance, to scan a chaotic messy room with a multitude of items sprawled around the place is a far more convincing approach where realism counts, what we refer to as a single pass strategy. The bouncing of light, natural placement of objects, the perfection of imperfection as they say. However, this dramatically limits interactivity as all these objects are baked into the scene. You can go down the other route and individually scan each and every asset, though you can lose the soul of the scene and the process will take much longer, this same principle goes for lighting. Simply said, one must weigh up what their desired outcome is. The middle-ground is always preferred. This rationalization allows us to produce content in rapid succession, we achieve from a team of three in weeks, what other studios with a team of many would take months or even years.
Lighting is never a one trick pony, something we are still perfecting in many ways. With the use of cross-polarization and software post-processing,, we are able to completely remove shadows / AO / Cavity and get very accurate PBR extrapolation from small environments and assets alike, even crude roughness extrapolation. Large environments still require clever software processes and 360 lightsphere capture rather than physical delighting, and these methods have room for improvement. However, our proprietary techniques for removing specular highlights really does aid. Reintroduction of specular highlights in scene via our hybrid lighting techniques are truly a point of difference to the competition. We can also import perfect PBR based models into our scene and superimpose the lighting of the environment onto them. Luckily for a scene like this, interactivity is miniscule as you’re unlikely going to be moving large boulders. If we wanted a higher level of day / night cycle, we can do so via world normals, shadow lift or reverse mapping of lighting. These techniques are complex, however, you can easily find this information online.
Our water shader is a modified version of the community ocean project. This shader is beautiful, however, we are wanting to expand it to include the use of distance field based flow maps. What I have noticed is that many people think of water as blue. It’s not, extensive matching to the original photograph was how realism was achieved. The slightest shift in colour throws you off completely, especially in VR. I might not see detail very well, but the overall picture, that's one of my strong suits.