Skip to content

In Depth

The basic idea behind deep images isn’t new – storing more data than just the standard RGBA frame buffer has been a standard workflow for some time.  As mentioned in the overview – what deep images try to solve is how this data is stored and referenced to avoid standard issues like edge aliasing etc.

2010 emerging from animal logic’s deep image implementation a paper was submitted to Siggraph, but got rejected. For anyone who wants to read up on it you can find a copy here.

Creating Deep Images

The initial implementation of the deep image tool set leverage’s Pixar’s deep shadow map as the file format due to ease of implementation and adoption within a production that was currently underway.  Although originally intended to be used as a way of providing more detailed shadows for the rendering engine ( see paper ) the addition of a deep shadow map pass as a secondary output for the hero camera is effectively capturing a major subset of the information needed. Houdini`s version of these kind of files, called RAT files, was used as well coning from this package. Though successful as an easy start for the intended goal we will discuss later why a custom format emerging from this pool of people would benefit the community. Implementing this is trivial and works pretty much ‘out of the box’. Initially the size of the files generated are much larger that what’s acceptable for CG production and become unmanageable, but different sorts of compression let the files come down to a reasonable size for the gain in flexibility. Compression is a huge part for the new file format.

Reading Deep Images

Currently there’s still a reliance on a proprietary, though documented, format, Pixar’s deep shadows, and the only way to read this format is to wrap up the dtex library that’s provided with Photorealistic Renderman. SideFx houdini has a similar file format for its deep camera maps, but has the same problems.  To get around the issues with those api`s such as threading, use of a license, etc.. we try to convert everything to a custom data format.  Ultimately the goal is to define and share an open format, or an extension to one already widely adopted, like OpenEXR. This format hast to encapsulate the requirements outlined below while allowing for future expansion of the concept.

Data Storage

Saving the deep data must ensure one important aspect. The date must be easily re-constructable at any arbitrary point in depth. When generating accurate hold out mattes the values must be accurate at any depth the holdout object intersects. The data can be stored in a multitude of ways, described in the various similar formats that currently exist. Compression is also vital for the success of the storage as the deep data is vastly bigger as the regular image data. Also compression is explored in the file formats out there. The trade offs are as always quality against data size, and ease of access versus elaborate data storage. We want to drive this process to collect as much requirements as possible and implement the new industry standard for deep image storage. This can be done building on existing packages like OpenEXR or hdf5.

These images show different storage types as graphs of value over depth.

Compositing operations

As a start we concentrated on the basic use of this additional data, to overcome traditional problems with depth buffers. Semi transparent edges cause big problems for these buffers with only one sample per pixel. We can hold out two different renderings against each other without any loss of quality on the edges.  This also means the order of operations aren’t as important – traditionally the order of compositing determines the look of the final image as each layer is composited in succession. As deep images contain this data the only determining factor for ordering is the initial setup in the 3D environment. As each sample can be seen as a solid piece of mater or as a block of density volumetric and hard-surface holdouts can be archives in a very simple manner. Also merging of deep images is trivial and does also not require any order of operations.
The other main compositing operation that benefits from the data is any kind of convolution over the depth of the image. Z-Blurs and Z-depth based defocus filters can be applies with greater accuracy around the edges of objects.

Rendering operations

The final data can be brought back to into the traditional 2D compositing workflow by integrating along the depth and combining ( “rendering” ) the samples. As the engine integrates through the samples depth shaders can be applied to offer additional effects, ie. atmospheric fog. This adds de-saturation and color shift over depth to the image and simply calculated – even offering the opportunity to project the sample into “world space” using embedded perspective and world space transformations for allowing even more detailed effects.

Next steps

As these data structures and the way we can access them evolve the possibility to enhance and expand on traditional compositing operations offering more accurate information about the pixels and elements which contribute to the overall appearance of the image.  Eventually this should also limit the number of times an element needs to be filtered as the entries in the pixel structure inherently encapsulate the filtered result produced by the rendering engine.
Ideally we’d like to offer a toolset to read/write and provide an initial subset of the operations one would need to utilize deep images within and production workflow.  As most modern compositing applications are developed around the concept of 2D layers it will be up to those software vendors to adopt these data structures internally to offer a more integrated solution.
Looking towards the needs of Stereoscopic productions, being able to encode and store disparity maps and other such data sets “at depth” could make resolving issues surrounding the stereoscopic workflow more straightforward.

Final thoughts

VFX workflows are extremely organic and as demands increase and the need for more flexibility is apparent – this has never been so visible than now with stereoscopic productions becoming more popular we have to re-think every aspect of our pipeline and come up with new solutions to make our life easier. Deep images help immensely in the compositing process, we can share more information from the 3D to the 2D part of our production and blur the lines more and more to create the best image possible obeying tighter and tighter deadlines