Richard Shackleton 28 January 2010
Just a few weeks ago, The Foundry announced that Sony Pictures Imageworks (SPI) and The Foundry agreed to enter into a reciprocal technology-sharing relationship, to mutually advance the state of the art in visual effects and digital production.
We’re really excited!
Not only have we gained access to the clever minds, technology and experience behind SPI's Katana software for 3D look development, lighting, rendering and compositing - we’ve taken another step on our journey with Nuke as the next generation of compositing and rendering tool and framework.
It’s a carefully considered step. Nuke has been pulled, squeezed, grown (and occasionally broken) during its journey from in-house tool at Digital Domain to Foundry product. We’ve taken big steps, like changing the UI and introducing Python scripting and we’re about to release significant new functionality including a re-written RotoPaint tool and new 3D Camera Tracker and Lens Distortion tools with Nuke and NukeX version 6.0.
Nuke’s 3D capability has help put it firmly in place as the staple of today’s compositing and visual effects tools. Compositing is no longer an A over B task. That is shown to us every day by the clever, often ‘not-as-the-design-intended’ application of Nuke’s 3D toolset to difficult (seemingly impossible!) compositing and visual effects challenges - always resulting in stunning pictures and new level of state of the art.
It is this 3D capability of Nuke and how it engages with compositing, we’re planning to strengthen and evolve. Access to the production-proven technology behind SPI’s Katana software will springboard our development plans for Nuke, specifically in 3D scene management, lighting and rendering.
But, the path is well lit. We won’t forget what the core of compositing is. If we do, we know you’ll shout at us very loudly.
Richard Shackleton
Head of Product Development at The Foundry
Q. What happened?
On the 1st November following many months of discussion The Foundry announced that Sony Pictures Imageworks (SPI) and The Foundry agreed to enter into a reciprocal technology-sharing relationship, to mutually advance the state of the art in visual effects and digital production.
Q. What does this really mean?
Most importantly, The Foundry has acquired the proprietary technology of SPI's Katana software tool and framework (if you don’t know what Katana is, stay with us until you reach
Q. What is Katana?). In the reciprocal part, SPI has acquired selected proprietary technology of The Foundry (including but not limited to the technology behind Nuke) together with a site-wide license to use commercial software tools produced and sold by The Foundry.
Q. Why did The Foundry and SPI agree to do this?
The Foundry and SPI are aligned on mutual plans relating to 2D and 3D compositing, lighting and rendering and how these might best fit together in the production pipeline of the future. The Foundry believes that access to the production-proven technology within SPI’s Katana software will springboard development plans for Nuke, specifically in 3D scene management, lighting and rendering. SPI and The Foundry share a view that a standardized and commercially available 3D compositing and rendering pipeline will be of great benefit to wider film and television production, seeding access to new capabilities and efficiencies currently not available to the majority and promoting availability of knowledge and expertise within our industry.
Q. Will The Foundry release Katana as a stand-alone product?
No, that is not currently part of our plan. Katana overlaps in many areas with Nuke (for example, it is an image processing/production framework and also a capable 2D compositor). Nuke will continue to be the leading compositor with a solid and supported future.
UPDATE: Our plans in this area have been revised - please see the blog entry.
Q. Will Katana appear as part of Nuke?
Yes, in a form yet to be determined. Selected technology and ideas behind Katana will be integrated into upcoming versions and optional components of Nuke. The Foundry aims to show the first results of integrating Katana technology in Nuke during 2010, followed by significant new 3D scene, lighting and rendering capability in 2011.
Q. That all sounds like a big change. I like The Foundry and Nuke, should I be worried?
No, this is a good thing and both The Foundry, SPI and many of our customers are very happy about it. The agreement involves no company ownership either way, nor any loss of jobs or reduction of any R&D staff. SPI staff will aid The Foundry in integrating the technology and The Foundry will add resources to the Nuke team to ensure core development continues. SPI will not get special or early releases of Nuke and there is no corporate special arrangements that are not fully open to any other major customers of The Foundry worldwide.
Q. What is Katana?
Katana is a production proven software tool and framework developed at SPI to aid artists in 3D look development, lighting, rendering and 2D compositing. Katana handles everything needed to complete a scene after the 3D modeling and animation is done and any processes are simulated. This includes 3D scene assembly, interactive look development, lighting and management of final quality 3D rendering. Katana has been in development at SPI since 2004. It has been used on almost every film at SPI since to deliver live action with CG visual effects (on productions like ‘Spiderman 3’, ‘Watchmen’ and ‘G-Force’) and CG-only animation productions like ‘Cloudy With A Chance Of Meatballs’. Katana has been developed by a competent software team at SPI who have achieved a remarkably polished tool considering it’s ‘in-house’ development. It shares many similarities with Nuke (for example, the nodebased graph interface and Python scripting engine). It is currently deployed within SPI on the Linux platform (although it can be cross-platform and development has taken that into account).
Q. Can I see Katana running?
Not today, although it is planned. Katana is integrated into the production pipeline at SPI. We will be talking with customers and exposing more visual and written information about Katana and our plans as we enter the 2010 New Year.
Q. What is Katana, in a bit more detail?
Katana is a single environment for 3D scene assembly, look development, lighting and final rendering and compositing. This allows artists to work interactively and iteratively on individual adjustments (for example a 3D character or lighting rig), in the context of the final image. The final image may contain several 3D and 2D CG and live-action elements, composited together. Katana provides a highly structured and collaborative framework, capable of addressing an entire show’s 3D rendering and compositing needs. For example, it allows the creation of baseline ‘look’ standards such as a specific mood lighting setup or character shader style, freeing end artists to concentrate on derived and specialist looks and finishing touches. They achieve this by inheriting and overriding standard attributes, rather than having to reproduce broad strokes and continuously manage consistency. The underlying complexity of assembling and rendering a scene is abstracted allowing artists to focus on creative interaction. This allows interaction with large and complex environments on demand, using principles such as selective level of detail and automatic update of rendered results. Only when the artists need to work with individual shapes and surfaces are the full resolution details loaded and processed and the rendered results brought back into the final composite. Efficient re-rendering uses incremental scene changes, saving artists valuable time and resulting in quicker change iteration and a more explorative process.
Q. What does Katana use for 3D rendering?
Katana does not have its own 3D renderer. The framework supports tight integration with commercial 3D renderers (such as Pixar's Renderman) and extends to other rendering solutions (such as the Arnold ray-tracing renderer, as used at SPI). Katana manages dependencies between render invocations and render-specific passes (such as shadows, effects, reflections and occlusion passes), necessary to complete the final image.
Q. Is Katana a 3D renderer and a 2D compositor?
Yes. Katana manages both 3D rendering of scenes and more traditional 2D compositing of render passes and 2D image layers. In this respect it is quite similar to and a good fit with Nuke. Katana supports integration of 2D and 3D workflows. Individual light passes from the 3D renderer can be rebalanced using traditional compositing operations (as familiar to many Nuke compositors today). Effects can also be a combination of 2D and 3D techniques. For example light rays rendered as volumetric 3D in an external renderer when near the camera, switched dynamically to use 2D compositing ‘cheats’ when further away. Or, the entire scene can be a combination of externally rendered 3D scene elements. A key Katana strength over Nuke is its assembly and management of large 3D scenes and associated lighting and surface attributes, and the tight integration with multiple external 3D renderers.
Q. Katana’s interface is node based, like Nuke?
Yes, Katana provides a procedural node-based approach to working, allowing artists to craft scenes of arbitrary complexity (a ‘recipe’). It also supports the creation of ‘Super Tools’ (similar to Nuke Gizmos) able to present a different interface mode if required (for example, a layer-based lighting interface, as preferred by some VFX supervisors and artists). Katana’s core currency of operation is the 3D scene graph, a collection of uniquely scoped attributes describing the scene to be rendered (3D objects, surface attributes, lights and cameras etc.). Node operations support concepts such as defining dynamic collections of scene attributes, direct modification, inheritance, override and the definition and pre-baking of template standards to optimize the application of large numbers of processes.
Q. I’ve heard Katana can handle very complex scenes?
Yes, handling complex scenes is one of Katana’s key strengths and something proven during production at SPI. Procedural operations are defined to address and modify arbitrary collections of attributes in the scene(for example, a surface property of all windows above the 3rd floor on all buildings facing northward). Evaluation of operations on the scene graph are deferred until render time and called on demand (for example only those elements needing evaluation will be invoked for given camera frustum or traced surface ray). This supports rendering of incredibly complex environments (such as an entire cityscape) that would otherwise be difficult to process due to memory or other constraints.
Q. Why is Katana and a future Nuke based on it so promising?
As we see computing power increase, so does the expectations of writers, directors and audiences. Production iterations, mistakes and decisions to change have a significant cost, especially with increasingly complex and multi-part rendering. Currently ad-hoc pipelines and disparate tools often require a need to iterate through separate departments, resources and production pipeline stages to complete the final image. Decisions are often made out of the context of the final deliverable. Katana offers efficient, traceable and repeatable ‘total scene’ processes (the scene-graph, shader, lights, 3D rendering and compositing), and puts decisions in context.
Q. Why is doing things in context - 3D and 2D in a single place so good?
A typical high-budget feature-film will involve the combination of live-action camera footage with CG material. The live-action and CG elements need, ultimately, to occupy the same visual space. Final overall composition is accentuated by lighting and how an object relates to other elements and its environment. Attention, scale, mood, depth and continuity are only a few of many considerations to ensure elements work together as a coherent whole and deliver on the director’s intent. That involves pre-vis, staging, look development, lighting and compositing. Knowledge and control of 3D in the final shot can influence artistic, technical and resource decisions (for example crop regions and render efficiency). This can result in increased creativity and higher quality with reduced time/cost. Managing look decisions as metadata and creating template ‘standards’, for a single shot, group of shots, scenes, characters and/or moods introduces control and efficiency throughout the production. Katana facilitates collaboration and consistency between artists and shots, from large team sequences to one-off cases.
Q. SPI and The Foundry seem to believe this is the future, but does everyone?
Some people don’t see it. It’s a bit like the day compositing moved from just 2D layers to include 3D projections - not everyone got it, some still don’t. Those that did, and now it’s the majority, have never looked back. They are producing more creative, higher quality work, more efficiently, using a wider range of techniques. Look development, lighting, 3D rendering and compositing have traditionally been separate tools, pipelines and people. Most large animation companies have combined their lighter/compositor positions together. Some VFX companies have also done this. The Foundry believes in the compositor and lighter working closer together. We often refer to them as ‘hybrid artists’. This new generation of artists are cross-functional, with a much wider appreciation of both 2D and 3D. To continue and support this shift requires frameworks and tools to abstract the complexity, so non-expert users benefit from previously large-studio restricted techniques.
Q. What happens next?
There is an industry wide move towards choosing standards, and streamlining process. We believe there is considerable scope now with Nuke and SPI’s Katana technology to design a new connective toolset providing a single coherent platform for scene assembly, look development, lighting and final rendering and compositing. Nuke ‘with’ Katana development will be staged, releasing increasing capability incrementally over time. The Foundry aims to show the first results of Katana technology in Nuke during 2010 followed by significant new 3D scene, lighting and rendering capability in 2011.
Q. Oh man! You are going to screw up my basic core compositing tool aren’t you?
No, we won’t. The Foundry is very aware that Nuke as an efficient and flexible compositor must not be sacrificed, by considering compositing, lighting and 3D rendering together.