Katana - FAQ

Richard Shackleton 28 January 2010

Let there be light!

Just a few weeks ago, The Foundry announced that Sony Pictures Imageworks (SPI) and The Foundry agreed to enter into a reciprocal technology-sharing relationship, to mutually advance the state of the art in visual effects and digital production.

We’re really excited!

Not only have we gained access to the clever minds, technology and experience behind SPI's Katana software for 3D look development, lighting, rendering and compositing - we’ve taken another step on our journey with Nuke as the next generation of compositing and rendering tool and framework.

It’s a carefully considered step. Nuke has been pulled, squeezed, grown (and occasionally broken) during its journey from in-house tool at Digital Domain to Foundry product. We’ve taken big steps, like changing the UI and introducing Python scripting and we’re about to release significant new functionality including a re-written RotoPaint tool and new 3D Camera Tracker and Lens Distortion tools with Nuke and NukeX version 6.0.

Nuke’s 3D capability has help put it firmly in place as the staple of today’s compositing and visual effects tools. Compositing is no longer an A over B task. That is shown to us every day by the clever, often ‘not-as-the-design-intended’ application of Nuke’s 3D toolset to difficult (seemingly impossible!) compositing and visual effects challenges - always resulting in stunning pictures and new level of state of the art.

It is this 3D capability of Nuke and how it engages with compositing, we’re planning to strengthen and evolve. Access to the production-proven technology behind SPI’s Katana software will springboard our development plans for Nuke, specifically in 3D scene management, lighting and rendering.

But, the path is well lit. We won’t forget what the core of compositing is. If we do, we know you’ll shout at us very loudly.

Richard Shackleton
Head of Product Development at The Foundry

Here are some answers to some Frequently Asked Questions about Nuke and Katana.

CVMP - European Conference on Visual Media Production

Simon Robinson, 14 November 2009

This week we're all recovering from The CVMP conference.

For the first year ever, we all took a gamble and ran the conference outside the IET, chosing instead to sponsor the event down at the BFI in London. And, even with our humble hats on, it really was a great success. CVMP aims to bring together academics and professionals working in all areas of visual media production - what one group learns from the other benefits us all.

There were lots of UK-originated highlights this year; but for contributions from outside the UK we were particularly fond of the key-note contribution from the legendary Professor Marc Pollefeys, and the talk given by Peter Hillman of Weta Digital. Peter Hillman dove into the intricacies of the stereo post-production pipeline on Avatar, still leaving us with the distinct feeling that he was protecting our little brains from the enormity of what Weta have pulled off.

Peter Hillman talking at the CVMP conference in November 2009.

So now we're booked in again for 2010, same time of year, same place. Our very own CVMP-organising chief, Abi Bowman, is clearly a sucker for punishment.

A Researcher let loose in Amsterdam

Jon Starck, 16 September 2009

Well I had a fantastic time at IBC this year - it's nice when they let you out of the office. I was even allowed to be near customers!

I only had one day and ended up spending the whole time at The Foundry stand. It's a friendly place to be and there was just so much interesting stuff going on. There were a couple of highlights - seeing the reaction to some of our new research technology was fantastic and chatting with customers about some of the Nuke production work gave us some great ideas for new plug-ins.

Simon (our Chief Scientist) and Ben (Demo Artist extraordinaire) were showing off the 2D to 3D technology we are working on in our i3dLive research project. In i3dLive we're generating stereo views from one or more non-stereo cameras. Ben and Simon tracked a single view sequence using our new CameraTracker for NukeX, calculated depth using a DepthCalculator plug-in we're working on and synthesised a stereo camera in Nuke's 3D environment. The cool trick is that once you're rendering using a 3D camera you can do 3D camera stabilisation as well as control interaxial convergence and separation to create a perfect stereo view. Neat.

Blenheim Palace, stabilised 3D stereo camera created from a single camera shoot.

Blenheim Palace, stereoscopic image generated from the 3D stereo camera.

Shervin Shoughian from Image Engine did a great demo of their work on District 9. They had some great tricks for image relighting using point geometry. Take a CG position pass (an XYZ render), convert it to point geometry and interactively update lighting effects in your comp by moving lights around in Nuke's 3D environment. Relighting is one of the problems we're looking at in our i3dPost research project. So thanks to the Image Engine guys and gals for sorting that out. They also showed some nice work on generating clean plates in tracked sequences. Wouldn't it be handy if there was a plug-in to automatically calculate a Nuke Camera for all the reference images they had?

So, as always, a hundred and one ideas on cool stuff to add to Nuke. It really is handy talking to clients and chatting through the possibilities.

EuroPython

Goncalo Carvalho, 30 June 2009

The Foundry took a trip to Birmingham this week. Or at least I did. Birmingham is a lovely city, quite lively and the Broad St area is nice to walk around. Oh, and there was a Python conference too.

EuroPython is a community run event about all things Python. The Foundry had a presentation on Python in VFX and Nuke's python bindings implementation. For all those who missed it, the talk went through an overview of Python in VFX and then some of the implementation ideas in Nuke. I also tackled some shortcomings of the current implementation and addressed how they can be improved. More specifically, some topics covered the use of custom UIs inside Nuke using 3rd party GUI toolkits and the node graph traversal that can simplify some tasks. Being a Python conference, there was Python code, and you will have it available in the next release of Nuke. The conference itself is buzzing with Python. There are plenty of talks and attendees from all over the world. There's a lot of knowledge going around and many of the talks are applicable to Nuke's bindings and The Foundry's workflow, which only adds to the whole experience. There is also a real sense of community with everyone being really helpful and quite social. I learned a great deal at the conference and look forward to incorporating it into forthcoming versions of Nuke...

Dirty Secret

Simon Robinson, 29 June 2009

Dan Alderman, Systems Manager

It's time we confessed. All this time we've been running our main London office internet with a heavily contended 2Mb SDSL line. Did you notice? To our credit, there was a point early on in our recent growth where we knew this wasn't going to work. So we put in an order for an uncontended fibre link. Or at least poor Dan did. Meet Dan (left) who keeps our systems ticking at 1 Wardour Street (below). It took Dan a year and a half to get us off the red cable he's clutching, and on to the other.

As Dan puts it: "18 months of phone calls, meetings and paperwork; 3 visits to dark, dank greasy basements; numerous conversations with property managers, landlords, BT technicians, solicitors, surveyors and ISP account managers (one left to go and have a nervous breakdown half way through) and we're there."

Dan's now got the cable up and running, from street level through KFC's kitchen and up to our offices. He's not going to do it again. But now you can email us more. Lots more.

The Foundry's London office, 1 Wardour Street, conveniently located above KFC.

End of an Era

Bill Spitzak, 05 June 2009

I have been offered a new position in Los Angeles at Rhythm and Hues and, after a good deal of consideration, I have decided to leave The Foundry and take up their offer.

Working with The Foundry engineers has been great, both as part of a team and on an individual level. I have also very much enjoyed staying in Europe for the first time, but the new position I have been offered will also be very interesting and is located where I live. I have enjoyed working remotely with the team, but the huge distance and 8 hour time difference has proved difficult. The Foundry's head office and the Nuke development team are located in London and, for various reasons, moving there was sadly impractical for me.

I have worked on Nuke for an (excessively :-) ) long time, and am looking forward to being a Nuke user myself. I will be working more directly with film production which is something I enjoyed. I will continue to contribute to the Nuke community and will certainly be following all development and news and answering questions on the mailing lists.

The Foundry has built an excellent team in London, who are responsible for all the new features of Nuke, such as the upcoming Roto/Paint system. I am confident they will progress Nuke in ways I have not even imagined and that many exciting things are in store for users.

I did not realize how hard it would be to leave everybody, and I am really looking forward to see what The Foundry does next.

I am looking forward to seeing you all again the next time I visit London and vice versa when you visit LA.

Thanks,

Bill Spitzak

Bill Spitzak & the rest of the staff at The Foundry's London office, 4 June 2009.

HPC Team is go!

Jay Cornwall and Bruno Nicoletti, 27 May 2009

The Foundry has formed a High Performance Computing (HPC) team to oversee the integration of hardware acceleration technologies into our products. Our goal is to improve the performance of The Foundry's visual effects plug-ins and compositing software, creating a smoother compositing experience and enabling effects with greater complexity and accuracy. In prototype research we have exploited high-throughput CPU and GPU technologies to deliver an order of magnitude performance improvement across two visual effects. Ongoing development of this work will lead to faster products for our customers' existing hardware, and create new cost-efficient hardware upgrade paths.

The focus of our initial work has been on graphics processing units (GPUs). GPUs have evolved from fixed function 3D rendering accelerators to massively parallel computational powerhouses. Their price, performance and power characteristics have all scaled much more favourably than traditional CPU architectures. The evolution of compute-oriented programming APIs - beginning with NVIDIA's CUDA, and more recently the cross-accelerator OpenCL standard - has driven the interest of many computationally intensive industries in GPU acceleration. OpenCL represents a step change for generic GPU programming, in the same way that OpenGL did for 3D programming.

Among the challenges we face in this venture lies the GPU's massively multithreaded programming model, which is very different from the serial CPU programming model (with a sprinkling of multicore parallelism) which we are used to. We have had to rethink and rewrite much of our base architecture to scale from a handful of threads to the tens of thousands needed to satiate the sprawling computational units of a modern GPU architecture. As a side effect of this work we hope to improve the scalability of our software on multicore CPU platforms as well; particularly those with eight or more cores.

Throughout the early stages of our research, in collaboration with the Software Performance Optimisation group at Imperial College London, we increased the performance of Furnace's F_DeGrain to beyond real-time on HD footage with an iMac's GPU. This accelerated plug-in remains unreleased while we work through deployment issues, but it is a good example of the performance gains we stand to make.

We are now committed to the OpenCL standard and our future products will build heavily upon it. AMD, NVIDIA and Intel have all pledged support for OpenCL in their ATI Radeon/FireStream, GeForce/Quadro and Larrabee accelerator products respectively. Apple's next release of OS X, Snow Leopard, ships with an OpenCL implementation for AMD and NVIDIA GPUs. NVIDIA has also begun distributing a beta OpenCL implementation for the Windows and Linux platforms. With this rapid progress in compute-oriented hardware and software many industries are experiencing a sea change in scalable programming and software interactivity. The Foundry's HPC team stands ready to lead the performance charge in visual effects!

Nuke Camera Tracker - now in stereo!

Jon Starck & Lucy Wilkes, 20 May 2009

As you saw in the last blog entry, we've been working on a new camera tracker for Nuke. Some of you might even have been lucky enough to get a sneak preview at NAB. For those of you who haven't seen it yet and perhaps haven't used one before, the camera tracker finds features in an image sequence taken by a moving camera and then tracks them through that sequence. By examining the paths taken by the features it can calculate their 3D locations and the path taken by the camera. Now, camera trackers are not new, but the advantage of having this all within Nuke is that it makes it very easy to manipulate the input and test the result in your comp. For example, prior to tracking you might want to put a bezier on the input to matte out moving objects, use Primatte to automatically matte a region or regrade so the tracker will find more features in dark regions. After tracking, the prototype camera tracker gives you a point cloud corresponding to the positions of the features it tracked in 3D space and a Nuke camera keyed to each frame of the sequence. It's straightforward to add these into your 3D scene, so unlike when using an external tracker you don't have to switch applications, export and import 3D data in order to get it all in the same place. Also if anything's not quite right you can tweak the tracks or the solve without having to go through the whole export, import, export, import cycle again.

Anyway, since that work is now progressing nicely our researchers have started to look for new challenges to keep them entertained. Now that we can reliably track and solve a sequence from a single camera, it seemed logical to move onto tracking and solving for multiple views of the same scene, starting with stereo views. There are two ways we could have done this: the first would be to solve for the left and right cameras separately, then combine the 3D data we get from them at the end. However, we decided it would be nicer to solve for both cameras at once, so at each point the solve is using all the data available to it from both views, which in theory should give a better result. This fits in nicely with Nuke's existing stereo workflow, where the left and right views can be treated as a single input stream.

We found some great footage to test this on - not great in the sense of being easy to track, but great to use as test footage because it displayed many of the problems that our stereo tracker would need to be able to cope with. Here are two views from the sequence:

In this example, you can see that the cameras are tilted and the colours don't quite match between the views. What's more, when we tracked and solved the sequence we found out that there was actually a frame offset between the left and right views - something we hadn't spotted at first. When we looked at the tracked results and saw how the cameras moved in 3D, it became obvious that there was a time-lag between the paths of the left and right cameras.

However, since we were still working in Nuke and therefore had a plethora of compositing tools at our disposal, this was easy to fix - we simply dropped in a TimeOffset node to get the views back in sync and then tracked again. This time, since the input streams were now properly aligned and we knew they had come from a stereo rig, we were able to use constant interaxial distance and constant interaxial convergence as constraints in the solver to get an even better result.

NAB

Simon Robinson, 29 April 2009

It was clear from the quiet on the NAB show floor that there was something afoot. Is there a recession on? Thankfully, the attendance at The Foundry stand was more buoyant than ever. For us, and other exhibitors we spoke to, the decision-makers were still there spending money and the quieter atmosphere was nothing but a bonus.

As with many vendors, we ponder the need for these events and weigh them up against alternatives, every show, every year. Are the big shows relevant? Will some of our favourites implode as more and more names occupy satellite positions in hotels outside the show proper? Witness the cancellation of this week's VisMasters DMVC in San Francisco ( http://dmvc.vismasters.com/ ) and rejoice that we're in the film business and not in the conference business. (Oops - forgot - we're sponsoring and organising CVMP in London later this year - see http://www.cvmp-conference.org/. Pressure on).

But we secretly love the shows. We tell our families and friends it's a chore, but the chance to meet that number of customers in one week and to see our colleagues in a new light is nothing but an uplifting experience. We're small enough that it's still fun, although we did spot this year that the stand now needs too many staff for us all to fit in the same stretch limo in the evenings – even though 8 seems an unreasonably low capacity for such a monstrous beast. A small burden we bear for you as part of the bigger picture.

For those of you who missed us, we built a card environment in Nuke on the show floor, using the new camera tracker prototype. You can download the files, roll your own and pretend you were there. Also keep an eye on the homepage for upcoming links to some of our customer demos.

Rolling Shutter

Simon Robinson, 17 April 2009

Here's a little something we've been working on in our spare time - correcting for CMOS rolling shutter artifacts. We've noticed that using the phrase "CMOS rolling shutter" in a conversation gets you one of three responses:

• Huh?
• Not really an issue is it?
• It has ruined my life.

For the Huh people, I'd recommend doing a search on the phrase using your favourite search engine. To save you time, though, here's an overview:

Modern digital cameras focus the image on a sensor chip. These chips fall into two categories, CCD sensors and CMOS sensors. As with most things to do with cameras, there is no right or wrong answer for the 'best sensor'. Here we're concentrating on one of the issues sometimes raised by using CMOS sensors.

Many CMOS cameras have a "Rolling Shutter". This means that the image is extracted from the sensor one scan-line at a time, leading to a time lag between the first scanline and the last. The alternative is a "Global Shutter", where the whole image is extracted simultaneously. Luckily for our experiments, my Sony consumer HD camera sports its CMOS label prominently, and has a rolling shutter lag to make me proud. So with no expense I can demonstrate the lag effect using this panning shot.

The shear on the image is caused by the bottom of the image being recorded at a later time than the top. Not so much of a problem when the camera pauses at the end of the pan, but very obvious in the middle of the pan.

Note that correcting for this is not as trivial as it first looks. Firstly the shear varies over time. Secondly for more interesting shots it's not always a shear nor is it the same all over the image, since it's entirely dependent on the relationship between the camera's motion and that of the objects in the scene. A vertical move, for example, leads to stretching or compression effects. But we've come up with an algorithm which is plausible fix a lot of the time. Here's a correction for the pan shot:

Pan shots are good for calibrating the lag, since we can manually adjust the correction algorithm to correct the things we know should be right-angles: the windows. For this camera, the lag is almost exactly one field. [ Note that I've simply dropped fields from the input to give us non-interlaced footage for this example ]

Once we've done that we can look at a more ambitious shot, with the same camera setup:

The correction for this shot then looks like this:

Having got this far - you'd be excused for not really being able to tell the difference on a shot at this speed. I can't really spot it. The Wendy House is slightly more wobbly in the uncorrected case - but is it enough for us to care?

Well, yes. For our current work, we now have the advantage that the objects in the scene can be made to have a reproject consistently over time. They are then easier to track in 3D, and easier to generate geometry for in 3D - two of the Research Team's pet long-term topics. It turns out that if you're interested in extracting dense data from moving images, it's nicer when everything in the world stays the same shape.