Nuke Camera Tracker - now in stereo!
Latest News
Jon Starck & Lucy Wilkes, 20 May 2009
As you saw in the last blog entry, we've been working on a new camera tracker for Nuke. Some of you might even have been lucky enough to get a sneak preview at NAB. For those of you who haven't seen it yet and perhaps haven't used one before, the camera tracker finds features in an image sequence taken by a moving camera and then tracks them through that sequence. By examining the paths taken by the features it can calculate their 3D locations and the path taken by the camera. Now, camera trackers are not new, but the advantage of having this all within Nuke is that it makes it very easy to manipulate the input and test the result in your comp. For example, prior to tracking you might want to put a bezier on the input to matte out moving objects, use Primatte to automatically matte a region or regrade so the tracker will find more features in dark regions. After tracking, the prototype camera tracker gives you a point cloud corresponding to the positions of the features it tracked in 3D space and a Nuke camera keyed to each frame of the sequence. It's straightforward to add these into your 3D scene, so unlike when using an external tracker you don't have to switch applications, export and import 3D data in order to get it all in the same place. Also if anything's not quite right you can tweak the tracks or the solve without having to go through the whole export, import, export, import cycle again.
Anyway, since that work is now progressing nicely our researchers have started to look for new challenges to keep them entertained. Now that we can reliably track and solve a sequence from a single camera, it seemed logical to move onto tracking and solving for multiple views of the same scene, starting with stereo views. There are two ways we could have done this: the first would be to solve for the left and right cameras separately, then combine the 3D data we get from them at the end. However, we decided it would be nicer to solve for both cameras at once, so at each point the solve is using all the data available to it from both views, which in theory should give a better result. This fits in nicely with Nuke's existing stereo workflow, where the left and right views can be treated as a single input stream.
We found some great footage to test this on - not great in the sense of being easy to track, but great to use as test footage because it displayed many of the problems that our stereo tracker would need to be able to cope with. Here are two views from the sequence:
In this example, you can see that the cameras are tilted and the colours don't quite match between the views. What's more, when we tracked and solved the sequence we found out that there was actually a frame offset between the left and right views - something we hadn't spotted at first. When we looked at the tracked results and saw how the cameras moved in 3D, it became obvious that there was a time-lag between the paths of the left and right cameras.
However, since we were still working in Nuke and therefore had a plethora of compositing tools at our disposal, this was easy to fix - we simply dropped in a TimeOffset node to get the views back in sync and then tracked again. This time, since the input streams were now properly aligned and we knew they had come from a stereo rig, we were able to use constant interaxial distance and constant interaxial convergence as constraints in the solver to get an even better result.
