Session Use of Color Separations for Preserving Visual Metadata Information in Digital Intermediates
Presenter Josh Pines
Technicolor Digital Intermediate

Jim Fancher
Thomson Corporate Research

ABSTRACT

The latest film-to-digital-to-film post-production process for motion pictures, the Digital Intermediate, represents a new problem for preservation, and perhaps also a new opportunity.  The data files which source these intermediates are often fugitive and difficult to recover or deploy by the time the film has entered the “library” phase of its existence.  Data files captured on LTO or DFT2 data tapes are sitting on vault shelves of production companies or in archives in untested form.  But these files do offer the possibility of recording out perfectly matched and registered black-and-white color separation masters at the same resolution as the digital intermediate, and these synthetic elements may comprise a part of a preservation regime for the Digital Intermediate.  The test material to be screened was scanned at 4k, processed at 2k and recorded back out to color separations at 4k, then recombined and printed conventionally. 

_______________________________

PRESENTATION

Josh Pines: I work for Technicolor Digital Intermediate, and I want to address some of the issues which relate to the engagement of traditional film and digital intermediate production processes.

Let’s start by looking at the way that traditional film works. We begin with the camera original negative. This original negative has a very, very high dynamic range in each of the red, green and blue color layers. This film is exposed and developed and it goes through a traditional timing process in the film laboratory. Every shot is timed or graded, that is color-corrected, to correspond to the aesthetic intent of the film-makers. Typically, they will say, “This scene needs to be darker,” or lighter or more blue or more green, and eventually every single shot is color corrected. This is an iterative process. There’s a first color correction and the result of that is an answer print. The director and the cinematographer view this print and they may say something like “That’s pretty close, but this sequence just didn’t work.” They make comments as to how they want the color modified, and then the adjustments are made to the printing lights and a second answer and perhaps subsequent answer prints are made. And these answer prints are made at a rate of one a day – the creative team screens the film and specifies changes, the timer modifies the printer lights on the timing tapes, the negative is printed and the new print is processed. So the turnaround time for each answer print is, for all practical purposes, about a day.

There’s always a time issue involved with a film production, so this process of refinement has a finite, terminal point, regardless of the degree of satisfaction. So at this point we have a final answer print, the final color correction. And since this is the old photochemical way of doing things, we will make interpositives and internegatives to service all of the requirements for distribution – first of all, those thousands of release prints (some of which have to be subtitled) – as well as providing mastering elements for telecine transfer that will be necessary for television, cable, and home video. The final color correction gets printed into the timed color interpositives and the internegatives made from them. Each of those internegatives is a printing element from which you may be able to make as many as a thousand release prints,

So this is the traditional process -- original film elements, a traditional color correction, which is done just with filtration, with simple red, green and blue filters that change the color of the light that exposes the film. And this filtration is the primary tool that is used to adjust the color in the answer print, and from the negative to the interpositive to the internegative.

As Phil Feiner pointed out, in the digital intermediate process, we take the same camera original negative and scan it into a computer as digital files, capturing, as much as possible of the full dynamic range, so we get the full latitude of the color information in the original negative.

Once the files are in the hard drive, we are able to do digital color correction. As with traditional film, this correction process may go through several iterations, but they are implemented instantly as you make them at the workstation, so the turnaround time, and the time it takes to see the changes you’ve made is substantially reduced. There are also many refinements in the area of digital color correction that you can’t do photo-chemically, so this step is now an important tool in the creative process of film-making, and most of the directors today seem to want to take advantage of it.

Let’s talk a little about think kind of film-making for a moment, because it is an important part of a complete system that we call digital intermediate, and it engages the contemporary production situation at many levels.

In a traditional production situation, an effects house would create visual effects – dinosaurs, let’s say – for a scene. These effects would be recorded out to film, and then the film would be composited optically and cut into the negative or cut in directly. This is a difficult and time- consuming process, and the primary tool for making these special effects match into the rest of the film was the process of photochemical timing we described earlier. With the DI, we can take digital files directly from the visual-effects facilities or the title houses and edit them as digital files into the continuity of the film. It becomes relatively easy to integrate optical and title components without loss of resolution and with much greater color stability and fidelity. It also becomes easier to integrate different formats, such as film-originated images and images originating as high definition digital video. And when it comes to the color correction session, it is much easier to experiment and to refine a look. Just by adjusting controls on a console, the director can try his ‘blue’ look, and see immediately whether it’s what he wants, and also see whether it works with the other segments of the film. He can easily abandon one look and try another, all within minutes. So this part of what used to be called the ‘post-production process’ actually becomes interactive and a key part of the creative shaping of the film. Of course having all this new freedom and control over the image can get us into trouble. Luckily, there’s that release date, though, and eventually, they do have to stop.

Now, what exactly is digital color correction? I’m only going to be able to touch on a few of the things we can do with digital color correction here. First, we can do standard primary corrections, much like what is done with traditional film timing: we can vary red, green and blue. We can also make ‘nonlinear’ changes; we can not only change the amount of red, green and blue in the image, but also change the contrast and gamma of each individual color independently of the others, which is very difficult to do photo-chemically. We can do secondary color correction; that is, we can actually isolate areas, like a wall, or a window, and so if the director upon viewing the scene realizes that a certain predominating color is not working for a scene, we have the ability to isolate and alter that color, to amplify or neutralize it or change it to another color altogether depending on the direction of the creative intent. In addition to cueing on color, we can draw or otherwise select specific areas that we want to affect. This is especially useful for certain kinds of scenes, for example a scene where the imagery outside a window might be too bright, blown out or distracting. We can draw an area around the window and make it darker. So digital color correction is able to address and correct specific areas in a frame, and carry this correction through the scene. Spatial corrections that were once possible only by means of expensive, time-consuming and often not altogether successful optical photography now become commonplace. Some of these functions include repositions, zooms, and defocusing certain areas. You know, when the director says “Oh, this is so sharp. I really want the focus to be on the actor and the actress, not on what’s going on in the background --could you defocus it?” These changes may be for technical or aesthetic reasons, but the point is they can be viewed, tested, and implemented in the digital files relatively easily and thus expand upon the range of choices a director has in bringing forth his vision.

Once the final color has been approved and the other elements of digital post production (titling, insertion of visual effects, etc) have been completed, you have digital files which contain the entire film with all of the changes and shadings and subtleties that the film-makers made and approved. And this data is recorded back out to film on a device such as the Arrilaser. And here, the process offers some interesting flexibility, because we can record out interpositives or we can record out internegatives. We’re not necessarily limited to the traditional ‘cascading photochemical printing process (negative to interpositive to internegative to print). We can shorten that cascade and improve the quality by outputting, for example, outputting printing internegatives, color separations, even subtitled foreign distribution internegatives. For most of the DIs that we handle today, make at least two printing negatives. One filmed out negative might be used to produce 1200 ‘first generation’ release prints for the high end “blue states” market – i.e., major city engagements, while the other might go through the IP/IN process, to produce secondary negatives for what we usually call the “red states,” venues with a lower quality requirement.

There are a range of stylized “looks” that are can be imposed on a film to give it an overall look, an overall color cast, contrast bias, a certain unity or consistency of tone that underscores or enhances the film’s mood. For example, recently, photochemical processes such as bleach bypass, or ENR, or the photochemical process used to create the look of the opening reel of Saving Private Ryan -- increased contrast with overall desaturation – have become popular. All of these photochemical looks can be emulated by manipulating data files in the DI process. As a director, you may want to achieve this look either by processing digital files or by actually photo-chemically altering the negative, but you can emulate and preview your look in the DI process regardless of how you physically implement the look. So once again, the DI process offers choices and supports the creative work of the film-maker.

The director can preview his experiments with a digital projector calibrated to emulate the look of a final print. And that is a lot of the “secret sauce” that each of these different facilities brings to the table – the ability to present a high-definition digital preview that closely emulates what the final film print will look like, and of course the ability to track and control color and other features from the completed DI data files through the film recording, processing, and printing stages to the final print.

The DI process offers another important feature. The final data files that are recorded out to film can also function as a source for digital HD “all media masters” – that is, as a source for downconversion to the traditional video formats (PAL, NTSC) but also for HD masters to serve broadcast, cable, satellite, VOD, and of course DVD mastering. Since the final approved color corrections and the director’s approved look is built in to the DI data files, all that is required is to correctly track color and other technical characteristics through down- or cross conversion, compression and encryption to ancillary media platforms. At least forty features were made last year, and it’s going to be between sixty and eighty this year. There are studios that have already said that within two years or so, it is very likely that every feature will be a digital intermediate. So this type of post-production is on the rise and is almost certainly here to stay.

The director’s like a kid in a candy store, this all sounds so great – we can do all this fabulous stuff in the DI system that we couldn’t do before, and while it’s not exactly free, it’s so easy and so natively part of the system and so intuitively right creatively speaking. What could possibly be wrong with this picture?

Well, guess what? Every single one of these remarkable, subtle aesthetic moves that we’ve created is stored in what I call “vendor-specific digital files.” Methods are not standardized from vendor to vendor, nor is there a target standard for DI data. Critical components of this system in both hardware (telecines, color correctors, film recorders) and software (data files, graphics and rendering programs, etc) are subject to modification without documentation. And each of these components becomes obsolete, is upgraded or replaced, and is generally not modified in a “backwards-compatible” fashion. When the new release of a particular component comes out, all of a sudden, those lists which contain all the information about how the original source material was modified and how the final product is meant to look won’t mean anything.

The color-correction lists for the first project that we did at Technicolor DI are already “Latin.” They mean nothing unless we reinstall a previous vendor’s color corrector, reinstall the operating system that runs it, and reinstall the version of their software that was running when made those files. So what seems to be a brilliant advance in the production of moving image is a total disaster when it comes to restoring the original achievement – that is, when it comes to preservation.

So with that happy thought in mind, let’s turn back to our archival concerns. Once again, our chart [REF: PowerPoint slide] outlines the way we archive a traditional film production as compared to a DI production.

We have our camera original negative, and through the timing process we eventually arrive at a final decision on the look of the film, and that look is represented in timed interpositive, which contains a record of the final aesthetic intent. From that original negative we also make black-and-white color separations – yellow, cyan and magenta color records as captured on black and white film. And if the original negative should fade, we can recombine these three color records to a new color internegative. Since the separations were made from the original negative, we will have to apply that same color correction. And from that modified color correction, we will be able to make a new timed interpositive or a restored print. Ideally, this represents the basic element set of tradition preservation; this is what you put in your cold vault for the future.

Why do we want to make separations? Because, as Phil so cogently noted, they last forever. Black-and-white film doesn’t fade like the color negative will, and polyester film has an estimated lifespan of hundreds or even thousands of years if store properly. There are bad things that can happen, if these separations are improperly made or improperly stored. But today, the black-and-white separation master has the longest life of any of the film elements we make.

So traditionally, these two element sets are critical for image preservation. But there’s something missing from this strategy, because along with these preservation elements, we need a realizable description of the artistic intent. What does this mean for films made by the traditional photochemical process? It’s a record of the “printing lights,” the instructions that control the printer with respect to how much red, green and blue light must be used to correctly expose a new print or interpositive, or how much light must be put through red, green and blue filters in order to correctly expose the separations in the printing of a recombined internegative. And how is this critical data stored? Normally, these rolls of punched paper are stored in the film cans with the original negative. Actually, it’s the oldest successful digital archiving medium known – punched tape with perforations which represent the red, green and blue printer lights. It still works, and that’s unusual for digital data. But if I look in the cans of the original negative of a film from the early 1960s – say, “From Russia, With Love,” -- I can still use those paper tapes which represent the timing lights. My point here is that in a traditional preservation scenario, along with the physical elements, we are able to archive critical data which informs us on how to use those physical elements to recover the original values of the film.

Now let’s turn to the digital scenario, again. The question of archiving digital data itself -- the ultimate source of the film, is very difficult, and although I’m not going to address it here, the issues concerning preservation of this data will be covered in other sessions at the JTS. For preservation elements, we have a timed interpositive that’s filmed out from the final data files, and we have digitally generated black-and-white color separation masters from the same source.

There are several different options being discussed with regard to the filmed out color separations. They can be either negative or positive, and they can be created as three separate reels, or as a sequential exposure element, with the red, green and blue frames alternating on a single reel of film. This method tends to decrease the problems of recombination which are due to shrinkage. We’ve found that filming out negative separations results in a more difficult recombination, but is more effective as a source if you want to scan the film elements back to digital. Our experimental knowledge of these methods is as yet incomplete, but in the final analysis, the physical elements that will be produced to archive a DI production will probably be slightly different from those used to protect a traditional production.

Now that we have a context for thinking about the coming shift in preservation methodology as a result of the rise of the DI, I want to introduce a concept for the tracking of critical data regarding the artistic intent of DI films.

After digital color correction, there’s the final film recording of the whole movie. We are proposing to film out what we call the “road map elements. In addition to the whole movie, there is a “proof print,” or “selects,” selected frames from each scene, from each event in the “CDL,” (color-decision list), which describe the look of the final color correction. Normally, we’d select the first, middle and last frame of each shot. In the case or more complicated sequences we may select more frames. For a 2000-foot reel, which normally will contain 200 or 300 shots, this process will add 40 or 50 feet of additional film to be recorded out. We also add control imagery at the head of the “road map:” gray scales, LAD (laboratory aim density) patches, etc. The gray scales and charts indicate a reference to establish what the colors of the entire feature should be. Variance from the known color of the reference allows us to accurately measure how much color is lost, and thus how to restore the color in the event of fading, or in the event that no other accurate indices remain on which to establish color restoration. So this “road map” of samples and references are at the end of the recorded out “original negative” of the digital intermediate.

We can also take the process a step further by making color separation masters of the film’s original negative. In the future, the negative, or a recombination of these separations can be used to produce a higher resolution version of the feature, since most digital intermediates today are made at the 2k level.

This is a way of archiving the camera negative with its full dynamic range with everything else, with a road map for how to reconstruct the color correction. Once again, it has the archival storage of the high dynamic range acquisition elements: your camera original negative is archived. There is a realizable description of the artistic intent; you have frames that will recombine and because of the controls built in with the “road map,” you know that the color of the recombine will be correct. And this process uses established restoration procedures, that is, we’ll still be recombining black-and-white separation masters.

I’ll be happy to entertain questions or I could probably come up with a few jokes. Oh, by the way, it’s really good that Canada is taking such a strong interest in archiving motion pictures, since now that you’re actually shooting most of them, you should probably also start archiving – had to get that one in.


Q & A


Leon Silverman (Kodak and Laser Pacific): You’ve mentioned this idea of a “road map.” I’m confused about whether the road map relates the original camera negative, or the files from the finished timed DI.

Josh Pines: The road map is a representative series of sample frames of the final color-corrected 2K Digital Intermediate with all the windows and saturation and special looks built into them. You could separation your original negative, and also separation the filmed-out color-corrected DI. This is a way of doing your separations of your original negative, but splicing onto it this short piece, which is a description of the artistic intent embodied in the original release.

Leon Silverman: --this goes back to an original cut negative? So the recommendation is, after the DI, cut your original camera negative-- with full opticals?

Josh Pines: Yes.

Leon Silverman: And then, store the end of that, an instruction set, which is a visual-- instruction set-to give you the road map back to future timing.

Josh Pines: Yes.

Ian Gilmour (National Screen and Sound Archive in Australia): SMPTE made an attempt to record information about processing that’s done in the digital or the analog domain, and tried to define this in the SMPTE Metadata Dictionary 330M. At that time they also allowed for what was fondly referred to as “dark metadata,” which was a provision for proprietary information, to be slotted in alongside the standardized information that everyone could read. Now, I know in Hollywood, there are mixed feelings about this dark metadata. I was just wondering if you are at all optimistic about the prospects of lightening some of this dark metadata?

Josh Pines: That’s actually a very good point. The American Society of Cinematographers, have a technology committee, and one of the things they are recommending is some form of collaboration among the vendors of the color correctors, to establish a minimal metadata set for interchange or archiving. The Holy Grail is to have all this supplementary data in a format we’ll all be able to understand. Some of the vendors are understandably nervous to do that, because they feel, “No, you’re going to buy our color corrector, because our Gaussian Blur algorithm is better than theirs.” And sometimes it is. But we’re going to take baby steps.

We’re going to have probably just a small subset of these corrections, primaries; maybe some form of secondary correction that will be in a global metadata (inaudible). So if your color corrections are limited to that, it’s okay. When we start getting into, “Oh, and I want to take all the doorknobs in the scene and change their colors,” it’s hard to know how far that’ll be able to go.

I’m optimistic we’ll be able to get a general subset of color corrections in some sort of re-readable form. I don’t know about the general thing. The cat’s out of the bag now. They can go in and you can do these effects. And they do, on every single shot. “Oh, I can reposition every single shot.” “Oh, I can go in and change colors of fingernails.” They’ll do it and try to actually be able to recreate data. So it’s going to be difficult to do that across vendors. But I am hopeful we’ll get a subset, which will be very, very helpful for a large portion of the work.

_______________________________

SPEAKER BIOS

James Fancher

Jim Fancher is currently developing next generation technology as Chief Science Officer for the Thomson Corporate Research facility in Burbank. As Chief Science Officer for Technicolor Creative Services, which is the post production arm of Technicolor, he was involved in the development of color management systems, image processing and media asset management. Jim has been a part of managing Technicolor's world-class Digital Intermediate facility (formerly known as Technique) as well as the deployment of DI processes to Montreal and New York. Recent titles served by Technicolor's Burbank facility include: Kill Bill; Looney Tunes: Back in Action; Thirteen; The Human Stain; Pirates of the Caribbean; Seabiscuit. He started his career as a freelance electronic news cameraman for the CBS affiliate in Cleveland. Prior to his engagement at Technicolor he was Chief Science Officer for Pacific Ocean Post and there he started POP Sound, POP visual effects, which won two Academy Awards for visual effects, and the POP - Cinram DVD center. Mr. Fancher holds a Bachelor degree in Chemistry from Princeton University, holds several patents, and has been a member of SMPTE since 1974.

Joshua Pines

Joshua is currently in charge of imaging and color science projects at Technicolor Digital Intermediates, which provides the motion picture industry with digital color correction processes for theatrically released films. He joined technicolor after more than 10 years at Industrial Light & Magic, where he supervised their film scanning/recording department from its inception, and worked extensively with both traditional and digital cinema technologies.

Joshua started his career teaching film courses at the Cooper Union in New York City after earning his degree in electrical engineering there. he began working in visual effects at MAGI in 1982 at the tail end of their work on "Tron", went on to lead the computer graphics division at r/greenberg associates in New York City, and then supervised film effects and film recording at Degraf/Wahrman in Los Angeles before working for ilm. he is a member of the academy of motion picture arts andsciences and has credits on several zillion feature films.

Joshua has always thought that computers could be a useful tool in making movies better, and he still hopes that one day this may come true.