Session	Digital Re-Registration of Separations
Presenter	Andrew Bonello Cinesite, Los Angeles
Abstract .. .. Presentation.. ..Download Powerpoint Presentation.. .Download Paper. .Speaker Bio

ABSTRACT

A recent project at Cinesite in Los Angeles involved full restoration of the film "Williamsburg: The Story Of A Patriot" from 35mm Vista Vision separation masters. This work involved defect and scratch detection and repair, dye-fade correction, color correction, and re-registration of the Vista Vision separations. Cinesite used existing digital techniques for some of this work, and developed new techniques to address other areas.

In particular, this presentation concentrates on recombining scanned film separations in an automated digital framework. Manual alternatives are labor-intensive and error-prone. By making use of image deformations and channel matching within an efficient sampling framework, the technique robustly re-registers scans from Academy to Vista Vision format.

_______________________________

PRESENTATION

Introduction

Film masters for older films exist in several formats. In some cases, work starts with separations. The process begins with three separate strips of film, all containing image data for the same piece of footage. These three pieces of film are created by optically exposing a color negative CMY film element with red, green and blue light respectively.

A full color image is obtained by optically recombining the information from these three strips of film on top of one another. Thus, the Cyan, Magenta and Yellow information from the original color negative is converted into the Red, Green and Blue channels of a positive color image.

The main contribution of this paper is to describe a technique for digitally reregistering separations in an automated framework. The document is organized as follows. In Section 2, a typical pipeline used for manual reregistration in a post-production environment is described. Section 3 introduces the new automated technique at a high level. In Section 4, details are given about the implementation of the new technique. Section 5 concludes by describing the overall function of the tool, as well as the different sources of input images which can be handled.

Section 2 – Manual Reregistration

The process of optically creating and reregistering film footage from separations can introduce unpleasant color aberrations in the image (see Figure 1). These are usually the result of poor pin-registration and film shrinkage. Such color fringes are referred to as misregistration in the image.

 

Figure 1 : Pin misregistration and film shrinkage between layers cause unpleasant color fringes in the resulting image

In the digital post-production environment, methods exist for eliminating the misregistration outlined above using a manual process.

The pipeline begins with three monochrome digitized frame sequences. For the remainder of this document, these will be referred to as the Red channel (R), the Green channel (G) and the Blue channel (B) respectively.

From within a digital compositing package, each separation frame sequence is first digitally stabilized over time. Typically, this is done using feature-tracking. Next, the first frame of the sequence is recombined manually. The (x, y) offsets of the Red and Blue layers are adjusted so that their image content matches that of the Green layer without any misregistration. The Green layer is designated as the “hero channel”. There seems to be a consensus within the film industry that, for the majority of film stocks, the Green layer exhibits low grain and high luminance content when compared to the Red and Blue layers. This makes the Green layer a good reference. It is kept fixed while the Red and Blue layers are moved around.

Once the first frame has been recombined in this way, those (x, y) offsets are propagated forward to every frame in the sequence. This is intended to correctly reregister all remaining frames. In practice, this only works some of the time. Any inaccuracies in stabilization mean that frames later in the sequence might still be misregistered. Furthermore, each shot in the movie must be tracked separately, since a new set of features must be specified by the compositor each time a shot transition occurs.

At Cinesite, research was undertaken to investigate methods of automating the process of reregistering Red, Green and Blue separations to form color images. Performing the task automatically allows the labor-intensive and error-prone process of manual reregistration described above to be circumvented. This enables major savings in both time and cost in the digital film restoration pipeline.

Section 3 – The New Technique

3.1
Automated Reregistration

Again, the starting point is three sets of monochrome digitized frame sequences R, G and B.

A hero-channel is designated as before. Whilst this is typically the Green channel, a different hero-channel can be designated if required. Two independent searches are performed. In the first, the Red channel is deformed to match it to the Green channel as closely as possible. A second search does a similar job to deform the Blue channel to match it to the Green (see Figure 2).

 

Figure 2 : Steps in recombining three monochrome separations into one color image

 

The problem essentially reduces to two optimizations. This strategy is outlined in the following sections.

3.2
Measuring Image Similarity  

Consider a similarity measure between two monochrome images A and B, referred to as D(A, B). The following properties are defined for the measure:

(1) for all A, B

 

(2)

Property (1) limits the measure to a real number which is always non-negative. Increasingly positive numbers correspond to increasingly dissimilar images. Therefore, D acts as an error measure.

Property (2) relates to the intuitive notion that comparing an image to itself results in a perfect score of 0.

3.3
Matching An Image To The Hero Image

Consider a monochrome image C. This image undergoes a transformation T(x) which takes image C and gives C’.This is a new, deformed version of image C. Hence:
................................. T(x)C = C’

 

The transformation T is specified by a multi-dimensional vector of parameters x. This vector is simply a set of values controlling several linear transformations. In a typical framework, these might include global translation, rotation about a floating centre, and scaling in the x- and y- directions (see Section 4.1 below). The idea is to define sufficient degrees of freedom in x to allow the image to deform enough to correct all misregistration.

Using the above definitions, transformation T is applied to each red image R. The goal is to find a vector x that, when applied to image R, results in a transformed red image R’ that is registered as closely as possible to the hero image G. This procedure is repeated for the blue image. More precisely, the aim is to minimize:
................................. D( T(x)R, G ) … [1]

 

And, in a separate stage:
................................. D( T(x’)B, G ) … [2]

Since R, G and B are fixed, the problem reduces to finding a set of values for the multi-dimensional vectors x and x’ so as to minimize expressions [1] and [2] above.

3.4
Choosing The Best Transformation ‘x’

The current implementation is based on an efficient sampling procedure, as commonly employed by particle filtering techniques for visual tracking. See [DEUT00], [DEUT02, Chapter 6] and [DAVI01].

A distribution containing N unique samples is defined. Each is an instance of vector x. For every sample vector x, each element of that vector is initialized by sampling from a zero-mean Gaussian with a predefined variance. Consider a vector element controlling the x or y component of the global 2d translation of an image. The range of values expected for that element will be different to that used to control, for example, the rotation applied to the image. For this reason, the variance of each element’s Gaussian is chosen independently.

Next, D( T(x)R, G ) is evaluated for every sample,resulting in a score for each vector x in the distribution. This yields a collection of similarity scores, which are then sorted in increasing order.

The top 20% of the samples from this sorted set are taken. Sampling from them, a new distribution is generated. This is done by perturbing about those top sample means through the application of the predefined Gaussians. The independent variance of each Gaussian is scaled by that element’s variance in the last distribution. This is repeated for several generations. In this way, the sample distribution is continually refined. The guesses in x gradually concentrate in the most promising areas of the multi-dimensional search-space (those with the best similarity scores).

Eventually, a point of maximum confidence is reached, and the search ends, outputting a single ‘best guess’ for x. Vector x’, which deforms image B to match image G, is found in the exact same way.

3.5
Random Misregistrations

By examining a broad selection of film separations, it is observed that the exact type of misregistration that occurs in any given frame is hard to predict. There is rarely a strong temporal correlation in misregistration from one frame to the next over time. Furthermore, the magnitude of misregistration seen in frames at the start of Reel 1 of a movie might be very different from those at the end of, say, Reel 5. Also, for a given frame, it is difficult to predict a correlation between the misregistration of the red image versus that of the blue image.

For these reasons, statistical independence of the misregistrations for each of the frames is assumed. Thus, an individual search is performed on each frame in turn.

Section 4 – Implementation Details

4.1
Details of 2D Image Deformation

The transformation T(x) is a combination of several linear transformations:

a) X- and Y- translation (global)

b) X-scaling with a variable centre of scaling

c) Y-scaling with a variable centre of scaling

d) Rotation of image about a variable centre

e) Top-left corner can be dragged around

f) Top-right corner can be dragged around

g) Bottom-left corner can be dragged around h) Bottom-right corner can be dragged around

 

Figure 3 : Transformations applied when reregistering an image

For (e) through (h) above, dragging each corner has the effect of pulling the image content in the direction of the drag. Imagine stretching the four corners of a patterned scarf made of stretchable material. The pattern on the material is the image content in the matching image.

The above are all linear transformations. They are applied in sequence, and combined to create a single output image for each candidate vector x. Through experimentation with many film sequences at Cinesite, this combination of transformations has been found to identify and correct for the majority of misregistrations that the three separate scans typically introduce.

Note that there is some redundancy in the combined set of transformations described above. For example, instead of steps (a) through (h), the deformation could consist of a global translation and a perspective transformation. The format presented here has been chosen for clarity.

4.2
Evaluating the measure “D( A, B )”

 The actual method for calculating the similarity measure could follow one of several schemes.

One possibility is to examine feature correspondences between images A and B.

Alternatively, the binary edge maps of the two images could be found. By computing the overall brightness of the image resulting from AND’ing those two edge maps together, a similarity measure is obtained. Note that this similarity measure must be inverted in order to comply with the properties defined in Section 3.2.

A second alternative is to compute some kind of image signature for images A and B in a new vector-space, and compute a difference measure within that new space. Wavelet decompositions have been used in this way, as explained in [STOL96, Chapter 5].Here, image similarity scores are generated and used for multi-resolution image querying from a known image database. This is one way of doing object recognition. The application is different to that described in this paper, but the principal is the same: the goal is to define a scalar measure which somehow quantifies the amount of similarity between two arbitrary images A and B.

The method of feature correspondences above is now explored. Assume that samples at n unique locations in the reference image are examined.

Each sample point defines an (x, y) location in the image. For each point, a difference measure between the luminance value at that location in the green image G and that same location in the transformed red image T(x)R is obtained. Call that single difference measure d. For a given sample point i (taken from the set of n sample points) and transformation vector x, the single difference measure d is defined as:

 

Where R iand G iare the luminance values in the red and green images respectively at sample point i. Measure D becomes:

 

 

The measure D is a sum of squares of functions. A search in x is performed to minimize the measure, as described in Section 3.4.

Section 5 – In Production

5.1
Overall Reregistration Pipeline

For each frame, three monochrome images (Red, Green and Blue) are taken and, by applying the minimizations described above, the Red and Blue images are deformed to match them as closely as possible to the Green image.

Once these corrections are complete, there is enough information to generate a reregistered color output image. The tool then moves onto the next frame in the sequence and repeats the entire process. So reregistration is carried out independently for EACH frame in the sequence. This allows for the random nature of misregistrations that commonly occur.

 

5.2
Different Sources for Input Separations

In addition to the “three-images-in (R,G,B) one-image-out (Color)” scenario described above, the automated tool can also be used to split a color image into its RGB components. These images are reregistered exactly as before. The color image is then regenerated. In this way, misregistrations which might have been built into the color image at the outset are corrected. Alternatively, different elements can be used to recombine. For example, two of the channels from a set of color images and a single channel from a (unique) set of separations could be reregistered together (see Figure 4).

 

Figure 4 : Source images can come from a variety of film formats

 

In the case of Sequential Exposures, the R, G and B frames all appear on a single strip of film. They are interleaved with one another such that each set of three monochrome frames actually comprises a single color frame. So the sequence on film might look like “…RGBRGBRGB…”, and will consist of three times as many frames as the actual number of color frames in the movie. Film data.

The idea is to allow input elements to come from a variety of different sources, giving the tool maximum flexibility. Several of these different scenarios have been put to use in the current production environment at Cinesite, Los Angeles.

The search-based strategy employed by the automated reregister tool makes it robust for a wide range of input image data. It has been demonstrated to be robust to camera pans, zooms, motion blur, subjects entering and leaving the frame, scene cuts and lighting changes. Manual reregistration procedures often require that these changing conditions are somehow handled explicitly.

See Figures 5-8 for examples of typical color film frames before and after automated reregistration has been applied.

Figure 5 : Example of a misregistered frame

 

Figure 6 : Same frame after automated reregistration was performed. In addition to the removal of color fringes, observe how image fidelity and grain definition are both improved in the reregistered frame.

Figure 7 : Example of a misregistered frame

 

Figure 8 : Same frame after automated reregistration was performed. Note the motion-blur present in this shot due to a rapid camera-pan. The automated technique is robust to pans, zooms, and scene cuts.

Acknowledgements

The research and development outlined in this paper was undertaken at Cinesite, Los Angeles. The author would like to thank colleagues and management at Cinesite for many useful, helpful and informative discussions during this research, as well as invaluable testing of the automated tool. The Entertainment Imaging Services group of Cinesite’s parent organization, the Eastman Kodak Company, is also acknowledged. Thanks go to Robert Harris of The Film Preserve, Ltd., and to Richard McCluney, Director of Colonial Williamsburg Productions, for their kind permission to use stills and film footage from the movie “ Williamsburg: The Story Of A Patriot” in this paper and the associated presentation.

Contact Information
Andrew Bonello
Phone: +1 323 251 4889
Email: abonello00@gmail.com

References  

[DAVI01] A. Davison, J. Deutscher and I. Reid, Markerless Motion Capture of Complex Full-Body Movement for Character Animation, Proceedings of the Eurographics Workshop on Animation and Simulation (2001), Springer-Verlag LNCS.

[DEUT00] J. Deutscher, A. Blake and I. Reid, Articulated Body Motion Capture by Annealed Particle Filtering, Conf. Computer Vision and Pattern Recognition (2000).

[DEUT02] J. Deutscher, 3D Human Motion Capture. DPhil thesis, Robotics Research Group, University of Oxford (2002).

[STOL96] Eric J. Stollnitz, Tony D. Derose, David H. Salesin, Wavelets for Computer Graphics – Theory And Applications, Morgan Kaufmann Publishers (1996).

SPEAKER BIO

Andrew Bonello (MEng)

Andrew was raised in the United Kingdom, and obtained a Masters degree in Engineering and Computer Science at the University of Oxford. After graduating in 1998, he spent two years working in a technical role for a financial institution in the City of London before moving into the film visual effects industry. Following two years working for a leading motion capture company in Oxford, Andrew relocated to Los Angeles, where he has spent the last 18 months researching digital film restoration techniques at Cinesite.