Imatest - RAW files

RAW files

The pixels in most digital image sensors are covered with a Bayer Color Filter Array (CFA), which makes each pixel sensitive to a single primary color, Red, Green, or Blue. The order of the colors is RGRGRG..., GBGBGB..., etc. There are four possible pixel arrangements (Red locations) for this order, shown in the diagram below, where pixel locations start with Row 1 Column 1 (R1C1) in the upper left hand corner. There is no univesal standard for mapping colors (R, G, or B) to pixel position (RmCn) in Camera RAW files: the unprocessed output of the sensor. The arrangement can be determined by examining a Bayer RAW image file with the Rawview utility, which can be called from a button on the right of the Imatest main window.

R1C1	R1C2	R1C3	R1C4
R2C1	R2C2	R2C3	R2C4
R3C1	R3C3	R3C3	R3C4
R4C1	R4C2	R4C3	R4C4

Red in R1C1

R1C1	R1C2	R1C3	R1C4
R2C1	R2C2	R2C3	R2C4
R3C1	R3C3	R3C3	R3C4
R4C1	R4C2	R4C3	R4C4

Red in R1C2

R1C1	R1C2	R1C3	R1C4
R2C1	R2C2	R2C3	R2C4
R3C1	R3C3	R3C3	R3C4
R4C1	R4C2	R4C3	R4C4

Red in R2C1

R1C1	R1C2	R1C3	R1C4
R2C1	R2C2	R2C3	R2C4
R3C1	R3C3	R3C3	R3C4
R4C1	R4C2	R4C3	R4C4

Red in R2C2

Most camera RAW formats are proprietary to the camera manufacturer (e.g., CR2 for Canon and NEF for Nikon) and must be converted into standard file formats (JPEG, TIFF, PNG, etc.) for most applications. This usually involves demosaicing, which is a form of interpolation that fills in the two missing colors in each pixel. Demosaicing is the primary function of RAW converter programs (which may be either built into the camera or implemented on a separate computer). In demosaicing, detail in the missing colors is inferred from detail in other, neighboring colors. Algorithms can be very mathematically sophisticated. Demosaicing (RAW conversion) programs usually incorporate a number of secondary image processing steps, including (1) applying a gamma curve, with a tonal response curve often applied on top of it, (2) sharpening, usually in the neighborhood of contrasty features, and (3) software noise reduction, typically in the absence of contrasty features. All these steps can affect Imatest results. To minimize their effects, Imatest uses dcraw, a freely available program for converting RAW images to useful file formats. Dcraw applies simple gamma curves (with no additional tonal response curves) and does not perform sharpening or noise reduction.

RAW files in Imatest

Imatest can use RAW files in two ways. Both use dcraw.

It can demosaic RAW files, producing standard 3-color RGB image files in TIFF, PPM, PNG, or JPEG format, with TIFF as the default. The advantage of using dcraw is that it applies no sharpening, no noise reduction, and a straight gamma curve (gamma = 0.45 for 24-bit output files and 1.0 for 48-bit files). If RAW files are available, cameras and lenses from different manufacturers can be compared on a consistent, fair basis.

Dcraw should be used with caution in Colorcheck and Stepchart because is performs autoexposure for 8/24-bit output. According to the dcraw FAQ, "No matter how dark an image is, dcraw's auto-exposure stretches it so that one percent of its pixels appear white. The "-b" option adjusts this behavior, while "-4" (16-bit output) avoids it entirely."
It can convert camera RAW files into Bayer RAW files stored in standard monochrome image file formats (TIFF or PGM). Starting with version 2.6.4 (February 2008) several Imatest Master modules (SFR, Rescharts, and Stepchart, with more on the way) can analyze Bayer RAW files. Bayer RAW is particularly valuable for measuring lateral chromatic aberration, which can be strongly distorted by the demosaicing process.

Dcraw does not currently run on Windows Vista. According to the dcraw FAQ, it should work when Service Pack 1 (SP1) is released.

To use RAW files in Imatest, either

Click on an Imatest module (SFR, Stepchart, etc.) and open the file just as you would any other image file. If a camera RAW image file (CR2, NEF, etc.— even Phase One RAW TIF files) is detected, the dialog box shown on the right for setting dcraw parameters is opened.
Click on the dcraw box on the right side of the Imatest main window (or click Modules, Run dcraw) to run dcraw as a standalone application: the output file will be saved but no Imatest module will run. The dialog box on the right is a graphic user interface (GUI) front-end to dcraw that allows you to select several of dcraw's options.

Dcraw can either (A) demosaic camera RAW files with minimal processing— no sharpening, no noise reduction, and a simple gamma curve, or (B) convert them to standard monochrome file formats (TIFF or PGM) that contain Bayer RAW (undemosaiced) data, which can be analyzed by several Imatest Master modules. This conversion is controlled by the Processing window, which has the following choices:

Normal RAW conversion (demosaiced)

RAW 8-bit (Bayer RAW; no demosaicing)

RAW 16-bit (Bayer RAW; no demosaicing)

If the second or third option above (RAW 8-bit or RAW 16-bit) is selected, the Demosaicing area (above) is grayed out and the output of dcraw is a monochrome TIFF or PGM file that contains undemosaiced (Bayer RAW) data from the image sensor.

Bayer RAW files

Whenever a monochrome file is opened by an Imatest Master module that supports Bayer RAW, the dialog box shown on the right appears. Standard monochrome file should be selected for regular monochrome files that do not contain Bayer RAW data. The color scheme of the buttons depends on the "Red in RmCn" selection, which affects some of the displays. The correct setting can be found using the Rawview utility.

If Bayer RAW R1C1, R1C2, ... is selected, one of the four image planes starting with R1C1 (Row 1 Column 1), R1C2, R2C1, or R2C2 is selected for analysis. (Recall that there is no universal standard mapping an image plane with a color.) The four planes are shown below. Note that the pixel height and width of the image in each plane is half that of the image as a whole and the spacing between pixels is twice that of the sensor pixel spacing.

R1C1	R1C2	R1C3	R1C4
R2C1	R2C2	R2C3	R2C4
R3C1	R3C3	R3C3	R3C4
R4C1	R4C2	R4C3	R4C4

↓
R1C1
↓

R1C1	R1C3
R3C1	R3C3

R1C1	R1C2	R1C3	R1C4
R2C1	R2C2	R2C3	R2C4
R3C1	R3C2	R3C3	R3C4
R4C1	R4C2	R4C3	R4C4

↓
R1C2
↓

R1C2	R1C4
R3C2	R3C4

R1C1	R1C2	R1C3	R1C4
R2C1	R2C2	R2C3	R2C4
R3C1	R3C2	R3C3	R3C4
R4C1	R4C2	R4C3	R4C4

↓
R2C1
↓

R2C1	R2C3
R4C1	R4C3

R1C1	R1C2	R1C3	R1C4
R2C1	R2C2	R2C3	R2C4
R3C1	R3C2	R3C3	R3C4
R4C1	R4C2	R4C3	R4C4

↓
R2C2
↓

R2C2	R2C4
R4C2	R4C4

The four image planes: one Red, one Blue, two Greens:
(There is no universal standard mapping colors to image plane.)

Demosaiced files

The best way to be sure an image file faithfully resembles the RAW file— that it has a straight gamma curve, no sharpening, and no noise reduction— is to read a RAW file into Imatest and convert it to a standard format (TIFF, PPM, etc.) using Dave Coffin's dcraw. If you select any of the standard RAW formats (CRW, NEF, etc.) or if raw data is detected (e.g., Phase One TIF files) the dialog box shown above appears. It is a front-end to dcraw that allows you to choose between several RAW conversion options.

dcraw conversion details:

The Processing box lets you select normal raw conversion (with demosaicing) or 8 or 16-bit Bayer RAW output(RGRGRG, GBGBGB, ... sequence; undemosaiced). The latter formats enable Imatest modules to examine the unprocessed (Bayer RAW) pixels. Choices:

Normal RAW conversion (demosaiced)

RAW 8-bit (Bayer RAW; no demosaicing)

RAW 16-bit (Bayer RAW; no demosaicing)

The Demosaicing box contains several options, most of which are explained in more detail in the dcraw Manpage. The following output color spaces are available for demosaiced output.

RAW (not true RAW pixels).
Demosaiced with no color space conversion or White Balance.
The camera's color profile is needed to correctly interpret the colors. Gamma = 0.45.

sRGB, Adobe RGB (1998), Wide Gamut RGB, ProPhotoRGB, and XYZ.

Profile tags are embedded in the output file for the four standard color spaces. For 24-bit conversion, gamma appears to be fixed at 0.45, independent of the output color space. (It should be different for ProPhoto, which is designed for display at gamma = 1.8.) For 48-bit conversion gamma is fixed at 1 (linear conversion; does not correspond to standard color spaces). Several demosiacing quality levels (using different algorithms) are available (and worth exploring). See the dcraw Manpage for details.

Auto-exposure is always applied for 24-bit conversion (but not for 48-bit). I've changed the default to make it slightly darker than the dcraw default (using -b 0.99 in the command line) to reduce the likelihood of clipping pixels. For now we're stuck with auto-exposure for 24-bit conversion: see dcraw Frequently Asked Questions.

Dcraw converts RAW files into TIFF files, with the obscure PPM format available as an option. If check the box to delete the converted (TIFF) file, you may save the image in one of two compressed formats: PNG (lossless compression; takes longer) or JPEG (lossy compression; maximum quality; 24-bit only).

A list of supported cameras appears on the dcraw site. New cameras are typically added within one to three months of their introduction. The dcraw version included with Imatest sometimes falls behind. According to Dave Coffin, "Francisco Montilla provides Mac OS and Windows executables on his website. And Benjamin Lebsanft has volunteered to maintain Windows executables optimized for specific CPUs."

Rawview utility

The Rawview utility allows you to determine the color arrangement of Bayer RAW files (Red in R1C1, R1C2, R2C1, or R2C2) by trying each and selecting the one that looks best. It can be opened by pressing the Rawview button on the far right of the Imatest main window, just to the right of dcraw. When Rawview is first opened, instructions similar to those shown below appear. (They are likely to be more up-to-date.)

Welcome to Rawview. Preview Bayer RAW files.You can find the relationship between colors and pixel locations.
The input file should be a monochrome file (r x n x 1 pixels) that contains Bayer Raw data.
The image is not fully demosaiced: it has only half-resolution.
There is no White Balance. Colors may appear faint.
16/48 bit images are lightened to improve visibility.
The input image should have some strong, identifiable colors.

Start by pressing "Read (Bayer) RAW image file."
Click on "Red in RmCn..." popup menu and determine which selection looks correct.

After the image file has been read, the Rawview window looks like this:

The Rawview window (no zoom)

The zoom function is always active: the image can be zoomed in to better view key details, like the red dots on the mirror, illustrated on the right, which only appear to be red when Red in R1C1 is selected. The Save button is enabled whenever the Red in RmCn selection is changed. It sets the default for opening Bayer RAW files in Imatest modules, affecting some displays.

Note that Rawview performs no White Balance, so the balance you see represents the color balance in the sensor, which may be of interest. It is rarely neutral.

Rawview detail

Rawview dropeown menus include display options and Save commands.

File

Read Bayer RAW file Read the Bayer RAW file (must be in a standard image file format: TIFF, etc.)
Save Screen Save the contents of the screen. Optionally display it if the Display screen checkbox is checked. You need to set the viewer before the first time you display the screen this way. Irfanview is an excellent free viewer.
Save image as RGB file Save the image as an RGB file, which has half the pixel dimensions (resolution) on the original file. This does not involve a demosiacing algorithm. The contents of the R, selected G (G(R) or G(B)), and B channels are identical to the undemosaiced channels.

Settings

Display G(R) as green Use the G(R) channel— the green pixels in the the same row as the red (R) pixels— for the green channel in the display and the RGB file save. You can tell if there is a difference between the two green channels (G(R) and G(B)) by toggling between these.
Display G(B) as green Use the G(B) channel— the green pixels in the the same row as the blue (B) pixels— for the green channel in the display and the RGB file save.
Adjust 16-bit images Brighten 16-bit images (which are linear or nearly so, i.e. gamma near 1.0) if needed to enhance display.
Brighten 16-bit images Brighten 16-bit images further for additional display enhancement.

Help

Online Help opens this page in your default web browser.