Documentation – Legacy v5.1 and below

Color/Tone Auto (Multitest): fixed (batch-capable) analysis

Current Documentation

All documentation versions

Analyze color and grayscale test charts
in a fixed batch-capable module

Color/Tone Auto (Multitest) incorporates all calculations in Color/Tone Setup (Multicharts), Colorcheck, and Stepchart.

News– Imatest 5.2: We’ve changed the name Multitest to Color/Tone Auto, but we’ll continue to use Multitest for a while (for example, for Imatest IT).
Imatest 5.1: 
(starting December 2018)  The exposure error calculation has been improved to be more consistent over a range of contrast (gamma) values. Details here
Multicharts and Multitest
 can now measure temporal noise and SNR. With this enhancement, these two new modules have all major features of the older Stepchart and Colorcheck modules.
Nonuniformity correction, available since Imatest 4.5, has been documented.
Imatest 4.2:
Color difference ellipses (MacAdam, ΔC 94, ΔC 2000, and ΔC ab) can be displayed in several displays: a*b*, xy, u’v’, vectorscope, and CrCb.
 

Introduction

Imatest™ Color/Tone Auto (Multitest) (not available in Imatest Studio) analyzes images of a large variety of test charts for color accuracy, tonal response, noise, SNR (Signal-to-Noise Ratio) and ISO sensitivity using fixed interface that can run batches of images. It can measure white balance and color response for a wide range of lighting conditions and scenes. It can also display the tonal response of monochrome charts (or monochrome sections of color charts). A color correction matrix can be calculated.

Color/Tone Auto can analyze all the charts supported by Color/Tone Setup (Multicharts) and can produce most of the results plots. It can also incorporates all important functions of the Colorcheck and Stepchart modules: it is similar in operation, but able to analyze more charts and display more results.

Major test charts supported by Multitest
X-Rite ColorChecker (24-patch)24-patch
Colorchecker
 
IT8 target (from Wolf Faust)IT8.7
 
Contrast Resolution
 
ColorChecker SG
Colorchecker SG
 
 

dynamic_35_E_200W
36-Patch DR chart

 
Q13 grayscale step chart
Q-13/Q-14
and other
grayscale
stepcharts
    ColorGauge_120W
ColorGauge
(Micro, Nano, etc.)
 
 
ISO-15739 Noise target crop
ISO-15739
 

General mxn
charts (L*a*b*
reference file
required)
 
TE269
 
ST-52 (ISO-14524) crop
ISO-14524
Also: SFRplus 20-patch color, SMPTE/Hale Color Bars (6 patches), TE226 (9×5) chart

A full list of supported charts can be found on Multicharts– Supported charts.

Getting started

Photograph or scan the chart, taking care to illuminate it evenly (±5%) and to avoid glare, which can be problematic in charts with semigloss (SG) and glossy surfaces such as the reflective IT8.7 and ColorChecker SG. Glare can be especially challenging to control with wide angle lenses. To minimize the effects of light falloff in lenses, the chart should typically not fill the image field.

A low-cost lighting setup is described Building a Low-Cost Test Lab. For testing white balance, you can photograph the chart in a scene under a variety of lighting conditions. Small versions of the ColorChecker (the Passport, the SpyderCheckr, or the ColorGauge charts) are especially suitable for this purpose. Some rectangular charts may be rotated by multiples of ±90° if they are geometrically symmetrical (these include the two Colorcheckers and the Kodak/Tiffen Q-13/Q-14, but not the IT8.7 (where top and bottom are different).

Before you run Multitest you should select the chart type to analyze. Click on the Settings button, just to the right of the Multitest button in the Imatest main window (shown on the right) or click on Fixed Modules, Multitest settings. These open the Multitest settings window, shown below.

The Chart type dropdown menu (in the Multitest settings window, shown on the right) contains all available chart settings.

After a selection has been made it is saved and displayed in the Multitest button, as shown above-right.

Additional settings can be changed at this time or after an image file has been read and the Region of Interest (ROI) selected (unless you’ve selected Express mode)— but since Chart Type always affects the next run, a selection should be made before a new chart type is run.

Chart selection dropdown menu

Chart type should be selected prior to running Multitest.

  multitest_button
  Chart type (for next Multitest run)
multitest_chart_type

Automatic region detection

This is also a good time to select automatic or manual region detection, located at the bottom of the Multitest settings window and enabled for charts that allow the automatic option (initially Colorcheck and the 36-patch Dynamic Range chart, with more to come).

multitest_autoroiManual/Automatic ROI selection, located at the bottom of the Multitest settings window

Running Multitest  To start Multitest, run Imatest, then click the Multitest button. A window will open for selecting the file (or batch of files).

To calculate temporal noise, select either two files (for the Imatest-recommended difference method) or 4-16 files (for the ISO 15739 method). Details of the calculation are can be found here.

If you have selected manual ROI detection, after you make the file selection, Imatest will check to see if the image(s) are the same size as recently-run images (it saves several sizes). If it is the same size you’ll be asked if you want to repeat the same Region of Interest (ROI) selection:

ColorTest_SQ_roi_repeat
ROI repeat window (appears when image is the same size as a previously-run image). Manual ROI detection.

If the image is not the same size as a previous image or if you click No, select new ROI(s) , a window will open asking you to select the region. Do this by drawing a rectangle from one to another corner of the chart you want to analyze. Don’t worry about getting the region exactly right— you’ll be able to refine it.

ColorTest_SQ_region_selection
Initial (coarse) region selection. Draw a rectangle from corner to corner of the chart.
Image from Imaging-resource.com (the Multi-Target from the Samples page for individual camera reviews).

Refine the selection in the fine selection window that opens after you make the coarse selection. The inner squares are different for different charts. (Note: this window will appear if you’ve selected Automatic ROI detection with confirmation with one of the charts that support automatic detection.)

ColorTest_SQ_fine_adjust
Fine region selection, shown for the ColorChecker SG

  • The entire ROI can be moved ( ^ > v < , upper-left).
  • The top, bottom, left, or right sides can be moved ( T^ Tv R> R< B^ Bv L> L< , middle-left).
  • Corners can be moved individually (any of the 8 buttons in each of the four corners, middle-left).
  • Pixel values for the four corners can be entered (X(tl), Y(br), …, where tl = top-left, br = bottom right, etc.). The origin is the upper-left. Be sure to press the Enter key after entering a value.
  • You can choose between Fine, Coarse, or Extra Coarse (XC-25) movement (1, 5, or 25 pixels of movement per click).
  • The display can be zoomed out or in.
  • The display can be lightened to view the dark regions more clearly.
  • For some charts (those with relatively large patches) a Fill factor (linear) slider to the left of the Distortion slider specifies the relative size of the squares. Its default value is 0.5. Smaller values can be used where lens distortion or pattern alignment makes alignment difficult.
  • The Distortion slider allows highly distorted images (barrel or pincushion) to be analyzed.

When you have completed the fine adjustment, click one of the buttons on the bottom of the window. If you click Yes, Continue, the Multitest settings window, shown below, will open. If you click Yes, Continue in Express mode Multitest will run, display selected results Figures, and save selected results files. 

Color/Tone Auto settings window

Chart type Selects the Chart type. Applied to the next run. If Chart type 3 (last grayscale) or 7 (Multi-Row grayscale Stepcharts (not linear)) is selected, the specific chart type should be selected in the Special (multi-row) grayscale chart type dropdown menu, below.
More settings: Color Matrix, noise, mirror/rotate, … Opens the Multicharts settings window, shown below. Lets you select Color Matrix settings (described in the Color Correction Matrix web page), noise settings, ISO speed calculation, mxn chart settings, and more, as described below.
Run Color Matrix Calculates and applies a Color Correction Matrix (CCM) if checked.
Reference Reference file. Can be standard or read in (typically from a CSV or CGATS L*a*b* or density file). Varies somewhat with the chart type.
Color space Color space
Noise calc. Noise calculations. Select 1. No (non-visual) noise calculations (fastest), 2. Elementary noise calculations (slower), or 3. Advanced noise calculations (slower).
Noise & Visual noise settings Opens the Noise and visual noise window, described in Multicharts/Multitest/eSFR ISO Noise. Lets you select visual noise and other settings.
Nonuniformity correction Allows you to correct for nonuniform illumination using a separate flat field image taken under identical conditions. Opens the Nonuniformity correction window described in Nonuniformity Correction in grayscale and color chart modules
Special (multi-row) grayscale chart type Visible only when Chart type 3 (last grayscale) or 7 (Multi-Row grayscale Stepcharts (not linear)) is selected. Use to select the appropriate grayscale chart type.
Special grayscale chart settings Opens a window that lets you select density increment for linear grayscale stepcharts.
Display thumbnails & EXIF on the right Display thumbnails of the entire image and crop as well as EXIF results to the right of the plot. Examples will be shown below.
Plot settings (on right) Selects output figures. Described below.

 

Multicharts settings window.

Color correction matrix Settings for the CCM calculation are described in the Color Correction Matrix page
Incident Lux Must be manually entered for ISO Sensitivity (Exposure Index) calculations. Aperture and Exposure (s) are normally obtained from EXIF data, but may be entered manually if the EXIF data is absent.
Noise Settings are described in the Multicharts/Multitest/eSFR ISO Noise page.
Rows, columns for
General mxn charts
Select number of rows and columns for general mxn (rectangular) charts. A reference file is required for this chart selection.
Saturation level Three settings that affect Dynamic Range calculations are available: Standard file: 255 (or 65535…),  ITU-R Rec. 601 (BT.601): 235,  Maximum detected patch level (useful when 255 can’t be reached).
Pixel offset Enter the pixel offset to subtract from the data. Pixel offsets are not uncommon: they cause a premature flattening of Density curves.
Nonuniformity correction Allows you to correct for nonuniform illumination using a separate flat field image taken under identical conditions. Opens the Nonuniformity correction window described in Nonuniformity Correction in grayscale and color chart modules
Image orientation (Applies to next read) Select among Standard image orientation, Mirror (H-flip), Mirror (V-flip), Rotate 180 degrees.

 

Reference files

Most charts allow reference files to be entered using the Reference dropdown menu in the Settings area. (IT8.7 and CMP DT003 charts require them.) The available settings depends on the chart type. The previous file is generally available in the dropdown list. Here is a summary of the available reference sources.

Chart Default values Options (data from files)
X-Rite ColorChecker
(24-patch)
Default values from GMB, BabelColor, or Danes Picta (a chart with the same geometry but different colors) L*a*b*, xyY D50, or xyY D65 files in CSV format.
IT8.7 Requires a reference file in IT8.7 format, available from the chart manufacturer, generally available on their websites.
Rezchecker You can use the standard default reference or NIST-traceable measurements provided by ISA. These measurements are in a difficult-to-use XLS/CGATS file format. To use this data, open the file in Google sheets (which can be done directly in Gmail) or in Excel. Select the 30 rows representing the patches and the 3 column representing LAB L, LAB A, and LAB B, and copy them into the clipboard. Past into a text editor and save the file (with txt or csv extension).
CMP DigitaL TargeT 003 Individually measured reference files are supplied with each target.
X-Rite ColorChecker SG Default values from GMB (L*a*b* D50); See Note below L*a*b*, xyY D50, or xyY D65 files in CSV format.
Stepchart (linear) Even density steps of 0.1, 0.15, 0.20, or 0.30. For the UTT only, L* = 95 to 5 in steps of -5 for the first 19 steps and 1 for step 20. (20-step UTT) should be selected. an ASCII density reference file with one density value per line can be entered.
Note: Several of the default reference values are derived from L*a*b* D50 values, i.e., L*a*b* values acquired under D50 illumination. When the color space has a different reference temperature (sRGB and Adobe RGB both have D65) a Bradford transformation is applied, resulting in a small change in the L*a*b* values. This is rarely a significant concern if you’re making images under a different illuminant: if the white balance algorithm is working properly, it should transform image colors to the appropriate values for the color space.

Except for the IT8.7 and CMP DT 003, reference files must be in CSV (comma-separated file) or CGATS format CSV files can be opened and edited in Excel. Color files (L*a*b*, xyY D50, and xyY D65) have three entries per line; density files have have one entry per line.

Here is an example of an L*a*b* file.

The format is CSV (comma-separated variables).
The format is the same for xyY files.

Added spaces are for visual clarity (readability) only.

55.261, -38.342, 31.37
28.778, 14.179, -50.297
65.711, 18.13, 17.81
51.038, -28.631, -28.638
51.935, 49.986, -14.574
81.733, 4.039, 79.819
42.101, 53.378, 28.19

If you have a spectrophotometer you can create a custom reference file (CGATS format) using techniques described in Measuring test chart patches.

Results figures

1. 3D Color Difference (with split)

In this plot, the height of the region is the selected color difference, in this case ΔE*ab (plain Delta-E). The tops of the patch platforms are shown as split colors with the reference color on the upper-left and the actual input color on the lower-right. This plot enables you to compare color measured with perceptual color differences. (They may not match as well as the theories claim.)

Panasonic_DMC-GF1_hVFAL_20MM_3D_diff

This plot is shown with the thumbnails and EXIF data on the right (set in a chckbox in the Multitest Settings window. Other plots will be shown without these results, i.e., with the box unchecked (so they take up less space).

2. Split Color (Reference/Input) plot

The Split Color plot shows the reference (ideal) color on the upper-left and the input color on the lower right.

If a color correction matrix has been calculated (if the checkbox has been checked), this plot and also the 3D Color Difference plot (above) will display the reference color on the upper-left, the input color on the upper-right, and the corrected color on the bottom.

ColorTest_split_3D_plot_corrected3D Color difference detail,
Color Correction Matrix calculated

ColorTest_split_plot_corrected

Split color detail, Color Correction Matrix calculated

 

Panasonic_DMC-GF1_hVFAL_20MM_split_diff

3. 2D a*b* plot

Displays the a*b* plane of L*a*b* color space. Chroma differences are (very) approximately the distance between the Reference and Camera values. (This difference is ΔCab. The more complex ΔC94 and ΔC00 (CIEDE 2000) equations (described here) are more accurate.

If a color correction matrix has been calculated, the corrected values will be displayed.

 

  Panasonic_DMC-GF1_hVFAL_20MM_2D_ab

4. xy Chromaticity

This plot is familiar, but less perceptually uniform than the a*b* plot, above. Green gets too much area in this plot.

If a color correction matrix has been calculated, the corrected values will be displayed.

  Panasonic_DMC-GF1_hVFAL_20MM_xy_Chrom

5. B&W Density & White Balance

The Upper plot displays Log(Pixel level/255) (maximum of 1) vs. input density (-log10(exposure)), i.e., uses density units.

The lower plot displays several types of ΔC as a function of patch density. See Colorcheck appendix.

If a color correction matrix has been calculated, the corrected values will be displayed.

  Panasonic_DMC-GF1_hVFAL_20MM_BW_density

6. EXIF data and Color Matrix

This figure shows selected EXIF data and the results of the Color matrix correction, including

  • the color matrix
  • a description of the settings used to calculate the matrix (Color difference used for optimization, weighting, linearization, …)
  • Mean and maximum olor differences between the input and reference image and between the corrected and reference image.
  Panasonic_DMC-GF1_hVFAL_20MM_EXIF

7. Additional 2D color plot

The vectorscope display is shown. One of several plots can be selected here:

  1. u’v’ Chromaticity
  2. Vectorscope
  3. 2D CbCr (from YCbCr)
  4. HS (Hue-sat) from HSV
  5. HS (Hue-Sat) from HSL

These displays are described in Using Multicharts – Displays and Options.

  Panasonic_DMC-GF1_hVFAL_20MM_2D_color

Noise/SNR analysis (not for all charts)

The Noise/SNR analysis is only available on charts that have patches large enough to provide good noise statistics. Noise analysis includes standard pixel noise, chroma noise, scene-referenced noise, sensor (raw) noise, and ISO-15739 visual noise. SNR (Signal-to-Noise Ratio) can also be displayed as a ratio or in dB.

The patches on the ColorChecker SG shown on this page are too small, but fortunately the Imaging-Resource image has a standard 24-patch ColorChecker to the left of the SG that we’ve analyzed to provide the results shown on the right.

Noise analysis is described in Multicharts/Multitest and eSFR ISO Noise. ISO 15739 visual noise was added in Imatest 4.0.

The following selections are available.

  1. Noise vs. input density (RGB)
    Noise in pixels or % of maximum pixel level
  2. Signal/Noise (S/N) vs. input density (RGBY) The luminance (Y-channel) is used for S.
  3. SNR (dB) vs. input density (RGBY) SNR (dB) = 20 log10(S/N).
    Shown on the right, above.
  4. Chroma noise vs. input density
  5. Chroma S/N vs. input density
  6. Chroma SNR (dB) vs. input density
  7. CIELAB (L*a*b*) noise
  8. CIELAB (L*a*b*) S/N
  9. CIELAB (L*a*b*) SNR (dB)
  10. Noise vs. pixel (all patches) These measurements are of primary interest in characterizing sensors from RAW (unprocessed or minimally processed) images
  11. S/N vs. pixel (all patches)
  12. SNR (dB) vs. pixel (all patches)
  13. F-stop (scene-ref) noise & Dyn Rng
  14. F-stop (scene-ref) S/N & Dyn Rng. Levels for Dynamic Range are shown as thick horizontal lines (yellow, orange, etc.).
    Shown on the right.
  15. F-stop (scene-ref) SNR (dB) & Dyn Rng
  16. ISO 15739 Visual Noise (L*u*v*)
  17. ISO 15739 Visual S/N (L*u*v*) SNR, as a fraction
  18. ISO 15739 Visual SNR dB (L*u*v*) in decibels (dB) = 20*log10(fraction)
  Panasonic_DMC-GF1_hVFAL_20MM_noise
GF3_DR36_narrow_i160_1010500_noise