Color/Tone & eSFR ISO noise measurements

Introduction

Noise is measured for charts with large enough patches to obtain good noise statistics, such as the 24-patch X-Rite Colorchecker or the circular grayscale near the center of ISO 12233:2017 Edge SFR charts. Prior to running any of modules that can analyze noise, you need to select the amount of noise detail to calculate (1. none; 2. elementary; 3. advanced; Visual: a classic tradeoff of detail versus speed; the numbers refer to the table below).

Settings

• In Color/Tone Setup, click on Settings, Noise & Visual noise settings…
• For Color/Tone Auto, click the Setup button (to the right of the wide Muticharts… button), then  press the Noise & Visual Noise settings button. Some settings are also available in the Color/Tone Auto Setup window, a portion of which is shown below.
  
• In eSFR ISO (run from Rescharts or eSFR ISO Setup), click the More settings button, then the eSFR ISO Noise settings button, or click on the Settings dropdown menu, Noise & Visual noise settings…

Noise and Visual noise settings, opened by any of the above actions

The first setting, Noise calculations, lets you trade off calculation detail and speed. Choices are

The second setting, Density & noise plot x-axis, lets you select the x-axis for noise plots. Choices are

The third setting, Visual noise display settings, lets you select the visual noise calculation.

To display noise,

Twenty-seven different noise plots are available. All measurements except for 10-13 are for the grayscale patches-only. Settings 10-13 apply to all patches (including color); these are of particular interest for analyzing sensor noise from raw images.

Notes:  2., 3.:  Noise calculation must be set to at least 2 =  Elementary or 3 = Advanced.
V.  The ISO 15739 Visual Noise checkbox must be checked.

Here is a typical noise result for a JPEG image of the X-Rite Colorchecker. The plot is for the grayscale patches (bottom row).

This plot shows noise, in units of percentage of maximum pixel level, for the R, G, B, and Y (luminance) channels for the grayscale patches (in the bottom row for the Colorchecker). 11. Noise plots is selected in the Display area, and the details of the plot (noise measurements and units) are selected just below Display.

A checkbox below the Settings button allows you to display tonal response (Log pixel level vs. Input density) above the noise (or S/N or SNR (dB)) plot. Here is a result from a JPEG image of the Imatest 36-patch Dynamic Range chart.

Log pixel level and Scene-reference (f-stop) noise vs. input density for grayscale patches.
R, G, B, and Y channels. Panasonic G3, ISO 160.

Noise fundamentals

Noise measurements typically refer to RMS (Root Mean Square) noise, which is identical to the standard deviation of the signal S.

RMS Noise = N =σ(S), where σ denotes the standard deviation.

S can be the signal in any of several channels: R, G, B, Y (luminance, typically 0.2125 R + 0.7154 G + 0.0721 B), or a derived channel such as R-Y or B-Y (both used for chroma noise) or L*, a*, b*, or others. See Color/Tone/eSFR ISO noise measurements.

Signal-to-Noise Ratio (SNR) is an important measurement derived from noise and is often considered more important than the noise itself. It is expressed as either a simple ratio (SNR = S/N) or in decibels (SNR(dB) = 20 log₁₀(S/N) (familiar to electrical engineers). In digital cameras noise consists of two parts: fixed noise (sensor dark current noise and thermal (resistive) noise) and shot noise, which increases with the square root of the mean number of photons striking the pixels. This relationship enables us to derive sensor dynamic range from raw (completely unprocessed) noise, as described below.

Imatest Dynamic range

Dynamic Range (DR) is the range of tones over which a camera responds with good image quality (typically specified by Signal-to-Noise Ratio, SNR) and contrast. It is usually measured in f-stops, which are equivalent to zones or EV (factors of two). It can also be measured in density units, where one density unit = 3.322 f-stops or decibels (dB, where 1 density unit = 20dB).

Imatest has several types of Dynamic Range calculation, which are cross-referenced here.

In Color/Tone (Interactive or Auto), Dynamic Range (based on Scene-Referenced noise or SNR) is defined as density range (in the original scene) over which the scene-referenced RMS noise measured in f-stops (the inverse of the f-stop signal-to-noise ratio, SNR), remains under a specified maximum value. The lower the noise (the higher the SNR, which is the inverse of f-stop noise), the better the image quality. SNR tends to be worst in the darkest regions. Imatest calculates the dynamic range for several minimum SNR levels, from 10 (high image quality) to 1 (low quality).

• The dynamic range corresponding to SNR = 1 is closest to the intent of the ISO Dynamic range measurement in section 6.3 of the ISO noise measurement standard: ISO 15739: Photography — Electronic still-picture imaging — Noise measurements. Since the Imatest measurement differs in several details from ISO 15739, the results are close but not identical. They may be compared in the ISO 15739 SNR plots, described below.

Transmission step chart

Several transmission step charts are listed in Using Stepchart. In this page, we focus on the recommended Imatest 36-patch Dynamic Range and High Dynamic Range charts.

The Imatest 36-patch Dynamic Range test chart, with density steps of approximately 0.1 from base density to base+3.4 (an 11.3 f-stop range), or the 36-patch High Dynamic Range (HDR) test chart, with density steps from 0.1 to 0.3 and a maximum density of at least base+6 (20 f-stops), are strongly recommended. A nearly circular patch arrangement ensures that vignetting has minimal effect on results. A CSV or CGATS reference file with actual densities (required with the HDR chart) is supplied. the chart has an active area of 7.75×9.25 inches on 8×10 inch film.

It also contains slanted edges in the center and corners with 4:1 contrast for measuring MTF. The edges have an MTF50 >= 16 cycles/mm, which is about 3 times better than the best inkjet charts. Registration marks make the regions easy to select, and fully automated region detection is available in Imatest 4.0+. A neutral gray background helps ensure that the chart will be well-exposed in auto exposure cameras (compared to charts with black backgrounds, which are sometimes strongly overexposed).

The standard 36-patch Dynamic Range chart can be used with the Dynamic Range postprocessor to measure dynamic ranges larger than 11 f-stops if several manual exposures are available. Dmax = 3.4 is sufficient for camera phones and digital cameras with small pixel sizes, but high-end DSLRs and HDR security and automotive cameras generally have higher dynamic ranges, which makes them well suited for measurements with either Dynamic Range or the HDR chart (which never requires the Dynamic Range postprocessor).

Imatest 36-patch Dynamic Range chart on 8×10 film. Dmax ≈ 3.4.

Imatest 36-patch High Dynamic Range (HDR) chart on 8×10 film. Dmax > 6.0.

This chart is produced with a high-precision LVT film recording process for the highest density range, lowest noise, and finest detail.

Lightbox

You’ll need a lightbox that can evenly illuminate the transmission step chart. 9×12 inches is large enough in most cases. Avoid thin or “mini” models, which may not have even enough illumination. Fluorescent light boxes should have high-frequency ballasts to eliminate flicker. The GLE-10 and GLE-16E, available in the Imatest store are recommended. You can find available lightboxes we offer here.

The best available lightbox is the Imatest LED Lightbox (shown on the right), available from the Imatest store. It has several advantages over the lightboxes mentioned in the previous paragraph.

• Much more uniform illumination.
• High-quality spectral response. The standard version allows you to choose between 3100K, 4100K, 5100K, 5500K, and 6500K color temperature with a Color Rendering Index (CRI) of 97. NIR channels of 850nm and 940nm are available. The box comes with a multi-channel mix mode so you can mix a combination of channels for a continuous range of color temperatures.
• Intensity is adjustable via a hardware knob, WIFI, or USB from 30-10,000 lux: a range of over 300:1, making it suitable for measurements from near-daylight to extremely dim light.

To measure dynamic range,

• Place the chart on the lightbox— or any source of uniform diffuse light. Be sure to block direct light from the lightbox outside of the chart: Stray light can reduce the measured dynamic range; it should be avoided.
• Photograph the chart in a darkened room. No stray light should reach the front of the target; it will distort the results. The surroundings of the chart should be kept as dark as possible to minimize flare light.
• Use your camera’s histogram to determine the minimum exposure that saturates the lightest region of the chart. Overexposure (or underexposure) reduces the number of useful zones. The lightest region should have a relative pixel level of at least 0.98 (pixel level 250 or 255); otherwise, the full dynamic range of the camera will not be detected. If the lightest zone is below this level, and a transmission chart is selected, a Dynamic range warning is issued.
•  

  For flatbed scanners with transparency units (TPUs, i.e., light sources for transparencies), you can simply lay the chart down on the glass. Stray light shouldn’t be an issue, though there is no harm in keeping it to a minimum.

• Photograph the chart following instructions in Using Color/Tone Interactive or Using Color/Tone Auto.

The dynamic range is the difference in density between the zone where the pixel level is 98% of its maximum value (250 for 24-bit color, where the maximum is 255), estimated by interpolation, and the darkest zone that meets the measurement criterion. The repeatability of this measurement is better than 1/4 f-stop.

Imatest Dynamic range results

Here are results for the Panasonic G3 (a Micro Four-Thirds camera with 3.75 micron pixel pitch) at ISO 160, converted from raw using dcraw with the following settings: Demosaicing: Normal RAW conversion (demosaiced), Output gamma: 2.2, White Balance: Camera, Output color space: 48-bit, Quality: Default.

Panasonic G3, ISO 160, Converted with dcraw, run with Color/Tone Interactive.
Compare with Stepchart results.

The Dynamic Range at low-quality level (scene-referenced SNR = 1) is 9.31 f-stops, decreasing to 5.15 f-stops at high-quality level (SNR = 10). These results are unchanged for 24-bit raw conversion and very close to the results from Stepchart.

The shape of the response curve is a strong function of the conversion software settings. The plot below is for the same exposure, saved as a JPEG file inside the camera. Note that the transfer curve is quite different: it has a “shoulder” in the highlights, which improves pictorial quality by reducing the tendency of highlights to saturate (“burn out”). Dynamic Range is improved due to software noise reduction (absent in the dcraw conversion).

Panasonic G3, ISO 160, in-camera JPEG. Note the “shoulder.”
Dynamic range is improved due to software noise reduction.
Compare with Stepchart results.

To convert dynamic range from f-stops into decibels (dB), the measurement normally given on sensor data sheets, multiply the dynamic range in f-stops by 6.02 (20 log₁₀(2)). The dynamic range for low quality (f-stop noise = 1; SNR = 1) corresponds most closely to the number on the data sheets. Measured dynamic range is often lower than specified sensor dynamic range because of lens flare and other factors.

Chroma noise

Two types of chroma noise display are available.

Standard ISO-based chroma noise.  Displays 4-6 show the noise and SNR for the Y, R-Y, and B-Y channels as well as total noise using two different weightings.  Luminance channel Y = 0.2125 R + 0.7154 G + 0.0721 B). The total weighted noise used in the ISO 12232-2006 and ISO 15739-2013 standards (based on the paper, ISO 12232 revision: determination of chrominance noise weights for noise-based ISO calculation (http://dx.doi.org/10.1117/12.582447) by Sean Kelly and Brian Keelan) is plotted as a solid black line (with ◊ markers). Total weighted noise is σ(D) = [ σ(Y)² + 0.279 σ(R-Y)² + 0.088 σ(B-Y)² ]^1/2.  The older total weighted noise ([ σ(Y)² + 0.64 σ(R-Y)² + 0.16 σ(B-Y)² ]^1/2) used in ISO 12232:1998 and 15739-2003 is obsolete and may be deprecated in a future Imatest release.

Signal/Noise for R-Y and B-Y (chroma) and Y (Luminance) channels,
as well as Total weighted noise from SO 12232-2006 and ISO 15739-2013 (—◊—).

L* a* b* (CIELAB) noise is potentially interesting because CIELAB was designed to be a perceptually-uniform space. Although it is less uniform than intended— which is why color difference metrics like ΔE-2000 were developed— it is still much more uniform than many other representations, such as RGB. CIELAB noise is not a standard measurement. There is very little literature on using CIELAB for noise measurements. even though a* and b* might be a better measure of chroma noise than R-Y and B-Y (above). We are not aware of any weighting factors (for combining L*, a*, and b* into a single number).

ISO 15739 and CPIQ Visual noise

Imatest 4.0+ includes a calculation of ISO 15739 visual noise as specified in Appendix B of the ISO 15739:2013 standard. And Imatest 4.4+ includes a calculation of visual noise from the IEEE CPIQ P1858 (Camera Phone Image Quality) 2016 specification, which is based on ISO 15739 with some added details. These are relatively complex calculations that include the effects of the human visual system. It is performed for all grayscale patches for up to three viewing conditions (viewpoints), each of which is specified by viewing distance and display (picture) height. The vertical display angle (in degrees) is

Viewing angle = 360/pi*atan(.5*picture height/viewing distance)

We summarize the calculation in the table below. Details can be found in the ISO 15739:2013 standard, Appendix B and in the IEEE CPIQ P1858 (Camera Phone Image Quality) 2016 specification.

Rube Goldberg would have been impressed. His spirit is with us.

To adjust visual noise settings

• In Color/Tone Interactive, click Settings, ISO speed, noise, mxn chart… Then click Visual noise settings.
• In Color/Tone Auto, click Setup (to the right of Multitest… in the Imatest main window), then click Noise & Visual noise settings.
• In eSFR ISO (run from Rescharts or eSFR ISO Setup), click the More settings button, then the eSFR ISO Noise settings button, or click on the Settings dropdown menu, Noise & Visual noise settings…

These selections open the Noise and Visual noise window, shown above, which allows you to select up to three viewpoints (viewing positions based on distance and picture height, both in cm).

To obtain visual noise results, the viewing distance and picture height in cm for the first viewpoint is required. The viewing angle is calculated from the distance and height.

If these numbers have been entered you can select a second viewpoint, and specify the second viewing distance and picture height. If a second viewpoint has been entered you can select a third, which will have distance and picture height exactly half-way between the first and second viewpoints.

Because visual noise is expressed as the standard deviations of the L*, u*, and v* channels, whose scaling is unfamiliar to most engineers and scientists, Imatest offers three displays of visual noise: one of the noise itself and two of signal-to-noise ratios (expressed as a fraction and as decibels (dB)).

16. ISO 15739 Visual Noise (L*u*v*)
17. ISO 15739 Visual S/N (L*u*v*) Signal-to-Noise Ratio, as a fraction
18. ISO 15739 Visual SNR dB (L*u*v*) Signal-to-Noise Ratio in decibels (dB) = 20*log₁₀(fraction)

We present two examples. The first is for an X-Rite Colorchecker for three viewpoints. All results are presented. (The plot is somewhat cluttered, but still readable).Visual SNR (Signal / Noise) for three viewpoints for an X-Rite Colorchecker.
On the y-axis, 10¹ corresponds to 20 dB; 10² corresponds to 40 dB.

The second example shows L* noise (σ_L*) and total visual noise (V) for the same three viewpoints for the Imatest 36-patch Dynamic Range chart. The absolute numbers are somewhat difficult to interpret due to their unfamiliarity: that’s why Imatest also displays Signal-to-Noise Ratios (as fractions and in dB). Noise levels for input densities above (to the left of) 2.0 or 2.5 have little meaning because the image is nearly black (though these levels may be increased by tone-mapping or digital “dodging”).Visual noise for three viewpoints for the Imatest 36-patch Dynamic Range chart.

ISO 15739 SNR and Dynamic Range

The ISO 15739 Dynamic Range (DR) calculation allows DR to be estimated from a patch with density = 2.0, which is available on many semigloss reflective test charts (though density = 2— where 1/100 of the incident light is reflected— cannot be achieved with matte media. The response is fairly accurate for standard (linear) sensors, but is not suitable HDR (High Dynamic Range) sensors. It does not take the effects of flare light into account.

Notes on ISO 15739 SNR and Dynamic Range calculations

• These calculations, which are described in sections 6.2 and 6.3 and Annex D of the ISO 15739:2013 Standard, are completely independent of the ISO 15739 Visual Noise measurements, described in Annex B.
• Imatest’s ISO 15739 SNR calculation is primarily for total SNR (§6.2.3). Temporal noise and SNR have been added in Imatest 5.1. There are two calculation methods:  (1) from the difference of two images (divided by √2) and (2) from multiple images (typically 8), following ISO 15739. We recommend method (1) (difference of two images), which is faster and equally accurate. Fixed pattern noise can be measured by averaging at least 8 identical images (by reading multiple images and specifying signal averaging).

ISO 15739 SNR (S/N) for the Imatest 36-patch Dynamic Range chart.
Plain pixel S/N (plot 2) in Cyan; Imatest (f-stop) scene-referenced S/N (plot 14) in Red.

The large magenta diamond ♦ (above) is located at luminance level \(L_{SNR} = 0.13 \times L_{ref}\), specified for measuring Signal-to-Noise Ratio (SNR) in the ISO standard §6.2.2, equation (4). L_ref is the luminance corresponding to pixel level 245 (of 255 for bit depth = 8), i.e., a little below saturation. 0.13 corresponds to 17.7 dB (0.886 Optical Density).

When ISO 15739 SNR is displayed in Color/Tone, or eSFR ISO, it is compared with the older Imatest f-stop-based calculation. Dynamic Range results tend to be fairly similar. The ISO calculation has two advantages. 1. It conforms to a standard and 2. it allows Dynamic Range to be calculated from SNR for a patch with density near 2— which is achievable with (semigloss) reflective media. This calculation uses extrapolation, which is less direct than finding the density where SNR = 1 (which generally requires transmissive media or several exposures). But in our experience, it produces useful results that are close enough to the direct measurement for practical work, i.e., for comparing different cameras in a standardized way.

F-stop noise and scene-referenced SNR

Scene-referenced SNR is the basis of Imatest dynamic range measurements.