Color/Tone & eSFR ISO noise measurements

Introduction

Noise is unwanted variations of the measured signal arising from a variety of causes — heat in electronic circuits, photons incident on the image sensor, and many more. 

It is usually modeled as a gaussian random process, but noise from photons, called photon shot noise, is a Poisson process (whose statistics strongly resemble a gaussian for mean values over 10 occurrences). Gaussian processes are based on the normal distribution, whose probability distribution function is

\(\displaystyle N(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \text{exp}\left(- \frac{(x-\mu)^2}{2\sigma^2} \right)\)   for mean μ and standard deviation σ. 

Noise measurements typically refer to RMS (Root Mean Square) noise, which is equivalent to the standard deviation σ of the signal S.

RMS Noise = N_RMS = σ(S), where σ denotes the standard deviation.

S can be the signal in any of several channels: R, G, B, Y (luminance, typically 0.2125 R + 0.7154 G + 0.0721 B), or a derived channel such as R-Y or B-Y (both used for chroma noise) or L*, a*, b*, or others. 

Signal-to-Noise Ratio (SNR) is often more useful than the noise itself. It can be expressed as either a simple ratio (SNR = S/N) or in decibels (SNR(dB) = 20 log₁₀(S/N) (familiar to electrical engineers). In digital cameras noise arises from several sources.

• fixed noise (sensor dark current noise and thermal (resistive) noise),
• photon shot noise, which increases with the square root of the mean number of photons striking the pixels,
• Dark Signal Nonuniformity (DSNU), which arises from variations in sensitivity of individual pixels, and
• Photo Response Nonuniformity (PRNU), which arises from variations in the gain of transistors in individual pixels, and increases linearly with illumination

Sensor noise and dynamic range can be measured from raw (completely unprocessed) images, as described in Image sensor noise from raw images.

Settings

The Noise & Visual noise settings window can be opened from the modules that support detailed noise measurements.

• from Color/Tone Setup, click on Settings, Visual noise settings.  (The Visual noise settings window contains several noise settings — not just visual noise.)
• from  Color/Tone Auto, click the Settings button in the Color/Tone Auto Chart area, then  press the Noise & Visual Noise settings button. Some settings are also available in the Color/Tone Auto Setup window, a portion of which is shown below.

• from eSFR ISO (run from Rescharts or eSFR ISO Setup), click the More settings button, then the Noise settings button on the left, or click on the Settings dropdown menu, Noise & Visual noise settings…

Noise and Visual noise settings, opened by any of the above actions

Noise & Visual Noise settings window

Noise calculations, lets you trade off calculation detail and speed. Choices are

Density & noise plot x-axis, lets you select the x-axis for noise plots. Choices are

Saturation Level — Standard file is recommended for most demosaiced files. Manually-entered is appropriate for most raw files. Details in the dropdown menu.

Plot Display Parameters —  for visual noise-only. Select details of sigma (noise) display.

Noise nonuniformity correction — The new calculation is strongly recommended. The alternative is no nonuniformity correction, which may be useful for comparing with external calculations on rare occasions.

Viewer Display Settings

CPIQ Display type Presets — 1. None lets you set Display Height (cm), Display Px/Cm, and Distance (cm). The others are presets for these settings.

1. 1. None (manual settings, assumed print at 600 DPI)
   2. Small Print
   3. Large Print
   4. Computer Monitor
   5. 4.5″ Cell Phone
   6. 30″ 4k UHDTV

Visual noise calculation — selects visual noise calculations, which can be slow.

1. 1. No Visual noise calculation
   2. Calculate ISO 15739 Visual noise
   3. Calculate CPIQ Visual noise
   4. Calculate both ISO, CPIQ Vis noise

Noise displays

Twenty-seven different noise plots are available. All measurements except for 10-12 are for the grayscale patches-only. Settings 10-12 apply to all patches (including color); these are of particular interest for analyzing Image sensor noise from raw images.

[ *Notes:  2., 3.:  Noise calculation must be set to at least 2 =  Elementary or 3 = Advanced;
V.  The ISO 15739 Visual Noise checkbox must be checked. ]

Here is a typical noise result for a JPEG image of the X-Rite Colorchecker. The plot shown normalized noise for the grayscale patches (bottom row).

This plot shows noise, in units of percentage of maximum pixel level, for the R, G, B, and Y (luminance) channels for the grayscale patches (in the bottom row for the Colorchecker). 11. Noise plots is selected in the Display area, and the details of the plot (noise measurements and units) are selected below Display. (The noise dropdown menu is where you get to select one of 27 plot choices.)

A checkbox below the Settings button allows you to display tonal response (Log pixel level vs. Input density) above the noise (or S/N or SNR (dB)) plot. Here is a result from a JPEG image of the Imatest 36-patch Dynamic Range chart.)

Dynamic Range and how to measurement is described in great detail on the Dynamic Range page.

Log pixel level (Tonal Response curve) and Scene-reference (f-stop) noise vs. input density for grayscale patches.
R, G, B, and Y channels. Sony A6000, ISO 100.

The Dynamic Range for this JPEG image seems optimistic because of the strong “shoulder” (low contrast region) on the right. The shoulder is not present in raw image captures, regardless of whether or not they are demosaiced.

Imatest Dynamic range

Dynamic Range (DR) is the range of tones over which a camera responds with good image quality (typically specified by Signal-to-Noise Ratio, SNR) and contrast. It can be measured in f-stops, which are equivalent to zones or EV (factors of two), in decibels (dB), or density units, where one density unit = 3.322 f-stops = 20dB).

It is described in great detail in the Dynamic range page, 

Imatest has several types of Dynamic Range calculation, which are cross-referenced here.

In Color/Tone (Interactive or Auto), Dynamic Range (based on Scene-Referenced noise or SNR) is defined as density range (in the original scene) over which the scene-referenced RMS noise measured in f-stops (the inverse of the f-stop signal-to-noise ratio, SNR), remains under a specified maximum value and the contrast is adequate. The lower the noise (the higher the SNR, which is the inverse of f-stop noise), the better the image quality. SNR tends to be worst in the darkest regions. Imatest calculates the dynamic range for several minimum SNR levels, from 10 (20 dB; high image quality) to 1 (0 dB; low quality).

• The dynamic range corresponding to SNR = 1 is closest to the intent of the ISO Dynamic range measurement in section 6.3 of the ISO noise measurement standard: ISO 15739: Photography — Electronic still-picture imaging — Noise measurements. Since the Imatest measurement differs in several details from ISO 15739, the results are close but not identical. They may be compared in the ISO 15739 SNR plots, described below.

Transmission step chart

Several transmission step charts are listed in the Dynamic Range page. In this page, we focus on the recommended Imatest 36-patch Dynamic Range charts. which come in several variants with density step of approximately 0.1 in light regions and maximum densities of approximately 3.6, 6, and 8.2. 

A nearly circular patch arrangement ensures that light falloff (vignetting) has minimal effect on results. A CSV or CGATS reference file with actual densities is required (An individually-measured density file is supplied with each chart.) the chart has an active area of 7.75×9.25 inches on 8×10 inch color film.

It also contains slanted edges in the center and corners with 4:1 contrast for measuring MTF. The edges have an MTF50 >= 16 cycles/mm, which is about 3 times better than the best inkjet charts. Registration marks make the regions easy to select, and fully automated region detection is available in Color/Tone. A neutral gray background helps ensure that the chart will be well-exposed in auto exposure cameras (compared to charts with black backgrounds, which are sometimes strongly overexposed). Darker backgrounds and masks are available.

Single-layer 36-patch Dynamic Range charts can be used with the Dynamic Range postprocessor to measure dynamic ranges larger than 11 f-stops if several manual exposures are available. But we don’t generally recommend this because the important effects of veiling glare (flare or stray light) are not measured. 

Imatest 36-patch Dynamic Range chart on 8×10 film. Dmax ≈ 3.4.

Imatest 36-patch High Dynamic Range (HDR) chart on 8×10 film. Dmax ≥ 6.0.

This chart is produced with a high-precision LVT film recording process for the highest density range, lowest noise, and finest detail. The HDR chart on the right has one or two extra layers covering the bottom 14 patches.

Lightbox

You’ll need a lightbox that can evenly illuminate the transmission step chart. 9×12 inches is large enough in most cases. Avoid thin or “mini” models, which may not have even enough illumination. Our lightbox product lineup is frequently updated.

Imatest lightboxes and other uniform light sources are listed here.

A typical Imatest LED Lightbox (shown on the right), Imatest lightboxes have several attributes.

• Highly uniform illumination.
• High-quality spectral response. You to choose color temperatures, typically 3100K, 4100K, 5100K, 5500K, and 6500K color temperature with a Color Rendering Index (CRI) of 97. NIR channels of 850nm and 940nm are available. The box comes with a multi-channel mix mode so you can mix a combination of channels for a continuous range of color temperatures.
• Intensity is adjustable via a hardware knob, WIFI, or USB from 30-10,000 lux: a range of over 300:1, making it suitable for measurements from near-daylight to extremely dim light.

To measure dynamic range, read the detailed instructions on Dynamic Range, which are summarized below.

• Place the chart on the lightbox— or any source of uniform diffuse light. Be sure to block direct light from the lightbox outside of the chart: Stray light can reduce the measured dynamic range; it should be avoided.
• Photograph the chart in a darkened room. No stray light should reach the front of the target; it will distort the results. The surroundings of the chart should be kept as dark as possible to minimize flare light.
• The chart does not need to fit the frame.
• Use your camera’s histogram to determine the minimum exposure that saturates the lightest region of the chart. Overexposure (or underexposure) reduces the number of useful zones. The lightest region should have a relative pixel level of at least 0.98 (pixel level 250 or 255); otherwise, the full dynamic range of the camera will not be detected. If the lightest zone is below this level, and a transmission chart is selected, a Dynamic range warning is issued.
• Photograph the chart following instructions in Using Color/Tone Interactive or Using Color/Tone Auto.

The dynamic range is the difference in density between the zone where the pixel level is 98% of its maximum value (250 for 24-bit color, where the maximum is 255), estimated by interpolation, and the darkest zone that meets the measurement criterion. The repeatability of this measurement is better than 1/4 f-stop.

Imatest Dynamic range results

Here are results for the Panasonic G3 (a Micro Four-Thirds camera with 3.75 micron pixel pitch) at ISO 160, converted from raw using LibRraw with the following settings: Demosaicing: Normal RAW conversion (demosaiced), Output gamma: 2.2, White Balance: Camera, Output color space: 48-bit, Quality: Default.

The Dynamic Range at low-quality level (scene-referenced SNR = 1) is 9.16 f-stops, decreasing to 4.39 f-stops at high-quality level (SNR = 10). These results are unchanged for 24-bit raw conversion.

The shape of the response curve is a strong function of the conversion software settings. The plot below is for the same exposure, saved as a JPEG file inside the camera. Note that the transfer curve is quite different: it has a “shoulder” in the highlights, which improves pictorial quality by reducing the tendency of highlights to saturate (“burn out”). Dynamic Range is improved due to software noise reduction (absent in the LibRaw conversion).

Panasonic G3, ISO 160, in-camera JPEG. Note the “shoulder.”
Dynamic range is improved due to software noise reduction.

Several options are available in the Display area and in the Settings window for selecting x and y-axis units. To convert dynamic range from f-stops into decibels (dB), the measurement normally given on sensor data sheets, multiply the dynamic range in f-stops by 6.02 (20 log₁₀(2)). The dynamic range for low quality (f-stop noise = 1; SNR = 1) corresponds most closely to the number on the data sheets. Measured dynamic range is often lower than specified sensor dynamic range because of lens flare and other factors.

The presence of a strong shoulder in the highlights—where the slope of the transfer curve decreases to near-zero, or flat—indicates a loss of contrast in the highlight regions. The shoulder may have some pictorial advantages: it reduces highlight “burnout”.

Quality-based dynamic range calculations are limited by the slope of the transfer curve in the highlight regions. Shoulder regions of the transfer curve where the slope is less than 0.05 (near-zero) do not contribute to the quality-based dynamic range. On noise analysis displays with quality-based dynamic range values, this manifests as a gap between the saturation region and the High/M-H/Med/Low quality level bars, as shown below. 

Shoulder regions of the transfer curve with near-zero slope indicate a severe loss of contrast in the highlights,
and therefore do not contribute to dynamic range.

Chroma noise

Two types of chroma noise display are available.

Standard ISO-based chroma noise.  Displays 4-6 show the noise and SNR for the Y, R-Y, and B-Y channels as well as total noise using two different weightings.  Luminance channel Y = 0.2125 R + 0.7154 G + 0.0721 B). The total weighted noise used in the ISO 12232-2006 and ISO 15739-2013 standards (based on the paper, ISO 12232 revision: determination of chrominance noise weights for noise-based ISO calculation (http://dx.doi.org/10.1117/12.582447) by Sean Kelly and Brian Keelan) is plotted as a solid black line (with ◊ markers). Total weighted noise is σ(D) = [ σ(Y)² + 0.279 σ(R-Y)² + 0.088 σ(B-Y)² ]^1/2.  The older total weighted noise ([ σ(Y)² + 0.64 σ(R-Y)² + 0.16 σ(B-Y)² ]^1/2) used in ISO 12232:1998 and 15739-2003 is obsolete and may be deprecated in a future Imatest release.

Signal/Noise for R-Y and B-Y (chroma) and Y (Luminance) channels,
as well as Total weighted noise from SO 12232-2006 and ISO 15739-2013 (—◊—).

L* a* b* (CIELAB) noise is potentially interesting because CIELAB was designed to be a perceptually-uniform space. Although it is less uniform than intended— which is why color difference metrics like ΔE-2000 were developed— it is still much more uniform than many other representations, such as RGB. CIELAB noise is not a standard measurement. There is very little literature on using CIELAB for noise measurements. even though a* and b* might be a better measure of chroma noise than R-Y and B-Y (above). We are not aware of any weighting factors (for combining L*, a*, and b* into a single number).

ISO 15739 and CPIQ Visual noise

Imatest 4.0+ includes a calculation of ISO 15739 visual noise as specified in Appendix B of the ISO 15739:2013 standard. And Imatest 4.4+ includes a calculation of visual noise from the IEEE CPIQ P1858 (Camera Phone Image Quality) 2016 specification, which is based on ISO 15739 with some added details. These are relatively complex calculations that include the effects of the human visual system. It is performed for all grayscale patches for up to three viewing conditions (viewpoints), each of which is specified by viewing distance and display (picture) height. The vertical display angle (in degrees) is

Viewing angle = 360/pi*atan(.5*picture height/viewing distance)

We summarize the calculation in the table below. Details can be found in the ISO 15739:2013 standard, Appendix B and in the IEEE CPIQ P1858 (Camera Phone Image Quality) 2016 specification.

Rube Goldberg would have been impressed. His spirit is with us.

 Noise settings window

To adjust visual noise settings

• In Color/Tone Interactive, click Settings… (on the right). This opens the Color/Tone Settings window, which has a Noise & Dyn Rng section in the middle that contains several settings. Click on Visual noise settings.
• In Color/Tone Auto, open the image, then press Continue (not express), or from the Imatest main window, click Settings (just below the Color/Tone Auto chart selection), then click Noise & Visual noise settings.
• In eSFR ISO (run from Rescharts or eSFR ISO Setup), click the More settings button, then the eSFR ISO Noise settings button, or click on the Settings dropdown menu, Noise & Visual noise settings…

These selections open the Noise and Visual Noise Settings window, shown on the right.

Noise Settings (Color/Tone…) near the top are for noise in general.

Except for Plot Display Parameters, settings are repeated from the Settings window.

• Noise Calculations – refers to non-visual noise: 1. None (fastest), 2. Elementary, and 3. Advanced (slowest). 2. Elementary is a good choice for most applications.
• Density & Noise plot x-axis – is relatively self-explanatory
• Saturation level – is significant when  the saturation differs from the maximum for the bit depth (65535 for bit depth = 16, etc.), for example, when 12-bit data is stored in a 16-bit file. 
• Plot Display Parameters – lets you choose which of the results (combinations of L*, u*, v*, and Visual noise) to display. When all are chosen, the plot is cluttered.
• Noise nonuniformity correction – Setting 1 is recommended because it compensates for nonuniform illumination, so it isn’t mistaken for noise.

Viewer Display Settings are for Visual noise.

Visual noise requires a viewpoint, which consists of display height, viewing distance, and device resolution in pixels/cm. The viewing angle is calculated from the distance and height.

    Older noise settings (no longer active)

CPIQ Display-type Presents contains 6 settings:

1 is for None (manual settings; no preset). You need to enter the viewpoint (Height, Px/CM, Distance Cm), and you can add one or two additional viewpoints. If a second viewpoint has been entered, you can select a third, which will be halfway between the first two viewpoints. You can also select Visual Noise Calculations, which is disabled (always set to 3. CPIQ) for presets 2-6. 

2-6 are presets for a for several displays: 2. Small print, 3. Large print, 4. Computer monitor, 5. Small cell phone,  and 6. UHD TV. Details are in the dropdowns.  Height, Px/Cm, and Distance are disabled and contain the preset settings. Visual Noise Calculations is set to 3. Calculate CPIQ Visual Noise, even if the input setting is different. Use 1. None (…) to select a different setting or keep the input setting.

Because visual noise is expressed as the standard deviations of the L*, u*, and v* channels, whose scaling is unfamiliar to most engineers and scientists, Imatest offers three displays of visual noise: one of the noise itself and two of signal-to-noise ratios (expressed as a fraction and as decibels (dB)).

16. ISO 15739 Visual Noise (L*u*v*)
17. ISO 15739 Visual S/N (L*u*v*)  S/N (Signal-to-Noise Ratio, as a fraction)
18. ISO 15739 Visual SNR dB (L*u*v*) SNR dB (Signal-to-Noise Ratio in decibels (dB) = 20*log₁₀(fraction))

We present two examples. The first is for an X-Rite Colorchecker for three viewpoints. All results are presented. (The plot is somewhat cluttered, but still readable).

Visual SNR (Signal / Noise) for three viewpoints for an X-Rite Colorchecker.
On the y-axis, 10¹ corresponds to 20 dB; 10² corresponds to 40 dB.

The second example shows L* noise (σ_L*) and total visual noise (V) for the same three viewpoints for the Imatest 36-patch Dynamic Range chart. The absolute numbers are somewhat difficult to interpret due to their unfamiliarity: that’s why Imatest also displays Signal-to-Noise Ratios (as fractions and in dB). Noise levels for input densities above (to the left of) 2.0 or 2.5 have little meaning because the image is nearly black (though these levels may be increased by tone-mapping or digital “dodging”).Visual noise for three viewpoints for the Imatest 36-patch Dynamic Range chart.

ISO 15739 SNR and Dynamic Range

The ISO 15739 Dynamic Range (DR) calculation allows DR to be estimated from a patch with density = 2.0, which is available on many semigloss reflective test charts (though density = 2— where 1/100 of the incident light is reflected— cannot be achieved with matte media. The response is fairly accurate for standard (linear) sensors, but is not suitable HDR (High Dynamic Range) sensors. It does not take the effects of flare light into account.

Notes on ISO 15739 SNR and Dynamic Range calculations

• These calculations, which are described in sections 6.2 and 6.3 and Annex D of the ISO 15739:2013 Standard, are completely independent of the ISO 15739 Visual Noise measurements, described in Annex B.
• Imatest’s ISO 15739 SNR calculation is primarily for total SNR (§6.2.3). Temporal noise and SNR have been added in Imatest 5.1. There are two calculation methods:  (1) from the difference of two images (divided by √2) and (2) from multiple images (typically 8), following ISO 15739. We recommend method (1) (difference of two images), which is faster and equally accurate. Fixed pattern noise can be measured by averaging at least 8 identical images (by reading multiple images and specifying signal averaging).

ISO 15739 SNR (S/N) for the Imatest 36-patch Dynamic Range chart.
Plain pixel S/N (plot 2) in Cyan; Imatest (f-stop) scene-referenced S/N (plot 14) in Red.

The large magenta diamond ♦ (above) is located at luminance level \(L_{SNR} = 0.13 \times L_{ref}\), specified for measuring Signal-to-Noise Ratio (SNR) in the ISO standard §6.2.2, equation (4). L_ref is the luminance corresponding to pixel level 245 (of 255 for bit depth = 8), i.e., a little below saturation. 0.13 corresponds to 17.7 dB (0.886 Optical Density).

When ISO 15739 SNR is displayed in Color/Tone, or eSFR ISO, it is compared with the older Imatest f-stop-based calculation. Dynamic Range results tend to be fairly similar. The ISO calculation has two advantages. 1. It conforms to a standard and 2. it allows Dynamic Range to be calculated from SNR for a patch with density near 2— which is achievable with (semigloss) reflective media. This calculation uses extrapolation, which is less direct than finding the density where SNR = 1 (which generally requires transmissive media or several exposures). But in our experience, it produces useful results that are close enough to the direct measurement for practical work, i.e., for comparing different cameras in a standardized way.

Scene-referenced noise and SNR are the basis of Imatest dynamic range measurements.