Modules

Imatest consists of several modules, each of which analyzes images of one or more test charts. Image files can be in any of several standard formats (TIF, high quality JPEG, PNG, etc.). Imatest Master also analyzes Bayer RAW files. A Test chart Cross-reference follows.

	SFR  measures the sharpness of cameras and lenses using a simple slanted-edge target (either the industry-standard ISO 12233 chart or a target you can print yourself on a high quality inkjet printer). Its standardized sharpening algorithm allows you to compare digital cameras on a fair basis. It also analyzes Chromatic Aberration and noise and calculates the Subjective Quality Factor (SQF)— a measurement of how sharp a print of a given size will appear, based on MTF and the eye's Contrast Sensitivity Function (CSF). It can analyze regions as small as 10x10 pixels.
	MTF Compare (Imatest Master only) is a post-processor for Imatest SFR that compares the MTFs of different cameras, lenses, and imaging systems using saved data from CSV files. These comparisions are far more detailed than comparisions of MTF50 or other summary results.
	Colorcheck  measures a camera's color quality, tonal response, exposure accuracy, and noise using the GretagMacbethTM ColorChecker®, which is available from Adorama and other sources.
	Stepchart  measures a camera's tonal response, noise, and dynamic range using grayscale step charts from Kodak (the Q-13/Q-14), Jessops, or Danes-Picta (Czech Republic), or transmission step wedges from Stouffer, Danes-Picta, or Kodak. It also measures exposure accuracy and veiling glare (susceptibility to lens flare) with reflection step charts. It also works with several charts (some ISO standards) from Applied Image. The Dynamic Range postprocesser for Stepchart, measures dynamic range from Stepchart results for several differently-exposed stepchart images.
	Multicharts analyzes several color charts for tonal response and color accuracy using a highly interactive interface. It also analyzes grayscale charts for tonal response. It currently supports the standard 24-patch GretagMacbethTM ColorChecker® and the IT8.7 in Imatest Studio and Master and the ColorChecker SG, several linear grayscale stepcharts, and patterns of squares arranged on a circle (for analyzing custom "pie" charts) in Imatest Master.
	Print Test  measures the quality (color response, color gamut, tonal response, and Dmax) of printers, inks, paper, ICC profiles, and rendering intents. A version of Print Test with additional capabilities is included in Gamutvision.
	Light Falloff (Uniformity) measures the light falloff (vignetting) of lenses. In Imatest Master it also measures several types of image sensor nonuniformity, including color shading, detailed noise distribution, and medium-sized (visible) nonuniformities, inculding dust smudges. It also flags stuck (dead and hot) pixels. A histogram display makes it easy to select dead/hot pixel thresholds. Display options include contours superposed over images and pseudocolor with color bar.
	Distortion measures lens distortion and calculates coefficients for correcting it using a square or rectangular grid that can be printed from files created by Test Charts.
	Rescharts analyzes several resolution-related test charts using a highly interactive interface. Functions include Slanted-edge SFR, which largely duplicates the calculations in SFR, Log frequency, which measures MTF and color moiré using a chart that varies in spatial frequency, and Log frequency-contrast (Imatest Master only), which measures MTF and loss in fine detail due to software noise reduction using a chart that varies in spatial frequency on one axis and contrast on the other.
	Test Charts creates image files for printing test charts on high quality inkjet printers. Charts include SFR slanted-edge images, star charts, patterns with varying spatial frequency and contrast, and zone plates. Several patterns can be used for observing aliasing, Color Moire, detail lost to software noise reduction, and other phenomena. Numerous options are available, including contrast ratio, highlight color, and sine or bar pattern (for certain charts). Both bitmap and Scalable Vector Graphics (SVG) charts, which can be printed any size without loss of quality, are available.
	Concise instructions for testing lenses using Imatest SFR.
	Gamutvision is a standalone program (sold separately from Imatest) for exploring the behavior of color management, gamut mapping, and rendering intents, incorporating Print Test. See precisely how gamut mapping affects colors. Assess ICC profile quality. Learn how working color space affects print quality. Determine the capabilities of printers and papers using downloaded ICC profiles. Compare them with scanned prints from your own printer. And much more...

Imatest is frequently updated. Major releases are announced on the Imatest main page. All releases are described in detail in the Change Log. You can download and install a new version at any time without uninstalling the old version.

Imatest is written in compiled Matlab, Release 13 (Version 6.5.1), an outstanding language for solving engineering problems. Algorithms can be rapidly coded and modified in response to user requests. It is a standalone program; Matlab does not need to be installed, but the Matlab runtime library needs to be downloaded and installed once (the first time Imatest is installed). In most cases this is done automatically.

Requirements:  Windows 2000, 2003, XP, Vista, or later, or Macintosh computers with Virtual PC 6 or 7. Minimum RAM is 256 MB (512 MB recommended), and minimum screen size is 1024x768 pixels.

Supported input file formats:  TIFF, PNG, or PPM (all 24 or 48-bit), JPEG, BMP, GIF, HDF, PCX, XWD, as well as RAW files from most digital cameras, using Dave Coffin's dcraw.

Imatest output is optionally written to two types of file: .CSV (comma-separated variable; Excel-readable), and XML.

To run Imatest in Windows Vista, you'll need version 2.3.3 or later.

Test chart cross-reference

The table below lists commercially-available charts. Additional charts that you can print a high quality inkjet printer are described in Test Charts. Cross references contains this material sorted by quality factor and module.

Try Imatest

You can download an evaluation version that allows you to make up to 20 runs of the individual modules. It has all the capabilities of the full versions except that (1) you can't save results, and (2) a watermark appears in the background of the figures.

You may purchase Imatest at any time by from our secure online store by going to the purchase page. The price includes one year of updates. You may also purchase Imatest via wire transfer or custom Paypal invoice. You can register Imatest as soon as the purchase is complete. Details can be found on Installing Imatest and getting started.

Learn more about Imatest

All Imatest documentation is available online. The Documentation page contains the table of contents, and the Cross-reference indexes image quality factors, modules, and test charts. The FAQ answers many questions.

Many of the concepts used by Imatest to measure sharpness and image quality are new to photographers and require some study. Sharpness: What is it and how is it measured? is a good place to start. It outlines the basic principles behind SFR. Standardized sharpening: why it's needed for comparing cameras, Chromatic aberration, and Subjective Quality Factor (SQF) contain additional concepts. Shannon information capacity is a relatively new concept for imaging, still under development— its measurement is distorted by ubiquitous software noise reduction.

Using Imatest contains instructions that are common to all modules. Instructions for SFR, Colorcheck, Stepchart, Print test, and additional modules are on individual pages.

Many of the terms used in the documentation are defined in the Glossary.

An Imatest forum has been established for posting questions and responses. The Change Log describes the Imatest version.

The Imatest license

The Imatest license for Light and Pro allows an individual user to register and use the software on (A) a maximum of three computers (for example, home, laptop, and office) used exclusively by a single individual, or (B) a single workstation used nonsimultaneously by multiple people, but not both. It is not a concurrent use license. The full license may be viewed here.

License holders are encouraged to publish test results in printed publications, websites, and discussion forums, provided they include links to www.imatest.com. The use of the Imatest Logo is encouraged. However you may not use Imatest for advertising or product promotion without the explicit permission of Imatest LLC. Contact us to learn more.

Imatest LLC assumes no legal liability for the contents of published reviews. If you plan to publish test results, you should take care to use good technique. See Using Imatest for more details.

Links

Volker Gilbert has written an excellent French language description of Imatest. (PDF version)

Colorcheck

Colorcheck analyzes images of the GretagMacbeth ColorChecker for tonal response, gamma, noise, and color fidelity.

Colorcheck: color accuracy and tonal response

Imatest™ Colorcheck analyzes images of the X-Rite (formerly GretagMacbeth^TM ) ColorChecker^® for color accuracy, tonal response, gamma, and noise. It is particularly useful for measuring the effectiveness of White Balance algorithms and settings under a variety of lighting conditions. The ColorChecker is available from ColorHQ, Adorama, and other suppliers.

You can select the Colorchecker reference values (using standard values or the contents of a data file) and one of six color spaces: sRGB, Adobe RGB (1998), Wide Gamut RGB, ProPhoto RGB, Apple RGB, or ColorMatch. Danny Pascale/Babelcolor's page on the Colorchecker contains nearly everything you want to know about the chart.

To run Colorcheck, photograph the ColorChecker, taking care to avoid glare. For best results, the chart image should be 500-1500 pixels wide, though smaller images produce satisfactory results. For testing white balance, you can photograph a ColorChecker in a scene. The ColorChecker image can be very small if a noise analysis is not required. Load the image file, crop it (if needed), then make any needed changes to the input dialog box (for color space, which defaults to sRGB, etc.).

The first figure: gray scale analysis

shows the tonal response and noise of the gray patches at the bottom of the Colorchecker and and selected EXIF data..

The second figure: noise detail (Imatest Master only)

shows the density response, noise in f-stops (a relative measurement that corresponds to the workings of the eye), noise for the third Colorchecker row, which contains primary colors, and the noise spectrum of the selected patch.

The third figure: color error shows the color error (the difference between the measured and ideal (reference) values) in the a*b* plane of the CIELAB color space, which is relatively perceptually uniform (not perfect, but far more uniform than the common xy Chromaticity diagram). The small squares are the ideal values; the large circles are the measured (camera) values. The chroma (related to the perception of saturation) of an individual color is proportional to its distance from the origin (a* = b* = 0). The mean camera chroma (the average of all camera chroma values) relative to the mean ideal chroma is displayed on the upper right. You can select one of three color error metrics: ΔE*ab (CIE 1976; the familiar Euclidian distance), ΔE*94 (CIE 1994), and ΔE*CMC. ΔE*94 and ΔE*CMC are more complex but more accurate. Mean and RMS (root mean square; σ) or maximum color errors (measured with and without a correction for camera chroma boost or cut) can be displayed. RMS error gives more weight to large errors. Calculation details are given in the Colorcheck Appendix.
Images can be analyzed in sRGB, Adobe RGB (1998), Wide Gamut RGB, ProPhoto RGB, Apple RGB, and ColorMatch RGB color spaces. The light gray curve is the boundary of the sRGB color space.
When you interpret the results, keep the following in mind. Camera manufacturers do not necessarily aim for accurate color reproduction, which often appears flat and dull. They recognize that there is a difference between accurate and pleasing color: most people prefer deep blue skies, saturated green foliage, and warm, slightly saturated skin tones. Ever mindful of the bottom line, they aim to please. Hence they often boost chroma (i.e., saturation), especially in blues, greens, and skin tones. Colorcheck can be used to check white balance algorithms and camera performance under different lighting conditions. John Beale's excellent article on Lighting and Color uses this Colorcheck figure to examine the effects of different lighting on color accuracy.
The fourth figure: color analysis is an image of the ColorChecker with the correct colors superposed in the center of each patch. The colors in the central squares are corrected for the difference in luminance between the exposed and the ideal values. The colors in the small rectangles to the right of central squares are uncorrected. The gray patches with exaggerated White Balance error are shown on the bottom. White balance error is displayed in HSV Saturation units, degrees Kelvin, and Mireds (1000/degrees K; a useful unit for filtration).
You can learn more about Colorcheck from Using Colorcheck

Stepchart

Measures tonal response, noise, and dynamic range using step charts

Imatest™ Stepchart analyzes the tonal response, noise, and dynamic range of digital cameras and scanners using

• reflection step charts such as the Kodak Q-13 and Q-14 Gray Scales (below, right), the Jessops Colour & Mono Separation Guide (UK), or any of several charts from Danes-Picta (Czech republic),
• transmission step charts from Stouffer, Kodak, Danes-Picta, Esser Test Charts, or
• Charts from Applied Image and additional ISO charts, shown below. (Imatest Master only; starting with version 1.7)

Stepchart can also measure veiling glare (lens flare) using reflection step charts. Results are more detailed than those provided by Colorcheck. Transmission step charts are recommended for measuring dynamic range. Full instructions can be found on Using Stepchart. Noise is explained in Noise in photographic images.

Applied Image and ISO charts: Imatest Master only
QA-61 ISO- 16067-1 Scanner	QA-62	ST-51 EIA Grayscale	ST-52 ISO-14524 12-patch OECF target	ISO-15739 Noise target
	20-patch OECF targets	ITE Grayscale

To run Stepchart, photograph the chart, taking care to avoid glare (shiny reflections). Scale the image for 50 pixels per zone for greatest noise analysis accuracy; 20 pixels per zone is adequate in most cases. Fewer are OK for tonal response curves. Load the image file, crop it (if needed), then specify the target density step and type (reflective or tramsmission). The default is a reflective target with density step = 0.1.

The two Figures below illustrate the results of analyzing a Q-13 image photographed with the Canon EOS-10D at ISO 100. A third figure (not shown here) displays density response for all channels (Y, R, G, and B) in color images.

The first figure contains basic tonal response and noise measurements. The horizontal axis for both plots is the chart zone— proportional to the distance along the chart. Density (–Log Exposure) increases by a fixed step (0.10 or 0.15, depending on the target) for each zone.   The upper plot shows the normalized pixel level of the grayscale patches (black curve) and first and second order density fits (dashed blue and green curves). Gamma is derived from the first order fit. The acvite areas used for the analysis is shown as think pale pink bars. The lower plot shows the RMS noise for each patch: R, G, B, and Y (luminance), expressed as the percentage of the pixel level difference corresponding to a target density range of 1.5: the same as the white - black patches on the GretagMacbeth ColorChecker. For this camera, the pixel difference is 197.5. Noise measured in pixels can be calculated by multiplying the percentage noise by 197.5. Noise can also be normalized to the maximum pixel level (255). The average noise for each channel (excluding the lightest and darkest zones) is displayed. See Noise in photographic images for a detailed explanation of noise.

The second Figure

contains the most important results:

• the response curve displayed on a log scale, similar to film response curves,
• noise expressed units of f-stops (the same as EV), explained below, or SNR (Signal-to-Noise Ratio), explained here,
• noise in normalized pixel levels
• dynamic range (for transmission step charts),
• the noise spectrum.

The horizontal axis for the three plots is Log Exposure, which equals (–) the nominal target density (0.05 - 1.95 for the Q-13/Q-14). This axis is reversed from Figure 1.

SNR (S/N = 1/f-stop noise), in middle-left plot in place of f-stop noise.

Why measure noise in f-stops? Because the human eye responds to relative luminance differences. That's why we think of exposure in terms of zones, f-stops, or EV (exposure value), where a change of one unit corresponds to halving or doubling the exposure. The eye's relative sensitivity is expressed by the Weber-Fechner law,     ΔL ≈ 0.01 L   –or–   ΔL/L ≈ 0.01 where ΔL is the smallest luminance difference the eye can distinguish. (This equation is approximate; effective ΔL tends to be larger in dark areas of scenes and prints due to visual interference from bright areas.) Expressing noise in relative luminance units, such as f-stops, corresponds more closely to the eye's response than standard pixel or voltage units. Noise in f-stops is obtained by dividing the noise in pixels by the number of pixels per f-stop. (I use "f-stop" rather than "zone" or "EV" out of habit; any of them are OK.) noise in f-stops = noise in pixels / (d(pixel)/d(f-stop)) = 1/SNR where  d(pixel)/d(f-stop) is the derivative of the pixel level with respect to luminance measured in f-stops (~log2(luminance) ).           SNR is the Signal-to-Noise Ratio. The above-right image illustrates how the pixel spacing between f-stops (and hence d(pixel)/d(f-stop)) decreases with decreasing brightness. This causes f-stop noise to increase with decreasing brightness, visible in the middle-left plot of the second figure, above. Since luminance noise (measured in f-stops) is referenced to relative scene luminance, independently of electronic processing or pixel levels, it is a universal measurement that can be used to compare digital sensor quality when sensor RAW data is unavailable.

Dynamic range

Dynamic range is the range of brightness over which a camera responds. It is usually measured in f-stops, or equivalently, zones or EV. It can be specified in two ways:

• The total range. Stepchart measures a camera's total dynamic range, including noisy dark areas.
• A range of tones over which the RMS noise, measured in f-stops, is under a maximum specified value. The lower the maximum noise value, the better the image quality, but the smaller the dynamic range. Noise tends to be worst in the darkest regions. Imatest calculates the dynamic range for several maximum noise levels, from RMS noise = 0.1 f-stop (high image quality) to 1 f-stop (relatively low quality).
  

Dynamic range is measured using transmission step charts, which have density ranges of at least 3.0: a 1000:1 ratio (10 f-stops). Reflective charts such as the Q-13 have a density range of only around 1.90; an 80:1 ratio (6.3 f-stops), insufficient for digital cameras. Two charts are recommended for digital SLRs, which can have over 10 f-stops of total dynamic range.

• The Stouffer T4110 transmission step wedge, which has a maximum density of 4.05 (41 steps, 0.1 density increment, 13.3 f-stops dynamic range)
• The Danes-Picta TS28D (on their Digital Imaging page), which has a maximum density of 4.2 ( 28 steps, 0.15 density increment, 13.6 f-stops dynamic range)

The dynamic range is the difference in density between the zone where the pixel level is 98% of its maximum value (250 our of 255 for 24-bit color), estimated by interpolation, and the darkest zone that meets the measurement criterion. The repeatability of this measurement is around 1/3 f-stop.

The figure below illustrates results for the Canon EOS-10D, taken from a JPEG image acquired at ISO 400 and converted with Canon Zoom Browser set for low contrast. A Kodak step tablet (density from 0.05 to 3.05 in steps of 0.15) was used. The total dynamic range is 8.58 f-stops (probably limited by the target; the Stouffer T4110 or Danes-Picta TS28D would have done a better job). It is displayed at the bottom of the upper-left plot (Density response). Total dynamic range changes little for 48-bit TIFF conversion or ISO 100. But 48-bit TIFF conversion has lower noise, hence higher dynamic range at any given quality level.

A medium quality image can ge achieved over a range of 8.21 f-stops out of a total dynamic range of 8.58 f-stops. When a high quality image is required (maximum noise = 0.1 f-stops), the dynamic range is reduced to 5.97 f-stops (indicated by the yellow line on the middle left plot). The best slide films have a total dynamic range of 5 to 6 f-stops.

The second plot contains all important Stepchart results.

The shape of the response curve depends strongly on the conversion software and settings.

Compact digital cameras have much higher noise levels, hence lower useful dynamic range, even though their total dynamic range may be quite large. Roger N. Clark, a space scientist and avid photographer has done a study of dynamic range. He finds digital (SLRs) to be superior to film.

Scanner results

Here are the results of scanning the Kodak step tablet with the Epson 3200 scanner (with transparency unit) set for negatives.

A few observations on the scanner results:

• All 20 steps of the Kodak step tablet were detected. The total dynamic range is greater than 3 density units (the 10 f-stop range of the tablet). To determine the true total dynamic range we would need the Stouffer T4110 step wedge, which has a maximum density of 4.
• The response closely follows an exponential curve with gamma = 0.506. No "S" curve has been superposed.
• The practical dynamic range is limited by noise. It is 9.62 f-stops (2.9 density units) for a medium quality image.
• The flat noise spectrum indicates that no software noise reduction has been applied. Most digital cameras have rolloffs in their noise spectra due to Bayer interpolation and noise reduction.
  

SFR

Measures image sharpness using a slanted-edge target

SFRplus

Automated measurement of image sharpness and other quality factors

Imatest™ SFRplus measures image sharpness and several additional image quality factors, including Lateral Chromatic Aberration (LCA), noise, distortion and tonal response, using a special test chart that provides a high degree of automation.

The primary sharpness indicator is MTF50, the spatial frequency where contrast drops to half its low frequency value. Spatial Frequency Response (SFR), also known as Modulation Transfer Function (MTF), is introduced in What is image sharpness and how is it measured?

Full instructions for using SFRplus and interpreting its results can be found in Using SFRplus Part 1: Setup, Part 2: Running, and Part 3: Results. 

SFRplus runs in two modes. To use SFRplus, Purchase an SFRplus test chart and mount it on a rigid substrate (typically 1/2 inch foam board).

Standard SFRplus test chart: 5x9 grid, 10:1 and 2:1 contrasts,
with 20-step 4x5 stepchart (0.1 density increment).

• Photograph it so that there is a small amount of white space on the top and bottom. It should be centered, but the sides don't matter. Lighting should be even and glare-free.

• Run the Rescharts SFRplus module. The SFRplus parameters & setup window allows you to choose criteria for automatic ROI selection as well as several additional settings. Pressing Display options brings up even more settings.

SFRplus parameters & setup window showing well-framed SFRplus chart image;
9 regions selected for analysis

SFRplus results in Rescharts window: Multiple region (ROI) summary

The multi-ROI summary results shown in the Rescharts window (above) is the best summary of SFRplus results. It is described in detail in Multiple ROI (Region of Interest) plot. The upper left contains the image in muted gray tones. The selected regions are surrounded by red rectangles and displayed with full contrast. Four results boxes are displayed next to each region. There is a legend below the image. Distortion statistics are shown in the lower left. SMIA TV distortion is the simplest overall measure of distortion.

Edge and MTF display in Rescharts window

The Edge and MTF display is identical to the SFR Edge and MTF display. MTF is explained in Sharpness: What is it and how is it measured? The average edge (or line spread function) is plotted on the top and the MTF is plotted on the bottom. There are a number of readouts, including 10-90% rise distance, MTF50, MTF50P (the spatial frequency where MTF is 50% of the peak value; differing from MTF50 only for oversharpened pulses), the secondary readout (MTF20 in this case; selectable to MTFnn or MTFnnP at any contrast level nn or MTF at a spatial frequency specified in cycles/pixel, line pairs/inch or line pairs/mm.), and the MTF at the Nyquist frequency (0.5 cycles/pixel).

Chromatic Aberration Lateral Chromatic Aberration (LCA), also known as "color fringing," is most visible on tangential boundaries near the edges of the image. Much of the plot is grayed out if the selected region (ROI) is too close to the center (less than 30% of the distance to the corner) to accurately measure CA. The area between the highest and lowest of the edge curves (shown for the R, G, B, and Y (luminance) channels) is a perceptual measurement of LCA. It has units of pixels because the curves are normalized to an amplitude of 1 and the x-direction (normal to the edge) is in units of pixels. It is displayed as a magenta curve.	Lateral Chromatic Aberration
Perceptual LCA is also expressed as percentage of the distance from center to corner, which tends to be more reflective of system performance: less sensitive to location and pixel count than the pixel measurement. Values under 0.04% of the distance from the center are insignificant; LCA over 0.15% can be quite visible and serious. Information for correction LCA (R-G and B-G crossing distances) is also given in units of % (center-to-corner) and pixels. LCA can be corrected most effectively before demosaicing. Results are explained in Chromatic Aberration ... plot.
SQF (Subjective Quality Factor) SQF is a perceptual measurement of the sharpness of a display (monitor image or print). MTF, by comparison is device sharpness (not perceptual sharpness). SQF includes the effects of the human visual system's Contrast Sensitivity Function (CSF), print (or display) size, and viewing distance (which is assumed to be proportional to print height, by default). See Introduction to SQF for more detail. SQF was developed by Eastman Kodak scientists in 1972. It has been verified and used inside Kodak and Polaroid, but it has remained obscure until now because it was difficult to calculate. Its only significant public exposure has been in Popular Photography lens tests	SQF (Subjective Quality Factor)
Tonal response & gamma This display is derived from the 4x5 stepchart pattern, located just below the central square of the SFRplus test chart. It resembles the third figure in Stepchart. The upper plot shows the tonal response for all colors. The lower plot shows instantaneous gamma— the slope (derivative) of tonal response. The value of gamma may differ slightly from the values in the Edge response and MTF display because it's calculated differently-- based on the average slope of the light to middle tone squares of the stepchart.	Tonal response & gamma
Histograms and noise statistics This display contains histograms of pixel levels for individual ROIs (original on top and linearized using input gamma on bottom). The black (background) histogram contains pixel levels for the entire ROI. The red histogram is for the light region, away from the transition, used in the noise statistics calculation. The cyan histogram is for the dark region. Sharpening may cause extra bumps to appear in the black histogram. A detailed signal and noise analyis, described here, is shown below the histograms. Noise calculations are made in portions of the ROIs (Regions of Interest) away from the transitions. The noise spectrum on the lower right contains qualitative information about noise visibility and software noise reduction, which generally reduces high frequency noise below 0.5, typical of demosaicing alone. The region on the right is shown in red; left is shown in cyan. See also Noise.	Histograms and noise stats

Uniformity (Light Falloff)

Measures light falloff (lens vignetting) and sensor uniformity

Imatest™ Uniformity (Light Falloff)  measures lens vignetting (dropoff in illumination at the edges of the image) as well as other types of image, illumination, and sensor nonuniformities. Examples include

• the uniformity of flash units (using light bounced off a white wall),
• the uniformity of flatbed scanners,
• sensor noise and spots of the type caused by dust on the sensor (Imatest Master only).

Display options include contours on top of images and pseudocolor with color bar. In Imatest Master , Light Falloff can also display a histogram of pixel levels, analyses of stuck (hot and dead) pixels and color shading, and a detailed image of fine nonuniformities.

Simplified instructions

To prepare an image for Light Falloff, set your camera lens for manual focus and focus it at infinity (the worst case for light falloff). Light Falloff works without a lens if the sensor is evenly illuminated. Photograph an evenly lit subject— a smooth sheet of matte paper (white or gray) is ideal. Get close to the subject, but not so close that you shade it. A diffuser near the lens can be helpful. Outdoors illumination (in the shade) often works best; it's easiest to get even illumination. Getting a uniform image can be a challenge with ultrawide lenses, such as the Canon10-22 mm digital zoom lens at 10mm (equivalent to 16 mm in the 35mm format); it's almost impossible to avoid shading part of the card. For such extreme lenses, a good quality light table with an additional diffuser in front of the lens may be necessary.

The subject does not need to be in focus; the goal is to measure lens light falloff and sensor uniformity, not features of the subject. For typical lens vignetting measurements, set exposure compensation to overexpose by about one f-stop. (You may, however, use any exposure you choose.) More details can be found in the Light Falloff instructions.

Channel contour plot

shows contours of the level (normalized or not) of the selected channel (Red, Green, Blue, or Y (luminance = Y = 0.30*R + 0.59*G + 0.11*B)) of the image file. The contours are superposed on top of the image. In this unnormalized display, the maximum value of 1 corresponds to pixel level = 255 in image files with bit depth = 8. Some lightning nonuniformity is evident in the plot: the top is brighter than the bottom. The contours are derived vrom a smoothed (lowpass filtered) version of the image.

The text displays the maximum luminance value, the worst and mean corner values (in normalized pixel levels and percentage of maximum), and the side values. Two hot and two dead pixels (which were simulated) were detected with thresholds of 246 and 10 (pixels), respectively. More details of the hot and dead pixels are in the CSV and XML output files. Selected EXIF data is shown on the right.

The setting for correcting light falloff in the Picture Window Pro Light falloff transformation is also given. The PW pro Light Falloff dialog box is shown on the right. Film Size (mm) remains at 36 (the PW Pro default value: the width of a 35mm frame). Picture Window Pro is the powerful and affordable photographic image editor that I use for my own work. The Lens Focal Length is rarely the exact focal length of the lens. Light falloff depends on the lens aperture (f-stop) as well as a number of lens design parameters. Lenses designed designed for digital cameras, where the rays emerging from the rear of the lens remain nearly normal (perpendicular) to the sensor surface, tend to have reduced light falloff.

For aesthetic purposes I generally recommend undercorrecting the image, i.e., using a larger Lens Focal Length. This makes the edges somewhat darker, which is usually pleasing. Ansel Adams routinely burned (darkened) the edges of his prints. Part of the reason was that he had to compensate for light falloff from his enlarger (when he wasn't contact printing).

"My experience indicates that practically every print requires some burning of the edges, especially prints that are to be mounted on a white card, as the flare from the card tends to weaken visually the tonality of the adjacent areas. Edge burning must never be overdone..."

Ansel Adams, "The Print," p. 66. 1966 edition.

F-stop contour plot

shows contours of the normalized channel levels (R, G, B, or Y) of the image file, measured in f-stops. A pseudocolor display with color bar has been selected. The colors in the color bar are fixed: colors always vary from white at 0 f-stops to black at -4 f-stops and darker. For this plot to be accurate, a good estimate of gamma (the camera's intrinsic contrast) is required. Gamma is measured by Stepchart, using any one of several widely-available step charts. (Reflection charts are easiest to use but transmission charts can also be used to measure dynamic range.)

Gamma can be tricky to measure for several reasons. (1) Many cameras have complex response curves, for example, "S"-curves superposed atop gamma curves. This means that gamma can vary with brightness. (2) Some cameras employ adaptive signal processing in their RAW conversion algorithms. This increases contrast (i.e., gamma) for low contrast subjects and decreases it for contrasty subjects. This improves pictorial quality for a wide range of scenes, but makes measurements difficult, especially since Light falloff targets have the lowest possible contrast.

Color shading (nonuniformity)  (Imatest Master only)

Light Falloff displays color nonuniformity. This image shows shading measured as the pixel ratio between the red and blue channels, normalized to 1.0 to minimize the effects of white balance errors. The background shows exaggerated colors (HSV saturation S has been increased by 10x for low saturation values; less for high values.) Several display options are available. Details here.

Uniformity profiles   (Imatest Master only) Uniformity profiles, introduced in Imatest Master 2.3.16 (August 2007), displays profiles of image levels along several lines: Diagonal Upper Left-Lower Right, Diagonal Lower Left-Upper Right, Vertical Top-Bottom (center), and Horizontal Left-Right (center). Several display options are available, including unnormalized levels (max 1), pixel levels (max 255), normalized levels (max 2), and R/G and B/G ratios (G constant). Details here.

Histograms  (Imatest Master only)

The histogram plot, introduced in Imatest Master 2.3.11 (July 2007), facilitates the detection and the setting of thresholds for stuck (hot, dead, etc.) pixels. Histograms of log₁₀(occurrences+1) are displayed for the red, green, and blue channels. Details here.

In this example, single stuck pixels are plainly visible near levels 0 and 252. You can quickly see if the thresholds are set correctly— if they are outside the valid density region and if the dead and hot pixels are above and below their respective thresholds. You can change threshold settings and rerun Light Falloff if needed.

Noise detail  (Imatest Master only)

shows an exaggerated image of the noise detail, a pseudocolor image of image variation, or a pseudocolor image emphasizing spots.. The algorithm is in the Light Falloff for Imatest Master instructions.

The first image (below) shows noise detail for the Canon EOS-20D at ISO 1600 with the 10-22mm lens set to f/4.5. No surprises here; electronic noise dominates.

The second image (below) shows noise detail for the Canon EOS-20D at ISO 100 with the 10-22mm lens set to f/8. Thanks to the small aperture, dust spots are visible. The dust is on the anti-aliasing/infrared filter/microlens assembly in front of the sensor. This assembly can be well over 1 millimeter thick. Stopping the lens down (increasing the f-stop setting) reduces the size of dust spots but makes them darker.

This image has a surprise in the form of concentric circles: bands where the noise appears to be higher or lower than the remainder of the image. These bands could be caused caused by the Analog-to-Digital (A-D) converter in the image sensor chip or by JPEG compression. Sensors can have small discontinuities when, for example going from level 127 to 128: binary 01111111 to 10000000. Recall that the nonuniformities are exaggerated by a factor of 10: they would be invisible or barely visible on most images, but you might seem then faintly in smooth areas like skies.

Here is the same image, displayed in pseudocolor (which shows the amount of variation) with a color bar, and including a histogram (of individual pixels, not the smoothed image used to generate the pseudocolor contour plot on the left). This histogram will usually be narrower than the histogram plot shown above because variations at low spatial frequencies have been (lowpass) filtered out.

The image below is an enlargement (a zoom) of the above image that includes the dust spot to the left of the center, shown with superposed contours.You can zoom into an image by using the mouse to draw an rectangle, or by simply clicking on a feature you want to enlarge. You can restore the original image by double-clicking anywhere on the image.

Distortion

Measures radial lens distortion

Imatest™ Distortion

• measures radial lens distortion, an aberration that causes straight lines to curve,
• calculates coefficients for removing it, and
• provides additional information on geometric distortions in digital images.

Distortion models assume circular symmetry about a central point, where undistorted and distorted radii r_u and r_d (distances from the center) are related by equations of the form r_u = f( r_d ), where f is one of several functions.

In the simplest distortion (third order) model,

r_u = r_d + k₁ r_d³.

Because this third-order equation is not adequate for all lenses, Distortion also calculates the coefficients for the fifth-order model,

r_u = r_d + h₁ r_d³ + h₂ r_d⁵.

Distortion also calculates the coefficients for Picture Window Pro, which uses a tangent/arctangent distortion model,  

r_u = tan(10 p₁ r_d )/ (10 p₁ ) ;      h₁ > 0   (barrel distortion)
r_u = tan^-1(10 p₁ r_d )/ (10 p₁ ) ;    h₁ < 0   (pincushion distortion)

Distortion has two forms, barrel (k₁ > 0) or pincushion (k₁ < 0), as illustrated below.

None

Barrel

Pincushion

Distortion tends to be most serious in extreme wide angle, telephoto, and zoom lenses. It is most objectionable in architectural photography and photogrammetry — photography used for measurement (metrology). It can be highly visible on tangential lines near the boundaries of the image, but it isn't visible on radial lines. Distortion by Paul van Walree is excellent background reading.

To measure distortion, you'll need a rectangular or (preferably) square grid pattern, which you can create using Test Charts. Print the chart, photograph it (taking care to avoid glare; matte surface recommended), and enter the image into the Imatest Distortion module.

An example of Distortion output is shown below for the Sigma 18-125 mm f/3.5-5.6 DC lens (designed for APS-C-sized sensors, such as the Canon EOS-30D, Digital Rebel, Nikon D100, D70, etc. The Sigma is an excellent lens — a bargain — except for its autofocus. Mine doesn't autofocus as reliably as Canon lenses, but it works beautifully on manual. The autofocus problem is plainly visible when working with the distortion chart.

The Sigma has modest amounts of pincushion distortion at 125 mm and barrel distortion at 18mm, its widest angle setting. Results displayed below the image, in the column on the left, include,

• TV Distortion from the SMIA specification, §5.20. Referring to the image on the right,
  
       SMIA TV Distortion = 100( A-B )/B ;
       A = ( A₁+A₂ )/2
  
  The box on the right is described in the SMIA spec as "nearly filling" the image. Since the test chart grid may not do this, Distortion uses a simulated box whose height is 98% that of the image. Note that the sign is opposite of k₁.
  

  Although any number in this list can be used as a measure of distortion, SMIA TV distortion may be the best choice because it's the easiest to visualize.

• Coefficient k₁ from the third-order equation, r_u = r_d + k₁ r_d³  where r is normalized to the center-corner distance. k₁ = 0 for no distortion; k₁ < 0 for pincushion distortion; k₁ > 0 for barrel distortion.
• Coefficients h₁ and h₁ from the fifth-order equation, r_u = r_d + h₁ r_d³ + h₂ r_d⁵.
• The Lens Distortion correction coefficient and scale factor for Picture Window Pro. The sign is the same as k₁.

Distortion also calculates decentering: the deviation of the center of distortion symmetry from the geometric center of the image. Decentering can result from poor manufacturing quality or mechanical shock (whoops!). Details are in the Distortion instructions.

The plot includes arrows that illustrate the change in radius when distortion is corrected. Distortion was too low on the above plot to make the arrows visible. They are illustrated in the plot below for a large amount of simulated barrel distortion.

Distortion can also be used with a single line if its curvature is visible. Long tangential lines near the edge of the image are preferred. The ISO 12233 chart contains two such lines, which are adequate but not ideal: they would be better if they were thinner and further from the image center. Here is an example, illustrating a modest amount of pincushion distortion.

In Imatest Master, Distortion  can also produce an intersection points figure, showing points where lines cross, and a radius correction figure, showing fine details of the distortion profile.

Print Test

Measures print quality factors: color response, tonal response, and Dmax

Introduction to Print Test

Imatest™ Print Test measures several of the key factors that contribute to photographic print quality,

• Color response, the relationship between pixels in the image file and colors in the print,
• Color gamut, the range of saturated colors a print can reproduce, i.e., the limits of color response,
• Tonal response, the relation between pixels and print density, and
• Dmax, the deepest printable black tone: a very important image quality factor,

using a simple test pattern scanned on a flatbed scanner. No specialized equipment is required. Print Test can also be used to study the effects of gamut mapping: transformations between different color spaces or devices.

Print quality depends on the

• Printer,
• Paper (manufacturer's or independent),
• Ink (manufacturer's or independent),
• Working color space,
• ICC profile,
• Rendering intent (one of four rules, embedded in ICC profiles, for mapping pixels between color spaces or devices such as monitors and printers
• RIP (Raster Image Processor), optional software packages available as alternatives to standard printer drivers and profiles, and
• Color engine (probably a minor factor); you can choose between Windows ICM 2, Adobe Color Engine (ACE), and Little CMS.

Even if you have only one printer, many options may be available. I use an Epson R2400 printer with standard Ultrachrome inks, but I can choose among a staggering variety of papers and profiles. And the working color space and profile rendering intent can also make a difference.

Print Test can answer questions such as,

• What range of colors can my printer/paper/ink can reproduce, i.e., what is its gamut?
• How dark a black tone can my printer/paper/ink make?
• How good are my ICC profiles? Do they provide smooth, uniform response? Under what conditions do colors saturate? What are their specific strengths and weakness?
• How do colors map between color spaces and devices? What difference does working color space and rendering intent make?

You may, of course, test a new ink/paper/profile combination using one or more test images. Such tests are important, even if you use Print Test. The problem is that one image (say, a landscape) may look fine while another (say, a portrait) may not. The reason lies in details of the tonal and color response revealed by Print test, which can spare you from unpleasant surprises and help you find superior ink/paper/profile combinations.

The test pattern, shown on the right, was generated using the HSL color representation.

• H = Hue varies from 0 to 1 for the color range R→ Y→ G→ C→ B→ M→ R (horizontally across the image on the right). (0 to 6 is used in the Figures below for clarity.)
• S = Saturation = max(R,G,B)/min(R,G,B). S = 0 for gray; 1 for the most saturated colors.
• L_HSL = Lightness = (max(R,G,B)+min(R,G,B))/2. (The HSL subscript is used to distinguish it from CIELAB L.) L = 0 is pure black; 1 is pure white. The most visually saturated colors occur where L_HSL = 0.5.

The key zones are,

1. S=1 square. A pattern consisting of all possible hues (0 ≤ H ≤ 1) and lightnesses (0 ≤L_HSL≤ 1), where all colors are fully saturated (S = 1), i.e., as saturated as they can be for the lightness value.
2. L=0.5 rectangle. A pattern consisting of all possible hues (0 ≤ H ≤ 1) and saturation levels (0 ≤ S ≤ 1) for middle lightness (L_HSL = 0.5), where colors are most visually saturated.
3. Two identical monochrome tone scales, where pixel levels vary linearly, 0 ≤ {R=G=B}≤ 1.

The zones labelled K, Gry, and W are uniform black (pixel level = 0), gray (pixel level = 127), and white (pixel level = 255), respectively.

Just by looking at the printed test pattern you can see response irregularities with greater detail than with some expensive instruments.

Results

Results below are for Epson 2200 printer with Ultrachrome inks, Premium Luster paper, and the standard 1440 dpi Epson profile.

Density

The first Figure contains the grayscale density response and Dmax, the maximum density of the print, where density is defined as –log₁₀(fraction of reflected light). The value of Dmax (2.24; 0.58% reflectivity) is the average of the upper and right black areas.

The upper plot shows –print density as a function of log₁₀(original pixel level (in the test pattern)/255). This corresponds to a standard density-log exposure characteristic curve for photographic papers. The blue plot is for the upper grayscale; the black plot is for the lower-right grayscale. Somewhat uneven illumination is evident. The print density values are calculated from Q-13 calibration curve (lower right). The thin dashed curves contain –log₁₀(pixel levels).

The lower left curve is the characteristic curve (print vs. original normalized pixel level) on a linear scale.

The lower right curve is the results of the (separate) Q-13 calibration run used to calibrate Dmax and the density plot.

S=1 La*b* Gamut map

The S=1 gamut map displays results for region 1, which contains all possible hues (0 ≤ H ≤ 1) and lightnesses (0 ≤ L_HSL ≤ 1) with maximum HSL saturation (S = 1).

The La*b* gamut map uses the device-independent CIELAB color space. In CIELAB, L is a nonlinear function of luminance (where luminance ≈ 0.30*Red + 0.59*Green + 0.11*Blue), a* represents colors ranging from cyan-green to magenta, and b* represents colors from blue to yellow. The colors of the a*b* plane are approximated in the background of the Figure below.

CIELAB is relatively perceptually uniform, meaning that the visible difference between colors is roughly proportional to the distance between them. It isn't perfect, but it's far better than HSL (where Y, C, and M occupy narrow bands) or or the familiar CIE 1931 xyY color space, where gamuts are represented as triangles or hexagons inside a familiar horseshoe curve. A CIE 1931 xyY gamut map proved to be useless because of its perceptual nonuniformity: values bunched up along edges in a way that made results difficult to interpret. Note: The L-values in the box inside the image below refer to L_HSL in the test file, not CIELAB L.

CIELAB color space is often displayed as a solid 3D volume. But although 3D displays can be visually impressive, they can be difficult to interpret. Print test displays an S=1 La*b* Gamut map as cross sections of the La*b* volume representing a*b* values for test file lightnesses (L_HSL) of {0.1, 0.3, 0.5, 0.7, and 0.9}, corresponding to near black, dark gray, middle gray, light gray, and near white.

Although this display contains less information than the HSL map (below), but the results are clearer and more useful. The solid shapes from white to black) are the measured response for values of L_HSL shown in the legend on the lower left. The dotted shapes represent the gamut of the color space (sRGB, in this case) at each L_HSL level. CIELAB gamut varies with lightness: it is largest for middle tones (L_HSL = 0.5) and drops to zero for pure white and black. This is closer to the workings of the human eye than HSL representation, where hues vary from 0 to 1, even for white and black.

The dotted concentric curves are the twelve loci of constant hue, representing the six primary hues (R, Y, G, C, B, M) and the six hues halfway between them.. They follow different curves above and below L = 0.5. The circles on the solid curves show the measured hues at locations corresponding to the twelve hue loci. Ideally they should be on the loci. The curves for L = 0.5 (middle gray) are outlined in dark gray to distinguish them from the background.

The hexagonal shape of the gamuts makes it easy to judge performance for each primary. The measured (solid) shape should be compared to the ideal (dotted) shape for each level. Weakness in magenta, blue, and green is very apparent, especially at L = 0.5 and L = 0.7. But the gamut is excellent for L = 0.3.

L=0.5 La*b* Saturation map

The L=0.5 Saturation map is for region 2, which contains all possible hues (0 ≤ H ≤ 1) and saturation levels (0 ≤ S ≤ 1) for middle lightness (L_HSL = 0.5), where saturation is highest. This plot,which displays the response to different levels of color saturation, is of interest for comparing different rendering intents (rules that control how colors are mapped when they are transformed between color spaces or color spaces and devices).

Colorimetric rendering intents should leave saturation unchanged unless the output device can't reproduce a color in the color space. Perceptual rendering intent compresses gamut when moving to color spaces with smaller gamut. There is no standard for perceptual rendering intent: every manufacturer does it in their own say. "Perceptual rendering intent" is a vague concept; it's hard to know its precise meaning unless you measure it, which you can do with Print test or with Gamutvision.

Gamuts are shown in the CIELAB a*b* plane for 0 ≤ S ≤ 1 in steps of 0.2. As in the the S=1 La*b* Saturation map, the dotted lines and shapes are the ideal values and the solid shapes are the measured values.

Gamut is excellent for S = 0.2 (brown) and 0.4 (red). But compression becomes apparent around S = 0.4 (red curve) for magenta and blue and S = 0.6 (green curve) for green and cyan, where the printer starts to saturate. Increasing S above these values doesn't increase the print saturation: in fact, saturation decreases slightly at S = 1. This is the result of limitations of the Epson 2200's pigment-based inks and the workings of the standard Epson ICC profile.

With Gamut maps, differences between printers, papers, and profiles are immediately apparent. These differences are far more difficult to visualize without Imatest Print test because each image has its own color gamut. One image may look beautiful but another may be distorted by the printer's limitations.

HSL contour plots

The HSL contour plots below display color response in great detail, but don't contain absolute color information. For that you need the La*b* or CIE 1931 xy plots described above. They work best when the same color space is used to print and scan the target. For these plots to be displayed, Display HSL coutour plots must be checked in the Print test input dialog box.

S=1 HSL Gamut map

The S=1 Figures are for region 1, which contains all possible hues (0 ≤ H ≤ 1) and lightnesses (0 ≤ L_HSL ≤ 1) with maximum HSL saturation (S = 1).

The Gamut map shows the print saturation levels for an image that goes from dark to light with maximum saturation. If the print and scanner response were perfect, S would equal 1 everywhere. Weak saturation is evident in light greens and magentas (L > 0.5). Strong saturation in darker regions, roughly 0.2 ≤ L ≤ 0.4, would seem to indicate that a better profile might perform better in the light green and magenta regions. Weak saturation in very light (L>0.95) and dark areas (L<0.1) is not very visible.

S-1 Lightness and Hue maps are also displayed. They are illustrated in the Print test instructions.

L=0.5 HSL Saturation map

The L=0.5 Figures are for region 2, which contains all possible hues (0 ≤ H ≤ 1) and saturation levels (0 ≤ S ≤ 1) for middle lightness (L_HSL = 0.5), where the greatest saturation takes place.

The ideal L=0.5 Saturation map would consist of uniformly spaced horizontal lines from 0.9 to 0.1. The weak saturation in the greens and magentas is visible here. The L*a*b* Saturation map is more generally usefu-- it also shows the weakness in greens and magentas, but in a clearer device-independent format.

An additional Figure with L=0.5 Hue and Lightness maps has been omitted.

Test Charts

Creates test charts for high quality inkjet printers

Imatest™ Test Charts creates test chart files that can be printed on high quality inkjet printers. A great many options are available. You can select chart type, contrast, highlight color, sine or bar pattern, printer gamma, and more. Bitmap and Scalable Vector Graphics (SVG) charts are available. Charts include

• Bitmap and SVG patterns for use in SFR slanted-edge MTF analysis.
• Star patterns and zone plate, useful for viewing angle-dependent effects and Moiré fringing (colored bands caused by aliasing).
• A log frequency-contrast gradient chart that is useful for viewing Moiré fringing and detail lost to software noise reduction. This loss typically takes place in high spatial frequency/low contrast regions.
• A grid pattern for use with Distortion.

You can either buy test charts or print your own.

Why buy test charts?

• Standardization. You can be confident that your chart is identical to the charts used by others.
• Calibration. Certain charts cannot be properly calibrated when printed on your own printer. These include step charts (the Kodak Q-13/Q-14, etc.) and color charts (the GretagMacbeth Colorchecker, IT-8, etc.). Commercially manufactured charts must be used.
• Resolution. Most commercial charts are printed at higher resolutions than inkjet printers can achieve, and hence may be used at shorter distances— microscopic in some cases. For example, the small Sine Patterns/Applied Image ISO-12233 Resolution Charts (.1X, .5X, and 1X) are printed on photographic paper, which is capable of much finer resolution than inkjet printers. The chart may safely fill the entire frame.

Why print your own?

• Cost. It is much less expensive.
• Convenience. You can use the charts as soon as you print them.
• Quality. Can be excellent with high quality inkjet printers if you photograph them from a sufficient distance.
• Versatility. Many options are available, some of which may not be available on commercial charts. You can use an image editor to modify charts to suit your needs. You can print the chart any size you want: the standard print size is merely a suggestion.

Imatest generates charts that may be impractically large for downloading, but present no problem if generated locally.

Chart patterns

Charts are designed to fit on letter-size (8.5x11 inch), A4, A3, and Super A3/Super B (13x19 inch) paper, but they can be printed any size. A number of options, described below, are available for each chart. The first three patterns, SFR:quadrants, SFR rectangles, and Grid, are available in all versions. The remaining patterns are available in Imatest Master only.

SFR: quadrants  (All versions) This is the standard chart for SFR measurements. It should be tilted when it is photographed. Instructions are found in Using SFR, Part 1 and in How to test lenses with Imatest.

SFR: two rectangles  (All versions) This chart is also used for SFR measurements. It should be cut into two segments (left and right sides) and tilted in a manner similar to the SFR quadrants chart, above, when photographed.

Grid  (All versions) Grid generates test charts for the Distortion module.

Star chart  (Imatest Master only) Star charts can be used for observing performance (SFR, Moire patterns, and other image processing artifacts) at a variety of angles. In addition to the star (which has a selectable number of bands and either bar or sine form), the chart contains tonal calibration patches and slanted edges for checking results with SFR. Star charts will be analyzed in a future version of Imatest.

Log frequency-contrast gradient chart  (1 and 2-band charts in Imatest Master only)

This chart displays bar or sine patterns of increasing spatial frequency. Several features resemble the Koren 2003 test chart described in Lens testing (pre-Imatest). This chart is valuable for viewing Moire fringing (colored bands caused by aliasing) and detail lost by software noise reduction, which takes place inside the camera and/or RAW converter, and which may remain even if you try to turn it off. Software noise reduction cannot distinguish noise from high spatial frequency low contrast detail, located in the lower right of the 1 and 2-band charts. The two and four band charts are designed to be printed on letter-size or A4 paper and cut into segments to be placed around the target, possibly in different orientations. The Log frequency-contrast chart will be analyzed in the Rescharts module, now under development .

	1-band chart
2 and 4-band charts

Each chart consists of the following patterns in 1, 2, or 4 bands. From top to bottom,

1. A bar of increasing spatial frequency (used to determine spatial frequency).
2. A sine or bar pattern of increasing spatial frequency (horizontally) and decreasing contrast (vertically) for 1 and 2-band charts. The contrast is proportional to the square of the distance from the bottom of this band. That emphasizes lower contrast areas, where noise reduction causes a loss of detail. It also corresponds to the way the eye sees, i.e., when the pattern contrast is one quarter of its maximum value, it appears to be about half the maximum. The contrast is constant for the 4-band chart, which is available in Imatest Studio.
3. (1-band chart only) 20 rectangular patches. 1 and 20 are black (0,0,0), 2 and 19 are middle gray, and 3-18 (16 patches) are a stepchart with density increment of 0.1 (if the chart is printed correctly, i.e., printer gamma mathes the Test Charts setting: some care is required).

Zone plate  (Imatest Master only) Zone plates (based on Fresnel diffraction) are useful for observing aliasing, particularly Moire patterns and other image processing artifacts. They have a much larger area with high spatial frequencies than the star chart. Thanks to Bart van der Wolf, who has done some interesting work with zone plates. In addition to the zone plate (which has a selectable number of bands and either bar or sine form), the chart contains tonal calibration patches and slanted edges for checking results with SFR.

Scalable Vector Graphics charts (SVG)   (Imatest Master only) Scalable Vector Graphics (SVG) images print at a printer's maximum quality regardless of size. They do not suffer from the pixellation that takes place when bitmap images are enlarged. The SVG chart shown on the right contains squares with slanted edges, "hyperbolic" wedges and bar patterns for viewing moiré caused by aliasing, and a step chart with density steps of 0.1 when the printer is in calibration. Five of these charts printed letter or A4 size and mounted at the center and corners of a 3x3 grid makes an excellent SFR test target, suitable for testing high quality DSLRs. Described here.

Several of the SVG charts consist of slanted squares arranaged on an m x n grid. A step chart (density increment = 0.1) and a pair of hyperbolic wedges may be optionally added. The chart's contrast, brightness, and several other details are adjustable. A typical SVG  SFR chart, consisting of a 5x9 grid, is shown below. This chart is optimized to be printed large: on 24 inch widebody printers. It offers significant advantages over standard ISO 12233 charts: less wasted area, better suited to automated testing, larger ROI location tolerance, and better contrast control. These advantages are discussed here.

SVG SFR chart: 5x9 grid, 20:1 contrast, with optional hyperbolic wedge
and step patterns (0.1 density increment) .

Options

A number of options are available, including

• Pixels per inch (PPI) for printing the chart. Affects the pixel size of the chart file.
• Chart height (cm) . Choose from 18 cm (for A4), 20 cm (for US letter; 8.5x11 in.), 28 cm (for A3), or 30 cm (for Super A3/Super B; 13x19 in.). The chart can be printed any size. The number of vertical pixels is PPI * Chart height (cm) / 2.54.
• Highlight color. In addition to the standard White/Black (or White/Gray) charts, you can replace white with any additive or subtractive primary color: R, G, B, C, M, or Y. Bayer sensors, used in most digital cameras, have half as many R and B sensors as G. Patterns with different highlight colors may perform differently in sharpness tests. A magenta star pattern is illustrated on the right.
• Chart lightness. Affects the highlight color in charts with grayscale images (Star, Log-frequency, Zone plate). Lightest uses the least ink, but response may be smoother when one of the darker values is chosen.
• Type (bar or sine). A bar star pattern is shown on the right; a sine star pattern is illustrated above.
• Star pattern bands. Multiples of 4. The magenta pattern (on the right) has 36 bands); the above White-Black pattern above has 20 bands.
• Contrast ratio. The ratio of the reflectivity of the light to dark areas (the upper zone only in the Log frequency chart). The maximum contrast ratio depends on the printer and paper. It is typically 100 or more for glossy (or luster) surfaces, but only around 60 for matte surfaces. Maximum print density = -log₁₀(minimum reflectivity) can be measured in Print Test. Reducing contrast ratio reduces clipping, which can be a problem in SFR measurements. The default value is 40 (the minimum recommended by the ISO standard), but 20 is recommended in most cases.
• Gamma. You should use the value of gamma for your typical color space and workflow (2.2 is standard for Windows; 1.8 for older Mac systems). The default value of 2.0 is close enough to both for most purposes. Gamma affects the accuracy of grayscale patterns and the actual printed contrast ratio: it must be accurate to get the correct ratio.
• Ink spread compensation corrects for ink spread (or "dot gain") in the Log frequency chart. Others may be added later.) The default is 0. The best value must be determined experimentally.

Multicharts

Interactive analysis of several test charts

Introduction to Multicharts: measure color accuracy and tonal response

Imatest™ Multicharts analyzes images of color test charts for tonal response and color accuracy using a highly interactive user interface. It can be used to measure white balance and color response in a wide range of lighting conditions and scenes. It can also display the tonal response of monochrome charts and calculate a color correction matrix (Imatest Master only). Multicharts currently supports

Colorchecker IT8.7 CMP DT 003 Colorchecker SG QPcard Q-13

the standard 24-patch X-Rite ColorChecker®, the industry-standard IT8.7 chart, the CMP DigitaL TargeT 003, the inexpensive QPcard, the 140-patch Colorchecker SG (Imatest Master only), linear step charts such as the Kodak Q-13/Q-14 (Imatest Master only), special charts, including a variety of monochrome test charts (Imatest Master only). Sources of charts are listed on Using Multicharts.

You can select either standard chart reference values or you can read in values from data files (Imatest Master only). Values for the two ColorCheckers have been supplied courtesy of X-Rite. You can select between six image file color spaces: sRGB, Adobe RGB (1998), Wide Gamut RGB, ProPhoto RGB, Apple RGB, or ColorMatch. Danny Pascale/Babelcolor's page on the ColorChecker contains nearly everything you want to know about the chart. Ian Lyons has a nice description of IT8.7 charts.

Before running Multicharts, photograph the chart, taking care to avoid glare. For testing white balance, you can photograph the chart in a scene (the mini ColorChecker is especially suitable). Then click on Multicharts in the main Imatest window. To load the image file, click the appropriate chart type in the box below Read image file. Chart type: Crop it, then enter any needed data in the input dialog box.

Example: the 24-patch ColorChecker, shown with the 2D a*b* plot

2D a*b * display for ColorChecker

The available displays, selected by the Display box, include Pseudocolor color difference. Shows the difference between the ideal and input color patches using any of several metrics. Probe is available for this display. 3D color difference. Split colors: ideal/input. Shows the patches split so that the ideal and input values are displayed in the upper left and the lower right or each patch region, respectively. Probe is available for this display. xy chromaticity. u'v' chromaticity. Black & White density. 3D L*a*b*. Shown on the right. L*a*b* coordinates can be difficult to visualize when viewed from a single angle. It can be rotated (automatically by pressing the A-rotate button) for improved visualization. EXIF data and Color matrix.  Shows EXIF data if available and the color correction matrix if it has been calculated.

3D L*a*b* display for ColorChecker

Examples of different charts and displays

Split display for IT8.7

This Multicharts split display is shown above for a reflective IT8.7 chart, with the Probe turned on. In each patch, the upper left portion is the ideal (reference) value, converted to sRGB for screen display, while the lower right is the input value, also converted to sRGB for screen display. The image is synthesized.

The Probe has been turned on by checking the Probe box on the right. When the probe is on, you can probe any patch in either image. The probe data (for the individual patch), shown below the split display, includes an image of the ideal and input patch colors, ideal and input RGB and L*a*b* values, and several ΔE* and ΔC* color difference metrics.  ΔE*ab, the geometric distance in L*a*b* space, is the most familiar, but ΔE*94, which is lower for saturation differences in saturated colors, may be a better measure of visual color difference. It's easy to correlate ΔE* metrics with visual color differences using this display. None of the metrics are perfect! The probe is turned off by clicking outside the images. The pseudocolor display is shown on the right. Differences can be displayed for several color metrics: ΔE*ab, ΔC*ab, ΔE*94, ΔC*94, ΔE*CMC, ΔC*CMC, ΔL*, ΔChroma, Δ|Hue distance| , Δ(Hue angle) , Chroma (input) and Chroma (output). Several color maps are available.

Pseudocolor display for IT8.7

IT8.7 charts are printed on photographic paper. They must be used with reference files. Here are a few sources: Monaco IT8 reference files  |  Kodak Q60 targets (FTP site)  | Wolf Faust (Coloraid.de)

xy Chromaticity display for IT8.7

The xy Chromaticity diagram for the IT8.7 chart is of special interest. To avoid clutter it doesn't display all the patches (this is also the case for the ColorChecker SG and the CMP DC003). Instead it displays values for one of five zones in the chart: columns 13-19 (primaries), 1-4 (dark), 5-8 (mid), 9-12 (dark), and 20-22 (miscellaneous— different for different manufacturer's charts). The primary region (columns 13-19, shown above) is the most interesting since it shows the response to the subtractive and additive primaries, and gives some indication (though far from perfect) of the color gamut of the device + color space under measurement.

B&W plot: Log(Pixels) vs. Input density for Q-13

The final example of the tour shows the grayscale response curve, in this case Log₁₀(Pixel level) vs. Input density. Gamma is the average slope of this curve for the lighter regions. This plot is also available for all of the color charts because all of them contain grayscale regions. Several additional B&W curves are available.

Rescharts

Analysis of resolution-related charts

Introduction

Imatest™ Rescharts analyzes images of several test charts for resolution-related parameters such as sharpness (i.e., spatial frequency response; MTF), color moiré, and fine detail lost to software noise reduction, using a highly interactive user interface is similar to Multicharts.

The following chart types are supported by Rescharts.

Applied Image QA-77 chart

SVG SFR chart: m x n grid

Slanted-edge SFR

Duplicates SFR, but with a far more interactive interface. It calculates and displays the average edge, SFR (Spatial Frequency Response or MTF), and Lateral Chromatic Aberration. Includes one figure not in SFR: a histogram with noise analysis.

It supports all charts that contain slanted edge patterns, including the ISO 12233 chart and its derivatives. Charts may be purchased from sources such as Applied Image or created with Test Charts (SVG charts are especially recommended) and printed on a high quality inkjet.

SFRplus chart

SFRplus Highly automated analysis of SFR (MTF), Lateral Chromatic Aberration, distortion, and tonal response. Introduced in Imatest 3.2. Uses a special test chart, shown on the left. Unlike other Rescharts modules, SFRplus does not require manual region (ROI) selection: it is performed automatically. Running SFRplus in Rescharts saves settings for fully automated runs. SFRplus instructions

Log frequency chart

Log Frequency (simple) Analyzes a sine or bar pattern of increasing spatial frequency. Measures SFR more directly, but less precisely, than the slanted-edge method. Also measures color moiré (Imatest Master only), which is a function of lens sharpness, anti-aliasing filter, and demosaicing algorithm. Spatial frequencies are detected automatically. Charts can be created by Test Charts and printed on a high quality inkjet printer. Also works with the old test chart in Lens testing. Log Frequency instructions

Log Frequency-Contrast chart

Log F-Contrast  (Imatest Master only) Analyzes a sine or bar pattern of increasing spatial frequency on the one axis and decreasing contrast on the other. Measures SFR for a range of contrast levels. Useful for measuring the loss of fine detail caused by software noise reduction— the effects of nonlinear signal processing. The chart can be created by Test Charts and printed on a high quality inkjet printer. Log F-Contrast instructions

Siemens Star chart

Star chart   (Imatest Master only) Analyzes a sinusoidally-modulated Siemens star chart (144, 72, or 48 cycles); proposed for the revised ISO 12233 standard. Measures MTF (SFR) for 8, 12, or 24 segments around the circle. Has a large variety of displays. The chart can be purchased from Image Engineering or created by Test Charts and printed on a high quality inkjet printer. Star Chart instructions

Getting started

Instructions for purchasing or creating test charts can be found on pages for individual tests: SFR, Log Frequency, and Log F-Contrast.

Photograph or scan the chart, taking care to avoid glare, which can be problematic in charts with semigloss and glossy surfaces. Glare can be especially difficult to control with wide angle lenses. A recommended lighting setup is described here and in How to build a testing lab.

To start Rescharts, run Imatest, then click the Rescharts button. The Rescharts window will appear with brief instructions, which may be more up-to-date than the ones shown below.

Select an image to analyze by clicking on Read image file if the correct Chart type is displayed, or by clicking on one of the entries in the popup menu below Chart type:. The Read image file button and Chart type popup menu are highlighted (yellow background) when Rescharts starts.

A standard Windows dialog appears with the chart type indicated in the title. Open the image file. Multiple files can be opened for averaging (but batch (sequential) runs are not supported by Rescharts).

If the folder contains meaningless camera-generated file names such as IMG_3734.jpg, IMG_3735.jpg, etc., you can change them to meaningful names that include focal length, aperture, etc., with the View/Rename Files utility, which takes advantage of EXIF data stored in each file.

For all modules other than SFRplus, if the image is the same size and type as the previous image analyzed by Rescharts, you'll be asked if you want to use the same ROI (region of interest). (ROI selection is entirely automatic with SFRplus.)

If the image size or type is different or if you answer No, a coarse cropping box appears. The initial crop doesn't have to be precise: you'll have a chance to refine it.

• For Slanted-Edge SFR, select the region you want. Recommended cropping.
• For Log Frequency (simple), select a narrow strip running from along the direction of increasing spatial frequency. Narrow strips of the Log Frequency-Contrast chart can be used.
• For Log F-Contrast, select most of the chart image, leaving small margins (1/2 to 1% of the image dimensions) at the edges.

After the rough crop has been selected, the fine adjustment box appears, showing the coarse crop as a rectangle. This dialog box can be enlarged or maximized to facilitate the adjustment. It offers numerous options.

Crop for the Log Frequency-contrast chart. Leave a small margin
(1/2 - 1% of the image height) at the top and bottom.

• The entire ROI can be moved ( ^ > v < , upper-left).
• The top, bottom, left, or right sides can be moved ( T^ Tv R> R< B^ Bv L> L< , middle-left).
• You can choose between Fine, Coarse, and Extra Coarse movement (1, 5, or 25 pixels of movement per click).
• The display can be zoomed out or in.
• Pixel values for the four corners can be entered (X(tl), Y(br), ..., where tl = top-left, br = bottom right, etc.). The origin is the upper-left. Be sure to press the Enter key after entering a value.

When you have completed the fine adjustment, click one of the buttons on the bottom of the window. If you click Yes, Continue , an additional dialog box may appear. The entries in this dialog box can be changed later by clicking on Image settings & options in the Rescharts window. If you click  Yes, Continue in Express mode , it will go directly to the Rescharts window, using saved (recent) values.

The Rescharts window

After the image file has been entered, calculations are performed and the most recent Chart view is displayed. The MTF/contrast (2D pseudocolor contour) display (actual MTF when a sine chart is used) for Log F-Contrast is shown below.

MTF contours for Canon EOS-20D, 24-70mm f/2.8L, 42mm, f/5.6, ISO 100

The main display image is on the upper left. Additional text results may appear below this image. An image of the chart is shown on the upper right. You can choose to display the whole image or the crop (ROI). The remainder of the right side is the control area.

The Zoom checkbox turns Zoom on and off.

The buttons to the right of Zoom set the display to either the Whole image or a Crop of the image (the selected ROI).

Image and Display areas

The image area is on the right of the Rescharts window, just below the Whole image and Crop buttons. It includes the the Read image file and Reload buttons, the Chart type dropdown menu (described above), and additional controls.

Reload reloads an image from a file or reacquires it from a device or video stream (depending on how the image was originally acquired). It is most valuable for versions of Imatest that can acquire images from devices or streams. Immediately below the Chart type menu is the New crop (same image) menu, which has identical entries. If an item is selected on this menu, the image is recropped for analysis by the selected chart type (which may be the same or different). This menu is useful when several regions of an image need to be analyzed or when the target contains more than one chart type. Image settings & options brings up the input dialog box for the selected chart type. This is the same box that appears when you press Yes, continue (not Express mode) in the ROI repeat or ROI selection window. Display, immediately below the image area, selects the results to display. Choices depend on the chart type.

The Display options area is immediately below Display. The contents, which depend on the Display selection, are described below in the sections for the individual displays.

Save screen (shown on the right) saves a snapshot of the current display (the entire Rescharts screen) as a PNG file (a widely-used losslessly-compressed format). It also allows you to immediately view the snapshot (if the Open window... box is checked) so it can be used a reference for comparing with other results. File name and Directory at the top of the window set the location of the file to save. When you check the Open window in an image viewer... box, the current screen will be opened either the system default viewer (if the box under Image viewer is blank) or a viewer/editor of your choice (if the box contains the path name to the viewer/editor). I recommend Irfanview, which is fast, compact, free, and supports an amazing number of image file formats. (It can be valuable for converting nonstandard images to Imatest-readable formats.) Its normal location in English language installations is C:\Program Files\IrfanView\i_view32.exe.

Save data Saves detailed results in CSV and XML formats. The exact content depends on the module.

Help opens this web page in an HTML browser window.

Exit terminates Imatest Rescharts, but the Imatest main window remains available.

Rescharts modules

This section describes the Rescharts modules, illustrating some of the available displays and options. More information is available in the pages for the individual modules (Chart type analyses).

Slanted-edge SFR

Slanted-edge SFR measures sharpness as Spatial Frequency Response, also called MTF (Modulation Transfer Function) from images of slanted edges, as described in Sharpness. ROI (Region of Interest) selection, calculations, and output largely duplicate SFR. Available displays: Edge and MTF (shown on the right) Chromatic Aberration (right, below) SQF (Subjective Quality Factor) Noise spectrum & Shannon capacity Histograms & noise stats (left, below) Summary & EXIF (for JPEGs) The principal differences from SFR are the highly interactive interface and an added histogram of the ROI levels, shown below. Black is the entire ROI. Cyan is the portion on the left and red is the portion on the right, both away from the edge. The upper histogram is for the original image (pixel levels/255); the lower histogram is for the linearized image (gamma applied).

SFR Edge and MTF display

SFR Histogram

SFR Lateral Chromatic Aberration

SFRplus

SFRplus summary display for 9 automatically-selected regions

Log Frequency (simple)

Log frequency image: right-click to download an image that can be used for testing.

Log Frequency measures image contrast of charts that increase in spatial frequency on a logarithmic scale (log frequency increases with x). When the image pattern is sinusoidal (not a bar pattern) the contrast is equivalent to SFR or MTF. This method is more direct than Slanted-edge SFR, but less accurate because it is degraded by noise and sampling phase variation. It also measures color moiré (Imatest Master only). The chart can be created by Test Charts and printed on a high quality inkjet. Also works with the test chart in Lens testing and narrow strips of the Log Frequency-Contrast chart..

Available displays:        Full instructions The Pattern (original pixel levels and linearized) MTF and Moire (linear frequency scale) MTF and Moire (log frequency scale) EXIF data The display on the right shows Contrast (MTF) on the top and color moire on the bottom as a function of spatial frequency (displayed linearly). Color moire can me measured in several ways: R-B (normalized Red − Blue channel) is shown on the right. The total color moire is the spread of values above 0.3 cycles/pixels (indicated by the red curve). Details of the color moire measurement can be found in Log Frequency. The Correct for color density box should be checked for best results. Pattern (original pixel levels and linearized)

Log Frequency MTF and moire display (linear x-axis frequency scale)

Log F-Contrast

Log F-Contrast (short for Log Frequency-Contrast) measures the image contrast of charts that vary in spatial frequency on one axis (log frequency increases with x) and in contrast on the other (contrast is proportional to ( y/h)2 for image height h). When the image pattern is sinusoidal (rather than a bar pattern) the contrast is equivalent to SFR or MTF. This module can be used to measure how much fine, low-contrast detail is lost to software noise reduction. The chart can be created by Test Charts and printed on a high quality inkjet printer. Available displays:       Full instructions The Pattern (original pixel levels and linearized) MTF (linear frequency scale) MTF (log frequency scale) 2D pseudocolor contour plots that can display any of several MTF-related parameters: MTF (envelope - standard) (standard MTF, normalized to 1 at maximum contrast and zero spatial frequency) Normalized contrast level (MTF normalized to 1 at zero spatial frequency for all contrast levels) Normalized contrast loss (MTF normalized to 1 at zero spatial frequency for all contrast levels as well as all spatial frequencies at the maximum contrast level: shows contrast loss from ideal) MTFnn or MTFnnP (spatial frequencies where MTF = nn% of the low frequency or peak value EXIF data The displays on the right show the Normalized contrast level (MTF normalized to 1 at low spatial frequencies for all chart contrast levels) for the Panasonic TZ1 camera with ISO speed set at 80 (top) and 800 (bottom). Spatial frequency is on a linear scale with a maximum displayed frequency of 0.5 cycles/pixel. The difference between the two images is quite striking. The TZ1 has much more noise reduction— and correspondingly less detail— at ISO 800. These plots are explained in detail in Log F-contrast.

Normalized contrast level (MTF) (1 at low frequencies for all chart contrasts): Panasonic TZ1, ISO 80 Normalized contrast level: TZ1, ISO 800

Star chart

Star Chart measures contrast (MTF or SFR) of sinusoidally-modulated star charts (also known as the Siemens star) proposed for inclusion in the updated ISO 12233 standard. Charts have 144, 72, or 48 cycles and can be analyzed in 8, 12, or 24 segments. Star charts can be purchased from Image Engineering or created by Test Charts and printed on a high quality inkjet printer. Available displays:       Full instructions Display Description MTF (original and linearized) MTF for up to 8 segments of the star. Both linear and logarithmic frequency displays are available. MTFnn or MTFnnP Display MTFnn (the frequencies where MTF equals nn % of the low frequency values) and MTFnnP (the frequencies where MTF equals nn % of the peak value) for nn = 70, 50, 30, 20, and 10. Both polar (spider) and rectangular plots are available. MTF contours (rectangular) Display MTF contours in a rectangular plot with linear or logarithmic frequency display. Similar to the MTFnn rectangular plot. MTF contours (polar) Display MTF contours in a polar plot whose geometry duplicates that of the target. EXIF data Show EXIF data if available. The displays on the right show MTF contours and MTF70 through MTF10 (spatial frequencies where MTF = 70 through 10% of the low frequency level). Spatial frequency is displayed in cycles per pixel, but Line Widths per Picture Height (LW/PH), cycles/inch, or cycles/mm can be selected by pressing Image settings & options.

MTF contours, rectangular display, Linear frequency scale.   MTF70 – MTF10: Polar coordinates, Linear frequency on radius.

Imatest IT

Executable programs for automated testing

Introduction

Imatest™ API* is a set of modules designed to
measure image quality in automated testing systems.
Their tight integration with Imatest Master makes it easy to
get them up and running quickly.
                 *API is an acronym for Application program interface.

The initial API release (April 2007) is a set of EXE (Windows executable) programs, called API/EXE, which includes the following modules:

How Imatest API/EXE works

Imatest API/EXE is a set of standalone programs initiated by DOS calls from the test system. They have the same functionality as the corresponding modules in Imatest Master, but they operate without user intervention, making them suitable for use in automated systems. API/EXE modules are represented by the blue box in the figure on the right. The test system interacts with the API/EXE programs through files specified in the DOS command line: the primary inputs are an image file and an INI control file (created and saved by Imatest Master during setup). The output is data files in XML or Excel-readable .CSV format. Figures can be saved if desired.

Although Imatest API/EXE operates independently of Imatest Master, we strongly recommend that API/EXE users have at least one Imatest Master installation on site.

See Imatest API/EXE instructions for detailed instructions for all modules.

Licensing

The license allows you one computer installation (one station) per module. Contact us to discuss quantity discounts. Purchasers should be familiar with the installation and operation of Imatest Master, which is necessary for setting up EXE runs. The license includes a reasonable amount of support (up to three hours) to get you up and running.

Contact to purchase an API/EXE modules or to request a free 30 day time-limited trial. We can issue Paypal invoices, and we can accept payments by several means, including wire transfer. (We are developing a more automated purchase/download system.)

Imatest API will not be available for public download. When we receive your request for a 30 day trial, your payment, or approve your purchase order, we will either send you instructions for downloading the installation files or send you the files themselves (typically around 800 kB per module). Contact us to discuss delivery details.

More about Imatest API

Imatest plans to introduce API/DLL programs about two months after the introduction of API/EXE in April 2007. API/DLL will be priced slightly higher than API/EXE. API/EXE users will be able to upgrade for the price difference. Many API/DLL installations will require some customization and support. Feel free to contact us with your specific requirements.

We also plan to introduce new test charts (using scalable vector graphics) along with modifications of sfr.exe that will support automatic region of interest (ROI) selection. This will allow for more framing tolerance in MTF tests.

Imatest API/EXE instructions

Image quality factors

Overview and Imatest measurements

Image quality is one of those concepts that is greater than the sum of its parts. But you can't ignore the parts if your goal is to produce images of the highest quality. Every image quality factor counts.

This page introduces the key image quality factors, describes how Imatest™ measures them, and explains what, if anything, can be done to improve them. It is a guide to Imatest organized by image quality factor. Other guides include the Tour (organized by module) and Imatest documentation (the Table of Contents).

To illustrate the quality factors, we use this early morning image of Monument Valley from Hunt's Mesa, near the Arizona-Utah border. A 13x19 print (available for purchase) is breathtaking, though it can't capture the experience of grabbing the camera gear and running for the truck as the storm broke. Hunt's Mesa isn't public land; you need a Navajo guide to get there. Tom Phillips does an excellent job.

The image was extensively edited. The unedited image, straight from the RAW converter (with some tonal adjustment) is shown below.

When people ask me about "digital manipulation" I delight in telling them that I would no more manipulate an image than Ansel Adams. Of course I'm jesting. In Ansel Adams: A Biography, Mary Street Alinder describes how Adams printed his famous Moonrise over Hernandez image. He'd spend the entire morning making test prints — dodging, burning, and (chemically) manipulating them until they met his exacting standards. It took him that long even though he kept meticulous notes. Then he'd spend the afternoon making prints. From a processing viewpoint he would have loved digital. But probably not from a business viewpoint. Thanks in part to his brilliant business manager he earned more in an afternoon of printing than most of us earn in a decade.

There are two key aspects of image quality. Factors affected by post-processing. Contrast, color balance, color saturation, etc. These are the factors that can be enhanced in the darkroom — digital or chemical — under the control of you, the photographer. For example, higher contrast images usually score higher in visual quality assessment tests, unless the image is clipped. Because of their great importance to photo finishers these factors are the focus of much image quality literature. But they're outside the scope of Imatest. Factors intrinsic to cameras, lenses, and printers. Sharpness, noise, dynamic range, color accuracy, color gamut, etc. Imatest focuses on these. Imatest modules are summarized on the right.

Summary table

This table summarizes the image quality factors described in detail below.

Image quality factors for cameras and lenses

Sharpness Sharpness is arguably the most important single image quality factor: it determines the amount of detail an image can convey. The image on the upper right illustrates the effects of reduced sharpness (from one application of the Picture Window Pro blur operation). Device or system sharpness is measured as a Spatial Frequency Response (SFR), also called Modulation Transfer Function (MTF). MTF is the contrast at a given spatial frequency (measured in cycles or line pairs per distance) relative to low frequencies. The 50% MTF frequency correlates well with perceived sharpness— much better than the old vanishing resolution measurement, which indicated where the detail wasn't. Sharpness and MTF are introduced in Sharpness: What is it and how is it measured? The perceived sharpness of a print or display is measured by Subjective Quality Factor (SQF), which is derived from MTF and the Contrast Sensitivity Function of the human visual system. Sharpness is measured with Imatest SFR, Rescharts Slanted-edge SFR, or the highly-automated SFRplus, using targets you can purchase or print with the Imatest Test Charts module. Concise instructions are found in How to test lenses with Imatest. Detailed instructions are found in Using SFR Part 1 - Setting up and photographing the target and Using SFR Part 2 - Running Imatest SFR. An alternative method of measuring MTF uses sine pattern charts that increase in frequency logarithmically. This method provides a check on the slanted-edge method; it is more direct but less accurate. It is described Rescharts Log Frequency and Log Frequency-Contrast. System sharpness is affected by the lens (design and manufacturing quality, focal length, aperture, and distance from the image center) and sensor (pixel count and anti-aliasing filter). In the field, sharpness is affected by camera shake (a good tripod can be helpful), focus accuracy, and atmospheric disturbances (thermal effects and aerosols). Some lost sharpness can be restored by sharpening, but sharpening has limits. It can't restore detail where MTF is very low (under about 10%). Oversharpening, illustrated on the right, can also degrade image quality (especially at large magnifications) by causing "halos" to appear near contrast boundaries. Images from many compact digital cameras are oversharpened.

Original | Blurred              Original | Oversharpened

Noise Noise is a random variation of image density, visible as grain in film and pixel level variations in digital images. It arises from the effects of basic physics— the photon nature of light and the thermal energy of heat— inside image sensors. Noise and its measurement are introduced in Noise in photographic images. Noise is measured by several Imatest modules. Stepchart produces the most detailed results, but noise is also measured in Colorcheck, SFR, SFRplus, and Light Falloff.

Original | Noise added

Noise scales strongly with pixel area. It can be very low in digital SLRs, which have pixels at least 5 microns square. But it can get ugly in compact digital cameras with small sensors, especially at high ISO speeds. It is also affected by sensor technology and manufacturing quality.

Typical noise reduction (NR) software reduces the visibility of noise by smoothing the image, excluding areas near contrast boundaries. This technique works well, but it can obscure fine, low contrast detail. Some specialized programs, such as Neat Image, which is trained to recognize the structure of noise, have fewer side effects. Several of the patterns in Test Charts can be printed and photographed (or entered directly into software) to examine the effects of NR algorithms.

Dynamic range, tonal response, and contrast Dynamic range (or exposure range) is the range of light levels a camera can capture, usually measured in f-stops, EV (exposure value), or zones (all factors of two in exposure). It is closely related to noise: high noise implies low dynamic range. It is also related to the tonal response— the relationship between light and pixel level (shown below). Contrast, also known as gamma, is the slope of the tonal response curve. High contrast (shown on the right) usually involves loss of dynamic range— loss of detail, or clipping, in highlights or shadows— when the image is displayed. (The image file often has a greater dynamic range than the displayed image.) Dynamic range, tonal response, and gamma are measured (A) by Stepchart using a transmission step wedge, preferably one with a maximum density at lest 4.0 (equivalent to 13.3 f-stops), or (B) by Dynamic Range, a postprocessor for Stepchart that uses results from up to four differently-exposed reflective stephart images. The latter technique is often the most convenient because it doesn't require a special light source in a darkened room.

original | clipped

Dynamic range is a strong function of pixel area, which is proportional to the number of electrons a pixel can store. It is invariably better in DSLRs (which have relatively large pixels; at least 5 microns square) than in compact digital cameras. It can be maximized by setting the camera at the lowest ISO speed.

Displaying images with large dynamic ranges (which can be well over 1000:1; 10 f-stops) can be problematic in printed media, which has a maximum dynamic range of about 100:1 (a little over 6 f-stops; 200:1 at the absolute maximum). Reducing contrast can make the image look flat and dull. Some processing is usually required, like applying an "S" curve or dodging and burning (lightening or darkening) portions of the image. Contrast masking is a particularly effective approach. Tonal quality and dynamic range in digital cameras offers additional tips.

Lenses have an intrinsic contrast— the higher the better. But lens contrast isn't exactly intrinsic. It results from flare light— light originating inside and outside the lens's field of view that bounces between lens elements and off the inside of the lens barrel. Flare light tends to fog the image and obscure shadow detail: it reduces dynamic range. To our knowledge there is no solid standard for measuring lens contrast. But you can measure it, or at least compare it for different lenses and apertures, by running Stepchart on a Q-13 (or equivalent) chart mounted on a white board that extends beyond the camera's field of view. (Recall, contrast = gamma.) There would be less flare light, hence higher contrast, with a black board. Actual lens contrast depends strongly on the scene.

Color accuracy Color accuracy is an important but ambiguous image quality factor. Many viewers prefer enhanced color saturation; the most accurate color isn't necessarily the most pleasing. Nevertheless it is important to measure a camera's color response: its color shifts, saturation, and the effectiveness if its white balance algorithms. Color accuracy is measured by Colorcheck, using the widely-available 24-patch GretagMacbethTM ColorChecker®  and by Multicharts using the 24-patch ColorChecker, ColorChecker SG, IT8.7, QPcard, CMP DC 003, and custom "pie" charts. Color accuracy is affected by the Bayer color filter array and by the signal processing and white balance algorithms in the camera or RAW converter. Flare light in lenses may reduce color saturation. Multicharts can calculate a color correciton matrix.

Original | Color-shifted

Color can be corrected in image editing programs. If a RAW image file has been captured it can be corrected with far more precision with the RAW converter.

Lens distortion Lens distortion is an aberration that causes straight lines to curve near the edges of images. It can be troublesome for architectural photography and metrology (photographic applications involving measurement). The simplest approximation is the equation, ru = rd + krd3 where ru is the undistorted and rd is the distorted radius. Depending on the sign of k, it can be either "barrel" (shown on the right) or "pincushion." Lens distortion and coefficients for correcting it are calculated in Distortion, which also includes 5th order and tangent/arctangent distortion models. SFRplus measures distortion in less detail, along with sharpness and several other factors. Distortion is worst in wide angle, telephoto, and zoom lenses. It often worse for close-up images than for images at a distance. It can be easily corrected in software. Picture Window Pro and PTLens have tools for removing it.

Original | Barrel-distorted

Light falloff (vignetting) and sensor nonuniformities Light falloff (vignetting) darkens images near the corners. It can be significant with wide angle lenses. It is measured by Light Falloff (Uniformity). Light falloff tends to be worst in wide angle lenses. It often improves when lenses are stopped down. It can be easily corrected in software. Picture Window Pro, PTLens, and several other programs have tools for removing it. Because moderate amounts of light falloff can be pictorially pleasing, it's not always advisable to remove it completely. Light Falloff (Uniformity) also measures other sensor nonuniformities, including stuck pixels, local sensitivity variations, spots (from dust), and noise. In Imatest Master it has a particularly rich set of displays.

Original | Vignetted

Exposure accuracy Exposure accuracy is not much of a problem with manually-adjustable cameras. You can usually determine it quickly (with the help of the histogram), and if you don't like it, you can change the exposure compensation or the way you meter. But exposure accuracy can be an issue with fully automatic cameras and with video cameras where there is little or no opportunity for post-exposure tonal adjustment. Some even have exposure memory: exposure may change after very bright or dark objects appear in a scene. Exposure accuracy can be measured by photographing a gray scale step chart such as the Kodak Q-13 or Q-14 in a scene and analyzing it with Stepchart, or photographing a GretagMacbeth ColorChecker and analyzing it with Colorcheck.

Original | Overexposed

Lateral chromatic aberration Lateral chromatic aberration (LCA), also called "color fringing" is a lens aberration that causes colors to focus at different distances from the image center. It is most visible near corners of images. It is explained in Chromatic aberration. LCA is measured by SFR, using edges at a distance from the image center. LCA is worst with asymmetrical lenses, including ultrawides, true telephotos and zooms. It is strongly affected by demosaicing. It can be fully corrected in software prior to demosaicing, but only partially corrected afterwards. Picture Window Pro has a fairly effective transformation. In the future, information provided by Imatest (detailed LCA profiles) may improve the degree of correction.

Original | Color-fringed

Veiling glare (lens flare) Veiling glare is stray light in lenses and optical systems caused by reflections between lens elements and the inside barrel of the lens. It predicts the severity of lens flare— image fogging (loss of shadow detail and color) as well as "ghost" images— that can occur in the presence of bright light sources in or near the field of view. Veiling glare is measured by Stepchart using a standard Kodak Q-13 or Q-14 step chart beside a "black hole," i.e., a box lined with black behind a small opening, mounted on a large white board, as described in detail in Veiling glare.	Original | Veiling glare
Color moiré Color moiré is artificial color banding that can appear in images with repetitive patterns of high spatial frequencies, like fabrics or picket fences. The example on the right is a detail of a shirt captured by the Canon Rebel XT with its excellent kit lens. The usual image wasn't used because it doesn't contain a repetitive pattern and because color moiré is difficult to simulate. Color moiré is the result of aliasing (image energy above the Nyquist frequency) in image sensors that employ Bayer color filter arrays, as explained here. It is affected by lens sharpness, the anti-aliasing (lowpass) filter (which softens the image), and demosaicing software. It tends to be worst with the sharpest lenses. Color moiré is measured by Rescharts Log Frequency, using a sine chart of logarithmically increasing spatial frequency.	Color Moiré
Software artifacts: noise reduction and sharpening Software (especially operations performed during RAW conversion) can cause significant visual artifacts, including oversharpening "halos" and loss of fine, low-contrast detail. These artifacts result from nonlinear signal processing (so-called because it varies with the signal). Images may be sharpened (MTF boosted) in the presence of contrasty features like edges and blurred (lowpass filtered) in their absence. This generally improves measured performance (both sharpness and noise/Signal-to-Noise Ratio (SNR)), but it may result in a degradation of perceived image quality, for example, a "plasticy" cartoon-like appearance of skin even though edges are strongly sharpened. This loss of detail cannot be measured with SFR. These artifacts can be measured by the Log F-Contrast module in Rescharts, which analyzes the chart shown on the right, which varies logarithmically in spatial frequency on the horizontal axis and in contrast on the vertical axis.	Original | NR+Sharpening
Data compression and transmission losses Data compression and transmission losses can have a significant effect on image quality. The right side of the image on the left has been saved as a low quality JPEG. Banding, loss of low-contrast detail, and "waviness" near edges are visible. A full analysis of data compression and transmission losses is not yet available in Imatest. Some of these losses, especially the loss of low-contrast detail, can be analyzed with the Log F-contrast module in Rescharts. A more detailed analysis of compression losses is under development.	Original | Low-quality JPEG

Quality factors for printers

Print Dmax Dmax = -log10(minimum print reflectivity) is a measure of the deepest black tone a printer/ink/paper combination can reproduce. It is an extremely important print quality factor. Prints with poor Dmax look pale and weak. Dmax = 1.7 is a good value for matte prints; 2.0 is a good value for glossy, semigloss, and luster prints. There have been reports that the new Epson Ultrachrome K3 printers have Dmax as high as 2.3 with Premium Luster paper. That would be outstanding. Dmax is measured by Print Test, along with the Printer's tonal response curve and the color factors described below. Print test requires a decent flatbed scanner, and for best accuracy, a Step chart. Dmax is affected by the printer, paper, and ink. The response curve is also affected by the ICC profile. Print Test can help with the selection of supplies and diagnosis of printing problems.

Original | Reduced Dmax

Print color gamut Print color gamut is the range of colors a printer/ink/paper combination can reproduce. It is an important quality factor, though its importance may be somewhat overrated. (This statement is bound to generate controversy.) Relatively unsaturated colors such as skin tones dominate our impression of print quality. Such colors must be reproduced accurately. Gamut affects only highly saturated colors. Overall color response, especially for low to moderately saturated colors, is more important than gamut. Print color gamut and overall color response are measured by Print Test, along with the density factors described above. Print gamut is affected by the printer, paper, ink, and working color space. Color response is also affected by the ICC profile and rendering intent. Print Test can help with the selection of profiles, software settings, and the diagnosis of color problems.

Original | Reduced gamut

Links

Bror Hultgren (ImagIntegration.com), an imaging scientist with 27 years experience at Polaroid, has developed algorithms and software that relates image quality factors (especially MTF, noise spectrum, white balance, and memory colors) to perceived image quality. Bror is available for consulting.

Direct Digital Image Capture of Cultural Heritage from RIT is a gold mine of information. Links to a number of reports on image quality targeted at museums and cultural institutions. The 78-page Final Project Report by Berns, Frey, Rosen, Smoyer and Taplin, July 2005, is probably the best summary unless you have time for Erin P (Murphy) Smoyer's 345 page Master's thesis (about twice the length of the average Ph.D. thesis).

The Research Library Group (RLG) has published an excellent series of articles, Guides to Quality in Visual Resource Imaging (2000). These articles are the predecessors to the above-mentioned RIT Direct Digital Image Capture work.

The European Broadcasting Union (EBU) has a library of technical papers (the EBU Tech 3000 series), some related to TV image quality, for example, T3249, Measurement and Analysis of the Performance of Film and Television Camera Lenses (1995), and T3281, Methods for the Measurement of Characteristics of CCD Cameras (1995).

Volker Gilbert has written an excellent French language description of Imatest. (PDF version)

SMIA (Standard Mobile Imaging Architecture), a consortium founded by Nokia and STMicroelectronics, has published a Camera Characterization Specification for image quality measurements in camera phones. To obtain the spec you must join SMIA. It's free; all you need to do is answer a brief questionnaire.

Paul van Walree has an excellent page on Optics, covering several sources of degradation.

The University of Texas Laboratory for Image & Video Engineering is doing some interesting work on image and video quality assessment, which approaches the problem using information theory, natural scene statistics, wavelets, etc. Challenging material!

Details of many Imatest algorithms are included in Appendix C, Video Acquisition Measurement Methods (pp. 91-125), of the Public Safety SoR (Statement of Requirements) volume II v 1.0, released by SAFECOM, prepared by ITS (a division of NTIA, U.S. Department of Commerce). No credit is given, but the style and illustrations will be recognizable.

Sharpness

What is it and how is it measured?

Image sharpness

Sharpness is arguably the most important photographic image quality factor: it's the factor most closely related to the amount of detail an image can render. But it's not the only important factor. Others measured by Imatest include chromatic aberration (closely related to sharpness), noise, dynamic range (closely related to noise), and color accuracy.

Sharpness is defined by the boundaries between zones of different tones or colors. It is illustrated by the bar pattern of increasing spatial frequency, below. The top portion represents a target used to test a camera/lens combination. It is sharp; its boundaries are abrupt, not gradual. The bottom portion illustrates the effect of a high quality 35mm lens on a 0.5 millimeter long image of the pattern (on the film or digital sensor plane). It is blurred. All lenses, even the finest, blur images to some degree. Poor lenses blur images more than fine ones.

One way to measure sharpness is to use the rise distance of the edge, for example, the distance (in pixels, millimeters, or fraction of image height) for the pixel level to go from 10% to 90% of its final value. This is called the 10-90% rise distance. Although rise distance is a good indicator of image sharpness, it has one limitation. It is poorly suited for calculating the sharpness of a complete imaging system from the sharpness of its components, for example, from a lens, digital sensor, and software sharpening algorithm.

To get around this problem, measurements are made in frequency domain, where frequency is measured in cycles or line pairs per distance (typically millimeters in film measurements, but may also be inches, pixels, or image height). Line pairs per millimeter (lp/mm) is the most common spatial frequency unit for film, but cycles/pixel is convenient for digital sensors. The image below is a sine wave— a pattern of pure tones— that varies from low to high spatial frequencies, in this case from 2 to 200 lp/mm, over a distance of 0.5 millimeters. The top portion is the original sine pattern. The bottom portion illustrates the effects of the same high quality 35mm lens, which reduces pattern contrast at high spatial frequencies.

The relative contrast at a given spatial frequency (output contrast/input contrast) is called the Modulation Transfer Function (MTF) or Spatial Frequency Response (SFR).

The upper plot displays the sine and bar patterns: original and after blurring by the lens. The middle plot displays the luminance of the bar pattern after blurring by the lens (the red curve). Contrast decreases at high spatial frequencies. The lower plot displays the corresponding MTF (SFR) curve  (the blue curve). By definition, the low frequency MTF limit is always 1 (100%). For this lens, MTF is 50% at 61 lp/mm and 10% at 183 lp/mm.

Both frequency and MTF are displayed on logarithmic scales with exponential notation (10⁰ = 1; 10¹ = 10; 10² = 100, etc.). Amplitude is displayed on a linear scale.

The beauty of using MTF (Spatial Frequency Response) is that the MTF of a complete imaging system is the product of the the MTF of its individual components.

Traditional "resolution" measurements involve observing an image of a bar pattern (usually the USAF 1951 chart) on film, and looking for the highest spatial frequency (in lp/mm) where a pattern is visible. This corresponds to an MTF of about 2-5%. Because this is the spatial frequency where image information disappears— where it isn't visible, it is not a good indicator of image sharpness.

Experience has shown that the best indicators of image sharpness are the spatial frequencies where MTF is 50% of its low frequency value (MTF50) or 50% of its peak value (MTF50P).

MTF50 or MTF50P are ideal parameters for comparing the sharpness of different cameras for several reasons: (1) Image contrast is half its low frequency or peak values, hence detail is still quite visible. (2) The eye is relatively insensitive to detail at spatial frequencies where MTF is low: 10% or less. (3) The response of virtually all cameras falls off rapidly in the vicinity of MTF50 and MTF50P. MTF50P may better for oversharpened cameras that have peaks in their MTF response.

Although MTF can be estimated directly from images of sine patterns (see Rescharts Log Frequency, Log F-Contrast, and Star Chart), a sophisticated technique, based on the ISO 12233 standard, "Photography - Electronic still picture cameras - Resolution measurements," provides more accurate and repeatable results. A slanted-edge image, described below, is photographed, then analyzed by Imatest SFR or Rescharts Slanted-edge SFR. (SFR stands for Spatial Frequency Response.)

Origins of Imatest SFR  The algorithms for calculating MTF/SFR were adapted from a Matlab program, sfrmat, written by Peter Burns () to implement the ISO 12233 standard. Imatest SFR incorporates numerous improvements, including improved edge detection, better handling of lens distortion, a nicer interface, and far more detailed output. The original Matlab code is available on the I3A ISO tools download page by clicking on ISO 12233 Slant Edge Analysis Tool sfrmat 2.0. In comparing sfrmat 2.0 results with Imatest, note that if no OECF (tonal response curve) file is entered into sfrmat, it assumes that there is no tonal response curve, i.e., gamma = 1. In Imatest, gamma is set to a default value of 0.5, which is typical of digital cameras. To obtain good agreement with sfrmat, you must set gamma to 1.

The slanted-edge test for Spatial Frequency Response

Slanted-edge test charts can be created with Imatest Test Charts (SVG charts are especially recommended) or downloaded from How to test lenses with Imatest. The bitmap chart has horizontal and vertical edges for best print quality. It should be tilted (about 2-8 degrees) before it is photographed.

Imatest SFR can also take advantage of portions of the ISO 12233 test chart, shown on the right, or a derivative like the Applied Image QA-77, or as a less expensive alternative from Danes-Picta in the Czech Republic (the DCR3 chart on their Digital Imaging page)). Two such portions are indicated by the red and blue arrows. ISO 12233 charts are used in imaging-resource.com and dpreview.com digital camera reviews.

Imatest SFR results

35mm camera lens tests use line pairs per millimeter (lp/mm) as the units of spatial frequency. This works fine for comparing lenses because all 35mm cameras have the same 24x36 mm picture size. But digital sensor digital varies widely, from under 6 mm diagonal in ultra-compact models to 43 mm diagonal for full-frame DSLRs— even larger for medium format backs. The number of pixels also varies. For this reason, spatial frequency should be measured in units that indicate the response over the total sensor height rather than the response per distance.

For this purpose we use line widths per picture height (LW/PH). LW/PH is equal to 2 * lp/mm * (picture height in mm). Where total picture height is involved, line widths is customarily used instead of pairs (where one line pair equals two line widths).

The use of picture height gives a slight advantage to compact digital cameras, which have an aspect ratio (width:height) of 4:3, compared to 3:2 for digital SLRs. Compact digital cameras have slightly more vertical pixels for a given number of total pixels. For example, a 5.33 megapixel compact digital camera would have 2000 vertical pixels— as many as a 6 megapixel DSLR.

Another measure of spatial frequency used with digital cameras is cycles or line pairs per pixel (c/p or lp/p). This gives an indication of how well individual pixels perform. There is no need to use actual distances (millimeters or inches) to evaluate digital camera image quality, although such measurements are available in Imatest SFR.

Imatest SFR program output contains results on the left and input data on the right (a thumbnail of the entire image, the region of interest (ROI), and selected EXIF data).

Results from the ISO 1233 image for the Canon EOS-10D.

(top) A narrow image that illustrates the tones of the averaged edge. It is aligned with the edge profile (spatial domain) plot, immediately below.

(middle) Spatial domain plot:  The average edge profile (linearized, i.e., proportional to light energy). The key result is the 10-90% edge rise distance, shown in pixels and in the number of rise distances per picture height. The red values are for standardized sharpening. Other parameters include overshoot and undershoot (if applicable). This plot can optionally display the line spread function (LSF: the derivative of the edge), or the edge in pixels (gamma-encoded).

(bottom) Frequency domain plot:  The Spatial Frequency Response (MTF), shown to twice the Nyquist frequency. The key result is MTF50, the 50% MTF frequency, which corresponds to perceived image sharpness. It is given in cycles per pixel (c/p) and line widths per picture height (LW/PH). Other results include MTF at NYQ, the MTF at the Nyquist frequency (0.5 cycles/pixel; sampling rate/2), which indicates the probable severity of aliasing. The Nyquist frequency is displayed as a vertical blue line.

For this camera, which is moderately sharpened, MTF50P (only shown when Standardized sharpening display is unchecked) is identical to MTF50.

SFR Results: MTF (sharpness) plot describes this Figure in detail.

Interpreting MTF50

What MTF50 do you need?  It depends on print size. If you plan to print gigantic posters (20x30 inches or over), the more the merrier. Any high quality 4+ megapixel digital camera (one that produces good test results; MTF50(corr) > 0.3 cycles/pixel) is capable of producing excellent 8.5x11 inch (letter-size; A4) prints. At that size a fine DSLR wouldn't offer a large advantage in MTF. With fine lenses and careful technique (a different RAW converter from Canon's and a little extra sharpening), my 6.3 megapixel Canon EOS-10D (corrected MTF50 = 1340 LW/PH) makes very good 12x18 inch prints (excellent if you don't view them too closely). Prints are sharp from normal viewing distances, but pixels are visible under a magnifier or loupe; the prints are not as sharp as the Epson 2200 printer is capable of producing. Softness or pixellation would be visible on 16x24 inch enlargements. The EOS-20D has a slight edge at 12x18 inches; it's about as sharp as I could ask for. There's little reason go go to a 12+ megapixel camera lie the EOS 5D, unless you plan to print larger. Sharpness comparisons contains tables, derived from images downloaded from two well-known websites, that compare a number of digital cameras. Several outperform the 10D.

The table below is an approximate guide to quality requirements. The equation for the left column is

Example of using the table: My Canon EOS-10D has MTF50 = 1335 LW/PH (corrected; with standardized sharpening). When I make a 12.3 inch high print on 13x19 inch paper, MTF50 is 1335/12.3 = 108 LW/in: "very good" quality; fine for a print that size. Prints look excellent at normal viewing distances for a print this size.

This approach is more accurate than tables based on pixel count (PPI) alone (though less refined than SQF, below). Pixel count is scaled differently; the numbers are around double the MTF50 numbers. The EOS-10D has 2048/12.3 = 167 pixels per inch (PPI) at this magnification. This table should not be taken as gospel: it was first published in October 2004, bandit may be adjusted in the future.

Subjective Quality Factor (SQF)

MTF is a measure of device or system sharpness, only indirectly related to the sharpness perceived when viewing a print. A more refined estimate of perceived print sharpness must include assumptions about viewing distance (typically proportional to the square root of print height) and the human visual system (the human eye's Contrast Sensitivity Function (CSF)). Such an formula, called Subjective Quality Factor (SQF) was developed by Eastman Kodak scientists in 1972. It has been verified and used inside Kodak and Polaroid, but it has remained obscure until now because it was difficult to calculate. Its only significant public exposure has been in Popular Photography lens tests. SQF was added to Imatest in October 2006.

A portion of the Imatest SFR SQF figure for the EOS-10D is shown on the right. SQF is plotted as a function of print size. Viewing distance (pale blue dashes, with scale on the right) is assumed to be proportional to the square root of picture height. SQF is shown with and without standardized sharpening. (They are very close, which is somewhat unusual.) SQF is extremely sensitive to sharpening, as you would expect since sharpening is applied to improve perceptual sharpness.

The table below compares SQF for the EOS-10D with the MTF50 from the table above.

An interpretation of SQF is give here. Generally, 90-100 is considered excellent, 80-90 is very good, 70-80 is good, and 60-70 is fair. These numbers (which may be changed as more data becomes available) are the result of "normal" observers viewing prints at normal distances (e.g.., 30-34 cm (12-13 inches) for 10 cm (4 inch) high prints). The judgments in the table above are a bit more stringent— the result of critical examination by a serious photographer. They correspond more closely to the "normal" interpretation of SQF when the viewing distance is proportional to the cube root of print height (SQF = 90, 86, and 80, respectively), i.e., prints are examined more closely than the standard square root assumption.

An SQF peak over about 105 may indicate oversharpening (strong halos near edges), which can degrade image quality. SQF measurements are more valid when oversharpening is removed, which is accomplished with standardized sharpening.

Some observations on sharpness

• Frequency and spatial domain plots convey similar information, but in a different form. A narrow edge in spatial domain corresponds to a broad spectrum in frequency domain (extended frequency response), and vice-versa.
• Sensor response above the Nyquist frequency is garbage— aliasing, visible as Moire patterns of low spatial frequency. In Bayer sensors (all sensors except Foveon) Moire patterns appear as color fringes. Moire in Foveon sensors is far less bothersome because it's monochrome and because the effective Nyquist frequency of the Red and Blue channels is lower than for Bayer sensors.
• Since MTF is the product of the sensor response, demosaicing algorithm, and sharpening, and since sharpening with a radius less than 2 boosts MTF at the Nyquist frequency, the MTF at and above the Nyquist frequency is not an unambiguous indicator of aliasing problems. It may, however, be interpreted as an indicator of potential problems.
• Results are calculated for the R, G, B, and Luminance (Y) channels, where Y = 0.3*R + 0.59*G + 0.11*B. The Y channel is normally displayed in the foreground, but any of the other channels can selected. All are included in the .CVS output file.
• Horizontal and vertical resolution can be different for CCD sensors, and must be measured separately. They're nearly identical for CMOS sensors. Recall, horizontal resolution is measured with a vertical edge and vertical resolution is measured with a horizontal edge.
• Resolution is not the only important criterion for evaluating image quality. Noise is nearly as important. The Shannon information capacity is an experimental metric that combines the two.

The ideal response would have high MTF below the Nyquist frequency and low MTF at and above it.

Links

Bob Atkins has an excellent introduction to MTF and SQF. SQF (subjective quality factor) is a measure of perceived print sharpness that incorporates the contrast sensitivity function (CSF) of the human eye. It will be added to Imatest Master in late October 2006.

This page has some interesting material on Fuji sensors and the history of the slanted-edge test.

Spatial Frequency Response of Color Image Sensors: Bayer Color Filters and Foveon X3 by Paul M. Hubel, John Liu and Rudolph J. Guttosch, Foveon, Inc., Santa Clara, California.  Uses slanted edge testing.

Optikos makes instruments for measuring lens MTF. Their 64 page PDF document, How to Measure MTF and other Properties of Lenses, is of particular interest.

Zeiss makes instruments for measuring MTF. You can't deep link to their site; you must go to the North America page, click on Camera/Cine Lenses in the dropdown menu, then search for MTF. The MTF-Tester K8 is used for many manufacturer's lens tests.

SQF

Subjective Quality Factor

Introduction to SQF

SQF (Subjective Quality Factor) is a measurement of perceived print sharpness that has been used for years in the photographic industry but has remained unfamiliar to most photographers. It includes the effects of

• MTF: the imaging system's Modulation Transfer Function, which is synonymous with Spatial Frequency Response (SFR),
• CSF: the human eye's Contrast Sensitivity Function,
• print height, and
• viewing distance.

SQF was introduced in the paper, "An optical merit function (SQF), which correlates with subjective image judgments ," by E. M. (Ed) Granger and K. N. Cupery of Eastman Kodak, published in Photographic Science and Engineering, Vol. 16, no. 3, May-June 1973, pp. 221-230. (If you do an Internet search, note that Granger's name is often misspelled Grainger.) It has been used by Kodak and Polaroid for product development and by Popular Photography for lens tests. This technical paper verified its correlation with viewer preference. But SQF is rarely mentioned on photography websites with the notable exception of Bob Atkins' excellent description, which includes an explanation of Pop Photo's methods.

SQF has remained obscure for only one reason. It was difficult to measure— until now.

SQF and MTF

The following table compares MTF and SQF. In essence, MTF is a measurement of device or system sharpness; SQF is a measurement of perceived print sharpness. SQF is derived from MTF, the contrast sensitivity function (CSF) of the human visual system, and an assumption about the relationship between print height and viewing distance.

Starting with Imatest 2.0.3, released in October 2006, SQF is calculated as a part of the SFR module. All you need to do is check the SQF checkbox in the SFR input dialog box. The SQF calculation is completely backwards-compatible: you can calculate SQF for all existing image files that contain slanted-edges for SFR analysis. Here is a sample result.

The large plot on top shows SQF without ( —— ) and with ( - - - ) standardized sharpening for print heights from 5 to 60 cm (~2 to 24 inches), assuming viewing distance (cm) = 30 √(picture height/10). For this mildly oversharpened camera, standardized sharpening decreases the sharpness (removes the strong overshoot), and hence reduces the SQF, which is highly sensitive to sharpening. The large plot also shows viewing distance ( - - - ). The plot on the lower right is the MTF without and with standardized sharpening, displayed in the main SFR/MTF figure, but repeated here to make this figure self-contained. The lower middle contains a thumbnail of the image showing the selected ROI in red. The text on the left contains calculation details and image properties.

What do the SQF numbers mean?

Ed Granger developed SQF to be linearly proportional to perceived sharpness. A change in SQF of 5 corresponds to a perceptible change in in sharpness— somewhat more than one "Just Noticeable Difference" (JND). Since SQF is a new feature of Imatest (version 2.0.3, Oct. 2006), we haven't had time to build our own database of SQF impressions. So we'll draw on the experience of others.

Popular Photography has been using SQF for testing lenses for years. They've developed the only generally-available SQF ranking system. Their scale isn't quite linear. {C, C+} takes up 20 SQF units; twice as many as {A, A+}, {B, B+}, or D. But it seems to be a good starting point for interpreting the numbers.

We encourage readers to examine Popular Photography's lens test results and to search its site for SQF. Imatest results should correlate with Pop Photo's results, but there are a few significant differences.

• Pop Photo measures SQF for lenses alone, while Imatest measures SQF for the entire imaging system. This means that Imatest results are sensitive to signal processing (sharpening and noise reduction; often applied nonlinearly) in the camera and RAW converter. This makes is difficult to compare lenses from measurements taken on different cameras. On the other hand, it means that you know what your camera/lens combination can achieve, and it's excellent for comparing lenses measured on one camera type (with consistent settings).
• Pop Photo's algorithm for calculating SQF (the SQF equation and the viewing distance assumption; both discussed below) is not known.

The scale on the right was developed by Bror Hultgren of Image Integration, based on extensive category scaling tests. According to Bror, the perceived quality level depends on the set of test images (particularly how bad the worst of them is) as well as the task (e.g., a group of cameraphone users would rank images differently from a group of art gallery curators), but the relationships between categories remains relatively stable. These levels are comparable to Popular Photography's scale.

Additional considerations in interpreting SQF:

• SQF has the same interpretation regardless of print size. That means a print with SQF = 92 would have the same quality "feel" for a 4x6 print as for a 24x36 inch print. This is in contrast to MTF measurements, where MTF measured at the print surface is interpreted differently for different sizes of prints: you tend to accept lower MTF for larger prints because you view them from larger distances (though the relationship, described here, is far from linear). Viewing distance is built into SQF.
• The SQF calculation omits printer sharpness (for now). It assumes that modern high quality inkjet printers can print as sharp as the unassisted eye can see at normal viewing distances— a fairly safe assumption for large prints (≥ 20 cm high).

Measuring SQF

SQF is measured as a part of Imatest SFR. All you need to do is check the SQF checkbox in the Imatest SFR input dialog box. Settings will be remembered in succeeding runs.

Clicking on Options to the right of the SQF checkbox opens this dialog box for setting SQF options. Most of the time you'll want to leave them (except for Maximum print height, which doesn't affect the calculations) at their default values, which you can always restore by pressing  Reset  .

CSF equation selects the equation for the Contrast Sensitivity function. There are two choices.

1. Flat from 3 to 12 cpd  (where cpd is cycles per degree) is the simplifying approximation originally used by Granger and Cupery in 1972, and by a number of authors since. (Granger actually used 10 to 40 cycles/mm at the retina, which translates to 3-12 cycles/degree when the eye's focal length of 17 mm is tossed into the equation.) This setting is recommended only for comparing results between the new and old explanation.
2. Equation: (a + bf) exp(-bf)  (the default setting) is discussed in detail in the section on Contrast Sensitivity function. This setting is recommended.

SQF integral is the form of the integral used in the SQF calculation. The default setting, 2. Integral( CSF(f) MTF(f) d(ln f) ), is recommended. The other values are experimental and not recommended; they are discussed below.

Viewing distance allows four choices.

1. Fixed  assumes a fixed viewing distance, which can be selected in the Base viewing distance... box.
2. Square root of PH (15 cm min.)  (the default setting) assumes that viewing distance is proportional to the square root of the picture height, (d = (base distance) (PH /10)^1/2 ), with a minimum of 15 cm. This is the recommended setting, appropriate for typical gallery viewing. We tend to look at large prints at greater distances than small prints. With this assumption, if you viewed a 4x6 inch print at 12 inches (a number often found in the literature), you would view a 16x24 inch print at 24 inches.
3. Cube root of PH (15 cm min.)  assumes that viewing distance is proportional to the cube root of the picture height, (d = (base distance) (PH /10)^1/3 ). With this assumption, if you viewed a 4x6 inch print at 12 inches (a number often found in the literature), you would view a 32x48 inch print at 24 inches. This seems to be a little close, but it could be appropriate in some situations.
4. Fixed print height; Distance from 1 cm to max.  This option is different from the others. Maximum viewing distance (cm) and Print height (cm) are entered instead of Base viewing distance and Maximum print height. SQF is plotted for viewing distances from 1 cm to the maximum for the fixed print height.

Of course Viewing distance is a broad average: we often move in and out when we critically examine a print. But some assumption must be made for the SQF calculation to proceed. We believe that choice 2, Viewing distance proportional the square root of the picture height, best represents the typical impression of sharpness for a range of prints. If we come across research that suggests a different fuction, we'll add it to the choices.

Base distance  Defined according to the Viewing distance setting. It should be left at its default value of 30 cm (12 inches) unless there is good reason to change it. 34 centimeters has also been used for measuring perceived quality in 10 cm high (4x6 inch) prints.

For Viewing distance = 1. Fixed, base distance is the viewing distance in cm.

For Viewing distance = 2. Square root or  3. Cube root (of picture height), base distance is the viewing distance for 10 cm (4 inch) high prints.

Maximum print height is the maximum to plot. (Note that the height setting assumes landscape orientation: wider than tall.) It has no effect on the calculations. The default is 40 cm (16 inches), which is about as large as prints from consumer digital cameras get. Picture heights of 60 cm (24 inches) and larger are of interest to users of professional-quality digital SLRs. 20 cm is a stretch for cameraphone images.

Reset resets settings to their default values. (It doesn't affect Maximum print height, which has no effect on the calculations.) We recommend keeping all settings at their default values, unless there is good reason to change them. This will help ensure standardized measurements and minimize confusion.

The SQF equation

Following the convention elsewhere in the Imatest site, we put the math in green boxes, which can be skipped by non-technical readers. The gist of the box below is that Granger presented the full equation for calculating SQF in 1972, but he used a simplified approximation for his calculations. Although the exact equation is strongly recommended, Imatest can use the simplified approximation where needed for comparing new and old calculations. In reviewing older publications, you should determine which calculation was used.

Contrast sensitivity function CSF

The human eye's contrast sensitivity function (CSF) is limited by the eye's optical system and cone density at high spatial (or angular) frequencies and by signal processing in the retina (neuronal interactions; lateral inhibition) at low spatial frequencies. Various studies place the peak response at bright light levels (typical of print viewing conditions) between 6 and 8 cycles per degree. We have chosen a formula, described below, that peaks just below 8 cycles/degree. You may learn something about your own eye's CSF by viewing the chart below at various distances and observing where the pattern appears to vanish. Chart contrast is proportional to (y/h)2, for image height h. To my eyes (which underwent Lasik refractive surgery in 1998 to correct for 10 diopter nearsightedness) it appears that the peak in the curve on the right should be considerably broader. But this isn't quite the case because of the eye's nonlinear response to contrast. Although the chart below appears to decrease in contrast linearly from top to bottom, the middle of the chart has 1/4 the contrast as the top.

Log frequency-Contrast chart, created by Test Charts.

Links

Bob Atkins' excellent introduction to MTF and SQF is highly recommended.

Bror Hultgren, with 26 years of experience as an imaging scientist at the Polaroid Corporation, has developed a number of tools that relate SQF to viewer preferences, using category scaling and subjective testing. Bror is available for consulting.

Popular Photography has been using SQF for testing lenses for years. Their test results are well worth exploring.

Noise in photographic images

Introduction

Noise is a random variation of image density, visible as grain in film and pixel level variations in digital images. It is a key image quality factor; nearly as important as sharpness. Since it arises from basic physics— the photon nature of light and the thermal energy of heat— it will always be there. The good news is noise can be extremely low— often impreceptably low— in digital cameras, particularly DSLRs with large pixels (5 microns square or larger). But noise can get ugly in compact digital cameras with tiny pixels, especially at high ISO speeds.

In most cases noise is perceived as a degradation in quality. But some Black & White photographers like its graphic effect: Many favor 35mm 35mm Tri-X film. The pointillist painters, most notably George Seurat, created "noise" (specks of color) by hand; a task that can be accomplished in seconds today with Photoshop plugins.

But by and large, the majority of photographers, especially color and large-format photographers, dislike noise with good reason.

Noise is measured by several Imatest modules. Stepchart produces the most detailed results, but noise is also measured in Colorcheck, SFR, and Light Falloff.

Appearance

	(A) Const. sensor noise	(B) Const. pixel noise	(C) No noise	(D) Canon EOS-10D ISO 1600

The appearance of noise is illustrated in the stepchart images on the right. Noise is usually measured as an RMS (root mean square) voltage. The mathematics of noise is presented in a green box at the bottom of this page.

The stepcharts in columns (A)-(C) are simulated. They are assumed to have a minimum density of 0.05 and density steps of 0.1, identical to the Kodak Q-13 and Q-14. They have been encoded with gamma = 1/2.2 for optimum viewing at gamma = 2.2 (the Windows/Internet standard). Strong noise— more than you'd find in most digital cameras— has been added to columns (A) and (B). Column (C) is noiseless.

The fourth column (D) contains an actual Q-13 stepchart image taken with the Canon EOS-10D at ISO 1600: a very ISO high speed. Noise is visible, but admirably low for such a high ISO speed (thanks, no doubt, to software noise reduction).

The noise in (A) is constant inside the sensor, i.e., before gamma encoding. When it is encoded with gamma = 1/2.2, contrast, and hence noise, is boosted in dark areas and reduced in light areas. The Kodak publication, CCD Image Sensor Noise Sources, indicates that this is not a realistic case. Sensor noise tends to increase with brightness.

The noise in (B) is uniform in the image file, i.e., its value measured in pixels is constant. This noise must therefore increase with brightness inside the sensor (prior to gamma encoding), and hence is closer to real sensor behaviour than (A). Noise appears relatively constant except for the darkest zones, where it's not clearly visible. Noise is often lower in the lightest zones, where a tonal response "S" curve superimposed on the gamma curve (or saturation) reduces contrast, hence noise.

For this reason the middle zones— where noise is most visible— are used to calculate the average noise: a single number used to characterize overall noise performance. We omit zones where the density of the original chart (hence display density in an unmanipulated image) is greater than 1.5 or less than 0.1.

Noise measurements

Noise measurements should ideally

• correlate with perceived appearance,
• (in many cases) be referenced to the original scene so the measurement is not affected by the tonal response (contrast) of the camera or RAW converter,
• be detailed enough to allow accurate assessment of sensor and camera performance, and
• be simple enough to interpret without difficulty.

Since noise is only meaningful in relationship to a signal, a Signal-to Noise Ratio (SNR or S/N) is often calculated.  SNR can be defined in many ways, depending on how Signal S is defined. For example, S can be an individual patch pixel level or the pixel difference corresponding to a specified scene density range (1.45 or 1.5 are often used for this purpose). It is important to know precisely how SNR is defined whenever it is discussed. SNR can be expresed as a simple ratio (S/N) or in decibels (dB), where SNR (dB) = 20 log₁₀(S/N). Doubling S/N corresponds to increasing SNR (dB) by 6.02 dB.

Since these requirements can be somewhat contradictory, Imatest modules have several noise and SNR measurements, some simple and some detailed. Available displays in Stepchart and Colorcheck include:

• Noise a function of pixel level or exposure, expressed in
  • Pixels, normalized to the pixel difference corresponding to a scene density range of 1.5 for Stepchart (comparable to the density range of 1.45 for the GretagMacbeth ColorChecker) or the White - Black zones in the ColorChecker (row 3, patches 1 - 6; density range = 1.45). Without this normalization, noise is a function of RAW converter response curve; it increases with converter contrast.
  • Pixels, normalized to the maximum pixel value: 255 for 8/24-bit files.
  • Pixels (maximum of 255 for 8/24-bit files)
  • f-stops (or EV or zones; a factor of two in exposure), i.e., referenced to the original scene. Noise measured in f-stops corresponds closely to human vision. See f-stop noise, below.
• Noise as a simple (average) number
  • A good single number for describing noise is the average noise of the Y (Luminance) channel, measured in pixels (normalized scene density difference of 1.5), shown in the lower plot in the figures, below. The lightest and darkest zones, representing chart densities greater than 1.5 or less than 0.1 are excluded from the average.
• S/N or SNR (dB) as a function of pixel level or exposure, where S is the pixel level of the individual test chart patch.
• S/N or SNR (dB) as a simple (average) number.
  • A good single number for defining SNR is the White - Black SNR, SNR_BW, where S is the difference in pixel levels between the White and Black patches in the GretagMacbeth ColorChecker in Colorcheck (density range of 1.45), or the pixel difference corresponding to a chart (scene) density difference of 1.5 in Stepchart. Noise N is taken from a middle level patch with density around 0.7.
    
    SNR_BW = 20 log₁₀(( S_WHITE – S_BLACK ) /&sl N_mid )
• a noise spectrum plot, described below.

In addition, Light Falloff displays a spatial map of noise.

Characteristic Stepchart results are shown below.

Column (B): Uniform pixel noise

This image has simulated uniform pixel noise (i.e., constant noise in the image file, measured in pixels). The upper plot is the tonal response (or characteristic curve) of the camera. It shows the expected ideal response for encoding with gamma = 1/2.2 = 0.4545: a straight line with slope = 0.4545 for the log-log plot. The middle plot shows noise measured in f-stops (or EV or zones). Noise increases more steeply for dark regions (large negative values of Log Exposure) in actual sensors. The lower plot shows noise measured in pixel levels, normalized to the difference in pixel levels for a density range of 1.5. It is relatively constant, showing only statistical variation, except for the brightest level (on the right), where the noise is reduced because some samples are clipped at pixel level 255. The lower plot also contains the single number used to characterize overall noise performance: the average Luminance channel noise (Y = 5.59%, on the right near the top). This number is very high; it corresponds to poor image quality.

This is SNR in dB for simulated uniform pixel noise. (This display option was introduced with Imatest 2.3.16). Because of the gamma = 2.2 encoding, SNR inproves by 6.02/2.2 = 2.74 dB for each doubling of exposure (0.301 density units); roughly 9.1 dB per decade (1 density unit).

Column (D): Canon EOS-10D at ISO 1600

Unlike the above figure (for the image in column (B)), real data for the Canon EOS-10D at ISO 1600 is used. The upper plot shows a slight tonal response "S" curve superimposed on the gamma curve (which is a straight line in this log-log plot). Some converter settings (such as "low contrast") result in a much more pronounced "S" curve. The middle plot shows f-stop noise, which increases dramatically in the dark regions. The lower plot shows the normalized pixel noise. It increases in the dark regions. This increase is due in part to the high ISO speed. Digital cameras achieve high ISO speed by amplifying the sensor output, which boosts noise, particularly in dark regions. This curve looks different for the minimum ISO speed: noise values are much lower and subject to more statistical variation.

This is SNR in dB for the Canon EOS-10D at ISO 1600. (This display option was introduced with Imatest 2.3.16). SNR improves by about 6 dB for each doubling of exposure (0.301 density units); roughly 20 dB per decade (1 density unit), which is what would be expected for constant sensor noise. (This curve would be dramatically different at lower ISO speeds.)

Noise summary

There are two basic types of noise.

• Temporal. Noise which varies randomly each time an image is captured.
• Spatial or fixed pattern. Noise cause by sensor nonuniformities. Sensor designers have made heroic and largely successful efforts to minimize fixed pattern noise.

Temporal noise can be reduced by signal averaging, which involves summing N images, then dividing by N. Picture Window Pro performs signal averaging using the Composite or Stack Images transformations, as described in Using Picture Window Pro in Astrophotography.

When individual images are summed N times, the signal pixel level increases by N. But since temporal noise is uncorrelated, noise power (rather than voltage or pixel level) is summed. Since voltage is proportional to the square root of power, the noise pixel level (which is proportional to noise voltage) increases by sqrt(N). The signal-to-noise ratio (S/N or SNR) improves by N/sqrt(N) = sqrt(N). When four images are averaged, S/N is improved by a factor of 2.

Several factors affect noise.

• Pixel size. Simply put, the larger the pixel, the more photons reach it, and hence the better the signal-to-noise ratio (SNR) for a given exposure. The number of electrons generated by the photons is proportional to the sensor area (as well as the quantum efficiency). Noise power is also proportional to the sensor area, but noise voltage is proportional to the square root of power and hence area. If you double the linear dimensions of a pixel, you double the SNR. The electron capacity of a pixel is also proportional to its area. This directly affects dynamic range.
• Sensor technology and manufacturing. The biggest technology issue is CMOS vs. CCD. We won't discuss it in detail here. Until 2000 CMOS was regarded as having worse noise, but it has improved to the point where the two technologies are comparable, differing only in detail. CMOS is less expensive because it is easy to add functionality to the sensor chip. Technology also involves other aspects of sensor design and manufacturing, all of which will improve gradually with time.
• ISO speed. Digital cameras control ISO speed by amplifying the signal (along with the noise) at the pixel. Hence, the higher the ISO speed the worse the noise. To fully characterize a sensor it must be tested at several ISO speeds, including the lowest and highest.
• Exposure time. Long exposures with dim light tend to be noisier than short exposures with bright light, i.e., reciprocity doesn't work perfectly for noise. To fully characterize a sensor it should be tested at long exposure times (several seconds, at least).
• Digital processing. Sensors typically have 12-bit analog-to-digital (A-to-D) converters, so digitization noise isn't usually an issue at the sensor level. But when an image is converted to an 8-bit (24-bit color) JPEG, noise increases slightly. The noise increase can be worse ("banding" can appear) if extensive image manipulation (dodging and burning) is required. Hence it is often best to convert to 16-bit (48-bit color) files. But the output file bit depth makes little difference in the measured noise of (unmanipulated) files.
• Raw conversion. Raw converters often apply noise reduction (lowpass filtering) and sharpening (see Noise frequency spectrum, below), whether you want it or not; even if NR and sharpening are turned off. This makes it difficult to measure the sensor's intrinsic properties.

General comments

• Imatest subtracts gradual pixel level variations from the image before calculating noise (the standard deviation of pixel levels in the region under test). This removes errors that could be caused by uneven lighting. Nevertheless, you should take care to illuminate the target as evenly as possible.
• The target used for noise measurements should be smooth and uniform-- grain (in film targets) or surface roughness (in reflective targets) should not be mistaken for sensor noise. Appropriate lighting (using more than one lamp) can minimize the effects of surface roughness.

F-stop noise The human eye responds to relative luminance differences. That's why we think of exposure in terms of zones, f-stops, or EV (exposure value), where a change of one unit corresponds to halving or doubling the exposure. The eye's relative sensitivity is expressed by the Weber-Fechner law,     ΔL ≈ 0.01 L   –or–   ΔL/L ≈ 0.01 where ΔL is the smallest luminance difference the eye can distinguish. (This equation is approximate; effective ΔL tends to be larger in dark areas of scenes and prints due to visual interference from bright areas.) Expressing noise in relative luminance units, such as f-stops, corresponds more closely to the eye's response than standard pixel or voltage units. Noise in f-stops is obtained by dividing the noise in pixels by the number of pixels per f-stop. (I use "f-stop" rather than "zone" or "EV" out of habit; any of them are OK.) noise in f-stops = noise in pixels / (d(pixel)/d(f-stop)) = 1/SNR where  d(pixel)/d(f-stop) is the derivative of the pixel level with respect to luminance measured in f-stops (log2(luminance) ). SNR is the Signal-to-Noise Ratio. The above-right image illustrates how the pixel spacing between f-stops (and hence d(pixel)/d(f-stop)) decreases with decreasing brightness. This causes f-stop noise to increase with decreasing brightness, visible in the figures above. Since luminance noise (measured in f-stops) is referenced to relative scene luminance, independently of electronic processing or pixel levels, it is a universal measurement that can be used to compare digital sensor quality when sensor RAW data is unavailable.

The mathematics of noise (just a taste) Amplitude distribution In most cases, the pixel or density variations that comprise noise can be modeled by the normal distribution. This is the familiar Gaussian or "bell" curve (blue on the right) whose probability density function is       f(x) = exp(-(x-a)2/2σ2 ) / sqrt(2πσ2 ) where exp(...) represents e = 2.71828... raised to a power, a is the mean (average value) of x, and σ is the standard deviation. σ is proportional to the width of the distribution: for the normal distribution, about 68% of samples are between a ± σ; 95.5% between a ± 2σ; 99.7% between a ± 3σ, etc. For a set of N samples xi with mean xm , the standard deviation is Noise is usually measured as RMS (root mean square) voltage or pixel level, where RMS is equivalent to standard deviation. RMS noise voltage is the square root of noise power. The value inside the square root is also called the variance. Noise varies with the pixel level, i.e., it differs in areas with different tonal values. The normal curve arises from a remarkable mathematical result called the central limit theorem: When a variable (such as voltage or pixel level) is affected by a large number of perturbations, the overall density function approaches the normal curve, regardless of the distributions of the individual perturbations. This is why the normal curve is by far the most common probability distrubution. But the normal distribution doesn't apply in all situations. For low light levels (low photon counts), where the normal distribution could result in negative counts, the Poisson distribution (red in the illustration above) gives the correct result.       f(s) = exp(-m)ms/s! where m is the mean, s ≥ 0 is an integer, and s! = "s factorial" = (s)(s-1)(s-2)...(1). The standard deviation is σ = sqrt(m). Shot (photon) noise, described in the Kodak publication, CCD Image Sensor Noise Sources, has Poisson statistics. For large values of m, the Poisson distribution approaches a normal distribution. Noise frequency spectrum In addition to an amplitude distribution, noise is characterized by a frequency spectrum, calculated by taking the Fourier transform of the spatial image. The spectrum is closely related to appearance. Two spectra are shown below, taken from the second Stepchart figure. The first is for the image from column (B), above, which contains simulated white noise. The second is for the image in column (D), above, taken with the Canon EOS-10D at ISO 1600. The images shown below have been magnified 2X (using the nearest neighbor resizing algorithm) to emphasize the pixel distribution. They are close approximations to the images used to calculate the spectra (but not the exact images). White noise, enlarged 2X (nearest neighbor) Blurred noise, enlarged 2X (nearest neighbor) White noise has two key properties. 1. The values of neighboring pixels are uncorrelated, i.e., independent of one-another. 2. Its spectrum is flat. The white noise spectrum (above, left) shows statistical variation and a small peak at 0.25 cycles/pixel, probably caused when the image in column (B) was resized for display. For spectral (non-white) noise, neighboring pixels are correlated and the spectrum is not flat. The spectrum and image (above, right) are the result of blurring (also called smoothing or lowpass filtering), which can result from two causes. 1. The Bayer sensor de-mosaicing algorithm in the RAW converter causes the noise spectrum to drop by about half at the Nyquist frequency (0.5 cycles/pixel), and 2. Noise reduction (NR) software lowpass filters noise, i.e., reduces high frequency components. NR usually operates with a threshold that prevents portions of the image near contrast boundaries from blurring. But NR comes at a price: detail with low contrast and high spatial frequencies can be lost. This causes the "plasticy" appearance sometimes visible on skin. Some people love it; I don't. (Plastic surgeons make a lot more income than I do.) The visibility of noise depends on the noise spectrum, though the exact relationship is complex. Noise at high spatial frequencies may be invisible in small prints (low magnifications) but very damaging in large prints (large magnifications). Because of the complex nature of the relationship, Kodak has established a subjective measurement of grain (i.e., noise) called Print Grain Index (Kodak Technical Publication E-58). Sharpening and unsharp masking (USM) are the inverse of blurring. They boost portions of the spectrum and cause neighboring pixels to become negatively correlated, i.e., they exaggerate the differences between pixels, making the image look noisier. Unsharp masking is often applied with a threshold that restricts sharpening to the vicinity of contrast boundaries. This prevents noise from degrading the appearance of smooth areas like skies. When poor quality lenses are used (or the image is misfocused or shaken), the image is lowpass filtered (blurred) but the noise is not. Some sharpness loss can be recovered with sharpening or USM, but noise is boosted in the process. That's why good lenses are important, even when digital sharpening is available. RAW converters often perform both sharpening and NR, whether you want it or not. This makes it difficult to compare the intrinsic performance of different cameras. Standardized sharpening is an imperfect attempt compensate for these software differences.

Links

CCD Image Sensor Noise Sources (a Kodak Application Note) An excellent, highly readable introduction to the different noise sources. Noise sources in CMOS sensors differ only in relative magnitude.

Neat Image is a remarkable program for reducing noise. Instead of lowpass filtering, the software is trained to recognize the structure of the noise. It can reduce noise with minimal side-effects. Particularly valuable at high ISO speeds.

Photography -- Electronic still-picture imaging -- Noise measurements (ISO 15739:2003) Murky, difficult to read, and expensive (106 Swiss Francs), but it's the official standard. If you'll try to read it you'll appreciate why the phrase "designed by committee" (ISO/TC 42) is so pejorative. Frequently refers to other expensive ISO standards. Its dynamic range measurement is bewildering: it uses a target with a black reference density of 2.0.

Sharpening

Why standardized sharpening is needed for comparing cameras

Introduction to sharpening

Sharpening is an important part of digital image processing. It restores some of the sharpness lost in the lens and image sensor. Virtually every digitized image needs to be sharpened at some point in its workflow— in the camera, the RAW conversion software, and/or the image editor.

Almost every digital camera sharpens images to some degree. Some models sharpen images much more than others— often more than appropriate. This makes it difficult to compare cameras and determine their intrinsic sharpness. Imatest has developed an approach to solving the problem— standardized sharpening.

Before we proceed, we need to describe the elementary sharpening process.

A simple sharpening algorithm subtracts a fraction of two neighboring pixels from each pixel, as illustrated on the right. The thin black curve is the input to the sharpening function: it is the camera's response to a point or sharp line (called the line or point spread function). The two thin dashed blue curves are replicas of the input multiplied by -ksharp/2 and shifted by distances of ±2 pixels (typical of the sharpening applied to compact digital cameras). This distance is called the sharpening radius. The thin red curve the impulse response after sharpening— the sum of the black curve and the two blue curves. The thick black and red curves (shown above the thin curves) are the corresponding edge responses, unsharpened and sharpened.

Sharpening increases image contrast at boundaries by reducing the rise distance. It can cause an edge overshoot. (A small overshoot is illustrated in the upper red curve.) Small overshoots enhance the perception of sharpness, but large overshoots cause "halos" near boundaries that can become glaringly obvious at high magnifications, detracting from image quality.

Sharpening also boosts MTF50 and MTF50P (the frequencies where MTF is 50% of its low frequency and peak values, respectively), which are indicators of perceived sharpness. But sharpening also boosts noise, which is not so good.

Standardized sharpening

The degree of sharpening performed by digital cameras varies greatly. Some cameras and many RAW converters allow you to change the amount of sharpening from the default value. If an image is undersharpened, the photographer will need to apply additional sharpening during the image editing process for best results. Many compact digital cameras oversharpen images, resulting in severe peaks or "halos" near boundaries. This makes small prints (4x6 or 5x7 inches) look good straight out of the camera, but it doesn't truly enhance image quality: halos can get ugly in big enlargements and noise can become objectionable.

Sharpening increases the 50% MTF frequency (MTF50). A camera with extreme oversharpening may have an impressive MTF50 but poor image quality. A camera with little sharpening will have an MTF50 that doesn't indicate its potential. For these reasons, comparisons between cameras based on simple MTF50 measurements have little meaning. Raw MTF50 is a poor measure of a camera's intrinsic sharpness, even though it correlates strongly with perceived image sharpness.

MTF50P, the frequency where MTF drops to half its peak value, is equal to MTF for weak to moderate sharpening but lower for strong sharpening. Hence it is a somewhat better indicator of perceived sharpness and image quality. Imatest displays MTF50P when standardized sharpening is not displayed (is unchecked in the input dialog box). It is also available as a secondary readout.

To obtain a good measure of a camera's sharpness— to compare different cameras on a fair basis, the differences in sharpening must be removed from the analysis. The best way to accomplish this is to set the sharpening of all cameras to a standard amount. This means sharpening undersharpened images and de-sharpening (blurring) oversharpened images.

The algorithm for standardized sharpening takes advantage of the observation that most compact digital cameras sharpen with a radius R of about 2 pixels (though R can be as low as 1 for DSLRs). This has been the case for several cameras I've analyzed using data from dpreview.com and imaging-resource.com. The algorithm for standardized sharpening is as follows.

1. Apply sharpening (or de-sharpening) with a default radius of 2 to make the MTF at f_eql = 0.3 times the Nyquist frequency (f_eql = 0.3 f_N = 0.15 dscan; a relatively low spatial frequency) equal to 1 (100%), which the MTF at very low spatial frequencies. MTF at higher spatial depends on lens and imager quality. The sharpening radius is adjustable in Imatest; 2 is the default.
2. Linearize the phase of the pulse by removing the imaginary part of the MTF. This results in an antisymmetrical pulse with a small overshoot (halo) near edges— typical of what you would get with carefully-done manual sharpening.

If the edge is seriously blurred (MTF50 < 0.2 f_N ), so that there is very little energy at  f_eql = 0.3 f_N , the sharpening radius is increased and the equalization frequency is decreased to f_eql  = 0.6 MTF50. The sharpening radius is not increased if Fixed sharpening radius in the Settings menu of the Imatest main window has been checked.

• Raw MTF50, without standardized sharpening, produces the most accurate results for comparing lenses and for comparing the sharpness at the center and edge of a single image.
• MTF50P is somewhat better when comparing cameras with strong sharpening. MTF50P equals MTF for slightly to moderately sharpened images, but it is lower for oversharpened images. It is a good sharpness indicator when images will not be post-processed, for example, in video cameras.
• Standardized sharpening is the best approach for comparing different cameras.

Examples

Oversharpening or undersharpening is the degree to which the uncorrected pulse is over- or undersharpened, i.e., sharpened relative to the standard sharpening value. It is equal to 100% (MTF( f_eql )-1). It if is negative (the case for the EOS-1Ds), sharpening is applied to the original response; if it is positive, de-sharpening is applied. (I use "de-sharpening" instead of "blurring" because the inverse of sharpening is, which applied here, is slightly different from conventional blurring.) Note that it is not the actual sharpening applied by the camera and software.

The raw MTF50 of the G5 is close to the EOS-1Ds, but the corrected MTF50 (with standard sharpening) is 73% of the EOS-1Ds, very close to the ratio of vertical pixels.

These results illustrate how uncorrected rise distance and MTF50 can be misleading when comparing cameras with different pixel sizes and degrees of sharpening. MTF50P is slightly better for comparing cameras when strong sharpening is involved.

Uncorrected MTF50 is, however, appropriate when comparing lens performance (different focal lengths, apertures, etc.) on a single camera.

Unsharp masking

"Unsharp masking" (USM) and "sharpening" are often used interchangeably, even though their mathematical algorithms are different. The confusion is understandable and far from serious because the end results are visually similar. But when sharpening is analyzed in depth the differences become significant.

"Unsharp masking" derives from the old days of film when a mask for a slide, i.e., positive transparency, was created by exposing the image on negative film slightly out of focus. The next generation of slide or print was made from a sandwich of the original transparency and the fuzzy mask. This mask served two purposes.

• It reduced the overall image contrast of the image, which was often necessary to get a good print.
• It sharpened the image by increasing contrast near edges relative to contrast at a distance from edges.

Unsharp masking was an exacting and tedious procedure which required precise processing and registration. But that has changed. USM can now be accomplished with only a few mouse clicks in an image editor.

Future versions of Imatest may use USM for more highly optimized Standardized Sharpening algorithms.

Sharpness comparisons

for several digital cameras

Sharpness results derived from Websites

Several websites publish images of ISO 12233 test charts that can be entered into Imatest SFR to obtain detailed sharpness measurements. This page contains sharpness results derived from charts downloaded from two sites:

• Dpreview.com (DPR)  
• Imaging-resource.com (IR)  

The measurements in the tables below come from two areas of the ISO 12233 image, illustrated by red rectangles in the image below.

• First row: near the center. The rectangle just above the center of the chart (about 15% of the distance to the corners).
• Second row: near the corner. The rectangle in the lower right of the chart (about 78% of the distance to the corner). Results include Chromatic aberration, which cannot be measured near the center.

These measurements are limited to

• One lens. For DSLRs this is usually one of the sharpest available, such as the 50mm f/1.4 prime (non-zoom).
• One focal length. Sharpness is a strong function of focal length for zoom lenses. The sites aren't very consistent in their choice of focal length for zooms in compact digital cameras; long, short or intermediate focal lengths may be chosen.
• One aperture, typically around the optimum: f/8-f/11 for DSLRs; f/4-f/5.6 for compact digital cameras.
• One ISO speed, which strongly affects noise and dynamic range, but has little effect on sharpness.
• One sharpening radius, typically 2, used in standardized sharpening calculations, indicated by (corr.) and red columns.
• One or two vertical edges (in most cases), resulting in horizontal MTF measurements. Sometimes vertical MTF (from horizontal edges) is different.
• One RAW converter (usually the JPEG converter built into the camera) with one setting, usually the default. The choce of converter and sharpness setting has a strong effect on sharpness measurements.
• Gamma assumed to be 0.5, typical for digital cameras. A 10% gamma error results in about a 2.5% MTF50 error. Though gamma error is rarely severe, gamma should be measured with Stepchart or Colorcheck for greatest accuracy.
• Exposure and lighting are not always optimum. If the image is too dim or bright there may be some clipping on data (where it goes pure black or white), which reduces the accuracy of the sharpness measurement. Lighting may not be as even as it should be.

If you want to compare your own camera or lenses with the published charts, test lenses at various focal lengths and apertures, or observe the effects of ISO speed and signal processing, you'll need to download Imatest. The trial version allows up to twenty runs.

Explanation of results

Tables of results

DSLRs: Digital SLRs, which take interchangeable lenses. Used by professionals and serious amateurs. Sensor size is at least 22 mm diagonal. Larger and heavier than the compact digital cameras, below. This table is more complete than the table for compact digitals because it's where my interest lies (and there are far too many compacts to keep up with). For each camera, the results in the upper and lower rows are for the image center and edge, respectively, as illustrated here.

Reminder: Remember the limitations listed above: one lens, one focal length, one aperture, one ISO speed, one RAW converter & settings (usually) built-in, one sharpening radius (2 unless otherwise indicated), etc., when interpreting these results. Also remember that several important image quality factors such as noise and dynamic range have been omitted.

Compact digital cameras: These are primarily consumer cameras, though many produce excellent image quality, especially for letter-size or A4 prints. Most have non-interchangeable zoom lenses. Maximum sensor size is 11 mm diagonal.

Interpretation of MTF50

What MTF50 do you need?  It depends on print size. If you plan to print gigantic posters (20x30 inches or over), the more the merrier. All the cameras in the above tables can make excellent 8.5x11 inch (letter-size; A4) prints. At that size the best of them wouldn't look much better than the 4 megapixel Canon G3 (or any other high quality 4 megapixel camera).

With fine lenses, careful technique, and a little extra sharpening, my 6.3 megapixel Canon EOS-10D (corrected MTF50 = 1335 LW/PH) makes very good 12x18 inch prints. Prints are quite sharp from normal viewing distances, but pixels are visible under a magnifier or loupe; the prints are not as sharp as the Epson 2200 printer is capable of producing. Prints made with the EOS-20D (corrected MTF50 = 1581 LW/PH) are have a slight edge: I'd call them excellent.

Softness or pixellation would be visible on 16x24 inch prints made from the EOS-10D. The EOS-20D would perform slightly better, and several of the cameras listed in the table— especially the full frame (24x36mm sensor) DSLRs— perform even better.

The table below is an approximate guide to quality requirements. The equation for the left column is

Example of using the table: My Canon EOS-10D has MTF50 = 1335 LW/PH (corrected; with standardized sharpening). When I make a 12.3 inch high print on 13x19 inch paper, MTF50 is 1335/12.3 = 108 LW/in: "very good" quality; fine for a print that size. Prints look excellent at normal viewing distances for a print this size.

This approach is more accurate than tables based on pixel count (PPI) alone. Pixel count is scaled differently; the numbers are around double the MTF50 numbers. The EOS-10D has 2048/12.3 = 167 pixels per inch (PPI) at this magnification. This table should not be taken as gospel: itwas first published in October 2004, andit may be adjusted in the future.

Some observations

There has been a good deal of skepticism about 7+ megapixel compact digital cameras. They live up to expectations for sharpness, but their noise may be objectionable at high ISO speeds. Dpreview.com has found the Sony DSC-828's noise at ISO 64 to be as high as the Canon EOS-300D (similar to the EOS-10D) at ISO 800.

The 6 megapixel raw/12 megapixel interpolated Fuji S7000, which uses Fuji's SuperCCD, has sharpness comparable to the Sony F828. It also has some problems with noise. The SuperCCD is based on an interesting concept. Thanks to gravity, most detail in nature is vertical or horizontal, and evolution has made our eyes are more sensitive to those directions. SuperCCD pixels are located on a diagonal grid where the spacing between x and y-pixel spacings is 1/sqrt(2) = 0.707 that of a normal grid the same number of pixels. The diagonal grid is interpolated to a horizontal grid that has twice the pixels, keeping the reduced pixel spacing. This results in an impressive gain in horizontal and vertical resolution, though sharpness is slightly below that of a sensor with the same number of uninterpolated pixels.

I analyzed the sides of Fuji F7000 because dpreview.com noted a problem with softness on the left side. Uncorrected MTF50 is the appropriate metric for comparing portions of a single image. The problem is indeed very evident. It seems to be slightly worse on the left, where MTF50 (uncorrected) is 62% of the center value. The lens alsohas a problem with chromatic aberration. The Sony DSC-F828 does much better: edge MTF is 93% of the center value. It apears to have a finer lens, at least at the focal length and aperture chosen for the test.

The Sigma SD9 has high resolution for its pixel count because it doesn't have, and probably doesn't need, an anti-aliasing filter. If only Foveon could break the 3.4 megapixel barrier...

Links

Scanner Bake-off 2004  Jim Hutchinson used Imatest results in his comparison of scanners.

Chromatic Aberration

AKA Color fringing

Introduction

Chromatic aberration (CA) is one of several aberrations that degrade lens performance. (Others include coma, astigmatism, and curvature of field.) It occurs because the index of refraction of glass varies with the wavelength of light, i.e., glass bends different colors by different amounts. This phenomenon is called dispersion. Minimizing chromatic aberration is one of the goals of lens design. It is accomplished by combining glass elements with different dispersion properties. But it remains a problem in several lens types, most notably ultrawide lenses, long telephoto lenses, and extreme zooms.

Lateral and Longitudinal CA; Tangential and radial lines

The two types of chromatic aberration are illustrated above. Longitudinal chromatic aberration causes different wavelengths to focus on different image planes. It cannot be measured directly by Imatest; it causes a degradation of MTF response-- by differing amounts for different colors. Lateral chromatic aberration is the color fringing that occurs because the magnification of the image differs with wavelength. It tends to be far more visible than longitudinal CA. Imatest measures Lateral CA. A software technique for reducing with is described in Eliminating chromatic aberration. Lateral chromatic aberration is best measured on a tangential edge near the sides or corners of an image. It's not visible on radial edges. Radial lines and tangential curves (which differ by 90 degrees) are shown in burgundy and blue on the right side of the above illustration. Because Imatest SFR requires edges to have an angle in the range of 3 to 7 degrees with respect to vertical and horizontal, only a limited number of locations in the ISO 12233 chart (on the right) are appropriate for measuring lateral CA. One is rectangle B, above. SFR/SVG charts, which can be printed any size on high quality photographic inkjet printers, are much better for measuring CA. The thumbnail on the right is from a 12 megapixel compact digital camera with fairly high chromatic aberration. The selected area is shown below. Red fringing, the result of lateral CA, is clearly visible. The black-to-white edge to the right side of this rectangle has equally vivid green fringing. Imatest analyzes the edge and produces a number that indicates the severity of the lateral chromatic aberration.

ROI for CA measurement

Imatest chromatic aberration measurement

The average transitions for the R, G, and B color channels, calculated by Imatest SFR for the above edge, are shown in the figure below. The edges have been normalized to have asymptotic limits of 0 and 1, i.e., they are dimensionless, approaching 0 and 1 at large distances from the transition center. Note that the three edges are not simply shifted, as you might expect if the focal lengths for the three colors were slightly different. They are distorted due to demosaicing (RAW conversion), as discussed below. When Bayer RAW images are analyzed, you do indeed see simple shifts.

The visibility of the chromatic aberration is proportional to the area between the edges with highest amplitude (in this case, red) and the lowest (in this case, blue). This area can be expressed by the following integral.

CA (area) = ∫ [ S_max(x) - S_min(x) ] dx

Since x has dimensions of distance in pixels and S is dimensionless, CA (area) has units of pixels— units of distance even though it is an area. CA defined by this equation is called the area chromatic aberration. It is displayed in magenta in the figure on the right. (CA (area) = 1.92 pixels). The distance between the crossings (the centers of the transitions) is also shown. It is less visually significant. While area in pixels is a good measure of CA, it has some shortcomings. It penalizes cameras with high pixel counts. The result depends on the measurement location. The chromatic aberration in most lenses is roughly proportional to the distance from the image center.

Chromatic Aberration figure (relatively high CA) (shown reversed: levels always increase from left to right)

To deal with these issues Chromatic aberration is also measured in percentage of the distance from the image center to the corner (percentage of sensor diagonal/2) , corrected for the angle of the ROI with respect to the center. In the above example, CA (area) = 0.115% of the distance from the image center. The correction is described in the green box, below. This measurement gives the best overall results, since it's relatively independent of the measurement location and the number of pixels. A table below presents rough guidelines for the severity of CA. Measurements displayed on the right of the figure are summarized in the following table.

Because Chromatic Aberration cannot be measured accurately near the image center, the chart is rendered in pale colors with the Region of Interest (ROI) is less than 30% of the distance from the center to the corner.

Demosaicing

Demosaicing is the process of converting Bayer RAW images, which have one color per pixel (RGRGRG...; GBGBGB...), to standard images, which have three colors per pixel. In the process of demosaicing, missing detail for each channel is inferred from detail in other channels. This is especially significant for Red and Blue pixels, which are half as common as Green. Demosaicing algorithms can be very mathematically sophisticated, but all of them can perform poorly in the presence of lateral CA, where detail is shifted from its expected location.

Demosaicing explains the shifted edges shown in the above example. Lateral CA cannot be reliably corrected after demosaicing, but it can be corrected to near-perfection prior to demosaicing. Correction coefficients can be calculated with Imatest Master, which can analyze Bayer RAW files created by converting manufacturer's Camera RAW files. Details are in the page on RAW files.

Here is an example illustrating the same region for a RAW and demosaiced file. Canon EOS-20D, 17-85mm IS lens, 53mm, f/11.

Chromatic aberration before demosaicing: easy to correct using a different
magnification for each color ( (1-0.00023)x for red; 1.000622x for blue; 1 for green).

Chromatic aberration after demosaicing: difficult to correct.

R-G and B-G are the CA correction coefficients. They are the spacing between the Red and Green and Blue and Green crossings, respectively, expressed as percentage of the center-to-corner distance. This mesurement is relatively independent of the location of the measurement.

Veiling glare (Lens flare)

Veiling glare is stray light in lenses and optical systems caused by reflections between surfaces of lens elements and the inside barrel of the lens. It is a strong predictor of lens flare— image fogging (loss of shadow detail and color) as well as "ghost" images— that can degrade image quality in the presence of bright light sources in or near the field of view.  It occurs in every optical system, including the human eye.

Lenses with low flare have been traditionally known for their excellent color performance. Color saturation is higher, especially in shadows. Low flare may be as responsible as sharpness for the exalted reputations of some classic Leica and Zeiss lenses. Even though lost color saturation and contrast can be recovered with digital processing, lenses with low flare will always have an edge in quality. There are several ways of dealing with lens flare.

• In lens design, veiling glare is controlled by using high quality lens coatings (multiple coatings are best), baffling in the lens barrel, and careful design.
• In the field it is controlled by lens hoods and anything that can shield the lens surface from the sun. Ansel Adams used his hat, as well as a bellows lens shade, made famous in a Datsun (Nissan) commercial.
• In the studio it is controlled by "barn doors" on light sources.

Obligatory bad photo showing severe lens flare caused by the sun (far left), and moderate flare with the sun blocked (near left). Canon EOS-20D, 24-70mm f/2.8L, 24mm, f/8, no hood. Taken during Colorado's notorious "Winter of 2007" (five feet of snow in three weeks in December and January)   The veiling glare measurement in Imatest Stepchart is designed to give a good prediction of a lens's susceptibility to lens flare in challenging situations: the larger the number, the worse it's likely to be. But it's not perfect in every detail. For example, it can't predict ghosting (visible behind the Stop sign in the image on the left). The measurement is fairly simple to perform, but several pitfalls can cause inaccurate or misleading results unless measurements are made with care. Some degree of exposure control— either a manual setting or exposure compensation— can enhance the accuracy of the results. Exposure may be a problem with cameraphones and other simple devices, where little, if any, control is available.

Veiling glare is measured by photographing a perfectly black object against a uniform white field that extends well beyond the image frame. It is defined as the ratio of the light reaching the sensor from the black object (which emits no light) to the light reaching the sensor from the surrounding white field.

Veiling glare = V = L(black object) / L(white surface), where L is the illuminance at the sensor.

Since digital image sensor output is linear, this measurement is straightforward

• if you have access to the linear (RAW) output of the sensor,
• if the white area is unsaturated,
• if the signal in the black area is large enough to be well above the noise, and
• if the black area is perfectly black.

We deal with the last of these issues first: how to create a perfectly black object— a black hole. Don't worry: it's more environmentally friendly than the type that sucks entire planets into its core.

Preparing the target

The black hole The darkest surfaces— materials or pigments— reflect about 0.5 to 1% of the incident light, i.e., they are are far from perfectly black. To obtain a surface suitable for measuring veiling glare you will need to construct a surface that little light reaches— a black hole— the darkest possible object that can be photographed in a bright environment. The black hole should be constructed inside a box or tube that is approximately 3x4 inches on its top (or 4 inches in diameter) and 4-8 inches deep. The bottom (inside back surface) should be lined with black velvet— the darkest material you can buy. The sides can be lined with any matte black material. We used black art paper because it was easy to work with. The top is a piece of black foam board with a 1x2 inch opening cut in the center. The structure should be kept as lightweight as possible. If necessary it can be constructed entirely out of black foam board, with the black velvet in the inside back surface.

Construction of the "black hole"

The black hole is mounted next to a standard step chart such as the Kodak Q-13 or Q-14, which performs two functions. It allows you to measure the camera's tonal response (pixel level as a function of scene luminance) and it provides a reference for determining the white level from a deep gray patch, so you don't have to measure it directly. This is important because many, if not most cameras, have nonlinear response in the bright regions, i.e., the response curve has a (film-like) "shoulder"— a good thing pictorially because without it digital cameras have a strong tendency to burn out highlights.

An alternative: Bart van der Wolf sent this link, which suggests that a cone painted glossy black can be used to make a superior black hole.

Gamma (image contrast) for measuring veiling glare is calculated somewhat differently from the average gamma shown in the figures. It is determined from the region where the step chart pixel level is between 0.1 and 0.6 of the minimum-to-maximum pixel range, i.e., the brightest areas, which are frequently nonlinear in digital cameras, are excluded. The white reference is inferred from the patch in the middle of this region. For example, suppose it is patch 11. Since the Q-13/Q-14 has density steps of 0.1, patch 11 is 1.0 density units darker than (1/10 as bright as) patch 1, which is the reference white, i.e., we infer the white level from the measurement of patch 11: the inferred white pixel level would be (10 times the pixel level of patch 11)^gamma. Since nonlinearity or even clipping of highlights is quite common, this technique results in a much more accurate veiling glare measurement than you'd get by measuring the white region directly. Tonal response for an in-camera JPEG and a linear (simple gamma curve) RAW conversion are shown here. Here is the modified veiling glare equation.

Veiling glare = V = ( Pixel level (black object) /Pixel level (white surface, inferred) )^1/gamma

Assembling the target A Kodak Q-13 step chart and the black hole box are mounted on a piece of mat board, roughly twice the width of the Q-13 chart— its dimensions are not critical. A 3/4x1 inch hole is cut in the mat board, just to the right of where of where the Q-13 chart will be mounted. The black hole box is mounted behind it. The Q-13 is mounted to slightly overlap the opening. The mat board is attached to the 40x60 inch white foam board with Velcro so it can be easily removed for safe storage. Black hole location detail

Typical framing for measurements

The entire target

Measuring veiling glare

• Illuminate the target evenly, with no more than about ±20% variation in illumination across the target, as described in Imatest test lab. Keep the ambient light near the camera as low as possible ti minimize the light entering the "black hole."
• Note whether you are using a lens hood or a filter (like the ubiquitous UV filter most photographers leave on their lenses for protection). Both affect veiling glare.
• Set the camera to the lowest available ISO speed to minimize the noise in dark patches.
• Adjust the exposure (or exposure compensation) so the white level is close to saturation, i.e., bias it in the direction of overexposure. It doesn't hurt if some highlights are blown out: the Imatest veiling glare algorithm is insensitive to highlight saturation.
• RAW output is preferred, though JPEG may be perfectly fine if the response is reasonably linear.
• Photograph the central portion of the target, leaving some white area around the Q-13 chart and "black hole." Typical framing is shown above, to the right of Assembling the target. The length of the white target to the left and right of the frame should be about equal the frame itself. Similarly, the length of the white target above and below the frame should be about equal to the frame itself. (Light entering the lens from outside the image frame is an important contributor to lens flare.)
• Run Imatest and select Stepchart. Open the image file. If Automatic Zone detection is unchecked and the image is the same size as the previous run you'll be asked if you want to repeat the same ROIs (Regions of Interest). If you enter No, you'll be asked to make an ROI selection. It's OK to make a rough selection: another window will enable you to refine it. After you've made the rough selection, the Stepchart input dialog box, shown below, appears. To measure veiling glare, Automatic Zone detection must be unchecked, Veiling glare must be checked, and the number of patches must be set to one greater than the number of patches in the step chart: 21 in this case.

Stepchart input dialog box, set for veiling glare

• Click OK. The fine adjustment box appears. Press Lighten, just to the right of Zoom out to view the dark patches clearly. The Lighten button changes to Normal when the image is in lightened mode. It also helps to maximize the window and click the Coarse radio button (middle, left).

Portion of Fine ROI adjustment window; Lightened view ( Normal box is displayed)

• Click Yes, Continue to run Stepchart and calculate veiling glare.

Results

The first illustration shows the results for the Canon EOS-20D with the 24-70mm f/2.8L lens set at 70mm, f/8. Veiling glare is shown on the middle-right of the first figure.

with UV filter (not multicoated)

No filter: 23% lower veiling glare

Overexposed about 2 f-stops, with UV filter: Results are insensitive to overexposure.

These plots show the difference in linearity between JPEG and RAW (to TIFF) density response. The EOS-20D only deviates slightly from linearity (a straight gamma curve).

JPEG density response, showing nonlinearity in highlights

RAW/TIFF density response, showing good linearity in highlights

Most of the results are not surprizing. The Canon 90mm has the lowest flare, while the cheap Canon 28-80 is the worst. The only surprise is the excellent performance of the Sigma 18-125: a lens know for its fine performance and value, though its autofocus performance is mediocre. I use mine on manual.

Limitations

The veiling glare may be underestimated for telephoto lenses— especially if they are measured without lens hoods— because telephoto lenses form an image of only a small fraction of the light reaching the front element (the portion is much larger with normal and wide angle lenses). The target, as shown above, may not be large enough to simulate all the light that reaches the lens. If a hood is used, this error is considerably reduced.

If there are nonlinearities in the camera response at low light levels, the results may be incorrect (although relative results, i.e., comparisons, will still be valid). It's always safest to work with RAW images and convert them with a "linear" (i.e., simple gamma curve) setting. You can read many RAW formats into Imatest, using the dcraw converter.

Color correction matrix

Introduction

Starting with Imatest 2.4.1, Multicharts can calculate a color correction matrix that can be applied to images to achieve optimum color balance, as defined by minimizing a color error parameter on the test chart of choice. (The default is the mean of (Delta-E 94)² for all patches where L*>10 and L*<95.) The matrix can be used by imaging system designers in their cameras or image processing algorithms. For example, Micron sensors use a 3x3 correction matrix.

Some of the background for the calculation can be found in Color Correction Matrix for Digital Still and Video Imaging Systems by Stephen Wolf, though the Imatest calculation differs in many respects: there is no issue with outliers and optimization is performed using one of the standard color difference metrics.

The color correction matrix is initially included only in Multicharts. In the future it may be added to Colorcheck and additional linearization options will be offered.

The Math

The matrix 

Color images are stored in m x n x 3 arrays (m rows (height) x n columns (width) x 3 colors). For the sake of simplicity, we transform the color image to a k x 3 array, where k = m x n. An Original (uncorrected) array O can be represented as

          | O_R1 O_G1 O_B1 |

      O = | O_R2 O_G2 O_B2 |

          |      ...       |
          |      ...       |
          | O_Rk O_Gk O_Bk |

where O_Ri, O_Gi, and O_Bi represent the normalized R, G, and B levels of pixel i. The transformed (corrected) array is called P, where

     P = O A         (case 1: A is a 3x3 matrix)  — or —
     P = [O 1] A    (case 2: A is a 4x3 matrix; the added column of 1s provides a dc-offset)

A is the 3x3 or 4x3 color correction matrix. For the 3x3 matrix,

          | P_R1 P_G1 P_B1 |    | O_R1 O_G1 O_B1 |  

      P = | P_R2 P_G2 P_B2 | =  | O_R2 O_G2 O_B2 |   | A11 A12 A13 |

          |      ...       |    |      ...       | X | A21 A22 A23 |
          |      ...       |    |      ...       |   | A31 A32 A33 |
          | P_Rk P_Gk P_Bk |    | O_Rk O_Gk O_Bk |

X denotes matrix multiplication. In this case, for row m,

P_Rm = O_Rm*A11 + O_Gm*A21 + O_Bm*A31      (* denotes multiplication.)
P_Gm = O_Rm*A12 + O_Gm*A22 + O_Bm*A32
P_Bm = O_Rm*A13 + O_Gm*A23 + O_Bm*A33

For the 4x3 matrix, a column of 1s is added to provide a dc-offset,

          | P_R1 P_G1 P_B1 |    | O_R1 O_G1 O_B1 1 |   | A11 A12 A13 |

      P = | P_R2 P_G2 P_B2 | =  | O_R2 O_G2 O_B2 1 |   | A21 A22 A23 |

          |      ...       |    |       ...        | X | A31 A32 A33 |
          |      ...       |    |       ...        |   | A41 A42 A43 |
          | P_Rk P_Gk P_Bk |    | O_Rk O_Gk O_Bk 1 |

In this case, for row m,

P_Rm = O_Rm*A11 + O_Gm*A21 + O_Bm*A31 + A41
P_Gm = O_Rm*A12 + O_Gm*A22 + O_Bm*A32 + A42
P_Bm = O_Rm*A13 + O_Gm*A23 + O_Bm*A33 + A43

The goal of the calculation is to minimize the difference (the mean square error metric) between P and the reference array (the ideal chart values) R. The initial values of A (the starting point for optimization) for the 3x3 and 4x3 cases, are

               | k_R 0  0  |               | k_R 0  0  |

      A(3x3) = |  0 k_G 0  | ;   A(4x3) =  |  0 k_G 0  |

               |  0  0 k_B |               |  0  0 k_B |
                                           |  0  0  0  |

where

k_R = mean(R_Ri ;  all i)/mean(O_Ri ;  all i)       for reference array R and original array O
k_G = mean(R_Gi ;  all i)/mean(O_Gi ;  all i)
k_B = mean(R_Bi ;  all i)/mean(O_Bi ;  all i)

These starting values are closer to the final values (have less mean square error) than the identity matrix (k_R = k_G = k_B = 1). They tend to converge slightly better.

Linearization

Although most digital image sensors are linear up to the point where they saturate, image files are highly nonlinear— they are designed for display at a specified gamma ( γ ), where display luminance = pixel level^γ. Gamma = 2.2 for the most commonly used color spaces (sRGB, Adobe RGB (1998) and Wide Gamut RGB (WGRGB)), although some well-known color spaces are designed for display at gamma = 1.8 (ProPhoto, Apply, ColorMatch; all RGB).

When cameras encode images (a part of the RAW conversion process), they apply a gamma that is the approximate inverse of the display gamma. Perhaps we should say very approximate: it may vary considerably from 1/γ, and it often includes a tonal response curve "shoulder" (an area of reduced contrast) to minimize highlight burnout. The shoulder makes the response more "film-like," improving pictorial quality in most instances.

The response should be linearized prior to applying the correction matrix, (though Imatest has the option of omitting linearization).

Optimization steps

• Linearize the input image (optional). If Color space gamma linearization is selected, O_L = O^γ.
• Call the optimizer, which
  • calculates a (temporary) corrected image T_L = O_L A.
  • Remove the linearization: T = T_L^(1/γ)
  • Finds the mean of squares of errors between T and the reference (ideal) array R. The error is one of the standard error measurements: ΔE*_ab, ΔC*_ab, ΔE₉₄, ΔE₉₄, ΔE_CMC, ΔC_CMC, ΔE₀₀, or ΔC₀₀, described here. ΔE₉₄ is the recommended default. Although the CIEDE2000 color error metrics (ΔE₀₀, ...) are more accurate, they contain small discontinuities that can affect optimization, and hence they should be used with caution. See Sharma for details. The ΔC errors are similar to ΔE with luminance (L*) omitted.
  • Adjusts A until a minimum value of the sum of squares of the errors is found, using nonlinear optimization.
  • Report the final value of A.

In applying A (generally outside of Imatest), a similar linearization should be used. A may be applied during the RAW conversion process, prior to the application of the gama + tonal response curve.

There is no guarantee that A is a global minimum. Its final value depends to some extent on its starting value.

The Color correction matrix in Multicharts

Options:  Color matrix calculation options can be set by clicking Settings, Color matrix in the Multicharts window. This brings up the dialog box shown on the right. The options are

• 4x3 or 3x3 matrix:  Color correction matrix size. The 4x3 matrix (the default) includes a dc-offset (constant) term. It may be slightly more accurate, but it takes more computation time.
• Optimize:  Select the color error parameter whose mean of squares over patches with L*>10 (nearly black; color is invisible) and L*<95 (little chroma) is to be minimized.  Choices include ΔE ab, ΔC ab, ΔE 94, ΔE 94, ΔE CMC, ΔC CMC, ΔE 00, and ΔC 00, described here. ΔE 94 is the default value, recommended because it gives less weight to chroma differences between highly chromatic (saturated; large a*² + b*² ) colors, which is closer to the eye's perception than ΔE ab (the standard ΔE value: the geometrical distance between colors in L*a*b* space).
• Weighting:  Set the weighting of the patches for optimization. Choices:
  • Equal weighting [default] Patches are equally weighted. May not be the optimum setting because highlight colors may be more visually prominent than shadow colors.
  • Emphasize highlights: weight according to CIELAB L*  Give more weight to visually-prominent hightlight patches.
  • Strongly emphasize highlights: weight according to L*^2.
    
• Linearization:  Choose the method of linearizing the image (or leave it unchanged). For linearization or gamma encoding, the value of gamma ( γ ) for the selected color space (sRGB, Adobe RGB, etc.) is used. The equation for linearization is O_L = O^1/γ. The equation for gamma application O = O_L^γ.
  • Linearize input, apply matrix, then apply gamma encoding. This is the best choice for images encoded in a standard color space.
  • No linearization: apply matrix to input pixels. Not generally a good choice for gamma-encoded images, but useful for experimentation and for images that start and remain linear.
  • Assume linear input, apply matrix, then apply gamma encoding.  A good choice for images that are not gamma-encoded, but need to be converted into a color space..

To calculate the color correction matrix, read the image into Multicharts, then press the Correction matrix button, shown on the right. The display will change, as shown below. The improvement for this image, which is quite good to begin with, is undramatic.

Split view, showing reference, input, and corrected patch colors

The image now shows the corrected colors on the bottom of each patch. The ideal (reference) color remains in the upper-left and the input (original) color remains in the upper right.

The Correction matrix button changes to Matrix calculated, highlighted with a pink background. The correction matrix cannot be recalculated until an image property changes (new image, color space, reference file, or color matrix setting). The Display input (or Corrected) dropdown menu, immediately to its left, is enabled. You can choose one of two selections.

The EXIF data and Color matrix display has a summary of results.

Exif and Color matrix view

The color correct matrix, results summary, and both [input - ideal] and [corrected -ideal] color difference summaries are shown. The initial and final error numbers shows how much the selected metric (in this case the sum of squares of Delta-E 94 for all patches with L*>10 and L*<95) has changed.

Blur units, MTF, and DXO Analyzer's BxU

Blur operations, MTF, and DxO^® Analyzer's BxU^®*

In early 2004, DxO Labs of Boulogne, France introduced a program for analyzing digital image quality called DxO analyzer^®. It has aroused a great deal of interest, but its price is far beyond the means of most photographers. (If you have to ask...) Several publications, including Luminous-landscape.com, pictchallenge.com (web), and France's outstanding Chasseur d'Images (print) use it for testing cameras and lenses.

DxO analyzer measures sharpness using the Blur Experience Unit (BxU^®), which was defined by Luc Marin of DxO in a PMA 2005 PowerPoint presentation.