Using eSFR ISO Part 1

This document introduces eSFR ISO and explains how to obtain and photograph the chart. Part 2 shows how to run eSFR ISO interactively inside Rescharts and how to save settings for automated runs. Part 3 and Part 4 illustrates the results for slanted-edge and other measurements, respectively.

Click on the thumbnails below (or the image itself, repeatedly) to explore the many capabilities of eSFR ISO.
Note the description below the image. Results are described in more detail in Part 3.

eSFR ISO setup window. Layout is similar to SFRplus setup. Settings on the lower-left select auto detection of the grayscale step chart, color patches, and wedges. Edge and MTF plot for a single region. Any region can be selected. Large view of edge with Edge and MTF measurements-- for visual comparison. Available in Edge & MTF and Image & Geometry displays. 3D plot. Has somewhat less spatial detail than SFRplus. You can select any of a great many results (including secondary readouts). Lateral Chromatic aberration (affected by the lens and demosaicing) is most significant near the edges of the image. CPIQ Acutance or Subjective Quality Factor: a perceptual measurement that includes viewing conditions and the human visual system. Tonal response: Log pixel level vs. log exposure. Gamma is the average slope in the highlight to dark gray areas. The lower curve can contain local contrast or white balance. Multi-Region display: can become cluttered if too many regions are selected. Options are available to simplify the display. Display the image and selected regions and show a number of geometric results, including distortion, tilt, shifts, convergence angles, and field of view. Checking Crop to ROI lets you compare edge appearance to Edge and MTF results. Edge roughness, which can be decomposed into periodic (aliasing-related) and aperiodic (noise-related) components. Reference colors on the upper-left; input colors on the lower-right of each patch. Probe can be used to examine each color patch in detail for color differences (Delta-E ab, 94, and 2000). Color differences on the a*b* plane of CIELAB color space. Noise measurement from grayscale patches, with tonal response on top. All noise measurements from Multicharts and Multitest are available, including chroma noise, scene-referenced noise, sensor (raw) noise, and ISO 15739 visual noise. Wedge MTF and Onset of Aliasing (where the number of detected bars drops). Wedges are automatically detected when the Wedge checkbox in the Setup window is checked.

E-SFR ISO test charts

Imatest E-SFR ISO test charts come in three variants: Standard (minimal), Enhanced and Extended. All conform to the ISO 12233:2014 standard. The standard E-SFR ISO test chart, which has only features called for in the ISO standard, is shown below.

Standard (minimal) E-SFR ISO chart: 3:2 aspect ratio, no added features
Click on any of these charts for larger versions.Enhanced E-SFR ISO test chart: 3:2 aspect ratio, 6 added squares on sides,
16 added color patches, four pairs of added wedge patterns. Extended E-SFR ISO test chart: 16:9 aspect ratio, 6 added squares on sides,
16 added color patches, four pairs of added wedge patterns.

Enhanced E-SFR ISO test chart: 3:2 aspect ratio, shown with extra wedges
(4 low frequency, 16 high frequency total) and pre-distortion for use with fisheye lenses.
A great many options are available.

Sharpness is derived from light/dark slanted edges at the boundaries of the squares, as described in Sharpness: What is it and how is it measured? eSFR ISO can handle a wide range of camera aspect ratios; the left and right sides of the chart may extend beyond the frame or be well inside it.

ISO 12233:2014 Compliance:  The Imatest standard (minimal) E-SFR chart is similar to the chart illustrated in the ISO 12233:2014 standard document, but with a few minor differences that maintain ISO-compliance. (We recommend the Enhanced or Extended charts.)

• There are four registration (fiduciary) marks instead of three. An additional black dot, just above the lower-left registration mark, is used to detect orientation. This change was made because the projective transformation used for automatic region detection requires four points. Annex C, Table C.1, I of the standard states, “Asymmetric features for automatic target detection. These features should be placed within the 4:3 active area aspect ratio, but may be placed for convenience as dictated by use cases.”
   
• The focus-aid patterns in the center of the squares are different from the patterns illustrated in the ISO standard document (which are not described in the standard): they are square (which allows more room for the slanted-edge regions used for measuring sharpness), and they have been designed to minimize interference with automatic region detection. They have no effect on measurement results.
   
• A 20-patch OECF gray scale pattern is used. Annex C states, “The test chart also includes grey scale patches which are used to determine the OECF. While the example chart in Figure C1 uses 16 OECF test patches; a 20 patch version as described in ISO 14524 may also be substituted.”

The Enhanced and Extended charts contain valuable features (additional squares, wedges, and color patches) in regions of the chart that are not specified by the ISO standard. These features do not violate the standard in any way.

Old ISO 12233:2000 chartAdvantages of the ISO 12233:2014 Edge SFR chart over the old ISO 12233:2000 chart Note: the ISO 12233:2000 standard and chart, shown on the right, is referenced in the new ISO 122433:2014 standard, but is no longer an official part of the standard. The new standard specifies three charts, one of which is a slanted-edge (Edge SFR or E-SFR) chart with much lower contrast (4:1; shown on the right, below). Compared to the old ISO 12233:2000 chart, The E-SFR chart has much less wasted area, especially in the Enhanced and Extended versions. About 90% of the old ISO chart is covered with patterns that have little value for computer analysis.  ISO 12233:2014 Edge SFR chart from the ISO document You can produce a detailed map of sharpness (MTF) over the image surface with the Enhanced and Extended charts. This cannot be done with the old ISO chart because there are not enough suitable edges— and they are not well-located. Automated region detection, based on location criteria you enter. This makes the E-SFR chart and eSFR ISO module well-suited for automated testing: With the old ISO chart, regions of interest (ROIs) had to be selected carefully whenever the image framing changed, even slightly.   The 4:1 contrast edges are less likely to clip than the edges in the old ISO chart, whose contrast is specified at ≥ 40:1. The camera operates in a more linear region, and hence results are more consistent and accurate— less affected by overexposure, underexposure, incorrect gamma estimate, or excessive software sharpening. 4:1 contrast edges are more representative of real edges that affect perceived image sharpness. eSFR ISO can measure additional image quality factors, including lateral chromatic aberration, distortion, gamma (contrast), tonal response, and color accuracy (in charts that have the color pattern). 4:1 contrast 5X7 SFRplus chart The old ISO 12233:2000 chart contains no information that can be used to measure gamma (the average slope of the Log pixel vs. log exposure curve. i.e., the contrast), which is needed to linearize the image for the MTF calculation. Accurate gamma measurement is particularly important with high contrast targets. In the E-SFR chart, gamma can be derived either from the individual squares (which have a known 4:1 contrast) or from the grayscale step chart. Compared to the SFRplus chart, The map of MTF over the image surface is less detailed, though it's sufficient for most purposes. Distortion measurements are slightly less detailed (especially for the Standard (minimal) chart) because there are no distortion bars and there are fewer features for measuring distortion. eSFR ISO contains a highly detailed noise calculation, including all noise measurements from Multicharts and Multitest. Framing is more flexible since you don't need white space above and below top and bottom bars. As long as the registration marks are reasonably well inside the image, region detection should work.

Slanted-edge algorithm The algorithms for calculating MTF/SFR were adapted from a Matlab program, sfrmat, written by Peter Burns () to implement the ISO 12233:2000 standard. Imatest SFR, SFRplus, and eSFR ISO incorporates numerous improvements, including improved edge detection, better handling of lens distortion, a nicer interface, and far more detailed output. The original Matlab code is available on http://losburns.com/imaging/software/SFRedge/index.htm. In comparing sfrmat results with Imatest, note that if no OECF (tonal response curve) file is entered into sfrmat, no tonal response curve is assumed, i.e., gamma = 1 (linear response). Since the default value of gamma in Imatest is 0.5, which is typical of digital cameras, you must set gamma to 1 to obtain good agreement with sfrmat.

Obtaining the chart

Summary

High resolution photographic eSFR ISO chart

The three Edge SFR charts– Standard (minimal), Enhanced, and Extended– can be purchased from the Imatest store in a variety of media: inkjet (reflective and transmissive), LVT film (transmissive), and photographic paper (reflective; much higher resolution than inkjet charts). (For Chrome-on-Glass charts, SFRplus is often preferred because Chrome-on-Glass can't support a grayscale.) Inkjet charts can be printed on widebody inkjet printers, but you must have fine materials, skill, and a knowledge of color management. We strongly recommend that you purchase a chart.

Inkjet and photographic charts should be mounted on 32x40 or 40 x60 inch sheets of 1/2 inch (12.5 mm) thick foam board with a spray adhesive (such as 3M^TM 77 or Photo Mount) or double-sided tape (such as 3M^TM #568 Positionable Mounting Adhesive). 1/2 inch foam board stays flatter than standard 1/4 or 3/8 inch foam board. PVC board is also very promising: it may be more durable and it comes in a variety of sizes.

Photographing the chart.

Framing requirements:   Ideal cropping The cropping on the right strictly conforms to the ISO standard, but is not required for practical measurements. With this cropping the calibration of the wedges will be correct (LW/PH is the printed number X 100). These numbers are not used for Imatest analysis.
Close Cropping: The registration marks and the left and right (center row) squares should be within the image. Use SFR if you can't obtain this.	Far cropping: the chart should occupy at least half the image height. This cropping is not recommended because regions near the image boundaries are not analyzed and edges and other features may be too small to provide optimum (consistent) results with low resolution images.

• The boundaries of the chart may extend beyond the image (as shown below) or be well within the image. The software is designed to accommodate a wide variety of framing and aspect ratios. Edges near the left and right boundaries will be properly located or omitted if necessary. If the sides of the chart are inside the image, you should try to minimize interfering patterns that could be mistaken for registration marks. Chart surroundings inside the image should be light to middle gray if possible. If there are problems due to the surroundings, you can crop the image in the eSFR ISO setup window.
• If possible, avoid patterns that could be mistaken for registration marks outside the chart area. In practice eSFR ISO is surprisingly robust in such situations, but it's best to keep surroundings simple, and if possible, close to neutral gray (15-25% reflectance).

Well-framed eSFR ISO image
(though not quite perfect— a little closer would have been better)
It's OK (and quite commonplace) for the sides of the chart to run off the image.

• The chart may be aligned using techniques shown in Building a Low-Cost Test Lab. eSFR ISO tolerates moderate misalignment: a tilt of ~2 degrees, some perspective (keystone) distortion, etc., but every effort should be made to align the chart properly. Moderate barrel or pincushion distortion (<8% SMIA) is also well tolerated, but a pre-distorted chart should be used with extreme fisheye lenses (with high barrel distortion).
• The image can be cropped (starting with) to remove interfering features near the edges, using the Crop borders button in the eSFR ISO setup window.

If exposure compensation is available, increasing exposure by +1 f-stop often produces a better exposure, especially on the grayscale stepchart pattern.

Lighting

The chart below summarizes lighting considerations. The goal is even, glare-free illumination. Lighting angles between 30 and 45 degrees are ideal in most cases. At least two lights (one on each side) is recommended; four or six is better. Avoid lighting behind the camera, which can cause glare. Check for glare and lighting uniformity before you expose. A description of recommended lighting setups can be found in The Imatest Test Lab. (We formerly recommended 4700K (near-daylight) SoLux quartz-halogen lamps, but we now recommend LED lamps, which are available with adjustable intensity and color temperature.) The BK Precision 615 Light meter (Lux meter) is an outstanding low-cost instrument (about $100 USD) for measuring the intensity and uniformity of illumination.

Simplified lighting diagram

Distance

A letter-sized (8.5x11 inch) chart printed on Premium Luster paper on the Epson 2200 (a high quality pigment-based inkjet photo printer) was analyzed for MTF using the 6.3 megapixel Canon EOS-10D. There was no change when the image field was at least 22 inches (56 cm) wide— twice the length of the chart. Performance fell off slowly for smaller fields.

Choose a camera-to-target distance that gives at least this image field width. The actual distance depends on the sensor size and the focal length of the lens. The minimum image field is illustrated on the right.

Cameras with more pixels, and hence higher potential resolution, should have a larger image field width, hence printed chart width.

Exposure

Good exposure is important for accurate eSFR ISO results. Neither the black nor the white regions of the chart should clip— have substantial areas that reach pixel levels 0 or 255. The best way to ensure proper exposure is to use the histogram in your digital camera. Blacks (the peaks on the left) should be above the minimum and whites (the peak(s) on the right) should be below the maximum.

The above image (taken from the Canon File Viewer Utility) is close to a perfect exposure. Some exposure compensation, typically around +1 f-stop, may be helpful.

Save the image as a RAW file or maximum quality JPEG. If you are using a RAW converter, convert to JPEG (maximum quality), TIFF (without LZW compression, which is not supported), or PNG. If you are using film, develop and scan it.

If the folder contains meaningless camera-generated file names such as IMG_3734.jpg, IMG_3735.jpg, etc., you can change them to meaningful names that include focal length, aperture, etc., with the View/Rename Files utility, which takes advantage of EXIF data stored in each file.

You are now ready to run Imatest eSFR ISO.

Links

How to Read MTF Curves by H. H. Nasse of Carl Zeiss. Excellent, thorough introduction. 33 pages long; requires patience. Has a lot of detail on the MTF curves similar to the Lens-style MTF curve in SFRplus. Even more detail in Part II. Their (optical) MTF Tester K8 is of some interest.

Understanding MTF from Luminous Landscape.com has a much shorter introduction.

Understanding image sharpness and MTF A multi-part series by the author of Imatest, mostly written prior to Imatest's founding. Moderately technical.

Slanted-Edge MTF for Digital Camera and Scanner Analysis by Peter D. Burns (2000) (alternate source). An excellent introduction to the ISO 12233 slanted-edge measurement. Closely related: Diagnostics for Digital Capture using MTF by Don Williams and Peter D. Burns (2001), Applying and Extending ISO/TC42 Digital Camera Resolution Standards to Mobile Imaging Products by Don Williams and Peter D. Burns (2007) (Contains an image of an early version of the low-contrast slanted-edge test chart proposed for the revised ISO 12233 standard.)

Bob Atkins has an excellent introduction to MTF and SQF. SQF (subjective quality factor) is a measure of perceived print sharpness that incorporates the contrast sensitivity function (CSF) of the human eye. It will be added to Imatest Master in late October 2006.

Optikos makes instruments for measuring lens MTF. Their 64 page PDF document, How to Measure MTF and other Properties of Lenses, is of particular interest.