Image Stabilization/Sharpness Compare

Operation

• Acquire the images. Lighting should be even and glare-free (see lighting recommendation). The lighting level is important since it affects shutter speed (in camera phones). The incident lux level and shutter speed should be recorded. The camera should be mounted on a shake platform that has at least pitch and yaw movements. Roll is desirable, but translation (x, y, and z) movements add relatively little value. At least ten (and perhaps twenty or more) shaken images without and with OIS should be acquired.
• Select the worst (least sharp) of the shaken images without and with OIS.

The fastest way to find the least sharp (or sharpest) of a batch of images is to run Find Sharpest Files, which lets you read a batch of image files, select a Region of Interest (ROI), then very rapidly rank the files. The numbers used by Find Sharpest Files are not standard and should not be used for anything but comparing a batch of similar images.

Run SFRplus on two images (for a simple sharpness comparison) or three images (for Image Stabilization). The images should have the same region settings. The region (ROI) selection shown on the right is a good choice for image stabilization: it has 5 vertical and 5 horizontal edges (enough to show the effects of shake in the center and corner regions, as well as distinguish pitch, yaw, and roll components). Good choices range from 5 through 13 regions (anything more is overkill). A custom Edge ID file can also be used.

SFRplus region selection

Note the region width selection (just under Crop borders). Wider (or Widest): Smooth for OIS is recommended for shaken images, which have very different blur statistics from standard Gaussian-like blurs (doubled due to the sinusoidal nature of the vibration.)

Be sure the Image Stabilization checkbox is checked in the SFRplus More settings window (shown on the right). This enables results needed for Image Stabilization to be saved in the JSON summary file.

If you are running from Rescharts (or SFRplus Setup ) click Save data (at the bottom of the Rescharts window). Be sure JSON summary file is checked in the Save SFRplus results? window. Note the location of the JSON summary files: they are used as an input to Image Stabilization.

• Run the Image Stabilization module by pressing OIS/Image Compare on the left side of the Imatest main window or by clicking Modules, Image Stabilization.

The window shown below will appear.

Image Stabilization opening window

The first time Image Stabilization is run the three windows at the bottom (to the right of 1. Unshaken, 2. Shaken NO OIS, 3. Shaken with OIS) will be unpopulated. On successive runs they will contain the most recent file name, which must be a JSON output file from SFRplus.

• Enter the names of the JSON files (saved output files from SFRplus) in the appropriate boxes.
• If necessary (if you didn’t enter new file names) press Calculate on the right to display the results.

Results are shown below.

Image Stabilization metrics

The key goal of Image Stabilization is to calculate the improvement in sharpness that results from Image Stabilization. A few things must be taken into consideration in making the calculation.

• Blur from the unshaken image should be removed from the calculation so that the comparison shows only the effects of image stabilization.
• Shaken images taken without and with image stabilization should be equally representative of the amount of shake so they can be properly compared. This generally requires that the worst of a large batch of images be selected, as described above. For now we’ll omit further detail, like the selection of shake apparatus and discussion of the number of images required for a good statistical sample.

The primary results of Image Stabilization are the average edge and MTF for unshaken, shaken without OIS, and shaken with OIS images. All other metrics are derived from these numbers. The glossary below contains key definitions needed for understanding the results.

Definitions

L_MTF (L_MTF) is a useful but unfamiliar metric that requires explanation. Essentially it is a length (i.e., distance) derived from MTF that correlates well with R1090. Recall that MTF is the modulus (absolute value) of a Fourier transform, which means that phase information has been removed. With shaken images there can be significant irregularities (asymmetries, ramps, etc.) in edges that can distort R1090 or R2080, making them unreliable as blur metrics. These irregularities, which tend to be a part of the phase of the edge, have a limited effect on MTF. The image below shows an irregular edge.

L_MTF = 1/(2*MTF50) when MTF has cycles in its units (e.g., Cycles/Pixel). (L_MTF = 1/MTF50 when MTF has line widths in its units, e.g., LW/PH.) In the display below, L_MTF = 1/(2*0.0399) = 12.5 pixels, which happens to be close to R1090. This is not always the case.

Average edge profile and MTF for a rather poorly behaved edge.
L_MTF = 1/(2*MTF50) = 1/(2*0.0399) = 12.5 pixels, which is close to R1090.

Summary— L_MTF (rather than metrics derived directly from the average edge, i.e., R1090 or R1080) is preferable for calculations of image stabilization effectiveness, but all calculations will be available.

Shaken and unshaken edge lengths

Equations for edge lengths (R1090, R2080, and L_MTF) corresponding to NoS, Shake, and OIS measurements are straightforward enough, but

to correctly measure OIS sharpness gain, the unshaken length must be removed
from Shake and OIS edge length measurements.

There are three equations for accomplishing this.

Results designations

Let L (in L_Shake and L_OIS, below) be one of the edge length measurements (L_MTF, R1090, or R2080).

For example, the OIS gain using L_MTF with unshaken blur removed by the division method expressed in dB is

OIS_gain_L_MTF_div_dB (This is the preferred metric, described below.)

Preferred calculation

The preferred calculation uses L_MTF with unshaken blur removed by MTF division (div). It’s more mathematically abstract than the difference techniques ((S1) and (S2)) but straightforward and accurate (not an approximation).

MTF(Shake-only) = MTF(Shake)/MTF(NoS) is the equivalent MTF of the shaken image (no OIS) with the unshaken blur removed.

MTF(OIS-only) = MTF(OIS)/MTF(NoS) is the equivalent MTF of the shaken image (with OIS) with the unshaken blur removed.

MTF50 derived from these two numbers can be used to calculate OIS gain, but care must be taken if the expected order of MTF50 (MTF50(NoS) > MTF50(OIS) > MTF50(Shake) is altered.

MTF50(Shake-only) is derived from MTF(Shake-only) only if MTF50(Shake) < MTF50(NoS).
Otherwise MTF50(Shake-only) = MTF50(NoS).
L_MTF(Shake-only) = 1/(2*MTF50(Shake-only)).

MTF50(OIS-only) is derived from MTF(OIS-only) only if MTF50(OIS) < MTF50(NoS).
Otherwise MTF50(OIS-only) = MTF50(NoS).
L_MTF(OIS-only) = 1/(2*MTF50(OIS-only)).

The OIS gain equations using L_MTF for OIS vs. Shake (no OIS) are,

OIS_gain_L_MTF_div_Delta = L_MTF(Shake-only) – L_MTF(OIS-only) = MTF50(Shake-only)/MTF50(OIS-only) (fraction, generally > 1)

OIS_gain_L_MTF_div_frac = L_MTF(OIS-only) / L_MTF(Shake-only) = MTF50(Shake-only)/MTF50(OIS-only) (fraction, generally > 1)

OIS_gain_L_MTF_div_pct = 100%(1-OIS_gain_L_MTF_frac) (percentage, generally >0)

OIS_gain_L_MTF_div_dB = -20 log10(OIS_gain_frac) (decibels, generally >0) (shown below)

Results

Three graphic displays are available.
The first display (below) is the detailed Single region results. Any of the regions (18 in this analysis) can be selected.

Image Stabilization window showing results for single region

MTF for the unshaken image, shaken image (no OIS), and shaken image (with OIS) are displayed along with the Shaken (no OIS)/Shaken (with OIS) transfer function. The table contains all OIS gain results for this edge (derived from L_MTF, R1090, and R1080; all unshaken blur removal algorithms, Delta, fraction, percentage, and dB).

The second display (below) contains Multi-region results for the selected metric. Choices are shown in the image below. The mean OIS gain in the Multi-ROI and 3D plot table can be set to display Vertical-only, Horizontal-only, or both (all edges).

Multi-ROI plot of OIS_gain_L_MTF_div_dB (the preferred result), showing Metric drowdown menu.
(Only a limited number of metrics are currently available for the Multi-ROI and 3D plots.)

The third display is a 3D plot of the selected metric.

3D plot of OIS_gain_L_MTF_div_dB (the preferred result)

A CSV output file can be obtained by pressing Save data. The CSV file that contains summary information about the input files and summary results for Vertical, Horizontal, or V&H edges. Here are some excerpts from the CSV file:

(This is repeated for the second and (if included) third files.

(This is repeated for Horizontal and H&V edges.)

Note: This appendix is not an integral part of the Image Stabilization instructions, but it may be of interest to people who want to know why and how the Imatest algorithm differs from the CIPA DC-X011 algorithm.

Appendix: Notes on the CIPA DC-X011-2012 specification

CIPA (Japan’s Camera & Imaging Products Association) has published a draft specification, CIPA DC-X011-2012, Measurement and Description Method for Image Stabilization Performance of Digital Cameras (Optical Method), that may be of some interest to Image Stabilization users. However, the Imatest Image Stabilization module does not follow the CIPA specification closely because it has some serious shortcomings.

• Imatest uses the SFRplus test chart (a grid of slanted-edges) while CIPA uses a high-contrast (Black & White) checkerboard with a fruit bowl still life image in the center of the black squares. The edges of the CIPA chart are straight vertical and horizontal with a minimum contrast of 20:1. SFRplus charts could be printed with a 20:1 contrast that would satisfy the description ii §4-2-1, but if they were they tend to clip or saturate, which reduces measurement accuracy. We don’t recommend printing SFRplus with over 10:1 contrast. (We currently recommend 4:1 contrast SFRplus charts, which are consistent with the upcoming revised ISO-12233 standard.)
• DC-X011 makes no mention of MTF, which has been the industry-standard of measuring sharpness for decades. Instead, it uses the 10-90% rise distance, which is a less reliable metric. The math applied to these distances (for removing the unshaken blur, etc.) is an approximation and is inconsistent (sometimes simple addition or subtraction, sometimes the square root of the sum of the squares). The vertical and horizontal edges can result in sampling phase errors (which is why slanted-edges are used in the ISO-12233 standard: see Sharpness – What is it and how is it measured?) The CIPA standard recommends an excessive number of image acquisitions. For example, it recommends acquiring ten images without camera shake (§4-4-1), but doesn’t say what to do with them (best case? worst? average?). Only one is needed for a properly designed slanted-edge target.

Below are some notes I took while reading the CIPA standard. They include a glossary of obscure definitions used in the standard.

Since some of the terminology in section 3-3 the CIPA document may be confusing to English speakers— and is important to understanding the remainder of the document, we have created the following glossary. Blur has units of distance: pixels or microns.

More notes

Little is said about the “vibratory apparatus” except that it should have yaw and pitch waveform components. (Where’s Roll? In the same place as the “Roll” in “Rock and Roll”?)

§4-2-7 states that “the shooting distance shall be about 20 times the 35mm equivalent focal length.” Why can’t it just say “field height should be about 19 inches (48 cm)? 24-inch (60 cm) high Imatest SFRplus charts are close to ideal.

§4-2-8 implies that the 10-90% rise distance (for the linearized image) is the key distance metric.

§4-4-1: Bokeh Offset Amount = unshaken blur. >= 10 images recommended. What do they mean by 6) … “until the necessary and sufficient shutter speed is attained”? (Probably some arbitrary maximum.)

§4-4-2: Measured Comprehensive Bokeh Amount = Blur measured with shake, IS ON.10 then >=200 images should be taken with shake ON for each shutter speed.Not clear how to select among the 200+ images.

§4-4-3: Measured Comprehensive Bokeh Amount = Blur measured with shake, IS ON. Very similar to 4-4-2. It would be a LOT clearer if they emphasized the differences.

§4-5-1: b) How can IS performance be measured with a camera incapable of turning the IS function OFF? Apparently the vibration angle of the vibratory apparatus is known. (This angle could be affected by the weight of the camera & lens.) A very high quality vibratory apparatus would be required. This approach won’t be implemented soon in Imatest.

1) Bokeh Offset Amount = unshaken blur. By sqrt(Yaw² + Pitch²) do they mean vertical and horizontal edges? Clear as mud.

5: To measure IS performance in “stops” (f-stops or EV), measurements must be made at several shutter speeds. For camera phones (with very limited adjustments) this will require different light levels.

6.8: Have the authors heard of MTF? After reading this section (which I’d characterize as a lot of confusion about the circle of confusion) I scanned the document for MTF. Nada. I am shocked. Unbelievable!