Introduction – Video – Documentation – Updates – White papers – Documentation web pages –
Conferences – EI 2024 paper – AutoSens Detroit 2024 talk – ISO 23654 – Call for participation
Introduction
The market for cameras that produce images for Machine vision (MV) and Artificial Intelligence (AI), in contrast to pictorial images for human vision, is steadily growing. Applications include automotive (driver assistance and autonomous vehicles), robotics, security, and medical imaging systems.
Two questions arise when designing camera systems for such applications.
- How best to select (or qualify) cameras for MV/AI applications?
- What image processing (ISP or filtering) is optimal?
To answer these questions, we must go beyond standard measurements of sharpness (MTF) and noise and apply metrics derived from information theory, including information capacity and related metrics for object and edge detection.
These metrics are important because Object Recognition (OR), MV, and AI algorithms operate on information, not pixels, making them far better predictors of system performance than MTF or noise.
Imatest has developed highly convenient methods for measuring information capacity and related metrics. The white papers (with varying degrees of detail) describe how the new metrics can be used to select (or qualify) cameras and determine the optimum Image Signal Processing (ISP) for Object Recognition, which is likely to improve the performance of MV and AI algorithms.
Video: Image Information Metrics in Imatest
Documentation, white papers, and instructions
Updates — We continue to learn new things about the information metrics.The units of Edge SNRi, which are frequency = 1/distance (typically 1/pixel), are non-intuitive and difficult to understand. To remedy this we have defined a new metric, Edge Location Standard Deviation (σ) = 1/Edge SNRi, which has units of pixels. It is the standard deviation of any point of the estimated edge location. The key metrics used to quantify machine vision system performance are Mean Average Precision (mAP) and Intersection over Union (IoU). We will be working on verifying the hypothesis that SNRi is a predictor of mAP and Edge Location σ is a predictor of IoU. We will be making some changes in the description of the noise calculation. We now look upon spatially dependent noise N(x) as the first step in the noise calculation, and the calculation of the Noise Power (Wiener) Spectrum, NPS(f), as the second step. The NPS(f) calculation is required to fully characterize the noise, which must be normalized so that ∫N(x)dx = ∫NPS(f)df . We are updating the documentation, but as of late June, 2024, it’s a long, tedious process. Many of the changes have been incorporated into the Autosens Detroit 2024 keynote talk (PDF). White papersThe white papers linked below contain similar material. Introduction to Image Information Metrics is a concise introduction to image information capacity and related metrics, with a minimum number of equations and technical detail. We may shrink it further for use in a brochure for marketing. Image Information Metrics and Applications: Reference (41 pages) has all the equations and technical detail, but it may be challenging to read. It’s the primary reference for the metrics. Image Information Metrics in Imatest is an intermediate-level introduction to image information capacity and related metrics with moderate technical detail. We are deprecating it to reduce the amount of redundant documentation. Camera Information Capacity from Siemens Stars (2020) describes a method that is slower and less convenient than the slanted edge, but better for observing the effects of image processing artifacts such as demosaicing, data compression, etc. Imatest software documentation web pages
ConferencesElectronic Imaging 2024Norman Koren presented a one-hour keynote talk on Image Information Metrics (covering much of the material in the white papers) at Electronic Imaging 2024 in Burlingame, California. The slides from the talk (PDF format) are available for download here. The EI2024 paper, “Image information metrics from slanted edges,” is an excellent introduction to the new metrics. Recommended. It is available for download here. I will change the link to the final paper when it is available on imaging.org. Autosens Detroit 2024 keynote talk (PDF of PowerPoint)The talk has been updated to include Edge location σ. Available for downoad here |
Try the Latest Metrics in the Imatest Pilot Program
Imatest 24.1 Alpha includes Edge SNRi, filter design, and many other significant enhancements over the 23.2 release.
Join the Imatest Pilot Program to try out these metrics in Imatest 24.1 Alpha.
Imatest 24.1 Beta will be available in early Spring. The full release of Imatest 24.1 will be in late Spring 2024.
International Standard (in development)
ISO 23654 began development in early 2023. It is based on ISO 12233 and defines how to calculate information capacity from a slanted edge. It is in the PWD (preliminary working draft) phase.
Call for Participation
We are interested in working with machine vision and computer vision experts who are experienced with studying how image quality metrics correlate with object detection performance in computer vision systems. Please contact infocap@imatest.com if you are interested.