Autofocus Analysis: Latency and Sharpness

In this study a new camera testing method is introduced to determine and analyze the autofocus latency of cameras. This analysis allows for objective comparison and tuning of autofocus algorithms in order to deliver both sharp images and the optimal user experience with a camera. Given images taken in variable illuminance conditions with different methods of focus reset, along with high-speed recordings of the camera viewfinder throughout the reset and capture, machine vision is used to extract three different types of latencies: ● The first latency is the autofocus time, which is measured from the end of the focus reset to full stability, as measured by slanted-edge sharpness in the camera viewfinder. ● The next latency is the user interface latency, which also comes from the viewfinder and is the time between the camera trigger and when the user interface of the camera indicates that a capture took place. ● The final latency is the captured image latency, which is taken from the captured image itself and is the time between the camera trigger and when the image is actually captured. In addition, we measure the sharpness of the final captured image in each test. Commercially available smartphone devices were tested using this method, showing significantly different results in both latency and sharpness measurements and uncovering trends in sharpness-latency trade-offs. Introduction In this study, we have explored a new camera evaluation method centered around the testing and analysis of autofocus algorithms. Exploring how autofocus algorithms on varied camera modules perform in comparison to each other and how they affect the optimal user experience with a smartphone camera can give insight to use in parameter tuning and benchmarking between cameras. This kind of analysis is best done in an automated and repeatable process, controlling as many test parameters as possible, in order to obtain the most objective and comparable results possible. Such a method, which has been developed and already used on commercially-available smartphones, is presented. Hardware and Control Software The first component to this fully automated autofocus testing system is customized testing hardware and corresponding control software. Our self-containing test station consists of a chart holder, chart magazine, adjustable light sources, LED tracer box, camera platform, autofocus reset devices, and a high-speed camera. The chart holder has a magazine containing a chart used as the target in the image capture and also a chart with illuminance meters to measure light levels in the center and edges of the chart during test setup and calibration. ​The LED tracer box, overlaying a portion of the chart, has a grid of LED lights that sequentially light up when triggered at the start of the test. The tracer is included in the captured image to timestamp the capture and exposure time based on the positions of the lit LED dots. Further, the autofocus reset devices include an occluder and rotation stage. The occluder moves a near target into the test camera field of view, blocking the view of the far target. The rotation stage rotates the test camera to different angles so that it can point towards different targets located at various distances. All of the hardware components are controlled by custom software, leading to a fully automated data collection system. This not only reduces the risk of human error during testing, as there are many components in play, but it also adds to the consistency and repeatability available in such a test. Testing Parameters There are certain parameters that we control that affect the latency and sharpness results. These include the camera-to-target distance, focus reset mode, light level, target type, reset delay, HDR+ mode, capture delay, and number of iterations. We have two general testing modes: 1. The first tests detection of objects and involves occluding the camera field of view with a near target consisting of a high-contrast image at 0.2 meters and keeping it there for a reset delay period of 3 seconds, then removing it to test the camera’s ability to switch focus to a target of slanted edges placed 2 meters away. 2. Our second test looks at how the smartphone’s gyro detection influences autofocus. The smartphone camera platform is rotated such that the camera has 3 seconds of reset delay to focus on a busy image 0.2 meters to the side, and then it is rotated back to be evaluated on its ability to refocus on a target of slanted edges placed 2 meters away. On top of the reset modes, we test different capture delays (0 milliseconds through 1000 milliseconds), which is the time between when the focus reset finishes and the camera is triggered to take a picture. Each of these capture delays are repeated 10 times to ensure the gathering of statistically significant data sets. Further, we run each of the above tests with all of the capture delays and iterations under different testing parameters, such as with HDR on or off, in addition to low and high illuminance levels. In addition, the chart with illuminance meters helps maintain and measure lighting across the tests, and the adjustable light sources throughout the isolated testing room allow for consistency between the near and far targets to eliminate an autoexposure (AE) latency variable. Further, the near and far targets have consistent color schemes such that auto white balance (AWB) variables do not add to the latency, either. This allows for a focus on the latency from the autofocus (AF) algorithm in the 3A (AE, AWB, AF) algorithm set. 2 Hampson et al.: Autofocus Analysis: Latency and Sharpness Published by Technical Disclosure Commons, 2017 Analysis Software Following the capture of the images and video recordings of the smartphone’s screen, the software uses machine vision algorithms to extract latency and sharpness measurements from the data set. These are plotted together to report the objective performance of the camera’s autofocus algorithm. These measurements include: ● the autofocus time, which is seen as the time from the end of the focus reset to full stability; ● the user interface latency, which is the time between the camera trigger and when the user interface of the camera indicates that a capture took place; ● the captured image latency, which is the time between the camera trigger and when the image is actually captured; ● and the sharpness of the captured image. Each of these measurements is examined in detail in the following sections. Autofocus Time The first measurement in the latency analysis is the autofocus (AF) time, which is the time between the focus reset and when the focus has stopped moving and thus stabilized. This focus time is directly caused by the speed of the autofocus algorithm to find the optimal focus position and move the lens to it, and in some cases it can be influenced to start or speed up by the capture trigger, hence the importance of varied capture delays in the testing parameters to look for this added factor. This analysis method uses a high speed video recording of the smartphone’s screen, computer vision, and a very thorough understanding of the control software’s timing of the trigger to extract this AF time. Using computer vision, the positions of fiducials in the camera field of view can be searched for in the image and then used to find a slanted edge in the target off of which to measure the sharpness per frame in the video. This is a complex task, as the varied spatial frequencies of the video camera and of the phone’s screen cause heavy aliasing and flickering in the recording. For this reason, conventional sharpness measures (for instance, the Modular Transform Function (MTF)) cannot be used, as they would be unreliable and noisy with such aliasing present around and on the slanted edge. Instead, the sharpness calculation must be done more rudimentarily, creating a shifted edge profile for the edge after first oversampling the pixels in the box surrounding the edge as follows. When the edge is vertical, this involves taking the individual rows of the region of interest surrounding the slanted-edge and re-sampling them to increase the size of the row. Then, horizontally shift each row such that the crossover from the dark pixels to the light pixels occurs at the same index in each row array. When the edge is horizontal, this is done column-wise. Then, the number of indices over which this crossover exists is used as a measure of sharpness. As this number decreases, the sharpness of the edge increases, as this means the crossover width from one side of the edge to the other took less pixels, indicating a sharper edge. Finally, these values are normalized across the frames to maintain a relative change uninfluenced by the raw number found, as in this case we are not interested in the raw sharpness but rather when the sharpness converges. Once this number converges, the autofocus has stabilized and the autofocus algorithm appears to have finished. Using the timestamp from the video frame at which this occurs and our knowledge of the timing of the hardware control software’s timing of the focus reset, we extract the autofocus time. User Interface Latency Next, we measure the latency that the user experiences with the camera user interface (UI) and viewfinder, which we will refer to as the UI latency. As an example, on many smartphones, image capture is indicated by the screen going dark. This is the latency that the user directly experiences, and the ideal situation involves the screen going dark immediately following the trigger to take a picture. Thus, the UI latency is measured from the moment the camera is triggered to take a picture to when the UI indicates capture, and the ideal latency is zero. Similar to the autofocus time above, we use a high speed video recording of the smartphone’s screen to extract this UI latency. Computer vision is used to find the location of the same slanted edge u