Enhancement of Thermal Imaging for Consumer Markets

According to the sampling theorem, if an array of pixels does not have a sufficiently small pixel pitch, spatial frequencies above the Nyquist frequency will be aliased into lower spatial frequencies. Consumer-grade thermal cameras often do not meet these requirements of the sampling theorem and so higher resolution detail is lost to aliasing. As the aliased frequencies are shift-dependent, multiple images with small non-integer pixel shifts can potentially contain further information from the underlying scene. The aim of this project is to improve the resolution of current consumer-grade thermal cameras using a multiple-image super-resolution technique. Multiple images with sub-pixel shifts are acquired and combined using knowledge of the exact shift to obtain a high resolution output image. The motivation for this work is that the performance of current low resolution cameras is limited by the pixel size, while the optics of the system allows for better performance. Through super-resolution, the resolving power can be extended to that of the system optics.

clynch_website_MTF
Figure 1: The modulation transfer function for a camera used in this project. The Nyquist frequency falls at 10 lp mm-1. Any spatial frequency reaching the detector above this is aliased into the lower spatial frequency range. All red shaded frequencies are therefore aliased. The aim of super-resolution is to recover these spatial frequencies to yield an output with a higher resolution.

 

Experimental images are obtained using consumer-grade thermal cameras. Two imaging platforms have been constructed with shifts applicable through either hand motion during capture or through controlled motion of a tip-tilt mirror. A co-aligned visible camera can be used to capture a simultaneous stream, with matching motion. As the pixels are much smaller in the visible camera, shifts can be measured to a much higher precision, with sub-regional registration also possible. A pre-computed scaling factor between streams is used to convert visible shifts into sub-pixel thermal shifts.
Shifts are currently measured through a frequency domain registration method using thermal images alone. Using this method is possible to measure shifts to the required sub-pixel precision. Alternatively, in the case of simultaneous visible capture, shifts can be measured to a high precision by first registering using the visible camera and conversion to a thermal shift using a scaling factor between cameras. A dual camera system is more robust in the presence of noise.
Resolution targets are used for justification of the methodologies used in both visible and thermal domains. As visible targets are not well-suited for imaging at thermal wavelengths, new test targets are required that consist of heated elements. Multiple targets have been designed for testing performance of the super-resolution method and registration.
A successful super-resolution method for the task is scene-independent, requiring little computation. Such requirements are considered in the design of this project.

clynch_website_apparatus
Figure 2: Experimental setup for simultaneous capture using thermal and visible cameras. Systems can be co-aligned using a dichroic beamsplitter. Alternatively, an offset between cameras can by simulated to match the physical separation between cameras in a consumer system.

 

clynch_website_results
Figure 3: (left) single low-resolution image interpolated, (right) super-resolution output using the experimental method developed. The resolution target used here was a target with two groups of three straight wires. A cooled surface was used to improve contrast in the target. In the original image, aliasing is present and wires are not well resolved. Upon application of the super-resolution method, the wires are better resolved.

 

Publications:
1. C Lynch, N Devaney, A Drimbarean, “Resolution enhancement of thermal imaging,” Irish Machine Vision & Image Processing Conference proceedings 2015, Irish Pattern Recognition & Classification Society (ISBN 978-0-9934207-0-2), 2015, pp 110 – 113.
2. C N Lynch, N Devaney, A Drimbarean, “Computational methods for improving thermal imaging for consumer devices,” Proc. SPIE 9485, Thermosense: Thermal Infrared Applications XXXVII, 94850P (May 12, 2015); doi:10.1117/12.2176566.