Core Infrared Image Processing Technologies

Chapter 1: Introduction - Why is infrared image processing important?

Infrared thermography allows us to "see" temperature by capturing the infrared radiation emitted by an object, thereby imaging invisible temperature differences in complete darkness. However, the raw data obtained from infrared detectors is inherently rough, blurry and full of noise. Compared to the visible light images we are accustomed to, raw infrared images often haveLow contrast, poor signal-to-noise ratio, and fixed pattern noise (i.e., non-uniformity) caused by defects in the detector itselfetc. questions.

Therefore, powerful image processing technology becomes the key to unlocking the potential of infrared thermal imaging. Its core goal is not just to make the image "look more beautiful", but to complete it through a series of sophisticated mathematical operations and algorithms.CorrectionDetector physical defects,EnhanceThe subtle temperature difference that is difficult to detect with the human eye is finally realizedIntelligentThe whole process of target recognition and abnormality diagnosis. This article will provide an in-depth analysis of the core technology pipeline of infrared image processing, revealing how it transforms frames of blurred raw thermal data into clear, accurate, and insightful decision-making basis.

Chapter 2: Front-end correction and preprocessing technology

Before any enhancement or analysis can be performed, the raw data must first be "cleaned" to remove errors introduced by the hardware itself. The processing accuracy at this stage directly determines the upper limit and ceiling of all subsequent advanced algorithms. This is the cornerstone to ensure the effectiveness of all subsequent processing.

2.1 Non-uniformity Correction (NUC - Non-uniformity Correction)

This is the most representative and critical step in infrared image processing. Due to manufacturing process limitations, each detection unit on the infrared focal plane array (FPA) has slightly different response characteristics to the same infrared radiation. We can describe the response of each pixel with a simple linear model:I = G * X + O, among which X is the true infrared radiation intensity, and G (gain) and O (offset) It is the unique and inconsistent response coefficient of each pixel. This difference is superimposed on the image, forming a layer of fixed pattern noise like "dirty glass" or "watermarks" that seriously obscures the true temperature details.

• Calibration-based methods: This is the most commonly used and most accurate method. The core idea is to back-solve the G and O coefficients of each pixel by observing a target with known radiation intensity (usually a black body with uniform temperature).
  • Single point correction: Only one blackbody is observed, mainly used to correct the bias O. It assumes that the gain G is the same for all pixels, which is not true in many cases.
  • two point correction: Observe a high-temperature blackbody and a low-temperature blackbody respectively, and by solving a binary linear equation system, the gain G and offset O of each pixel can be accurately calculated at the same time. This is the current mainstream high-precision solution in industrial and scientific research applications.
  • Multi-point correction: In applications that require extremely high accuracy, more than two blackbodies will be used for calibration to more accurately fit the possible nonlinear response areas of the detector.

  It should be noted that the response characteristics of the detector will drift with operating temperature and time, so the calibration-based method requires regular "shutter correction" (for example, every time it is turned on) using a built-in baffle or shutter as a temporary uniform source to refresh the correction coefficient.

• Scene-based adaptive correction: In situations where blackbody calibration cannot be used (such as when the camera is continuously working), this type of algorithm can dynamically estimate and remove non-uniformity based on the statistical characteristics of the image sequence. The basic assumption is that the statistical properties (such as mean and variance) of the real scene radiation received by each pixel should be similar over a long enough period of time. Any long-standing, fixed statistical deviations are considered "non-uniformities" and are gradually eliminated by the algorithm. Common algorithms includeConstant Statistics、Kalman Filterand in recent years based onneural networkmethod. Although this type of algorithm is not as accurate as the calibration method, it is more flexible and convenient and does not need to interrupt the observation.

2.2 Detection and replacement of dead pixels/blind pixels

There may be a small number of pixels on the detector that are completely failed (constant response) or have abnormal response (extremely noisy), that is, "bad pixels". Dead pixel detection is usually completed during the factory calibration stage, and its judgment criteria include:

• static defects: The response value of a pixel is much higher or lower than its neighborhood mean.
• dynamic defects: The temporal noise of the pixel (i.e. the standard deviation of the response value) is much larger than that of the normal pixel.

Once the location of the bad pixel is determined and stored in the form of a "Bad Pixel Map", it is replaced with the output of its surrounding "good" pixels during image processing. The easy way is to useMedian or mean of 3x3 neighborhood, while more advanced methods will consider the edge direction and perform directional interpolation replacement to better protect the texture structure of the image.

2.3 Original data denoising

In addition to the fixed pattern noise mentioned above, the original infrared signal is also affected by random, time-varying noise, mainly including thermal noise and readout circuit noise. Denoising is usually performed combining both spatial and temporal dimensions.

• spatial denoising: Filter a single frame image, such as using Gaussian filter, median filter, etc. But more commonly used isedge preserving filter, such as Bilateral Filter or Guided Filter, which can avoid blurring the edge contour of the target while smoothing the noise.
• Temporal Denoising: This is the most effective way to suppress random noise. It is achieved by taking a weighted average of multiple consecutive frames of images. For example, simpleRecursive FilterThe formula is:Y(t) = α * X(t) + (1-α) * Y(t-1), where Y(t) is the current output frame, X(t) is the current input frame, and Y(t-1) is the output of the previous frame. The α value determines the intensity of denoising. The challenge with this approach is to balance denoising with motion blur: the smaller the alpha value, the better the denoising, but the easier it is to produce "ghosting" or smearing when moving objects appear in the scene. Therefore, advanced temporal denoising algorithms often include motion detection modules that use strong denoising in static areas and weaken or turn off denoising in dynamic areas.

2.4 Radiation calibration and temperature conversion

For consumer-grade thermal imaging cameras, the above steps are sufficient to output a clear relative heat map. However, for professional applications such as industrial temperature measurement and medical diagnosis, it is also necessary to convert the unitless digital signal (ADC value) received by the detector into a temperature value with physical meaning. This process is called radiometric calibration.

Its core is to establish an accurate relationship between signal value and target temperature. This usually requires a complex model that not only takes into account the internal response curve of the camera (calibrated in the laboratory by observing black bodies at different temperatures), but also must compensate for the effects of external environmental factors, including:

• Target emissivity (Emissivity): Different objects have different abilities to reflect and emit infrared radiation. In order to measure accurately, the emissivity of the object being measured must be set correctly.
• Atmospheric attenuation: Infrared radiation is absorbed and scattered as it travels through the air, especially when observing over long distances. Compensation needs to be made based on distance and air temperature and humidity.
• reflection compensation: The surface of the measured object may reflect radiation from surrounding high-temperature objects, causing the reading to be high. Advanced thermometers will ask for input of the reflected temperature of the environment to compensate.

Only after accurate radiation calibration can a thermal imager be transformed from an imaging device that can only "see" heat distribution into a scientific measurement instrument that can "measure" temperature.

Chapter 3: Image Enhancement Algorithm - Let the details "emerge"

After front-end correction, the fidelity of the image is guaranteed, but its dynamic range is usually very narrow, the visual effect is poor, and the contrast is very low, making it difficult to observe directly. The purpose of image enhancement is to use a series of algorithms to map the original 14-bit or 16-bit linear data to an 8-bit display range (0-255) that the human eye can comfortably perceive, and in this process, maximize the highlighting of the details we are interested in.

3.1 Grayscale transformation and dynamic range compression

This type of algorithm uses a mapping function to change the input gray value into a new output gray value one-to-one to change the overall brightness and contrast of the image.

• Histogram equalization (HE): This is a classic global contrast enhancement method. It redistributes gray levels by counting the grayscale histogram of the entire image and using the cumulative distribution function as a mapping curve to make the final output image histogram as flat as possible. However, for a typical infrared image—the histogram of which usually has only one huge background peak and several small target peaks—global HE will give the background with the largest proportion too much dynamic range, but instead compress the contrast of the small target area that we really care about, while also amplifying the noise in the background area.
• Plateau Histogram Equalization: In order to solve the problem of HE, platform equalization sets a "platform" upper limit for the original histogram before calculating the cumulative distribution function. Any pixel statistics above this limit will be "flattened". This is equivalent to limiting the excessive influence of large-area areas such as the background on the transformation function, leaving precious dynamic range "budget" to sparse but important areas such as the target, thus effectively suppressing the amplification of background noise while enhancing the target.
• Contrast-limited adaptive histogram equalization (CLAHE): This is a more advanced and effective local enhancement algorithm. It does not calculate the global histogram, but divides the image into several non-overlapping "tiles" (Tiles), and independently performs histogram equalization with platform restrictions in each small tile. In this way, each area of ​​the image can obtain the optimal contrast that matches its local characteristics. In order to eliminate the boundary effects between blocks, CLAHE also uses bilinear interpolation to smoothly transition the transformation function of each pixel. It achieves an excellent balance between retaining the global light-dark relationship and enhancing local details, and is currently one of the best infrared image enhancement algorithms.

3.2 Spatial domain filtering enhancement

This type of algorithm enhances local details of an image, especially edges and textures, by analyzing the relationship between a pixel and its neighbors.

• Smoothing and sharpening: Smoothing (such as Gaussian filtering, median filtering) can suppress residual random noise, but at the cost of the image becoming blurry. Sharpening is the reverse process and its purpose is to enhance edges. The most classic sharpening algorithm isUnsharp Masking (USM), the process is as follows:
  1. Copy the original image and perform a Gaussian blur on it, resulting in an "unsharp" image (i.e., a "mask").
  2. Subtracting this blurred mask from the original image results in a "difference map" that contains only edges and details.
  3. This difference map is added back to the original image with a certain weight, thereby greatly enhancing the contrast of the edges and making the image look "clearer".
• Bilateral Filter: This is an extremely excellent "edge-preserving denoising" filter. Traditional smoothing filters (such as Gaussian filtering) only consider the inter-pixelspatial distance, so while smoothing the noise, the edges will be blurred indiscriminately. The bilateral filter is a smart filter that "sees what others are doing". It considers two dimensions at the same time:spatial distanceandPixel value difference (color/brightness distance). When it calculates a new value for a central pixel, a pixel in its neighborhood will only be given a high weight if the two conditions of "close" and "similar" (pixel values ​​​​are close) are met at the same time. Therefore, it can smooth out noise in large flat areas while perfectly bypassing and protecting edge areas where pixel values ​​change drastically.

3.3 Pseudo color enhancement

The human eye has a limited ability to resolve grayscale (about dozens of levels), but its ability to resolve colors is much stronger (thousands of levels). Pseudo-color technology takes advantage of this and maps different grayscale values ​​into different colors through a "Look-Up Table (LUT)", thus amplifying the grayscale differences that the human eye is insensitive to into significant color differences, greatly improving the readability of the image.

• Selection and interpretation of color palettes (LUTs): Choosing the right color palette is critical for a specific application.
  • Grayscale palette (white hot/black hot): The most faithful to the original data and suitable for image analysis and feature recognition.
  • Continuous color palette (such as Iron, Lava): This type of palette has smooth and intuitive color transitions (such as from blue to red to yellow). It is very suitable for observing the continuous gradient process of temperature and can provide a natural visual experience.
  • High contrast palette (e.g. Rainbow/Rainbow, Jet): This type of palette uses a large number of jumping, highly saturated colors within a small grayscale range. They are extremely effective at quickly discovering and locating tiny abnormal hot or cold spots, where a small temperature difference can cause drastic changes in color. But its shortcomings are also obvious: it can easily give people the illusion of "temperature faults" and exaggerate the severity of temperature differences, and is not suitable for accurately judging the magnitude of temperature changes.
• Color Bar: A professional false color heat map must be accompanied by a color reference strip. This reference bar indicates the specific gray value or temperature value corresponding to different colors, which is the basis for accurate interpretation and quantitative analysis of pseudo-color images, otherwise the image will lose most of its quantitative meaning.

Chapter 4: Advanced Infrared Image Processing and Analysis

4.1 Detailed explanation of scene detail enhancement (detail layer enhancement) and DDE algorithm

In infrared thermal imaging, we often face a tricky challenge: the overall dynamic range of the image can be very large (for example, from the ground at -20°C to an engine at +300°C simultaneously in one frame), but the local contrast within the object of interest (whether it is a footprint on the ground or a tag on an engine) is very low. If traditional global contrast enhancement (such as histogram equalization) is used, either the details in the highlight area (engine) will be completely "saturated" into a white, or the details in the dark area (the ground) will be completely "submerged" in black.

Digital Detail Enhancement (DDE) Algorithms are the core technology designed to solve this problem. Its fundamental goals are:While maintaining the overall light and dark structure of the image, the visibility of local details is greatly improved.The essence of the DDE algorithm lies inmulti-scale decompositionthought, which sees images as consisting of"base layer"and"Layer of detail"Made of two parts.

The basic process of the DDE algorithm can be broken down into the following steps:

1. Generate base layer (base layer): The algorithm first needs to extract the base layer from the original image (Original). This is usually achieved through a low-pass filter. More advanced DDE algorithms will useedge preserving filter, the most famous of which isBilateral Filter。
2. Extract detail layer (detail layer): Once you have the base layer, extracting the detail layer becomes very simple. Simply subtract the base layer from the original image. This detail layer contains all edge and texture information in the image.
3. Non-linear gain amplifies detail layer: This is the "enhanced" core of the DDE algorithm. The algorithm will use aNonlinear, adaptive gain function (Gain Function), giving higher gain to areas with low contrast and lower gain to areas with high contrast, thereby amplifying details while suppressing noise.
4. Reconstruct the final image: Finally, the nonlinearly enhanced detail layer is added back to the base layer generated in the first step to obtain the final output image.

DDE enhancement comparison (left: original image; right: DDE-enhanced result), showing clearer local detail in the subject, road, and surrounding trees.

4.2 Image fusion: visible light and infrared

Visible light images have rich texture and contour information, while infrared images contain temperature distribution information. Image fusion technology combines the advantages of both to generate a composite image with high-definition detailed contours and a clear display of heat distribution.

4.3 Moving target detection and tracking

In fields such as security monitoring, detecting moving targets from a static background is a core task. Commonly used methods include background modeling and subtraction, inter-frame difference methods, etc., which can effectively extract and track targets in low signal-to-noise ratio environments.

Chapter 5: Intelligent Thermal Imaging Analysis Based on Artificial Intelligence

Artificial intelligence, especially deep learning, is revolutionizing the way infrared image processing is done, moving it from "enhancement" to "understanding".

5.1 Target detection and recognition (CNN/YOLO)

By training a convolutional neural network (CNN) on a large infrared image data set, such as the YOLO series model, real-time, high-precision detection and classification of specific targets (such as pedestrians, vehicles, animals) in thermal images can be achieved.

5.2 Image segmentation

Unlike object detection, which only provides a bounding box, image segmentation models are able to classify every pixel in the image. This provides the possibility to accurately measure the area, shape and temperature distribution of abnormal areas.

5.3 Automated anomaly detection and diagnosis

This is the ultimate goal of AI applications. By letting the model learn a large number of thermal images under normal and abnormal conditions, a system can be built that can automatically perform fault diagnosis. For example, it can directly report "the temperature of capacitor C103 is abnormal and there is a risk of short circuit."

Chapter Six: Processing Technology in Application Cases

6.1 Industrial inspection

An engineer uses a thermal imaging camera to inspect a substation.Non-uniformity correctionIt ensured that what he saw was not the equipment noise but the real temperature difference.Platform histogram equalizationandDDE algorithmThis allowed him to clearly see a tiny but loose overheating signal from a tiny connector bolt on a huge, otherwise normal-temperature transformer.

6.2 Medical diagnosis

Doctors use high-resolution thermal imaging cameras to assess patients' circulation conditions.Bilateral filteringSubtle noise in the skin is smoothed while preserving the clear outlines of blood vessels.False color enhancementUsing significantly different colors to distinguish normal body temperature from the tiny temperature rise caused by inflammation (perhaps only 0.5°C) provides an intuitive basis for diagnosis.

6.3 Security monitoring

In the port at night, a surveillance system that combines visible light and thermal imaging is working.Image fusion algorithmAllowing security personnel to clearly see the outline of the terminal and container numbers, while a highlighted, human-shaped heat signature moves within it. backgroundAI target detection modelIt was immediately identified as a "human" and an alarm was triggered, effectively avoiding the failure of traditional visible light monitoring at night.

Chapter 7: Summary and future prospects

Infrared image processing technology is a bridge that connects the invisible heat radiation of the physical world with human or machine perception and understanding. From basic non-uniformity correction, to complex detail enhancement and pseudo-color mapping, to revolutionary artificial intelligence analysis, this series of technologies work together to make the thermal imaging camera evolve from a simple "thermometer" to a powerful "insight tool."

Looking to the future, two major trends are leading the development of this field: First,Edge computing and real-time processing, integrate more algorithms into the camera front-end; the second isContinuous optimization of AI models, developing lighter and more efficient models to bring intelligent thermal imaging technology to a broader civilian and consumer market.

Ultimately, the goal of infrared image processing remains the same: to make the invisible world clearer, smarter and more controllable than ever before.