HDR Technology White Paper

HDR technology overview

High Dynamic Range (HDR) imaging technology aims to improve the ability of images to present both bright and dark details simultaneously. The expansion of the dynamic range of camera sensors and display devices allows HDR images to be closer to the light and dark levels of the scene seen by the human eye. Traditional camera systems are often unable to take into account both the brightest and darkest areas in high-contrast scenes. HDR, through multiple exposures and intelligent synthesis, is expected to achieve the goal of "underexposed highlights and detailed shadows".

HDR imaging initially uses multi-frame image synthesis: taking a series of photos with different exposures (such as normal exposure, underexposure, overexposure), and then fusing the best details of each frame into one HDR image in the post-processing stage. This "multi-frame differential exposure" HDR idea was widely used in early digital photography (such as smartphones with HDR mode launched in 2011). Its core concept is to "learn from each other's strengths and complement each other's weaknesses, and combine them into one", that is, using short exposure frames to retain highlight details, long exposure frames to retain dark details, and finally synthesizing them into a high dynamic range result.

However, multi-frame HDR also exposes some problems: if there is a displacement in the shooting timing of multiple photos, moving objects will be misaligned in different frames, and ghosts or blur artifacts will easily occur after synthesis. At the same time, multi-frame acquisition and processing increase shutter delay and computational load, which may cause momentary images to be missed. Low-performance hardware also has difficulty processing complex alignment and fusion in real time. In order to improve these shortcomings, HDR technology is expanding from pure software algorithm synthesis to sensor hardware level fusion, combined with AI ghost removal and detail enhancement, and moving towards a new architecture of "Sensor+ISP+AI" collaborative processing.

In general, the development of HDR technology has experienced an evolution from multi-frame synthesis on the ISP side to single-frame/multi-frame hardware fusion on the Sensor side. The following will introduce in detail the HDR solution on the ISP side, various HDR implementation technologies on the Sensor side, and the technical paths of mainstream image sensor manufacturers in HDR. Finally, a comparative analysis of the two solutions will be conducted and future trends will be looked forward to.

ISP side HDR technology

ISP-side HDR refers to a technical solution that mainly relies on the image signal processor (ISP) or back-end software in the imaging system to align, fuse and optimize multiple differently exposed images to generate HDR photos or videos. Its characteristic is that based on the standard frame output by the sensor, the expansion of the high dynamic range is completed by ISP or software algorithm. A typical ISP-side HDR process includes: the sensor captures multiple frames (usually 2 to 3 frames) of images with different light and dark in rapid succession. After the ISP acquires the RAW or YUV images, it first performs image alignment (correcting misalignments caused by hand shake or movement), then weights and fuses the pixels of different exposure frames, and finally outputs the HDR result through tone mapping.

The HDR mode commonly used in early smartphones falls into this category. For example, the double-frame exposure + ISP synthesis method: continuously shoot a short exposure and a long exposure photo in a very short period of time, and ISP fuses the bright and dark information to output a wide dynamic range image. Some platforms support three-frame exposure fusion to obtain higher dynamic range, but additional frames also bring higher computational overhead and latency. The HDR+ algorithm proposed by Google is another innovative ISP HDR path: multiple short-exposure (under-exposure) photos are continuously captured and stacked for fusion. Different from the traditional combination of long and short frames, HDR+ not only controls the highlights but also reduces the noise in the dark areas through the superposition of multiple under-exposure frames, thereby obtaining HDR imaging that is close to normal exposure effects. This multi-frame simultaneous exposure fusion idea is widely used in night scene mode, which greatly improves the dynamic range and image quality in low light.

The advantage of a typical ISP HDR solution is that it does not require special sensor hardware support and is easy to implement on existing devices through algorithms. However, its disadvantages are also obvious: there is a time difference in multi-frame acquisition, and moving objects will cause ghost artifacts, requiring complex alignment and deghosting algorithms. In addition, continuous shooting of multiple pictures increases the delay, and users often feel "half a shot slow" from pressing the shutter to imaging. To this end, the industry has introduced improved solutions such as ZSL (Zero Shutter Lag) HDR: by allowing the camera to continuously capture frames at high speed and cache them, when the shutter is pressed, multiple frames at the latest moment are directly extracted from the cache for HDR fusion to reduce perceived latency. ZSL HDR is still essentially ISP multi-frame synthesis, but it achieves near-real-time output through clever frame scheduling, which greatly improves the user experience.

In short, ISP-side HDR takes algorithms as the core and achieves high dynamic range enhancement through multi-frame image fusion. This method is flexible and controllable and suitable for various existing sensors, but its effect is limited by the dynamic range of the sensor output and the ability of the multi-frame algorithm to handle motion/scenes. With the development of sensor technology, more and more HDR processing is being moved to the sensor side to make up for the limitations of pure algorithm HDR.

Sensor-side HDR technology

Sensor-side HDR refers to a technology that utilizes the special design of the image sensor hardware to obtain images or multi-channel signals with extended dynamic range during the sensor output stage, and then simply processes the ISP to obtain HDR results. Compared with ISP HDR, which requires multi-frame post-processing, Sensor HDR uses innovative pixel circuits and readout methods to fuse or output high dynamic range data at the sensor level. It has the advantages of good real-time performance and no shutter delay. Common sensor-side HDR technologies can be divided into the following categories:

time-sharing exposure

Time-sharing exposure (also called time-domain multi-frame HDR) means that the sensor sequentially captures multiple frames of different exposure images at different times and then fuses them. This is similar in concept to ISP multi-frame HDR, but here the sensor typically provides hardware support for consecutive exposures, such as quickly switching exposure parameters and outputting aligned multi-frames. The classic implementation is Sony's DOL-HDR (Digital Overlap HDR) technology: the sensor completes short, medium, and long exposures in one frame period, and outputs the data of these three frames in an interleaved manner. Since each exposure overlaps in time, sampling and readout are completed "quasi-simultaneously", greatly reducing the time difference between different exposures and reducing the risk of misalignment in moving scenes. For example, the Sony IMX290 sensor supports DOL-HDR and can output three RAW images with different exposures in one frame for ISP fusion. OmniVision and other manufacturers also have similar Staggered HDR solutions. The principle is similar to DOL. The difference lies in the number of exposure frames supported (for example, Sony DOL supports up to 4 frames of output, and OV Staggered supports up to 3 frames).

Sensors that use time-sharing multi-frame HDR increase data throughput compared to ordinary sensors, but by completing multiple exposures at the same time, the overall efficiency is higher, and because all exposures capture the scene at the same time, ghosting and motion blur problems are greatly reduced. It should be noted that this solution will still output multiple frames and require ISP to perform fusion in the RAW domain. However, due to the small inter-frame alignment error, the fusion quality and real-time performance are significantly improved. In addition, multi-frame overlapping exposure requires the sensor to have high-speed readout and group hold functions to ensure that each exposure parameter is applied synchronously.

divided gain sampling

Divided-gain sampling refers to a method that uses different settings of sensor analog gain/conversion gain to perform multiple gain readings on the same scene signal, thereby obtaining both light and dark signals and synthesizing HDR. This type of technology is completed within a single frame and does not rely on different exposure times. Instead, it uses different gain amplification for the same exposure to expand the dynamic range. Typical implementations are:

• Dual Conversion Gain Sensor (DCG): Each pixel supports high and low conversion gain modes. In bright scenes, the sensor switches to low conversion gain (LCG) mode to increase full well capacity to avoid saturation; in dark scenes, it switches to high gain (HCG) mode to increase sensitivity and obtain low-light details. DCG mode essentially selects the appropriate gain under different shooting conditions to improve the overall dynamic range and signal-to-noise performance of the sensor. It should be noted that traditional DCG only uses one gain mode at a time. It does not output HDR images, but only widens the adaptation range of a single frame.

• Simultaneous dual gain output (Dual Analog Gain, DAG): The sensor simultaneously performs two analog gain amplifications (high gain and low gain) on the signal of each pixel, reads out two sets of data respectively, and then combines them into HDR output in the sensor or ISP. The high-gain channel enhances dark signals, and the low-gain channel retains bright details. The combination can expand the dynamic range by about 10dB. DAG technology is also called single-frame dual-channel HDR. The advantage is that there is no time difference, no ghosting, and a single frame of HDR image is directly output. For example, GalaxyCore's recently released GC13A2 sensor uses a DAG single-frame HDR solution, achieving a nearly 10dB dynamic range improvement through "dual analog gain" in the 13-megapixel output. The real-time preview effect of DAG is also consistent with the final imaging, what you see is what you get.

• Dual conversion gain combining: Similarly, Samsung’s Smart-ISO Pro technology can be seen as a split-gain scheme that combines digital and analog. The principle is that the pixels collect signals in two modes: high ISO (high conversion gain) and low ISO (low conversion gain) at the same time, and then combine the two results inside the sensor to output a 12-bit HDR image. High conversion gain ensures details in dark areas, while low conversion gain preserves color in highlights. After synthesis, it covers a wider dynamic range and reduces rolling shutter distortion. Smart-ISO Pro is a single-frame, single-shot HDR solution that can complete HDR processing on the sensor side independently of the AP. Its upgraded version, Dual Slope Gain, implements HDR by applying two analog gain reads to a single signal when the pixel signal is insufficient for dual-channel conversion gain, and is applied to high-resolution cropping scenes.

The common advantage of split-gain sampling technology is that HDR is completed in a single frame without frame-by-frame alignment, and motion artifacts are completely avoided. However, its dynamic range improvement is limited by the gain range, and the high-gain path will increase noise. Therefore, it is often used in combination with other HDR methods. For example, the sensor supports Staggered multi-frame and DAG dual gain at the same time to further improve HDR performance. It is worth mentioning that the sensor uses conversion gain at the front end (pixel level), which is more conducive to reducing noise than back-end analog gain - this is why Smart-ISO Pro has better HDR signal-to-noise performance than Dual Slope.

Sensor-side PWL compression and ISP inverse PWL decompression (matched with divided gain sampling)

Multi-gain class built-in HDR (such as DAG/DCG/SmartISO Pro) is fused to obtain high bit depth linear RAW (such as 18–22-bit equivalent bit depth). In order to reduce link bandwidth and storage throughput, PWL (PieceWise Linear, piecewise linear) compression is often applied on the sensor side: the dynamic range is partitioned with several knees, each segment is quantized with different slopes, and the bit width is reduced to 10/12 bits for output via interfaces such as MIPI; the ISP side uses a matching inverse PWL lookup table to restore the code value to linear HDR RAW, and then enters processes such as denoising, white balance, color and tone mapping. This mechanism can also be used with Sensor HDR such as multi-slope/Quad Bayer.

The knee point/slope needs to be calibrated together with the gain chain, taking into account both highlight non-over-compression and dark quantization accuracy;

Common bit width: 10/12 bits; number of segments: 2–4 segments; optional differential jitter reduction quantization step;

The ISP inverse PWL version needs to be strictly matched with the sensor side to avoid color/brightness shift.

Dual/Multiple Slope Integration

Multi-slope pixel integration refers to dividing the pixel integration process into two or more time periods (slope/slope segments) within the same frame. Each segment uses a different exposure time or equivalent integration slope (can be combined with reset/clamp/conversion gain switching), thereby taking into account both highlight and dark information in a single frame. Common implementations include DualSlope (double slope) and MultiSlope (multiple slope, 3 to 4 segments), which are widely used in scenarios that require high motion robustness, such as automotive, security, and machine vision.

The basic idea is to sequentially execute multiple sub-integration segments such as short/medium/long in the same line: first use a very short segment to capture highlight details, and then use a longer segment to accumulate dark signals; at the end of each segment, clamping or partial reset can be performed to prevent highlight pixels from being oversaturated. After the row/frame is read out, the sensor completes the linearization and weight merging of each segment in the chip, and outputs one frame of HDR (some devices also support the output of multiple segments of data, which are merged by the ISP).

Advantages:Completed in a single frame, with almost no time misalignment, and no ghosting in moving scenes; more friendly to highlight overflow, and easier to retain highlight textures than pure gain-based solutions; no resolution loss, strong real-time performance, and suitable for videos and high-speed scenes.

Limitations:The position and weight of the "knee/knot" between segments need to be accurately calibrated. Improper algorithm will cause tonal discontinuity or local halo; the rolling shutter version still has linear distortion; anti-flicker strategies may need to be used under PWM lighting; the hardware complexity and power consumption are slightly higher than pure single slope.

Implementation points:It is often used in conjunction with gain methods such as DCG/DAG. It first uses multiple slopes to expand the effective full well and linear area, and then uses conversion/analog gain to optimize SNR; it can work in rolling or global shutter architecture. Taking dual slopes as an example: if the integration time ratio of the two segments is Tlong/Tshort=k, it can theoretically bring about a dynamic range expansion of about 20·log10(k) dB; multiple slopes (3 to 4 segments) can achieve a system DR of 90–120 dB in engineering.

Typical applications and deployment:Vehicle-mounted CIS (to cope with strong contrast in sunlight/tunnels, etc.), security backlighting, machine vision detection of high-brightness reflective parts, etc. The output form is divided into: (a) Single-frame HDR (Builtin HDR) synthesized at the sensor end; (b) Split data of each slope segment for ISP fusion in the RAW domain.

spatial interlacing

Spatially interleaved HDR is a method of obtaining light and dark information in parallel by alternating/distributing different exposures in the spatial dimension of the sensor. It includes specific implementations such as interlacing and interlacing pixels. The core is that different pixels in the same frame image bear different exposures, so that multiple exposure results can be obtained in one shot, and then fused into HDR images through reconstruction. The main spatial interleaving schemes are:

• Interlaced HDR (iHDR): The sensor alternates between long and short exposures in rows (or groups of frame rows). For example, Sony's early BME-HDR technology means that for every two rows of pixels, one row is short-exposed and one row is long-exposed, and finally the HDR output is obtained by fusion. This line alternation allows each frame to contain light and dark information simultaneously, avoiding the time difference of multiple frames, but at the cost of losing half the vertical resolution (because the effective information is only interlaced). OmniVision has also adopted a similar Alternate Row HDR (alternate row exposure) solution. iHDR can be regarded as an early attempt at sensor-level hardware HDR, which solved the problem of motion artifacts, but it gradually faded out of the mainstream due to reduced resolution and obvious image rasterization.

• Checkerboard exposure (Spatially Multiplexed Exposure, SME): This is an improvement on iHDR, and represents Sony's SME-HDR technology. It staggers long-exposure and short-exposure pixels in a checkerboard pattern at the pixel level. For example, the Sony IMX214 sensor uses SME, which arranges adjacent pixels in a spatial checkerboard for different exposures. In this way, light and dark pixels are evenly interspersed in each frame of the image, and HDR can be obtained after fusion. SME sacrifices less spatial resolution than interlacing, with Sony claiming that the resolution loss is about 20%. Through more advanced interpolation fusion algorithms, definition loss can be further reduced. SME achieves a compromise between HDR hardware sampling and high resolution, and is regarded as an upgraded version of iHDR. It is also called Zigzag HDR (zigzag HDR) by some literature.

• Quad Bayer HDR (Quad Bayer HDR): With the emergence of high-pixel density image sensors, the Quad Bayer structure (four sub-pixels of the same color constitute one "large" pixel) is widely used. This structure can divide four sub-pixels into two groups in HDR mode, perform short exposure and long exposure respectively, and then fuse and output HDR signals according to pixel positions. When not HDR, four-in-one output is used to improve the signal-to-noise ratio, and when HDR is used, two-in-one output is used to expand the dynamic range. 48-megapixel Quad Bayer sensors such as Sony IMX586 support this mode and complete HDR synthesis within a single frame. Compared with SME, Quad Bayer HDR utilizes sub-pixel redundancy to achieve both HDR and high resolution, and is suitable for real-time HDR scenes such as videos. SK Hynix’s research points out that Quad HDR has the advantages of high speed and good effects, and is especially suitable for high-resolution video shooting.

• Large and small pixel structure HDR: This is another spatial domain scheme, that is, large pixels and small pixels are designed in pairs in the sensor pixel array. Large/small pixel pairs are very close, almost sensing the same location, but have completely different photosensitive characteristics and saturation capacity due to different sizes. When shooting, the two are equivalent to simultaneous long exposure (large pixels are more sensitive and easy to saturate) and short exposure (small pixels resist saturation), and the HDR image is fused after readout. This method makes full use of the differences in physical structure, does not require multiple frames in time, and has no resolution loss problem. As early as 2003, Fujifilm launched the Super CCD SR sensor, which uses large/small pixel dual photodiodes to achieve HDR shooting. Sony's vehicle-mounted sensor IMX490 also applies a similar large/small pixel HDR architecture to achieve simultaneous acquisition of light and dark details in a single frame, which is particularly suitable for HDR imaging in high-speed scenes.

In-pixel storage

In-pixel storage HDR refers to integrating storage nodes in the sensor pixel circuit, allowing a single pixel to collect and save multiple copies of charge information during a single exposure, thereby achieving multi-exposure capture without time differences. To put it simply, pixels can store multiple sub-exposures in a time-sharing manner in one shot, and then read them out uniformly. This technique is usually implemented in conjunction with a global shutter or special driver. For example, some sensor pixels are designed with main and auxiliary photocapacitors. When the exposure starts, the charge is first transferred to the auxiliary container for storage in a short accumulation time, and then the remaining long-exposure charge is continued to be accumulated in the main container. Finally, the two parts of data are combined to output HDR. Under this architecture, short exposure and long exposure are actually performed almost simultaneously (completed within the same frame without any interval), completely eliminating the problem of artifacts caused by movement during shooting. In-pixel multi-storage node technology has been used in some academic prototypes and high-end sensors. For example, the early lateral overflow capacitor structure (LOFIC) can be regarded as a simple in-pixel storage scheme: after the pixel is full, the overflow charge automatically enters the bypass capacitor, which is equivalent to recording a short exposure information for highlighting details. For another example, Sony uses multi-sampling in some global shutter HDR sensors to read the pixel charge multiple times and reset it during the exposure process. This requires a storage shutter or memory capability in the pixel. In the same way, SmartSite's new generation In-Sensor HDR™ technology claims to be able to achieve high dynamic output in a single frame. It also obtains multiple exposure information without time difference through pixel circuit innovation. Generally speaking, in-pixel storage HDR technology is currently mostly used in professional and special applications (such as automotive photography, high-end video, etc.) due to its complex pixel structure and high cost. However, it represents the ideal form of HDR imaging - truly acquiring multiple levels of exposure simultaneously without any subsequent alignment correction.

To sum up, sensor-side HDR technology includes multiple paths from multi-frame in time domain to multi-pixel in spatial domain to circuit-level gain and storage. Their common goal is to solve the pain points of HDR shooting at the sensor hardware level: not only improve the dynamic range, but also reduce the delay and ghosting caused by multiple frames. Different solutions have their own trade-offs: the time domain solution focuses on compatibility but still has a slight time difference, the gain solution is simple to implement but has limited improvement, the spatial interleaving solution has no time difference but needs to deal with resolution loss, and the in-pixel storage solution has the best effect but is the most difficult to implement. In actual products, these technologies are often used in combination to complement each other's strengths and achieve the best HDR imaging effect.

Figure 5: Sensor-side HDR hardware technology family tree (showing the layers and typical representatives of mainstream solutions).

Detailed explanation of HDR implementation technology from mainstream sensor manufacturers

The major CIS (CMOS image sensor) manufacturers currently on the market have launched unique HDR technology solutions. The following is an analysis of HDR implementation based on products from manufacturers such as Sony, OmniVision, Samsung, Onsemi and SmartSens.

Sony

As a leader in the field of image sensors, Sony has accumulated rich experience in HDR technology and has launched multiple generations of hardware HDR solutions:

• Multi-frame overlay HDR (DOL-HDR): Sony introduced digital overlay HDR in consumer-grade sensors earlier. Its IMX290 and other devices support multiple interleaved exposure outputs within one frame. Sony DOL-HDR can output long, medium, short or even very short four-channel exposure frames for ISP fusion, and is widely used in security, vehicle and other video scenes that require high dynamic range. DOL mode effectively reduces the delay and ghosting problems of traditional frame-by-frame HDR, and is an important milestone for sensor-level HDR.

• Row/column alternating HDR (BME-HDR, SME-HDR): Sony used spatial interleaved HDR technology in early mobile phone CIS (such as IMX135, IMX214). BME-HDR achieves HDR through interlaced short and long exposures, but the disadvantage is that the vertical resolution is halved. SME-HDR uses checkerboard pixel interleaving instead, only losing about 20% of resolution. SME is considered to be Sony’s improved solution to iHDR, achieving a better compromise between clarity and HDR. Sensors using SME such as IMX214 can directly output fused HDR images or output double exposure frames for processing. These innovations from Sony have explored valuable directions for sensor hardware HDR. However, due to resolution loss issues, BME/SME is now less common in new main cameras and more used in specific modules.

• Quad Bayer HDR: Sony is the first to apply the Quad Bayer pixel structure to high-pixel sensors in mobile phones and use this structure to achieve HDR. IMX586, IMX689 and other models combine four pixels into one output in normal mode to improve sensitivity; in HDR mode, diagonal pixels are grouped to perform different exposures and then fused to generate a full-resolution HDR image. This solution takes into account high pixels and HDR effects, and has become a standard feature of Sony's high-end mobile phone CIS. In addition, Sony has also developed in-pixel double conversion gain technology to achieve single-frame HDR on industrial sensors such as IMX661.

• Large pixel + small pixel architecture: The IMX490 launched by Sony in the automotive field adopts a HDR solution for large and small pixels in the same field of view. Each unit of this sensor contains two pixels, one large and one small. They share an optical viewing angle. The large pixels improve low-light performance, while the small pixels are responsible for the bright light part. The combination achieves a wide dynamic range and no ghosting. This architecture improves HDR while maintaining full resolution output, and is also very friendly to high-speed sports scenes. It is a new idea for automotive HDR.

Overall, Sony has built a comprehensive HDR technology matrix from mobile terminals to automotive security: mobile phone CIS focuses on multi-frame overlap and Quad Bayer fusion, and professional fields explore special pixel structures and global shutter HDR. In addition, Sony also cooperates with AI ghosting, Local Tone Mapping, etc. in its imaging algorithms to maximize the effectiveness of hardware HDR data.

OmniVision (OmniVision Technology)

As a major global CIS manufacturer, OmniVision also has multiple plans in HDR:

• Alternate Row HDR: Howe’s early sensors used an alternate row HDR solution similar to Sony BME, namely Alternate Row HDR. For example, automotive-grade CIS such as OV10640 achieves dynamic range expansion through different exposures of odd and even rows to adapt to high-contrast scenes. Although the interlaced scheme has a resolution loss, it was an effective means of in-vehicle HDR at the time.

• Staggered HDR: Howe’s mid-to-high-end sensors in recent years generally support staggered multi-frame HDR. For example, OV48C, OV64B, etc. support triple exposure Staggered output, similar to Sony DOL. It is worth mentioning that the OV series of high-end mobile phone CISs (such as OV48C) launched by OmniVision around 2020 already have Staggered HDR capabilities and were launched almost at the same time as the Sony IMX766 (the first Sony staggered HDR mobile phone sensor). This shows that Howe is not lagging behind in Sensor HDR, and its solution can also output multiple channels of RAW with different exposures in one frame.

• Dual Conversion Gain / Dual Analog Gain: OmniVision has long applied DCG technology (called DCG™) to enhance dynamic range. Security sensors such as OS08A and OS04A achieve industry-leading low-light and HDR performance through dual-conversion gain modes. Recently, Howey has further introduced the Dual Analog Gain (DAG) solution to achieve single-frame dual-gain HDR. Official information shows that high-end sensors such as OV50A/OV50X support DAG mode, which simultaneously reads out high and low gain signals on-chip and fuses them to improve dynamic range and take into account LED flicker suppression. Howe's DAG technology is similar to Samsung's Dual Slope or Geke Micro DAG solutions, focusing on the field of mobile video HDR.

• Combined HDR solution: Haowei often combines multiple HDR technologies on its flagship mobile phone CIS. For example, OV50X is said to support triple exposure HDR + dual analog gain - the sensor can output three frames of different exposures at the same time, and also applies high analog gain to the dark channel to achieve maximum dynamic range coverage. This strategy of combining software and hardware reflects Haowei’s pursuit of HDR performance. In addition, OmniVision's marketing term zHDR is commonly used in its mid-range products (such as OV13A10/16B10, etc.), which refers to the acquisition of multiple exposure data in a special arrangement within a single frame. It is speculated to be similar to Sony's SME-HDR.

Overall, OmniVision's HDR technology includes both Staggered/DAG single-frame HDR for mobile platforms and multi-frame HDR+LFM (anti-flicker) solutions for automotive security. Howe is actively catching up with Sony and Samsung in terms of mobile phone sensors, enhancing its competitiveness through high pixels and high dynamic range performance. With its acquisition and integration by Vail, OmniVision has continued to invest in HDR innovation and continues to launch new HDR functions on its high-end flagship CIS to meet market demand.

Samsung

Samsung Semiconductor has made rapid progress in the smartphone image sensor market in recent years, and has also developed its own unique HDR technology:

• Staggered HDR: Samsung’s latest generation ISOCELL sensors (such as GN2, HP1, etc.) all support Staggered (triple) HDR shooting, which completes multiple exposures in a single frame and outputs multiple signals for ISP fusion. Samsung calls it progressive HDR, and it does the same thing as Sony/OV’s interlaced HDR, which is used to improve dynamic range performance in videos and photos. One example is the GN2 sensor, which has powerful interleaved HDR capabilities and can provide a dynamic range of more than 100dB without sacrificing frame rate.

• Smart-ISO / Smart-ISO Pro: This is Samsung’s original single-frame HDR solution. Ordinary Smart-ISO means that the pixels can automatically switch between high ISO (high conversion gain) and low ISO (low conversion gain) modes to adapt to light and dark scenes and improve the overall photosensitive performance of the sensor. On this basis, Samsung launched Smart-ISO Pro, which can simultaneously read high and low ISO signals and synthesize them into HDR images. The specific process is that when taking pictures, the sensor generates high-gain and low-gain data in parallel, and then intelligently fuses them to output a 12-bit color depth image. Because it completes two types of gain sampling in a single exposure, Smart-ISO Pro can avoid the delays and artifacts of multi-frame HDR and provide more realistic high dynamic range imaging in scenes such as backlighting and portraits. Samsung revealed that by merging two 10-bit signals, Smart-ISO Pro can display approximately 68.7 billion colors (12-bit), far exceeding traditional 10-bit HDR. This technology is also called iDCG (intra-scene Dual Conversion Gain) and has been adopted on image sensors such as HM3 and GN5.

• Dual Slope Gain: In order to solve the problem that Smart-ISO Pro cannot be used in high-resolution mode (such as the pixel signal at full resolution is too small to support dual conversion gain at the same time), Samsung introduced the Dual Slope Gain mechanism. DSG applies two analog gain readings to the same signal during pixel reading, and obtains high and low gain data. In this way, even if dual conversion gain cannot be used in single-pixel mode, HDR can be achieved through dual analog gain. Samsung uses Dual Slope to achieve HDR functionality available at full resolution on ultra-high pixel sensors such as HP2. However, Samsung officials also pointed out that Smart-ISO Pro is still better than Dual Slope in terms of signal-to-noise performance because the former amplifies the signal at the pixel end and has lower noise. In the future, Samsung may automatically switch between the two modes according to the scene to take into account HDR needs under different focal lengths/modes.

Taken together, Samsung is following Sony in HDR technology and taking advantage of the advantages of SoC manufacturers to deeply integrate HDR solutions with ISP/AI. For example, its mobile phone platform supports real-time fusion of multi-channel HDR data output by sensors in the ISP, or implements ZSL HDR at the software layer to output photos with zero delay. Samsung’s strategy is to provide solutions for single-frame sensor HDR (Smart-ISO Pro, etc.) and multi-frame ISP HDR (ZSL, AI HDR) in parallel. On the HP2 200-megapixel sensor released in 2023, Samsung has combined triple ISO modes (low, medium, high) and improved Smart-ISO Pro, which can output 14-bit HDR in a single frame, and the dynamic range is extremely amazing. It is foreseeable that Samsung will continue to take advantage of its sensor and processor collaborative design to launch innovative solutions with higher integration and stronger performance in the HDR field.

Onsemi

As an important supplier of global automotive and industrial CIS, Onsemi has long-term accumulation in wide dynamic range (WDR/HDR) technology:

• Multiple exposure + LFM: ON Semiconductor's automotive-grade image sensors generally support triple or even quadruple exposure output, as well as the accompanying LED flicker suppression (LFM) function. For example, the classic AR0231AT has a three-frame HDR architecture that can output long, medium, and short frames at the same time, and uses a special algorithm to reduce the flicker of the LED light source. In an autonomous driving environment, it is necessary to be highly dynamic and deal with the stroboscopic problem of LED traffic signals. Onsemi solves the problem of HDR and LFM compatibility by adding a short exposure to the sensor specifically to capture LED pulses, and then blending it with other normal exposure frames. This HDR solution is widely used in automotive cameras, allowing Onsemi to occupy most of the automotive CIS market share.

• Resolution and HDR trade-off: In the early days, ON Semiconductor (Aptina at the time) had introduced HDR similar to alternating rows, such as the DR-Pix architecture, which could switch between two sets of pixels, high and low light. But its mainstream products prefer full-resolution progressive interlaced HDR. For example, the new Onsemi XGS series global shutter sensor achieves smear-free HDR output by sampling and storing multiple times within a pixel. Onsemi also provides an HDR mode that can adjust the frame interval by itself (called interleaved frame HDR), allowing customers to find the optimal balance between dynamic range and resolution/frame rate.

• Double conversion gain: ON Semiconductor also masters DCG technology. For example, its security sensor AS0140 has used pixel dual gain mode to improve low-light performance and dynamic range. Although Onsemi does not promote HDR selling points in the mobile phone market like Sony/Samsung, in the professional field, many of its sensor data sheets are marked as supporting HDR mode, which usually refers to time-sharing multiple exposures combined with corresponding algorithms.

Generally speaking, Onsemi's HDR technology emphasizes reliability and practicality, and is mostly used in automotive ADAS cameras, security monitoring, etc. As cars become smarter, the real-time requirements for HDR videos are higher. Onsemi is also developing next-generation sensor HDR solutions, including pixel-level simultaneous double sampling, high dynamic and infrared fusion, etc., to maintain its leading position in the field of automotive imaging. However, in consumer markets such as mobile phones, Onsemi’s presence is weak, so HDR technology is mainly reflected in professional-grade product lines.

SmartSens (SmartSens)

As a rising CIS manufacturer in China, SmartSign has also made unique innovations in HDR technology, gradually narrowing the gap with international giants:

• PixGain®/SuperPixGain HDR: This is a single-frame HDR technology independently developed by Smartway. The so-called PixGain means that the pixel has both high and low conversion gains (High/Low Conversion Gain), similar to the aforementioned double conversion gain. But SmartSite goes a step further and launches SuperPixGain HDR™ in 2023, which can generate three frames of equivalent images in a single exposure and fuse them to achieve ultra-high dynamic range. The main camera sensor of the SC5A5XS mobile phone equipped with this technology has a dynamic range of up to 110dB and can effectively suppress motion artifacts.

The principle of SuperPixGain HDR is speculated to be to read the pixel signal three times within the sensor (or three groups of pixels with different gains), which is somewhat similar to a comprehensive solution with both Staggered and multi-gain. In terms of actual results, SmartSite claims that it can present HDR images without smearing under 4K 60fps video, reaching a new height of hardware HDR on mobile platforms.

• Line overlap HDR: SmartSite’s sensor product line also has a traditional interleaved exposure solution. For example, the SC630 series of IoT cameras supports 2-exposure staggered HDR line interleaving mode, with a dynamic range of more than 87dB. SmartSite uses progressive HDR similar to Sony DOL in security sensors, and combines it with its own QCell™ LED flicker suppression to simultaneously prevent LED flicker and improve dynamic range. It can be seen that SmartSite has followed the mature line-interleaved HDR idea of ​​major international manufacturers to meet the needs of security vehicles.

• Hierarchical HDR architecture: SmartSite proposed a complete HDR technology portfolio in its technical promotion, including: "line-by-line multiple exposure HDR, line-overlapping HDR, single-frame HDR without time interval" and many other technologies. Line-by-line multiple exposure corresponds to regular multi-frame HDR, line overlap corresponds to Staggered, and single-frame HDR refers to SuperPixGain. This shows that SmartSite has built an HDR solution stack covering software to hardware, and can provide different levels of HDR support according to application scenarios. In particular, its single-frame HDR technology gives domestic sensors a selling point to compete with Sony and Samsung in the high-end mobile phone market.

• Application and cooperation: SmartSite actively cooperates with mobile phone manufacturers to integrate HDR technology into terminals. For example, the main camera of a domestic flagship mobile phone uses a SmartSwei 50-megapixel sensor to achieve pixel-level HDR video shooting. Its night scene and backlight video effects are said to be comparable to international brands. Smartway also promotes its own HDR sensors in the fields of automobiles, machine vision and other fields, emphasizing the ability to obtain "clear light and dark" under complex light. As a latecomer, SmartSite is providing HDR solutions with a higher cost performance and is gradually gaining market recognition.

Overall, SmartSens's HDR strategy is "two-pronged": on the one hand, it imitates and catches up with the mainstream level of the industry (such as row-interleaved HDR, etc.); on the other hand, it achieves differentiated breakthroughs through innovation (such as single-frame three-fusion HDR). As its technology matures and its customer base expands, we have reason to expect that more flagship domestic imaging products will use SmartSite's HDR technology in the future.

Comparative analysis of ISP HDR and Sensor HDR

ISP-side HDR and Sensor-side HDR are two different implementation paths, each with its own advantages and disadvantages, and is suitable for different application scenarios. The following is a comparative analysis of the two from multiple dimensions:

• Dynamic range and image quality: Traditional ISP multi-frame HDR can continuously improve the dynamic range by increasing the number of frames, but it is limited by alignment accuracy and noise accumulation, and there is a bottleneck in actual improvement. Sensor hardware HDR often uses a lower-level signal acquisition method to avoid many post-processing losses and can retain more original details and higher bit depth in a single frame (such as Samsung Smart-ISO Pro output 12-bit RAW). In addition, because Sensor HDR does not have severe alignment stretching, the edge transition of the picture is more natural and the color consistency is better. Generally speaking, at the limit of dynamic range, the combination of multi-frame + hardware HDR has the best effect, but at the same number of frames, Sensor HDR can usually achieve better HDR imaging quality.

• Motion artifacts: This is an important consideration in HDR imaging. Due to the time difference between multiple frames, ISP HDR will cause ghosting of moving objects, which requires algorithm detection and elimination. Although the introduction of AI assistance can alleviate Ghost to a certain extent, algorithm complexity and risk of failure still exist. The Sensor HDR method (such as single-frame multi-gain, interleaved exposure) has no motion artifacts or ghosting problems because each exposure is almost simultaneous and the spatial position is consistent. Even for sensor multi-frame Staggered HDR, since the frame interval is microseconds within one frame, it is far smaller than the time difference of traditional multi-frame HDR, and motion consistency is significantly improved. Therefore, in sports scenes and video shooting, the advantages of the Sensor HDR solution are very obvious.

• Shutter response and delay: Multi-frame ISP HDR often requires the accumulation of multiple images before processing and output. Users may perceive obvious shutter delay and preview desynchronization. Since Sensor HDR outputs real-time HDR data, "what you see is what you get" can be achieved with ISP. For example, when using a single-frame HDR sensor, the preview image itself is an HDR effect. Imaging is almost instantaneous after the shutter is pressed, greatly improving the user experience. Even for the sensor's three-frame interleaved HDR, its output delay is only on the order of tens of milliseconds, which is far lower than the hundreds of milliseconds or even higher than software HDR synthesis. Therefore, Sensor HDR has more advantages in applications that require high real-time performance (such as car photography or mobile phone capture).

• Implementation complexity and cost: ISP HDR mainly consumes algorithm research and development and chip computing power. It does not require special sensors. Therefore, the cost is relatively low and the flexibility is high. Different algorithm upgrades can be implemented on general hardware. Sensor HDR requires more complex pixel design, readout circuitry and process support. For example, checkerboard pixels, dual-gain ADCs, pixel storage nodes, etc., all increase the difficulty and unit cost of sensor development. Therefore, for entry-level products or old devices, using ISP HDR software upgrade is a more economical solution, while high-end flagships often introduce sensor HDR at the expense of hardware to obtain differentiated performance.

• Power consumption and efficiency: Multi-frame HDR requires multiple exposures and multi-frame processing, and the overall energy consumption is high. Some Sensor HDR solutions can complete multiple sampling within the same frame period, reusing the readout process and consuming better power. For example, Sony has calculated that Staggered HDR reduces energy consumption by about 15–25% compared to continuous multi-frame exposure. Single-frame HDR also reduces the number of shots and storage I/O. In addition, ISP hardware can be optimized for Sensor HDR, making the fusion process simple and efficient, further reducing the power consumption of the entire machine. For battery-powered devices, reducing the power consumption of HDR shooting is very important to improve user experience, and Sensor HDR is even better in this regard.

• Applicability and flexibility: ISP HDR has great flexibility in algorithm, and the number of frames, exposure ratio, and fusion strategy can be adjusted according to the scene. For example, night scene mode can stack more frames for noise reduction, while backlight scenes require only two frames for fast HDR. Sensor HDR is relatively fixed. For example, the sensor may only support 3 frames of Staggered or single frame dual gain, and may not be as flexible as the algorithm in extreme scenarios. In addition, Sensor HDR is usually optimized for specific scenarios during design (for example, automotive HDR emphasizes that LEDs do not flicker) and is slightly less versatile. However, as the Sensor side and the ISP side gradually collaborate (the hardware outputs multi-channel HDR, and the ISP flexibly chooses the fusion method), this boundary becomes blurred. In the future, we will see multi-mode HDR - a variety of HDR data provided by the Sensor, combined on demand by ISP/AI, thus taking into account flexibility and real-time performance.

Table: Comparison of ISP HDR vs Sensor HDR solutions

Contrast Dimensions

ISP side multi-frame HDR

Sensor side hardware HDR

Dynamic range improved

Depends on the number of algorithm frames and can theoretically be expanded infinitely, but is limited by noise and alignment

Depends on sensor hardware capabilities, has physical upper limit, but high signal quality

ghost artifacts

Misalignment of multiple frames is prone to ghosting, which needs to be removed later.

No time difference or very small time difference, basically no ghosting

imaging delay

Shutter lag is obvious and multi-frame synthesis is time-consuming

Real-time HDR output, almost zero-lag preview

Hardware cost

No special requirements, can be achieved by upgrading the algorithm

Requires special sensors, high pixel/process complexity, and high cost

tr>

Power efficiency

Multi-frame acquisition + high power consumption for processing

Single frame completion, multiplexing, lower power consumption

Applicable scenarios

The software and hardware are highly versatile and can implement HDR on low-end devices.

High-end professional applications, pursuing ultimate HDR performance

Flexibility

The algorithm is adjustable and has good scene adaptability.

The hardware mode is fixed and relies on the preset HDR mode.

technology trends

Tend to integrate AI post-processing to improve quality

Tends to cooperate with ISP to provide multi-mode raw HDR data

It is worth noting that today's high-end imaging systems often combine two solutions: the sensor provides basic HDR hardware support, and the ISP superimposes multiple frames or AI optimization to form a "1+1>2" effect. For example, when a flagship phone turns on HDR, it uses both the sensor Staggered output and the ISP's ZSL multi-frame synthesis to obtain higher dynamic range and more robust imaging. It can be seen that ISP HDR and Sensor HDR are not antagonistic, but a cooperative relationship: the sensor hardware provides better raw materials, and the ISP algorithm adds icing on the cake, ultimately achieving the best HDR imaging performance together.

Development trends and prospects

Looking to the future development of HDR technology, the following trends deserve attention:

Sensor-ISP-AI integrated HDR: As mentioned above, HDR processing is evolving into a collaborative system engineering across Sensor, ISP, and AI. Future cameras will dynamically select HDR modes based on the scene: a still landscape may trigger multi-frame synthesis, while a moving scene will enable single-frame HDR on the sensor, with AI correcting details on the back end. This multi-mode parallel architecture will enable HDR to achieve optimized results under various shooting conditions, without users having to perceive internal process switching.

Higher bit depth and color fidelity: As the image signal path supports 14-bit or even 16-bit, HDR solutions from various manufacturers are increasing the output bit depth. For example, Samsung Smart-ISO Pro has reached 12-bit, and the next step will be to challenge 14-bit HDR output. Higher bit depth means that the brightness levels and color combinations that can be expressed increase exponentially, making HDR images closer to real scenes. In addition, color science will also be integrated into the HDR process to ensure accurate color and natural transitions under wide dynamic range. The HDR formats (HDR10+, Dolby Vision) promoted by companies such as Dolby also emphasize metadata tone mapping to make full use of high dynamic range and wide color gamut display capabilities, which in turn will promote the shooting end to provide richer HDR information.

Fusion of computational photography and HDR: In the future, HDR will no longer be an independent feature, but will be deeply embedded in the computational photography framework. For example, the night scene and portrait mode of mobile phones will automatically call the HDR sub-process; a multi-camera system may implement HDR between different cameras (one is responsible for high exposure, one is responsible for low exposure, and then merged). In the video field, frame-by-frame HDR and frame-by-frame denoising, anti-shake and other algorithms will also be integrated into unified multi-sensor, multi-algorithm collaborative processing. It is foreseeable that HDR will be intertwined with other technologies in the torrent of computational photography to jointly improve imaging quality.

Smarter scene adaptation: AI will play a greater role in determining when and how to apply HDR. For example, the machine learning model analyzes the dynamic range of the viewfinder and decides which HDR mode of the sensor to use or whether to superimpose multiple frames with one click. AI can also learn the human eye's subjective perception of high-contrast scenes and adjust the HDR results (such as local contrast optimization, simulating human eye adaptation). This smart HDR, which introduces the characteristics of human vision, is expected to produce more eye-catching and realistic pictures. Manufacturers such as SK Hynix even envision using neural networks to directly model human eye perception within sensor chips to optimize HDR output in real time.

HDR and new pixel architecture: In pursuit of higher HDR performance, new sensor architectures are emerging one after another. For example, stacked sensors provide more circuit space for HDR, and back-illuminated/double-layer transistors increase full well capacity; technologies such as SPAD (single photon avalanche diode) are combined with event HDR; quantum dot filters increase SNR and thereby improve HDR noise floor, etc. These new technologies will be gradually applied to HDR imaging, so that future sensors will have natural wide dynamic properties and no longer require complex acquired processing.

Generally speaking, HDR imaging is developing in the direction of "more real-time, smarter, and closer to the human eye". We predict that in the near future, HDR will become a standard feature of all cameras and become ubiquitous. Users do not even need to switch modes, and the camera can automatically capture HDR images based on the scene. At the same time, the effect of HDR will be further improved. The ultimate goal is to make the dynamic range and look and feel of digital imaging close to or even surpass the human eye, so that what you see is what you get in photos and videos.

Summary and conclusion

After years of development, HDR technology has entered a new era of combining software and hardware from the initial software synthesis stage. This white paper systematically sorts out the two major ways to achieve HDR: the ISP-side multi-frame fusion solution relies on powerful computational photography algorithms to realize the early popularization of HDR functions; the sensor-side hardware fusion solution obtains higher dynamic range data from the source through innovative sensor design, greatly improving the timeliness and effect of HDR imaging. Major mainstream sensor manufacturers have launched a variety of technology schools around HDR - from Sony's interleaved HDR and SME checkerboard pixels, to Samsung's Smart-ISO Pro single-frame HDR, to the multi-gain fusion of OmniVision, SmartSign, etc. - the blooming technologies have jointly promoted the continuous improvement of HDR imaging capabilities.

Through comparison, we can see that ISP HDR and Sensor HDR each have their own advantages and disadvantages, and they do not replace each other but complement each other. The former has high flexibility and low cost, while the latter has good real-time performance and good quality. Today's high-end imaging systems often combine the two in order to achieve better dynamic range, lower ghosting and latency. Looking to the future, with the integrated collaboration of sensors, processors and AI algorithms, HDR shooting will be more intelligent and efficient, eventually breaking through the limits of human vision. In this process, manufacturers that master core HDR technology will lead imaging innovation, bringing unprecedented freedom and possibilities to photography and video creation.

The ultimate vision of HDR is to allow the images we capture to be unafraid of any light ratio challenge. From dazzling sunlight to dark shadows, every detail is clearly visible and the colors are as real as they come. This vision is gradually becoming a reality. It is foreseeable that HDR technology will continue to evolve and become an indispensable cornerstone of digital imaging, bringing greater breakthroughs for humans to record and share the visual world.