Image Denoising in ISP

Noise is an unavoidable “destructive factor” during sensor image capture. Without proper processing, it degrades image quality and impairs the accuracy of visual‑perception tasks. Therefore, the image‑denoising module inside ISP plays a decisive role in final image output quality.

Sources of Noise

A sensor receives photons as input and outputs digital signals through photoelectric conversion, electron accumulation and voltage quantization. Multiple factors introduce noise throughout these signal‑conversion stages. For instance, the quantum nature of photons brings Poisson noise during photoelectric conversion; thermal noise from CMOS circuitry occurs during voltage readout; ADC quantization noise arises in voltage quantization. These noises are inherent to current CMOS sensors and must be suppressed by dedicated denoising modules.

2DNR

2DNR stands for Spatial‑Domain Noise Reduction, which does not utilize temporal information. Conventional 2DNR covers spatial‑domain and transform‑domain algorithms. It can be implemented at RAW, RGB or YUV stages within the ISP pipeline. Algorithm and color‑space selection depend on actual noise distribution and image‑quality requirements. Widely‑used 2DNR algorithms in ISP chips are listed below:

NLM (Non‑Local Means)

The Non‑Local Means algorithm pioneers denoising by exploiting non‑local self‑similarity within images. Its core idea is noise suppression via similar image patches. For each input pixel, similarity calculation is performed inside a given search window to generate pixel weights for filtering. Achieving a good balance between performance and hardware area overhead, NLM and its variants are adopted by many chip vendors.

Wavelet Thresholding

Wavelet thresholding belongs to transform‑domain methods. With properly selected wavelet bases and decomposition levels, wavelet transform isolates different frequency components of an image so noise can be processed frequency‑specifically. Since noise mostly resides in high‑frequency bands, wavelet thresholding mainly applies threshold operations to high‑frequency components.

BM3D

BM3D combines strengths of spatial‑domain and transform‑domain approaches. It consists of preliminary estimation and final estimation stages, incorporating similar‑patch matching, 3D frequency‑domain transform, threshold processing and Wiener filtering before aggregating the final denoised output. BM3D delivers superior performance among conventional algorithms. However, high algorithm complexity and heavy hardware‑implementation cost limit its practical deployment in real ISP chips.

3DNR

3DNR adds temporal information upon 2DNR, processing data from both current and historical frames. Benefiting from extra temporal information, 3DNR achieves better denoising results than 2DNR. Its advantage is especially prominent in low‑light scenarios where 2DNR performs poorly.

A simplified 3DNR example is given to illustrate its principle. 3DNR relies on motion estimation: simple implementations use frame difference to detect motion regions; more advanced versions adopt optical‑flow information for further optimization. The exemplified algorithm also integrates noise estimation by partitioning the image: higher weight is assigned to the current frame for low‑noise and motion‑active regions; higher weight is assigned to historical frames for high‑noise static regions.

Since 3DNR interacts with DDR memory, high‑resolution cases consume substantial DDR bandwidth. Image compression is therefore often applied before storing frames into DDR. Despite its advantages, 3DNR may introduce artifacts such as motion noise, ghosting and motion blur. Engineers need to trade‑off denoising performance against hardware cost to tune algorithms for target requirements.

Conclusion

This article introduces ISP image‑denoising modules, focusing on classic 2DNR and 3DNR algorithms. Due to signal‑dependent real‑world noise and non‑linearities introduced by ISP hardware, multiple denoising modules working across different color spaces are usually required for optimal output. Furthermore, AI‑based denoising outperforms traditional algorithms, so ISP chip designs increasingly adopt AI‑powered NR modules. AI‑driven image denoising will be covered in future articles.