Exploiting the point spread function for optical imaging through a scattering medium based on deconvolution method

1. Introduction

Propagation of light field through turbid medium, like fog, smoke, biological tissues, suffers from scattering.^1,2 Scattering can severely disturb the imaging process, leaving only a messy speckle. This speckle in the imaging system was previously considered as noise without information. Hence, imaging through a scattering medium was always a big challenge for visible light for a long time. With the development of technology, people got to find methods to overcome this problem in the recent decade.³ The most widespread reported techniques are wavefront shaping,^4,5,6,7 time-of-flight imaging,^8,9 time reversal or phase conjugation,^{10,11,12,13,14,15,16} transmission matrix measurement,^17,18,19,20 holography,^21,22,23 speckle correlation,^{24,25,26,27,28,29,30,31,32,33,34,35} and so on. Among them, speckle correlation is thought to be a promising solution because it usually does not need coherent light sources, expensive wavefront modulators, complicated interference setups, nor a time-consuming calibration. The key to realizing imaging reconstruction with speckle correlation is the so-called “Optical Memory Effect” (OME),^36,37 which reveals the high correlation between two-speckle patterns generated from two small angle tilting incident lights. Within the OME range, the scattering medium can be considered as a shift-invariant system, which is similar to a lens system. In this case, the speckle can be described as a convolution of the original object and system’s Point Spread Function (PSF).³⁸ To realize image reconstruction from the speckle of an object, auto-correlation or PSF deconvolution methods are the most frequently used speckle correlation methods. Although the auto-correlation method can realize single-shot, noninvasive imaging reconstruction through a scattering medium, it usually suffers from a large computation load phase retrieval calculation and a complicated calibration calculation. The deconvolution approach,^{34,39,40,41,42,43,44,45,46} instead, can reconstruct the image of the object from its speckle and the PSF of the scattering system⁴⁷ in real-time. By harnessing the spatial, spectral properties of the PSF of a scattering system, large Field-of-View (FOV) beyond the limit of the OME,^34,45,46 extended Depth-of-Field (DOF) for 3D imaging,³⁹ color image reconstruction,³⁴ or spectral-resolved imaging recovery^40,41 can be realized by a simple deconvolution between the speckle and the PSF. If a priori knowledge of a reference object,⁴⁴ spatial-correlation of the speckles,⁴⁵ or estimation of the speckle pattern⁴² can be attained, noninvasive imaging recovery through a scattering medium can be realized. For its easy-to-use and robust qualities, the speckle correlation scattering imaging technique based on deconvolution is undergoing a rapid development. It is thought to be a promising technique in imaging through a scattering medium and may find useful application in the field of security monitoring, biomedical imaging, and other circumstances encounter scattering in thin turbid medium.

In this paper, we review recent progress in the imaging reconstruction technique through scattering medium with deconvolution-based speckle correlation method. In Sec. 2, the principle of this method is briefly introduced. Section 3 will discuss the techniques on how to exploit the PSF to achieve high quality imaging through scattering medium. Section 4 summarizes the paper.

2. Brief Introduction to the Imaging Principle Through Scattering Medium with Deconvolution Method

According to the OME,^36,37 the speckle pattern from a thin scattering medium follows the trend of the incident light when it tilts a small angle. For example, when pinhole $P_1$ is illuminated, speckle pattern $I_1$ is detected on the imaging plane (Fig. 1). If a pinhole $P_2$ at the adjacent of $P_1$ is illuminated solely, speckle pattern $I_2$ is detected. $I_2$ is almost the same appearance as $I_1$, but just shifts literally according to the tilting angle $\theta$ between $P_1$ and $P_2$ related to the center of the scattering medium. Furthermore, if $P_1$ and $P_2$ are illuminated by an incoherent light simultaneously, the speckle pattern on the image plane will be simply the superposition of $I_1$ and $I_2$ (that is $I=I_1+I_2$). So a thin scattering medium turns out to be a linear shift-invariant system within the OME range when it is illuminated by incoherent light.³⁴ The transmission property of the thin scattering system can be described by a PSF. Suppose the tested object is considered as an assembling of point sources. Thus, the speckle pattern on the imaging plane is a superposition of all PSFs from each point source of the object. The imaging process can be denoted as a convolution function,

where subscripts $i$ and $o$ denote the image plane and the object plane, respectively.^24,48$F$ ($x_i$, $y_i$) is a form factor function defined as an angular correlation of the emerging speckle patterns of two incident wavefronts.^36,37 The $F$ function has a bell shape curve of angular correlation and is more often referred to OME range which acts as a field stop placed in the focal plane to limit the observed FOV. The convolution equation can be rewritten in the form of $I=O\ast \mathrm{\text{PSF}}$, where * denotes the convolution operator. So if the PSF of a thin scattering medium is detected or estimated, the image of the original object can be reconstructed by a deconvolution of its speckle pattern $I$ and PSF of the system.

3. Exploit the PSF of a Scattering System to Achieve High Quality Scattering Imaging Applications

The precondition of the successful image recovery with deconvolution-based speckle correlation method is the OME. Only tested objects which are located within the OME range can be detected and reconstructed clearly. The OME is mainly restricted by the effect thickness $L$ of the scattering medium, which leads to a limited FOV. The form factor $F$ in Eq. (1) indicates the transverse FOV. It is commonly defined as $\mathrm{\text{FOV}}\sim \lambda d_o/\pi L$, where $\lambda$ is the wavelength of the illumination source, $d_o$ is the distance between the object plane and the scattering medium (Fig. 1).³⁵ DOF of the scattering lens system can be deduced according to its FOV and it turns out to be $\mathrm{\text{DOF}}\sim \mathrm{\text{FOV}}\cdot d_o/D$, where $D$ is the diameter of the illuminated spot on the scattering medium.^35,39 The FOV and DOF of the scattering imaging system are inversely proportional to the effective thickness of the scattering medium. The effect thickness of a common thin film diffuser is around several micro-meters (for example: Newport 5^∘ circular light shaping diffuser). The FOV and DOF of the thin scattering system should be around tens of milliradians and several millimeters (under the condition of $d_o=15$cm, $\lambda =550$nm, $D=5$mm).^34,38 So in order to resolve a macroscopic object in a scattering light system with deconvolution method, the first challenge is to break the limitation of the OME and enlarge system’s FOV and DOF. Secondly, the deconvolution method requires to know the PSF of a thin scattering medium. While for some applications, access to the object space is not possible. Thus, noninvasive acquisition of the PSF will be another challenge. Thirdly, the most commonly used deconvolution algorithm is Wiener Deconvolution Filtering. However, the Signal Noise Ratio (SNR) of a scattering process is usually hard to estimate correctly. A proper modeling of the scattering imaging process and estimation of the SNR of it are needed to improve the quality of Wiener Deconvolution. Last but not the least, the spectrum property of a tested object also contains significant information. Functions like spectral-resolved imaging should be realized with a deconvolution-based scattering imaging method.

To solve all the challenges and demands mentioned above, researchers have developed many techniques to exploit the PSF properties of the thin scattering system and made great progress in this field. Achievements like extended FOV and DOF, noninvasive acquisition of the PSF, OME-based filter to improve the deconvolution quality,⁴³ spectral-resolved reconstruction are introduced in what follows.

3.1. Manipulation the PSF for FOV extension and DOF extension

Usually, the most simplified setup of a scattering lens (shown in Figs. 1 and 2(a)) cannot make the best use of the correlated scattering light. For one thing, CCD can only capture part of the correlated speckles with limited sensor area (red rectangle shown in Fig. 2(a)) which is smaller than the OME range viewed from the image space (black circle shown in Fig. 2(a)) and exit pupil of the system (white dash circle shown in Fig. 2(a)). For another, the light diffracted from object points whose distance are beyond the OME range might lead to a less overlapped area on the scattering medium. It results in less correlated scattering light. So the reported angular FOV by the simplified setup was much smaller than the OME range. To solve this problem, a PSF manipulation strategy is taken by Zhuang et al.³⁴ By inserting an additional lens into the optical path, most of the scattering light is collected for imaging and the image in the entrance pupil is demagnified to fit the CCD size (Fig. 2(b)). So the correlation of the speckle pattern within the exit pupil is increased and the FOV is enlarged to approach ME range (Fig. 2(f)).

For a target $O$ whose spatial range is beyond the FOV of a thin scattering system, the object can be divided into $N$ parts according to different viewing zones of the scattering system $O={\sum }_i{O}_i$. The speckle pattern $I$ of the object can be considered as a composite response of the scattering light coming from the various regions of the object. Thus, $I$ can be expressed as $I={\sum }_i{I}_i={\sum }_i(O_i\ast {\mathrm{\text{PSF}}}_i)$. Theoretically, the speckle patterns produced by two point sources beyond the OME regions are considered to be completely uncorrelated PSF${}_i\ast {\mathrm{\text{PSF}}}_j={\delta }_{ij}$, where*denotes a correlation operator. So the reconstructed images of viewing zones $O_i$ can be attained by a deconvolution of $I$ and the corresponding PSF_i. Finally, all the recovery images of different viewing zones are synthesized to completely reconstruct the whole scene. According to this principle, Tang et al. and Li et al. recover a large scene through a scattering medium exceeding memory effect (Fig. 3)^45,46 by detecting or measuring PSFs from different areas beforehand.

Besides the literal information, the axial information is also a very important spatial property of an object. To overcome the limited DOF due to small OME range, one can detect a series of PSFs in different axial positions and demultiplex the axial information. However, determination of PSFs for different objective planes complicates the system hence recovery of 3D image appears to have low efficiency and is less practical. Xie et al. study the PSFs corresponding to different axial locations and found that they are highly correlated.³⁹ So by adjusting the scaling factor of one PSF, other PSFs of different object planes can be deduced. The scaling factor is derived to be $m=f/f^{\prime }$, where $f$ and $f^{\prime }$ are the focal length of the “scattering lens” with object distances of $d_o$ and $d_o^{\prime }$, respectively. Experimental results show an approximate 5 times of improvement in depth resolved ability compared to the original method without PSF manipulation (Figs. 4(a)–4(b)). Three-layer objects located separately beyond the original DOF are reconstructed through a thin scattering medium slide by slide with the deconvolution technique (Fig. 4(c)). If a reference object place is known beforehand, the PSF scaling technique can be even made use of detecting the distance in axial direction through a thin scattering medium.

3.2. Acquisition, calculation and estimation of PSF for a thin scattering system

Direct acquisition of the PSF of a thin scattering system can be done by placing a pinhole on the center of the object plane and measuring its scattering speckle pattern. However, for some special applications, the object space cannot be accessed directly. The PSF should be calculated or estimated before doing a deconvolution reconstruction. Xu et al. proposed a novel method to reconstruct a tested object through a thin scattering medium with the help of a known reference object.⁴⁴ As it is shown in Fig. 5, the model is to recover the target “T” from the scattering pattern $I_{\mathrm{\text{sum}}}$ of the whole scene (letter mask of “TH”) when the distribution $O_R$ of the reference object “H” and its speckle $I_R$ are known in advance. For a fixed scattering system, the PSF can be retrieved according to Eq. (1) by doing deconvolution between $I_R$ and $O_R$. The deduced PSF can be used for a deconvolution process mentioned in Sec. 2. However, the error will be accumulated during these two processes. Actually, these processes can be simplified into one by $O={{ℱ}}^{-1}\{[{ℱ}\{O_R\}\times {ℱ}\{I_{\mathrm{\text{sum}}}\}]/{ℱ}\{I_R\}\}$, where symbols ${ℱ}$ and ${{ℱ}}^{-1}$ denote the Fourier transform and Inverse Fourier transform. This method is valuable in real-life applications. The knowledge of a certain shaped object under a scattering medium can be achieved. For example, the organ detected by X-ray technique, ultrasonic imaging⁴⁹ or other techniques below the skin can act as Refs. 50 and 51. Meanwhile, in many medical applications, the surroundings of a tested object also play an important role in diagnosis and treatment.⁵² So the development of a scenario with a reference object is potentially possible and necessary.

If a prior knowledge of a reference object is hard to attain, people can also calibrate the thin scattering medium before a deconvolution reconstruction. Compared to a complex process with scanning or interferometry detection to measure the transmission matrix of a scattering system, the deconvolution scattering imaging method only needs to get the response of one object point after it passes through a scattering system. It is convenient to calibrate the PSF before measuring. For example, Li et al. proposed a spatial-correlation architecture shown in Fig. 6 to obtain intensity transmission matrices of diffusers and provided flexible access to PSFs.⁴⁵ The key to success measuring the PSFs is to detect the light field intensity on the conjugate plane CCD_r and make use of the correlation of the transmission speckle and the incident light. With this technique, the PSF corresponding to any position on the object plane can be attained. And a wide FOV reconstruction can be attained by spatially demultiplexing (Figs. 3(c)–3(h)).

Furthermore, if neither a priori knowledge of a reference object can be attained, nor can the scattering system be invasively calibrated, the PSF of a thin scattering system can be estimated. The related technique can be referred to in the paper reported by Wang et al.⁴² In their speckle pattern estimation process, a series of targets are utilized as the training input. They first directly reconstruct the images of the training targets by speckle correlation and phase retrieval. And then, a speckle modeling and constrained least square optimization are applied to estimate the distribution of the speckle pattern. Finally, the image of the to-be-observed target is reconstructed by deconvolution of the estimated speckle pattern from the acquired integrated intensity matrices.

3.3. Improvement on deconvolution algorithm with an OME-based filter

The common deconvolution algorithm of speckle correlation scattering light imaging technique is Wiener deconvolution method. It has widespread used in image deconvolution applications. Its form⁵³ is,

where ${ℱ}\{O\}$, ${ℱ}\{I\}$ and ${ℱ}\{\mathrm{\text{PSF}}\}$ are the Fourier transforms of $O$, $I$ and PSF, respectively, SNR is the signal-to-noise ratio. The SNR is usually set to be a constant⁴³ in scattering imaging application because the present noise models are inadequate to depict the imaging processing through diffusers. Also, the value of SNR is hard to predict and relies on experience. Recently, Chen et al. found that the speckle pattern is not noise but signal correlated to its tilted angle in ME range.⁴³ They built up a model to study the process of scattering light transmission and take into account the influence of OME. An OME-based filter deconvolution operation is derived, where $S_n/S_f=|1-{ℱ}\{F\}{|}^2/|{ℱ}\{F\}{|}^2$, and ${ℱ}\{F\}$ is the Fourier transforms of the form factor $F$ in Eq. (1) which relates to the OME. Experimental results show the proposed method of the ME-based filter has better consistency in restoring the center and edge of objects, and the effect of background suppression is notable (Fig. 7). The proposed OME-based filter is believed to be more close to the real scattering situation by considering the noise model of the scattering process when doing deconvolution imaging calculation.

3.4. Spectral-resolved imaging through a scattering medium with deconvolution technique

Besides the spatial information, the spectrum property of a tested object also contains significant information. Light scattering is thought to be a high dispersion process. So the correlation between speckle patterns from different wavelengths is usually of a narrow bandwidth. To achieve multispectral imaging or color reconstruction through a scattering medium with PSF deconvolution method, one can individually acquire PSF with different wavelengths and reconstruct the original object with the corresponding wavelength. For example, Zhuang et al. use the built-in color filter to separate the light from a point source and a tested object into RGB channels.³⁴ The reconstruction results of the three primary-color channels combine to recover the color image of the tested object.

Furthermore, the spectrum-dependent behavior of scattering medium spectral can be descripted as the decorrelation effect of the speckle pattern and expressed as PSF${}_{\lambda 1}\ast {\mathrm{\text{PSF}}}_{\lambda 2}={\delta }_{\lambda 1\lambda 2}$. So Sahoo et al. proposed a single-shot multispectral imaging with a monochromatic camera.⁴¹ The schematics of this technique can be illustrated in Fig. 8. Serial multispectral PSFs are acquired by a pinhole illuminated with different spectral bands. And images with different spectral bands can be reconstructed by the correlation of the monochromatic speckle and the corresponding spectral PSFs. The full-spectrum image can be attained by a synthesis of all the spectral images. Thus, a cost-effective approach for multispectral imaging is achieved.

Usually, the PSF of a scattering system is spectrum-dependent. While Xu et al. did a fine study on it and found that PSFs according to different spectral bands are still correlated only if a proper spatial scaling is applied to them.⁴⁰ The correlation of the original PSF and the rescaled PSF can be calculated by $C({\lambda }_1/{\lambda }_2,m)=\int dxdy{\mathrm{\text{PSF}}}_{{\lambda }_1}(mx,my)\text{-}{\mathrm{\text{PSF}}}_{{\lambda }_2}(x,y)$, where $m$ is a scaling factor. A proper scaling factor $m$ can lead to a maximum of the correlation. So, the PSF of ${\lambda }_2$ can be approximately deduced by spatially rescaling the PSF of ${\lambda }_1$. As it is shown in Fig. 9, if a reference object is illuminated by a yellow LED, the tested object is illuminated by a red LED, the PSF of the yellow wavelength can be calculated or measured. The PSF of the red wavelength can be estimated by spatially rescaling the PSF of the yellow one. The reconstructed image of the tested object is much clear by rescaling the PSF than without rescaling. Experimental and theoretical results show that for a thin scattering medium the wavelength-shifted PSF correlates strongly with a coordinate-scaled PSF. The use of a spectrally separated reference object allows for single-shot imaging where the reference and object are recorded in a single exposure.

4. Conclusion

This paper summarized the latest progress of deconvolution-based speckle correlation method on imaging through a thin scattering medium. The OME of a scattering medium and incoherent illumination enable the scattering system to be considered as a linear shift-invariant system. So if the PSF can be attained, a simple deconvolution is enough to reconstruct image of an object through the scattering system. As for its easy-to-use and flexible qualities, many techniques can be joined and integrated into it and push it to gain further development. By tailoring or scaling the PSF or spatially demultiplexing the multi-PSFs, a large FOV and large DOF reconstruction can be achieved. By a reference object assisted, spatial correlation or a series target training, PSF of the scattering system can be acquired, calculated, or estimated for noninvasive detection application. By exploiting the spectral property of the scattering system, multiple-spectrum reconstruction can be realized. New physical phenomena such as axial PSF scaling effect, scaling effect of PSF with different wavelength, and so on are discovered during the study of the scattering imaging technique based on deconvolution method.

Although studies and applications on the technique of deconvolution scattering imaging method have made a lot of progress, there are still some problems that need to be solved before its wide application. For example, as the thickness of the scattering medium gets larger, the OME range will decrease dramatically which might lead to a failure of the deconvolution method. A solution for the deconvolution method when the thickness of a scattering medium is larger than tens of mean free path is still a challenge. Secondly, how to speed up the current process to adapt for the dynamic scattering environment is still a problem. Last but not the least, although spatial and spectral properties are exploited for imaging, other parameters of the light such as temporal, polarization are rarely mentioned. To solve these problems, it requires a deep study on the scattering physics, development of technique and theory, and multidisciplinary research.

Conflict of Interest

The authors declare no competing interests.

Acknowledgments

This work is supported by National Natural Science Foundation of China (Nos. 61705035, 61575223, 11534017 and 61475038), the Project of Department of Education of Guangdong Province (No. 2018KTSCX241), State Key Laboratory of Optoelectronic Materials and Technologies (Sun Yat-sen University), and STU Scientific Research Foundation for Talents.