SOFFLFM: Super-resolution optical fluctuation Fourier light-field microscopy

1. Introduction

In order to understand the basic principles of biological systems, it is essential to observe the three-dimensional (3D) structure of intracellular organelles with high spatial and temporal resolution. In traditional fluorescence microscopies, 3D information of samples is always collected in a sequential or scanning way. This process inevitably increases the photo damage to living cells and reduces the temporal resolution. Light-field microscopy (LFM) technology can simultaneously record two-dimensional spatial and angle information of light, allowing the 3D structure of a sample to be reconstructed from a single shot of the sample without any scanning.¹ The high scalability and high temporal resolution of LFM make it useful in both functional brain imaging and single-cell imaging.^2,3,4 However, the inevitable reconstruction artifacts and huge computational costs limit the application of LFM.⁵

Recently, Fourier light-field microscopy (FLFM) has been developed to obtain four-dimensional light field information in the Fourier domain, so that the point spread function (PSF) of the system can be described by a unified 3D PSF.⁶ This method effectively removes the limitation of LFM because of its reconstruction artifacts and computational cost, and furthermore, the image quality is improved compared with the traditional LFM, which makes it more applicable in some fields such as endoscope imaging.⁷

However, it should be noted that, the resolution of FLFM is directly related to the aperture division of the system.⁶ With an aperture partition coefficient $N$, the resolution is $N$ times worse than the diffraction-limited resolution. Recently, FLFM systems have recently been demonstrated to achieve resolution close to the diffraction limit in three dimensions with appropriate aperture division.⁸ In fact, $N$ is numerically greater than 1, which enables FLFM to obtain multiple 2D perspective views (Angle information) to extract 3D information. Only by adjusting the aperture division, the resolution that FLFM can achieve is inevitably lower than that in wide-field microscope. In order to solve this problem, an effective strategy was proposed, which replaced the on-axis central perspective collected by FLFM with high-resolution image acquired by wide-field microscopy.⁹ This method effectively improved the lateral resolution of the system. However, because the off-axis perspective images which determine the axial resolution of FLFM system remained unchanged, the axial resolution was not improved. More recently, a super-resolved FLFM was proposed,¹⁰ where FLFM was combined with single-molecule localization microscopy (SMLM) to achieve 3D super-resolution imaging. It achieved a localization accuracy of 20 nm in three dimensions within a depth range of 3$\mu$m, but it needs to collect tens of thousands of raw images for localization and reconstruction. Besides SMLM, there are also other super-resolution microscopies.^{11,12,13,14,15} Among these techniques, super-resolution optical fluctuation imaging (SOFI) overcomes the diffraction limit by analyzing fluorescence intensity fluctuations of statistically independent emitters in a time series of images. The final images are background-free and show enhanced spatial resolution, which are two wonderful features of FLFM. We propose a 3D super-resolved FLFM, named super-resolution optical fluctuation Fourier light-field microscopy (SOFFLFM). As an approach combining SOFI and FLFM, SOFFLFM can improve the resolution of FLFM in three dimensions and even break through the diffraction limitation by calculating high-order cumulants.

2. Materials and Methods

As an approach combining FLFM and Sofi, SOFFLFM could use the same imaging system as used in previous FLFM.⁶

The light field propagation model of the original FLFM can be described as shown in Fig. 1(a). First, the 3D information of the object’s imaging domain is mapped into the native imaging plane through Debye diffraction theory of the high numerical objective lens.¹⁶ Then, the optical Fourier transform is performed through a Fourier lens on the native imaging plane, which means the spectral plane of the objective lens is relayed to the rear focal plane of the Fourier lens. In the rear focal plane of the Fourier lens, a micro-lens array (MLA) performs aperture segmentation and modulation on the frequency information collected by the objective lens. Finally, the light field from MLA is propagated to the rear focal plane to form the perspective images on the sensor. Obviously, there are two kinds of effects derived by spectral information segmentation and modulation through an MLA. First, with the division of aperture, the frequency information collected by the objective lens is equalized by the two-dimensional perspective images produced by each sub-lens in MLA. The lateral resolution of the FLFM system can be expressed as $R_{xy}=\frac{\lambda }{2{\mathrm{\text{NA}}}_{\mathrm{\text{FLFM}}}}=\left(\frac{\lambda }{2{\mathrm{\text{NA}}}_{\mathrm{\text{obj}}}}\right)N$, where $\lambda$ is the emission wavelength, ${\mathrm{\text{NA}}}_{\mathrm{\text{FLFM}}}$ is the equivalent numerical aperture of the FLFM system, and ${\mathrm{\text{NA}}}_{\mathrm{\text{obj}}}$ is the numerical apertures of the objective.⁶ Therefore, compared with the traditional wide-field microscopic system, it has lower resolution. Second, each sub-lens captures its local wavefront on the spectral plane, and its position and direction on the rear focal plane are proportional to the average gradient of the captured wavefront.¹⁰ Therefore, the images of emitters at different axial positions on each perspective image are transversely translated to different positions through each independent sub-lens. FLFM system uses these lateral offsets to retrieve the axial information through 3D deconvolution. It is reasonable to take the images of emitters at different axial positions as Gaussian spots with different transversal offsets, and the axial resolution of the FLFM system is directly dependent on its lateral resolution.

Here, the FLFM’s 3D PSF could be approximately described as follows:

where $(x_{zj},y_{zj})$ is the lateral offset relative to the image center for the $j^{\mathrm{\text{th}}}$ sub-lens. ${\omega }_0$ is the standard deviation of the Gaussian distribution.

The basic idea of SOFI is to improve the resolution of the whole image by calculating high-order cumulants of a time image sequence of the emitters with random fluctuations.¹⁵ Due to the independence of fluctuations, signals from different emitters in 3D space are not correlated. Therefore, here in SOFFLFM, high-order cumulants calculation is carried out on a time series of perspective images collected by the above FLFM system to improve the resolution of each perspective image. Based on cumulants formula in Ref. 15 and Eq. (1), we give the $n^{\mathrm{\text{th}}}$-order cumulant formula of FLFM as follows:

where $\delta (r-r_k)$ is the distribution of emitters, ${\epsilon }_k$ is the constant molecular brightness, $\tau$ is the time lag, $w_k({\tau }_1,\dots ,{\tau }_{n-1})$ is a correlation-based weighting function, depending on the specific fluctuation properties of each emitter in 3D space.

The value of the $n^{\mathrm{\text{th}}}$-order cumulants defines the $n^{\mathrm{\text{th}}}$-order SOFFLFM raw image whose PSF is the $n^{\mathrm{\text{th}}}$ power of the original PSF. Therefore, the resolution of the $n^{\mathrm{\text{th}}}$-order SOFFLFM raw image consisting of perspective images can theoretically be improved by $\sqrt{n}$. Finally, the 3D volume of the sample is reconstructed by deconvolving the $n^{\mathrm{\text{th}}}$-order raw Sofi image with the $n^{\mathrm{\text{th}}}$-order cumulative PSF, based on Richardson–Lucy algorithm.

Although SOFFLFM and HR-FLFM could share the same schematic design, they are totally different in imaging strategy. In SOFFLFM, samples must be labeled with proper fluorophores applicable to Sofi, which means each fluorophore’s fluorescence should have reversible, random, and independent intensity fluctuations. Moreover, a time series of raw images are collected for the consequent cumulants calculation.¹⁷

There are two obvious benefits of Sofi-assistant FLFM. First, Sofi can effectively improve the resolution of each perspective image, so the lateral resolution of the final SOFFLFM image reconstructed by the subsequent deconvolution based on these perspective images will theoretically improve by $\sqrt{n}$ times, compared with that of the original FLFM. Furthermore, the improvement of the resolution of the perspective images will also improve the axial resolution of SOFFLFM by $\sqrt{n}$ times theoretically. Second, since the cumulant of Gaussian background noise is equal to zero after high-order cumulants processing, the influence of Gaussian background noise on original FLFM can be automatically suppressed in SOFFLFM, thus making reconstruction through 3D deconvolution more robust, and reducing artifacts. It should be noted that, compared with FLFM, the improvement of resolution is based on the sacrifice of temporal resolution since there should be a time series of images for the SOFFLFM. Moreover, as is indicated by Eq. (1), the raw images that are collected from a SOFFLFM system are formed by the superposition of images of all emitters in three dimensions, which means that the emitter density in the raw images is higher than the images collected by a traditional SOFI system, resulting in artifacts and information loss appear more easily after high-order cumulants processing.¹⁸ As a result, the image quality of deconvolution reconstruction will be degraded.

3. Results and Discussion

The SOFFLFM system for the following simulations is similar as what is used in high-resolution Fourier light-field microscopy (HR-FLFM).⁸ Three regular hexagonal micro-lenses are used to achieve the minimum segmentation of the full Fourier aperture. According to the calculation in Ref. 6, with the parameter settings shown in Table 1, the lateral ($R_{xy}$) and axial ($R_z$) resolution of such an original FLFM system are expected to be 0.508$\mu$m and 0.741$\mu$m respectively, and its depth of focus (DOF) is expected to be 3.78$\mu$m.

Referring to the theoretical framework of FLFM,⁶ we simulate the 3D PSF of the FLFM system, shown in Fig. 1(b). Each raw image in the time series of SOFFLFM is obtained by convolving the 3D PSF and the distribution of emitters whose fluctuations are modeled following the Bernoulli processes. To make the simulations more practical, all the raw images are processed by applying Poisson noise, and adding Gaussian noise whose expected value and variance are set to 0 and 0.01, respectively.

In SOFFLFM, raw images are processed as follows. First, the time series of raw images are processed following the same procedure used in Sofi,¹⁹ i.e., calculating the $n^{\mathrm{\text{th}}}$ order cumulants to generate an image, i.e., $n^{\mathrm{\text{th}}}$ order SOFFLFM raw image, with improved resolution. Then, Richardson–Lucy iterative deconvolution is applied to this SOFFLFM raw image with corresponding PSF which is actually the $n^{\mathrm{\text{th}}}$-order cumulative PSF, i.e., the $n^{\mathrm{\text{th}}}$ power of the original PSF shown in Fig. 1(b). As two examples, the second and the fourth-order cumulative PSFs are shown in Figs. 1(c) and (d), respectively. In the following simulations, if there is no specific mentation, the raw images collected for SOFFLFM are sequences consisting of 100 frames, and the time series of images are processed with the second and the fourth-order SOFI algorithm,¹⁹ then processed with FLFM algorithm⁸ where the number of deconvolution iterations is 20. It should be noted that higher orders SOFFLFM are also workable (refer to Fig. S1 in Supplement A). However, for the calculation of higher-order cumulants, especially when the fluorophore density is relatively high, time series of more frames are required to ensure high-fidelity images after SOFI processing.¹⁸ It means the acquisition time has to be longer. Meanwhile, for SOFFLFM, the computation cost of cumulants and deconvolution also increases exponentially with the increase of the cumulants order. So, in the following simulations, considering the cost of calculation and temporal resolution, second and fourth-order SOFFLFM were selected in the simulation process to verify the feasibility of the method.

To test the performance of SOFFLFM, a series of simulations and analyses were carried out. First, to verify the improvement in lateral resolution, four pairs of emitters spacing 0.2, 0.3, 0.4, and 0.5$\mu$m were assumed at the same depth, $z=0\mu$m. According to the parameter settings in Table 1, the effective pixel size is 0.1$\mu$m. Reconstructed images by means of the original FLFM and the SOFFLFM are shown in Fig. 2. In order to estimate the resolving power quantitatively, a parameter denoted as $I_{min}∕I_{max}$ is calculated, when there are two peaks appear in the curves. $I_{min}$ and $I_{max}$ are the valley and the peak intensities in the curve respectively, as is shown in the last curve in the fourth row in Fig. 2. The value of $I_{min}∕I_{max}$ is shown at the bottom of each curve. Higher value means weak resolving power. The value of ‘1’ means the two emitters cannot be resolved at all, while the value of ‘0’ means the two spots of the two emitters are totally separated. As shown in Fig. 2, FLFM can distinguish two emitters spacing 0.5$\mu$m, which is consistent with the theoretical prediction. For the two emitters separated by 0.4$\mu$m, they cannot be distinguished by FLFM, but obviously distinguishable by second-order SOFFLFM. As to the fourth-order SOFFLFM, two emitters spacing 0.3$\mu$m can easily be distinguished, and two emitters spacing even 0.2$\mu$m could also be distinguished. Compared with the lateral resolution of the FLFM, that of second and fourth-order SOFFLFM improved to 1.25 times and more than 2 times, respectively.

Next, to verify the improved axial resolution, six pairs of emitters were set to different axial separations, from 0.4$\mu$m to 0.9$\mu$m. Reconstructed images by means of the original FLFM and the SOFFLFM are shown in Fig. 3. Similar and solid improvements in axial resolution are observed. FLFM is just able to distinguish emitters spacing 0.9$\mu$m, while second and fourth-order SOFFLFM are able to distinguish emitter pairs spacing 0.5$\mu$m and 0.4$\mu$m, respectively. Compared with the axial resolution of FLFM, that of second and fourth-order SOFFLFM improved to 1.60 times and more than 2.25 times, respectively.

Then, a sample was simulated, which consists of emitters randomly distributed at four fixed layers with interval of 0.5$\mu$m, as is shown in Fig. 4(a). Raw image collected by FLFM and the corresponding reconstructed FLFM image are shown in Figs. 4(b) and 4(c). In SOFFLFM, a series of raw images (Fig. 4(d)) are collected, then deal with second and fourth Sofi algorithm (Figs. 4(e) and 4(f)), and 3D deconvolved to reconstruct the 3D structure of the sample which are presented as projections on planes $x$–$y$, $y$–$z$, and $x$–$z$ (Figs. 4(g) and 4(h)). Compared with the perspective image of FLFM (Fig. 4(b)), that of second and fourth SOFFLFM (Figs. 4(e) and 4(f)) show a progressively improved resolution and quite high signal-to-noise ratio (SNR).

Now, we would like to compare the effect of adjacent emitters on the appearance of a certain emitter in the reconstructed images. First, a sparsely distributed spot (indicated with a blue arrow) from a single emitter was selected for analysis (Fig. 4(i)). In the FLFM image, its full width at half maximum (FWHM) along $x$, $y$, and $z$ axis is 0.222$\mu$m ($x$), 0.214$\mu$m ($y$), and 0.348$\mu$m ($z$), respectively. In the second-order SOFFLFM image, they are 0.142$\mu$m ($x$), 0.152$\mu$m ($y$), and 0.229$\mu$m ($z$), which means an improvement in the lateral resolution by 1.41 times and the axial resolution by 1.52 times compared with that in FLFM. In the fourth-order SOFFLFM image, the FWHMs are even squeezed to 0.077$\mu$m ($x$), 0.076$\mu$m ($y$), and 0.125$\mu$m ($z$), increased by 2.82 times and 2.78 times compared with that in FLFM. According to the results above, in FLFM image, an emitter should be distinguishable from its neighbor emitter which is more than 0.222$\mu$m ($x$ or $y$ direction) or 0.348$\mu$m (axial direction) away. However, for the two emitters with even 0.400$\mu$m apart in lateral directions, which are highlighted with yellow rectangles, they could not be resolved (Fig. 4(j)). Two other emitters (highlighted with white rectangles) with 0.500$\mu$m apart in axial direction could not be resolved either (Fig. 4(k)). It seems like the discrete emitters exhibit a smaller FWHM than the nondiscrete emitters under the same number of iterations. Since deconvolution is a standard step for the final reconstructed image, such conclusion is reasonable. And it is another advantage of SOFFLFM because two emitters which could not be distinguished in raw image of FLFM might be resolved in second- or fourth-order SOFFLFM raw image. That also explains that, compared with the resolution of FLFM, why the resolution improvement of second- and fourth-order SOFFLFM is more than $\sqrt{2}$ and $2$ times (Figs. 2 and 3). The performance of SOFFLFM in improving resolution is outstanding, after all, theoretically, based on the second and fourth-order SOFI algorithm here, the resolution can only be improved to $\sqrt{2}$ and $2$ times, compared with the traditional microscopy. But this is also reasonable, because of the fact that the Gaussian noise is suppressed after SOFI processing, such a raw image is more conducive to deconvolution. Notably, in some regions where the density of the emitter is higher, artifacts may appear, as shown in Fig. 4(h). We believe that this artifact can be effectively suppressed by increasing the number of raw image frames (refer to Fig. S2 in Supplement 1).

Lastly, we use a diverging star-shaped structure as a sample (Fig. 5(a)) to test the influence of different numbers of raw images on the performance of SOFFLFM. Figures 5(c), 5(e) and 5(d), 5(f) are reconstructed from 100 and 400 frames, respectively. Two adjacent lines which could not be resolved in FLFM image (Fig. 5(b)) can be recognized as two lines in second-order SOFFLFM images (Figs. 5(c) and 5(d)) and fourth-order SOFFLFM images (Figs. 5(e) and 5(f)). However, when the number of frames is small, some information might be lost, especially in high-order SOFFLFM, which can be ascribed to the statistical error of cumulants.

4. Conclusion

In this paper, we propose an approach to enhance the resolution by introducing high-order cumulants analysis into FLFM. Performances of second-order SOFFLFM and fourth-order SOFFLFM are tested by simulations. The results demonstrated that more than $\sqrt{2}$ and 2 improvements in resolution are found in all three dimensions, compared with that in FLFM. Besides, SOFFLFM improves resolution by calculating the cumulants of image sequences collected by normal FLFM systems, which can be directly applied to most FLFM systems without any changes in hardware. In the future, we anticipate SOFFLFM to be an important tool in 3D super-resolution application scenarios.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant Nos. 11774242, 61605127, 61975131, 62175166, and 61335001), the Shenzhen Science and Technology Planning Project (Grant Nos. JCYJ20210324094200001, JCYJ20200 109105411133, and ZDSYS20210623092006020).

Conflicts of Interest

The authors declare that there are no conflicts of interest relevant to this paper.