Research ArticleOpen Access

Automated cone photoreceptor cell identification in confocal adaptive optics scanning laser ophthalmoscope images based on object detection

Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, Jiangsu 215163, P. R. China

Search for more papers by this author

Yi He

Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, Jiangsu 215163, P. R. China

E-mail Address: heyi_job@126.com

Corresponding author.

Search for more papers by this author

Jing Wang

https://orcid.org/0000-0001-7751-8607

Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, Jiangsu 215163, P. R. China

Department of Biomedical Engineering, University of Science and Technology of China, Hefei, Anhui 230041, P. R. China

Search for more papers by this author

Wanyue Li

Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, Jiangsu 215163, P. R. China

Department of Biomedical Engineering, University of Science and Technology of China, Hefei, Anhui 230041, P. R. China

Search for more papers by this author

Lina Xing

Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, Jiangsu 215163, P. R. China

Search for more papers by this author

Xin Zhang

Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, Jiangsu 215163, P. R. China

Search for more papers by this author

, and

Guohua Shi

Jiangsu Key Laboratory of Medical Optics, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, Jiangsu 215163, P. R. China

Department of Biomedical Engineering, University of Science and Technology of China, Hefei, Anhui 230041, P. R. China

Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai 200031, P. R. China

Search for more papers by this author

https://doi.org/10.1142/S1793545822500018Cited by:3 (Source: Crossref)

This article is part of the issue:

Contains the Special Issue on Advances in Biophotonics and Biomedical Optics: Part II
Guest Editors: Polina Dyachenko, Tingting Yu, Dan Zhu and Valery V. Tuchin

Abstract

Cone photoreceptor cell identification is important for the early diagnosis of retinopathy. In this study, an object detection algorithm is used for cone cell identification in confocal adaptive optics scanning laser ophthalmoscope (AOSLO) images. An effectiveness evaluation of identification using the proposed method reveals precision, recall, and $F_{1}$ -score of 95.8%, 96.5%, and 96.1%, respectively, considering manual identification as the ground truth. Various object detection and identification results from images with different cone photoreceptor cell distributions further demonstrate the performance of the proposed method. Overall, the proposed method can accurately identify cone photoreceptor cells on confocal adaptive optics scanning laser ophthalmoscope images, being comparable to manual identification.

Keywords:

1. Introduction

Vision is essential for human activity and thriving in modern life. Therefore, its decline and loss due to eye diseases can cause major losses to individuals, families, and society. In recent years, retinopathy has become increasingly common, being a major cause of blindness. Nevertheless, most patients suffering retinopathy can prevent blindness with early diagnosis and treatment. Retinopathy is mainly diagnosed by observing pathological changes and the degree of damage to the retina through imaging. The main instruments used for retinal imaging are the fundus camera,^1,2 scanning laser ophthalmoscope (SLO),^3,4 and optical coherence tomography system,^5,6 which only achieve tissue-level resolution under ocular aberrations. For early diagnosis of retinopathy, adaptive optics (AO) has been introduced to correct ocular aberrations during retinal imaging.^7,8,9 By integrating AO, the fundus camera,⁷ SLO,⁸ and optical coherence tomography system⁹ can achieve cellular-level resolution. Among these instruments, adaptive optics scanning laser ophthalmoscope (AOSLO) provides real-time and clear imaging of cone photoreceptor cells.^{8,10,11,12,13} Moreover, AOSLO can contribute to cone photoreceptor cell identification, which is important for the early diagnosis of retinopathy.^14,15,16,17 Although manual cone photoreceptor cell identification on AOSLO images is reliable, it is time-consuming and subjective. Therefore, many semi-automatic and automatic identification algorithms have been developed.^{18,19,20,21,22,23,24,25,26,27,28,29,30,31,32} These algorithms are based on either centroid detection,^18,19,20,21 segmentation of the cone photoreceptor cells,^{24,25,26,27,28,31,32} or image patch classification step with threshing the probability map.^22,23,29,30

In this study, we develop our cone photoreceptor cells identification algorithm based on another kind of technique called object detection which is introduced to the cone photoreceptor cells identification task for the first time as far as we know, and apply it to confocal AOSLO images. The object detection algorithm provides not only centers but also bounding boxes of cone photoreceptor cells which may be helpful for the diagnosis of retinopathy, and its computational complexity is lower than segmentation algorithm generally. Among the available object detection algorithms, we adopt YOLO (You Only Look Once)³³ as the backbone for cone photoreceptor cell identification, given its high performance. To confirm the effectiveness of the proposed method, its identification results were compared with manual identification results in terms of precision, recall, and $F_{1}$ -score in the confocal AOSLO images. To further demonstrate the performance of the proposed method, we obtained cone photoreceptor cell detection and identification results from images with different cone photoreceptor cell distributions in the confocal AOSLO images.

2. Methods

The proposed method was evaluated on a publicly available dataset³⁴ that contains 840 confocal AOSLO images. The ground truths of training process are taken from the results of graph-based method.²⁵ They contain not only centroid selections but also very accurate cone photoreceptor cells segmentation results which can be used to generate circumscribed rectangles of cone photoreceptor cells for the training process. We used 720 confocal AOSLO images for training, 80 confocal AOSLO images for validation, and the remaining 40 confocal AOSLO images for testing. The proposed cone photoreceptor cell identification algorithm comprises the three main steps illustrated in Fig. 1: training, testing, and post-processing. First, labeled training and validation datasets containing confocal AOSLO images³⁴ and bounding boxes of cone photoreceptor cells generated by circumscribed rectangles of the segmentation results using a graph-based method²⁵ are used to train the YOLO v3 model.³³ Second, testing confocal AOSLO images are input to the trained YOLO v3 model³³ to generate the object detection results. Third, the object detection results are sequentially processed using nonmaximum suppression³⁵ and abnormal length–width ratio filtering, obtaining the final cone photoreceptor cell detection results. Using the bounding boxes of these outputs, the cone photoreceptor cells are individually identified by calculating their centers.

Fig. 1. Diagram of the proposed cone photoreceptor identification method in confocal AOSLO images based on object detection.

2.1. Training

First, $k$ -means clustering³⁶ was used to determine anchor boxes for nine clusters and three scales as the requirement inputs of YOLO v3. The sizes of the nine clusters determined by $k$ -means in the training datasets are listed in Table 1. Second, YOLO v3³³ pretrained on the COCO dataset,³⁷ which is a large labeled dataset for object detection and image segmentation, was further trained on the labeled confocal AOSLO images with anchor boxes. Training consisted of two stages. In the first stage, DarkNet-53, which is the backbone of YOLO v3, was frozen, and the rest of the YOLO v3 architecture³³ was trained with a batch size of 32 for 50 epochs. In the second stage, DarkNet-53 was unfrozen, and the complete YOLO v3 was trained using the same datasets as in the first stage with a batch size of 8 for 50 epochs. Early stopping³⁸ and an adaptive learning rate³⁹ were applied for training. The loss curves of the training and validation process are shown in Fig. 2. As shown in the figure, the loss value decreased and then converged in the two stages, and early stopping happened at the 36th epoch in the second stage.

**Table 1. Sizes of nine clusters obtained by the k-means method.**
Cluster	1	2	3	4	5	6	7	8	9
Size (pixels)	30 × 30	24 × 30	36 × 36	36 × 42	36 × 30	42 × 36	42 × 42	48 × 48	54 × 48

Fig. 2. Loss curves of the training and validation process.

2.2. Testing and post-processing

For testing, the trained YOLO v3 was used to obtain the object detection results per testing confocal AOSLO image. These object detection results contain the initial cone photoreceptor cells detection information with over-detection as the other YOLO v3 applications. To solve the over-detection problem, we first applied nonmaximum suppression³⁵ to the object detection results. Then, as a cone photoreceptor cell has an approximately circular shape, the length–width ratio should be close to 1. Accordingly, we filtered out object detection results with abnormal length–width ratios after nonmaximum suppression.³⁵ In detail, object detection results with the length–width ratios below 1/3 or above 3 were discarded, obtaining the final cone photoreceptor cell detection results. An example of nonmaximum suppression³⁵ and abnormal length–width ratio filtering using a representative image patch is shown in Fig. 3. As shown in the figure, the over-detection problem is solved to a large extent by the two processing steps. To individually identify the cone photoreceptor cells, the centers of the bounding boxes in the final detection results were calculated.

Fig. 3. Effectiveness of post-processing: (a) Input confocal AOSLO image, (b) original object detection results, and (c) object detection results after nonmaximum suppression and abnormal length–width ratio filtering.

3. Results

The computations took 16.4h for training and 189.7s for testing and post-processing. These computation times were obtained from a computer running 64-bit Python and equipped with an Intel Xeon E5-2620 processor (2.10GHz), 32-GB memory, and an NVIDIA GeForce GTX 1080Ti graphics card.

To confirm the effectiveness of the proposed method and compare it with similar methods,^{24,25,26,29,31,32} the identification results were evaluated by calculating the overall precision, recall, and $F_{1}$ -score on the testing dataset considering manual identification as the ground truth, obtaining the results listed in Table 2. The proposed method achieves more accurate cone photoreceptor cell identification than our previous methods,^24,26,31,32 which confirms its effectiveness. However, it did not quite reach the performance of methods such as graph-based method²⁵ and confocal convolutional neural network.²⁹

**Table 2. Performances of various methods for the cone photoreceptor cell identification.**
Method	Precision	Recall	$F_{1}$ -score
Proposed method	95.8%	96.5%	96.1%
Confocal convolutional neural network²⁹	99%	99.1%	99%
Graph-based method²⁵	98.2%	98.5%	98.3%
DeepLab-based method³¹	96.7%	94.6%	95.7%
Watershed-based method²⁴	93.2%	96.6%	94.9%
The $k$ -means clustering-based method³²	93.4%	95.2%	94.3%
Superpixel-based method²⁶	80.1%	93.5%	86.3%

To illustrate the identification performance of the proposed method, Fig. 4 shows the cone photoreceptor cell detection and identification results of three typical examples with different cone photoreceptor cell distributions from the testing dataset. The cone photoreceptor cells were identified on the three confocal AOSLO images with high accuracy using the proposed method.

4. Discussion

To the best of our knowledge, this is the first method based on object detection to identify cone photoreceptor cells, and it has achieved highly accurate results. Although we adopted YOLO v3, other object detection algorithms may be more suitable for cone photoreceptor cell identification. Thus, other object detection algorithms should be evaluated to determine those with possibly higher identification accuracy.

In previous methods,^{18,19,20,21,22,23,24,25,26,27,28,29,30,31,32} there are two types of results which can be obtained: identified results^{18,19,20,21,22,23} [Fig. 5(b)] and segmentation results^{24,25,26,27,28,29,30,31,32} [Fig. 5(d)]. In this paper, we provide the third types of results — object detection results [Fig. 5(c)]. These results provide more geometry information of cone photoreceptor cells by the output of height and width of each cone photoreceptor cell, which may be helpful for the diagnosis of retinopathy. Although segmentation results provide even more geometry information of cone photoreceptor cells, but the accurate segmentation process needs generally more complex computation results in more computational time.

Fig. 5. Three types of results: (a) Input confocal AOSLO images, (b) identification results, (c) object detection results (our method), and (d) segmentation results.

Like every application of deep learning object detection, the proposed method is expected to fail when the intensity distribution of cone photoreceptor cell in the testing dataset is too much different than that in the training dataset. Although we cannot apply our method to pathological images, vessel shadow images, rod and cone photoreceptor cell images, and low-quality images due to shortage of large dataset, we believe that our method can provide accurate object detection results if we have enough number of these kinds of images for training.

5. Conclusions

We propose an automated cone photoreceptor cell identification method based on object detection. The proposed method achieved precision, recall, and $F_{1}$ -score of 95.8%, 96.5%, and 96.1% considering manual identification as the ground truth. To illustrate the performance of the proposed method, cone photoreceptor cell identification and detection results of three typical examples with different cone photoreceptor cell distributions were obtained. The experimental results show the effectiveness and high accuracy of the proposed method for cone photoreceptor cell detection and identification. The proposed method may assist health professionals in the early diagnosis of retinopathy.

Conflict of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

The authors wish to thank the anonymous reviewers for their valuable suggestions. This work was supported in part by the Natural Science Foundation of Jiangsu Province (BK20200214); National Key R&D Program of China (2017YFB0403701); Jiangsu Province Key R&D Program (BE2019682 and BE2018667); National Natural Science Foundation of China (61605210, 61675226, and 62075235); Youth Innovation Promotion Association of Chinese Academy of Sciences (2019320); Frontier Science Research Project of the Chinese Academy of Sciences (QYZDB-SSW-JSC03); Strategic Priority Research Program of the Chinese Academy of Sciences (XDB02060000); and Entrepreneurship and Innovation Talents in Jiangsu Province (Innovation of Scientific Research Institutes).

References

1. A. Sharma et al., “Smartphone-based fundus camera device (MII Ret Cam) and technique with ability to image peripheral retina,” Eur. J. Ophthalmol. 26(2), 142–144 (2016). Crossref, Web of Science, Google Scholar
2. M. Monjur et al., “Smartphone based fundus camera for the diagnosis of retinopathy,” Smart Health 19, 100177 (2021). Crossref, Google Scholar
3. R. H. Webb, G. W. Hughes, “Scanning laser ophthalmoscope,” IEEE Trans. Biomed. Eng. 7, 488–492 (1981). Crossref, Web of Science, Google Scholar
4. R. H. Webb, G. W. Hughes, F. C. Delori, “Confocal scanning laser ophthalmoscope,” Appl. Opt. 26(8), 1492–1499 (1987). Crossref, Web of Science, Google Scholar
5. W. Drexler, J. G. Fujimoto, “State-of-the-art retinal optical coherence tomography,” Prog. Retin. Eye Res. 27(1), 45–88 (2008). Crossref, Web of Science, Google Scholar
6. A. Unterhuber et al., “In vivo retinal optical coherence tomography at 1040 nm-enhanced penetration into the choroid,” Opt. Express 13(9), 3252–3258 (2005). Crossref, Web of Science, Google Scholar
7. B. Xue et al., “Photoreceptor counting and montaging of en-face retinal images from an adaptive optics fundus camera,” J. Opt. Soc. Am. A, Opt. Image Sci. Vis. 24(5), 1364–1372 (2007). Crossref, Web of Science, Google Scholar
8. A. Roorda et al., “Adaptive optics scanning laser ophthalmoscopy,” Opt. Express 10(9), 405–412 (2002). Crossref, Web of Science, Google Scholar
9. Y. Zhang et al., “High-speed volumetric imaging of cone photoreceptors with adaptive optics spectral-domain optical coherence tomography,” Opt. Express 14(10), 4380–4394 (2006). Crossref, Web of Science, Google Scholar
10. S. A. Burns et al., “Large-field-of-view, modular, stabilized, adaptive-optics-based scanning laser ophthalmoscope,” J. Opt. Soc. Am. A 24(5), 1313–1326 (2007). Crossref, Google Scholar
11. R. D. Ferguson et al., “Adaptive optics scanning laser ophthalmoscope with integrated wide-field retinal imaging and tracking,” J. Opt. Soc. Am. A 27(11), A265–A277 (2010). Crossref, Google Scholar
12. D. X. Hammer et al., “Adaptive optics scanning laser ophthalmoscope for stabilized retinal imaging,” Opt. Express 14(8), 3354–3367 (2006). Crossref, Web of Science, Google Scholar
13. D. C. Chen et al., “High-resolution adaptive optics scanning laser ophthalmoscope with dual deformable mirrors,” J. Opt. Soc. Am. A 24(5), 1305–1312 (2007). Crossref, Web of Science, Google Scholar
14. J. Lammer et al., “Cone photoreceptor irregularity on adaptive optics scanning laser ophthalmoscopy correlates with severity of diabetic retinopathy and macular edema,” Invest. Ophthalmol. Vis. Sci. 57(15), 6624–6632 (2016). Crossref, Web of Science, Google Scholar
15. S. P. Park et al., “Early structural anomalies observed by high-resolution imaging in two related cases of autosomal-dominant retinitis pigmentosa,” Ophthalmic Surg. Lasers Imaging Retina 45(5), 469–473 (2014). Crossref, Web of Science, Google Scholar
16. S. Nakatake et al., “Early detection of cone photoreceptor cell loss in retinitis pigmentosa using adaptive optics scanning laser ophthalmoscopy,” Graefe’s Arch. Clin. Exp. Ophthalmol. 257(6), 1169–1181 (2019). Crossref, Web of Science, Google Scholar
17. Y. Chen et al., “Cone photoreceptor abnormalities correlate with vision loss in patients with Stargardt disease,” Invest. Ophthalmol. Vis. Sci. 52(6), 3281–3292 (2011). Crossref, Web of Science, Google Scholar
18. K. Y. Li, A. Roorda, “Automated identification of cone photoreceptors in adaptive optics retinal images,” J. Opt. Soc. Am. A 24(5), 1358–1363 (2007). Crossref, Google Scholar
19. A. Turpin et al., “Automated identification of photoreceptor cones using multi-scale modelling and normalized cross-correlation,” ICIAP 2011: Image Analysis and Processing, G. Maino, G. L. Foresti, Eds., Lecture Notes in Computer Science, Vol. 6978, pp. 494–503, Springer, Berlin (2011). Crossref, Google Scholar
20. D. Cunefare et al., “Automatic detection of cone photoreceptors in split detector adaptive optics scanning light ophthalmoscope images,” Biomed. Opt. Express 7(5), 2036–2050 (2016). Crossref, Web of Science, Google Scholar
21. J. Liu et al., “Automated photoreceptor cell identification on nonconfocal adaptive optics images using multiscale circular voting,” Invest. Ophthalmol. Vis. Sci. 58(11), 4477–4489 (2017). Crossref, Web of Science, Google Scholar
22. D. Cunefare et al., “Deep learning based detection of cone photoreceptors with multimodal adaptive optics scanning light ophthalmoscope images of achromatopsia,” Biomed. Opt. Express 9(8), 3740–3756 (2018). Crossref, Web of Science, Google Scholar
23. C. Bergeles et al., “Unsupervised identification of cone photoreceptors in non-confocal adaptive optics scanning light ophthalmoscope images,” Biomed. Opt. Express 8(6), 3081–3094 (2017). Crossref, Web of Science, Google Scholar
24. Y. Chen et al., “Automated cone photoreceptor cell segmentation and identification in adaptive optics scanning laser ophthalmoscope images using morphological processing and watershed algorithm,” IEEE Access 8, 105786–105792 (2020). Crossref, Web of Science, Google Scholar
25. S. J. Chiu et al., “Automatic cone photoreceptor segmentation using graph theory and dynamic programming,” Biomed. Opt. Express 4(6), 924–937 (2013). Crossref, Web of Science, Google Scholar
26. Y. Chen et al., “Automated superpixels-based identification and mosaicking of cone photoreceptor cells for adaptive optics scanning laser ophthalmoscope,” Chin. Opt. Lett. 18(10), 101701 (2020). Crossref, Web of Science, Google Scholar
27. D. M. Bukowska et al., “Semi-automated identification of cones in the human retina using circle Hough transform,” Biomed. Opt. Express 6(12), 4676–4693 (2015). Crossref, Web of Science, Google Scholar
28. J. Hamwood et al., “Automatic detection of cone photoreceptors with fully convolutional networks,” Transl. Vis. Sci. Technol. 8(6), 10 (2019). Crossref, Web of Science, Google Scholar
29. D. Cunefare et al., “Open source software for automatic detection of cone photoreceptors in adaptive optics ophthalmoscopy using convolutional neural networks,” Sci. Rep. 7(1), 6620 (2017). Crossref, Web of Science, Google Scholar
30. B. Davidson et al., “Automatic cone photoreceptor localisation in healthy and Stargardt afflicted retinas using deep learning,” Sci. Rep. 8(1), 7911 (2018). Crossref, Web of Science, Google Scholar
31. Y. Chen et al., “DeepLab and bias field correction based automatic cone photoreceptor cell identification with adaptive optics scanning laser ophthalmoscope images,” Wirel. Commun. Mob. Comput. 2021, 2034125 (2021). Web of Science, Google Scholar
32. Y. Chen et al., “Automated cone cell identification on adaptive optics scanning laser ophthalmoscope images based on TV-L1 optical flow registration and K-means clustering,” Appl. Sci. 11(5), 2259 (2021). Crossref, Google Scholar
33. J. Redmon, A. Farhadi, “YOLOv3: An incremental improvement,” preprint, arXiv:1804.02767 [cs.CV] (2018). Google Scholar
34. R. Garrioch et al., “The repeatability of in vivo parafoveal cone density and spacing measurements,” Optom. Vis. Sci. 89(5), 632–643 (2012). Crossref, Web of Science, Google Scholar
35. M. D. Buil, “Non-maxima suppression,” Technical Report No. ICG-TR-xxx, Institute for Computer Graphics and Vision, Graz University of Technology, Austria (2011). Google Scholar
36. T. Kanungo et al., “An efficient k-means clustering algorithm: Analysis and implementation,” IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 881–892 (2002). Crossref, Web of Science, Google Scholar
37. COCO Consortium, “Common Objects in Context,” http://cocodataset.org (2021). Google Scholar
38. L. Prechelt, “Early stopping — but when?, ” Neural Networks: Tricks of the Trade, Lecture Notes in Computer Science, Vol. 7700, pp. 53–67, Springer, Berlin (1998). Crossref, Google Scholar
39. M. D. Zeiler, “ADADELTA: An adaptive learning rate method,” preprint, arXiv:1212.5701 [cs.LG] (2012). Google Scholar

Vol. 15, No. 01

Metrics

Downloaded 14,751 times

History

Received 30 June 2021

Accepted 30 August 2021

Published: 25 September 2021

Information

This is an Open Access article. It is distributed under the terms of the Creative Commons Attribution 4.0 (CC-BY) License. Further distribution of this work is permitted, provided the original work is properly cited.

Keywords

PDF download