Open Access

Comparison of two reader modes of computer-aided diagnosis in lung nodules on low-dose chest CT scan

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

Qiongjie Hu and Shaofang Wang contributed equally to this work.

Search for more papers by this author

Shaofang Wang

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

Qiongjie Hu and Shaofang Wang contributed equally to this work.

Search for more papers by this author

Chong Chen

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

Search for more papers by this author

Shichao Kang

Department of Radiology, Shenzhen Maternity & Child Healthcare Hospital, Affiliated to Southern Medical University, Hongli, Shenzhen 518028, P. R. China

Search for more papers by this author

Ziyan Sun

https://orcid.org/0000-0002-0713-957X

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

Search for more papers by this author

Yujin Wang

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

Search for more papers by this author

Min Xiang

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

Search for more papers by this author

Hanxiong Guan

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

Search for more papers by this author

Yiwen Liu

https://orcid.org/0000-0003-0350-6472

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

E-mail Address: xialiming2017@outlook.com

Corresponding author.

Search for more papers by this author

, and

Liming Xia

Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Jiefang Dadao 1095(#), Wuhan 430030, Hu Bei, P. R. China

E-mail Address: zoeyenjoy@163.com

Corresponding author.

Search for more papers by this author

https://doi.org/10.1142/S1793545822500134Cited by:3 (Source: Crossref)

Abstract

Low-dose computerized tomography (LDCT) scanning is of great significance for monitoring and management of pulmonary nodules on chest computerized tomography (CT). Nevertheless, the malignant potential of these nodules is often difficult to detect, especially for some smaller pulmonary nodules on LDCT images. Recent advances using the state-of-art computer-aided detection (CAD) system have attempted to address this problem by identifying small nodules that can be easily missed during clinical practice. CAD is used in two reading modes: Concurrent-reader (CR) mode or second-reader (SR) mode. In this study, we prospectively evaluated the efficiency of a CAD system’s SR and CR modes in detecting pulmonary nodules on LDCT. We found that the SR mode improves pulmonary nodule detection regardless of the dose and experience level, especially for interns in the low-dose setting. The CR mode maintains the sensitivity of SR mode while significantly decreasing reading times.

Keywords:

1. Introduction

Lung cancer is one of the most common fatal diseases, which has killed millions of people worldwide. As known, pulmonary nodules often occurred in early-stage lung cancer when it could be treated effectively. Additionally, pulmonary metastases that often manifested as pulmonary nodules occur in approximately 1/3 of extra-thoracic malignancies.¹ However, the malignant potential of these nodules is often difficult to detect, especially for some smaller pulmonary nodules, because it is impractical or difficult to establish a definitive diagnosis by obtaining pathological tissues.² In this case, it is important to identify and monitor pulmonary nodules as early as possible.

Due to the advantages of noninvasion and low cost, chest CT became a routine checkup method to detect lung cancer and a very effective method for detecting small pulmonary nodules.³ For example, reports had shown a high detection rate of pulmonary nodules in patients with extra-pulmonary malignancies on thin-section computed tomography CT.⁴ To decrease cumulative radiation exposure from repeated routine CT exams, Low-dose computerized tomography (LDCT) can be employed as an effective option for routine monitoring of metastatic pulmonary nodules.^5,6 However, low-dose and thin-slice thicknesses lead to increased image noise in LDCT images, which may make it more difficult for the radiologist to find lesions.⁷ Also, the workload of viewing a large number of images causes the fatigue and exhaustion of radiologists, contributing to the generation of false-negative results in the diagnostic decision.^8,9,10 Recent advances in using the state-of-art CAD system based on DL have attempted to address this problem by identifying small nodules that can be easily missed during clinical practice.^8,11 Several studies have demonstrated promising results of CAD improved the detection of pulmonary nodules and inter-reader agreement, leading to more consistent follow-up recommendations across radiologists with different levels of experience.^{8,11,12,13,14,15} Previous studies.^1,16,17,18 have evaluated CAD in the detection of pulmonary nodules in patients with extra-thoracic malignancies on stand-dose CT (SDCT). However, studies on the evaluation of the detection of pulmonary nodules in patients with extra-pulmonary malignancies using DL-based CAD on low-dose CT images are still few and insufficient.

CAD is often used in two different reading modes: CR mode (deployed simultaneously with the unassisted interpretation).¹⁹ or SR mode (only deployed after the radiologist has performed a full unassisted analysis of the data set first).²⁰ The SR mode is more effective and accurate.^{11,13,14,15,21,22} except for longer reading time, since it has to read images twice.²³ Comparatively, the CR mode is potentially more attractive for its high efficiency despite that its effectiveness remains controversial.^23,24 Matsumoto et al.²⁴ reported that the sensitivity in the CR mode increased significantly compared with unaided reading. On the contrary, Beyer et al.²³ found that the CR mode led to no increase in the sensitivity.

The purpose of this study was to prospectively assess the value of a CAD system in SR and CR modes for the detection of pulmonary nodules on LDCT images of patients with extra-thoracic malignancies as evaluated by radiologists with a wide spectrum of experience in the clinical setting. We hypothesized that, on both SDCT and LDCT images, CAD would improve the detection of pulmonary nodules and make detection rates more comparable across radiologists of all experience levels. We also hypothesized that the CR mode would decrease reading time.

2. Materials and Methods

The local institutional review board approved this prospective study. Written consent was obtained from all participants.

2.1. Patients

From July–December 2017, 129 patients with extra-thoracic malignancies scheduled consecutively for chest CTs were recruited. Exclusion criteria were severe pulmonary fibrosis,severe emphysema, extensive pulmonary infection, tuberculosis or sarcoidosis, and massive pleural effusion. One hundred and seventeen patients were ultimately included. The present study was approved by the ethics committee of Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology (TJ-IRB20180615).

2.2. CT acquisition

All patients underwent two noncontrast chest CT scans during an end-inspirational breath-hold: (1) SDCT scan (120 kvp with 150 mAs, NI = 13); (2) LDCT scan (120 kvp for BMI ≥ 22, n = 88; 100 kvp for BMI < 22, n = 41, with 20mAs, NI = 30) on a 64-row multidetector system (Discovery CT750 HD; GE Healthcare).The dose is calculated based on the anti-true human body model and the Monte Carlo method.²⁵ The mean size-specific dose estimate (SSDE) was calculated to be 10.54 ± 1.6mGy and 8.43 ± 1.54mGy for SDCT images and 2.24 ± 0.14mGy and 1.55 ± 0.07mGy for LDCT images.²⁶ The effective dose after ICRP guideline 103 of LDCT versus SDCT was 1.23 ± 0.14 versus 6 ± 0.89mSv and 0.76 ± 0.1 versus 4.71 ± 0.97mSv. Data were reconstructed to 1.25mm-thick sections, with ASIR 60% and “Standard” kernels, which provide higher spatial resolution and are frequently used for the lung.^14,27 Importantly, they are recommended for nodule management in current recommendations.^27,28,29

2.3. CAD software

In this study, we implemented a commercially available AI system for nodule detection and characterization (inferRead CT Lung Research, Beijing, China). The principle of this AI system has been previously described, and the performance of which was vigorously validated in several studies.^30,31,32,33 In addition, this AI system has been officially approved by the Food and Drug Administration (FDA) in the USA (510(k) K192880), Conformite Europeenne (CE) in Europe (ISO 13485: 2016), and National Medical Products Administration (NMPA) in China (Registration No. 20203210839) for clinical use. The system is trained on a large data set of over 450,000 chest CT images. Each lesion identified by the software was marked on the image showing the lesion’s largest diameter.

2.4. Detection study

All readers were trained on the CAD system. Radiologists were blinded to patient identity and evaluated images independently. Radiologists were allowed to adjust the window level, zoom in and out, invert gray-scale, and use maximum intensity projection (MIP) thick slab images. Images were randomized.

Acceptable image quality and the reference standard for the pulmonary nodules³⁴ on both SDCT and LDCT images were determined by the consensus of three chest radiologists, each with >20 years of experience. They read images twice in the CR mode in three months between sessions. CAD-detected lesions were included in the reference standard if all radiologists agreed they represented true pathology. None of these radiologists participated in the below studies.

Nine observers completed the first study: Three interns (< 1 year of experience), observers A–C; three radiology residents (2–3 years of experience), observers D–F; and three experienced chest radiologists (5–10 years of experience), observers G–I. Each observer identified the number of nodules per patient and nodule size (≤ 4mm, > 4mm to < 6mm, 6mm to 8mm, and > 8mm) and characteristics (solid, subsolid, or calcified) on 117 LDCT image sets first without CAD (unaided mode) and then with CAD in the SR mode. Eight weeks later, the same readers repeated the above task with the SDCT images.

To prevent reading fatigue, 40 of the 117 LDCT sets were randomly chosen for the second study. Eight different observers (3 interns, 3 radiology residents, and 2 experienced radiologists) identified pulmonary nodules on 20 datasets without CAD; they used the CR mode for the remaining 20 datasets. Eight weeks later, the same observers read the images that they had initially read unaided in the CR mode and vice versa. Fourteen weeks after the first session, all 40 datasets were read in the SR mode. Observers assigned confidence levels to each nodule from 1 (probably not a nodule) to 4 (definitely a nodule).³⁵ Reading times were recorded.

2.5. Data analysis

All statistical analyses were performed using SPSS 21.0. True positive (TP) rates/sensitivity, and false positive (FP) rates were calculated for all conditions, including mode, dose, experience level, and nodule descriptors (size, characteristics). The FP rate was computed as the number of FP detections per CT. Differences in sensitivity and FP rates were compared between SDCT and LDCT images, between the unaided and SR mode on both SDCT and LDCT images,and between the unaided reading, SR, and CR reading modes on LDCT images. Reading times were also compared among three reading modes on LDCT images. The sensitivity was further compared according to the nodule diameter and characteristics. Normality was tested using the Kolmogorov–Smirnov test. Normally distributed variables were compared using analyses of variance (ANOVA); the remaining variables were compared using the Wilcoxon signed-rank test. The Mann–Whitney test was used to compare performance (i.e., sensitivity, FPs) between experience levels on both SDCT and LDCT images and in all reading modes.

Free-response receiver operating characteristic (FROC) curve analyses in implementations (JAFROC 4.2.1; www. devchakraborty.com) were used to analyze true-positive versus false-positive rates for all reading modes^35,36,37,38 on LDCT images. Figures of merit (FOM) representing the nonparametric estimate (ANOVA) of the area under the curve and its 95% confidence interval were calculated using the jackknife method³⁸ FOMs were also calculated by experience level.

Significance was set at P < 0.05. However, when multiple comparisons were conducted among the experience groups and three reading modes using the Wilcoxon signed-rank test,the Bonferroni-corrected P-value was set to alpha = <0.05∕3.

3. Results

3.1. Patient characteristics

LDCT was completed at 120 kVp for the first 81 consecutive patients (44 women, 37 men; mean age 50.63 ± 9.54 years, range 29–78 years, BMI 23.82 ± 1.5kg/m², range 22–28kg/m²). After preliminary data analysis, the radiation dose for LDCT was lowered to 100kVp for the final 36 consecutive patients (18 women, 18 men; mean age 48.53 ± 13.25 years, range 22–76 years, BMI 20.58 ± 0.87kg/m², range 18.5–22kg/m²). Image quality was determined to be acceptable in 100% of patients. All the enrolled patients were followed up for one year and 27 nodules were found to have enlarged pulmonary nodules, of which 6 were smaller than 4mm, 6 were between 4 and 6mm, 8 were between 6 and 8mm, and 7 were larger than 8mm in diameter.

3.2. Observer performance

3.2.1. Study 1

Unaided Reading technique (Without CAD) versus SR Mode

Table E1 (online) presents the nodules as a reference standard both in the standard-dose and the low-dose settings for study 1 (117 patients). Readers, regardless of experience level, were more sensitive at detecting nodules in the SR mode than without CAD on both standard-dose and low-dose images (all P < 0.001). Observer A (intern) had a much lower detection rate than the other participants (Table E2). To explore the effect of observer A on the results, a sensitivity analysis in which observer A was excluded was performed (Table 1). The average sensitivities of the 8 observers continued to be significantly higher when using the SR mode compared to the unaided reading technique on images at both dose settings (all P < 0.001). The experienced radiologists and residents showed significantly better detection of pulmonary nodules in the unaided reading technique than interns regardless of the dose (all P < 0.001), while experienced radiologists, residents, and interns had comparable sensitivities in the SR mode regardless of the dose (all P > 0.05). In a subgroup analysis, the sensitivity in the SR mode was also statistically higher than in the unaided reading technique for nodules with a diameter ≤ 8mm and in all characteristics on both SDCT and LDCT images (all P < 0.05) (Table 2).

**Table 1. Results of nodule detection in the unaided reading and second-reader modes: sensitivity and false positive rate per CT (exclude observer A).**
	Sensitivity						False positive rate
	UR			SR*			UR			SR**
Groups	SD	LD	P	SD	LD	P	SD	LD	P	SD	LD	P
Interns
B	22.77%	20.36%		45.09%	45.70%		3.63/CT	2.05/CT		7.45/CT	5.34/CT
C	30.28%	26.17%		56.42%	61.21%		1.51/CT	1.82/CT		2.74/CT	3.36/CT
Average	26.47%	23.21%	0.461	50.68%	53.33%	0.884	2.56/CT	1.89/CT	0.002	5.06/CT	4.23/CT	0.034
Residents
D	36.36%	36.62%		53.48%	54.46%		1.21/CT	1.13/CT		1.88/CT	1.90/CT
E	46.82%	37.23%		59.39%	63.08%		1.49/CT	1.39/CT		1.89/CT	2.30/CT
F	45.45%	44.24%		61.97%	62.37%		3.82/CT	3.58/CT		5.91/CT	4.42/CT
Average	42.88%	39.36%	0.004	58.28%	59.97%	0.858	2.17/CT	2.03/CT	0.007	3.23/CT	2.87/CT	0.066
Experienced radiologists
G	39.55%	30.15%		68.18%	64.31%		0.76/CT	0.88/CT		1.15/CT	1.57/CT
H	57.12%	54.31%		71.21%	70.31%		1.66/CT	2.02/CT		2.24/CT	2.62/CT
I	50.15%	47.08%		68.79%	61.69%		1.42/CT	1.17/CT		1.95/CT	1.54/CT
Average	48.94%	43.85%	0.008	69.39%	65.44%	0.007	1.28/CT	1.36/CT	0.411	1.78/CT	1.91/CT	0.318
Average of all	43.96.%	39.76%	< 0.000	62.52%	61.06%	0.111	1.81/CT	1.79/CT	< 0.000	2.76/CT	2.57/CT	< 0.000

Notes: UR = unaided reading technique, SR = the second-reader mode, SD = standard-dose CT LD = lowdose CT, and X/CT: The average number of false-positive pulmonary nodules detected on each CT image.

**Table 2. Characteristic of nodules in unaided reading and the second-reader modes of CAD on standard-dose and low-dose CT images**
		Attenuation						Size range						Location
		Solid		Subsolid		Calcified		< 4		(4, 6)		[6, 8]		> 8		Subpleural		Nonsubpleural
		SD	LD	SD	LD	SD	LD	SD	LD	SD	LD	SD	LD	SD	LD	SD	LD	SD	LD
Groups		(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)	(%)
Interns	UR	22.59	19.63	21.33	39.68	6.67	12.5	19.3	17.86	25.81	34.62	35.29	47.62	93.94	90.91	16.4	18.7	24.39	22.44
	SR	41.85	44.4	45.33	53.97	37.78	41.67	39.43	42.74	58.06	61.54	52.94	57.14	100	100	32.8	40.65	47.56	47.8
	P	< 0.001	< 0.001	0.001	0.007	< 0.001	0.001	< 0.001	< 0.001	0.002	< 0.001	0.003	0.014	0.157	0.083	< 0.00	< 0.001	< 0.001	< 0.001
Residents	UR	41.98	40.63	47.56	40.52	45.93	21.71	40.6	36.68	47.3	52.08	70.59	57.45	96.67	96.67	39.87	39.92	44.72	39.05
	SR	57.47	62.77	62.67	55.6	60.74	33.33	56.91	58.58	56.99	60.42	70.59	65.96	100	100	54.8	56.72	60.41	61.84
	P	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	0.003	0.005	1	0.046	0.317	0.317	< 0.001	< 0.001	< 0.001	< 0.001
Experienced radiologists	UR	48.95	43.98	49.78	42.47	47.41	44.44	46.31	40.83	64.52	62.37	68.63	66.67	100	100	50.53	48.97	47.97	40.79
	SR	70.43	66.92	69.78	59.36	56.3	57.78	68.28	64.02	72.04	67.74	72.55	76.47	100	100	68.4	64.2	70	66.18
	P	< 0.001	< 0.001	< 0.001	< 0.001	0.001	< 0.001	< 0.001	< 0.001	0.008	0.025	0.157	0.025	1	1	< 0.001	< 0.001	< 0.001	< 0.001
All	UR	42.2	39	44.76	4125	40.95	30.13	40	35.74	51.61	54.42	64.71	59.66	96.67	95.96	41.09	40.73	43.21	37.41
	SR	60.79	61.86	63.24	57	55.56	45.19	59.28	58.61	63.59	63.72	68.91	68.91	100	100	57.49	57.61	62.68	61.68
	P	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	< 0.001	0.025	< 0.001	0.083	0.046	0.001	< 0.001	< 0.001	< 0.001

Notes: UR = unaided reading technique, SR = the second-reader mode, SD = standard-dose, and LD=low-dose.

Regardless of the experience level and dose, FP rates in the SR mode were significantly higher than in the unaided reading technique (all P < 0.0001). Additionally, significant differences in FP rates existed between experienced radiologists, residents, and interns in the SR and unaided reading technique on both SDCT and LDCT images (all P < 0.05).

SDCT versus LDCT

Readers, regardless of the experience level (excluding observer A), were more sensitive at detecting nodules on SDCT images than LDCT images in the unaided mode. However, the SR mode resulted in comparable sensitivities between SDCT and LDCT images, and interns even had a higher sensitivity on LDCT images (P > 0.05) (Table 1).

FP rates were comparable regardless of the dose within each mode for experienced radiologists (P = 0.411 and 0.318), and residents and interns even had a lower FP rate on LDCT images compared to SDCT.

3.2.2. Study 2

Sensitivity and FP rates in three reading modes

Table E3 (online) shows the nodules as a reference standard both in the standard-dose and the low-dose settings for study 2 (40 patients). The average sensitivity of reading with CAD in the CR mode (64.76%) was significantly higher than reading without CAD (44.71%, P < 0.001) and was comparable to reading in the SR mode (66.86%, P = 0.033 > 0.05∕3) on LDCT images. In a subgroup analysis, the sensitivity in the CR mode was also statistically higher than in the unaided reading technique and was comparable to reading in the SR mode for nodules with a diameter less than 6mm and for solid and subsolid nodules (all P < 0.001). There were no statistical differences regarding the sensitivity among all experience levels regardless of reading modes (all P > 0.05). FP rates in the CR mode (average, 4.09/CT) were significantly higher than in the unaided reading technique (average, 2.67/CT, P < 0.0001)) and were comparable to FP rates in the SR mode (average, 4.11/CT, P = 0.26) (Tables 3 and 4, Fig. 1).

Fig. 1. Examples of a pulmonary nodule that was misdiagnosed by observers without CAD. (a–c) Male patient, 51-year-old, with metastatic adenocarcinoma of the neck. The nodule (1–2mm in size, red circle) was detected by only one observer without CAD; in the concurrent-reader mode, the tumor was correctly identified by six observers. (d–e) Male patient, 48-year-old, with rectal cancer. (d) Nodule (2mm in size) was detected by CAD. However, it was detected by only one observer without CAD in the low-dose settings; in the concurrent-reader and second-reader mode, the tumor was correctly identified by six observers. (e) Seven months later, the size of the pulmonary nodule (yellow circle) increased obviously.

**Table 3. Results of nodule detection in the unaided reading, the concurrent-reader and second-reader modes of CAD: sensitivity, false positive rate per CT.**
	Sensitivity						False positive rate
	UR	CR	SR	P1	P2	P3	UR	CR	SR	P1	P2	P3
Experienced chest radiologists
1	39.35%	62.13%	65.98%	< 0.0001	< 0.0001	0.081	1.38/ct	3.38/ct	3.425/ct	0.000020	0.000010	0.899782
2	58.88%	70.41%	76.92%	0.009	< 0.0001	0.057	4.23/ct	3.23/ct	4.275/ct	0.017276	0.835722	0.002641
Average of 1–2	49.56%	66.27%	71.45%	< 0.0001	< 0.0001	0.030	2.8/ct	3.3/ct	3.85/ct	0.106124	0.00064	0.020213
Residents
3	39.94%	66.57%	62.43%	< 0.0001	< 0.0001	0.142	2.28/ct	5.05/ct	4.3/ct	< 0.0001	< 0.0001	0.053
4	34.91%	57.69%	67.16%	< 0.0001	< 0.0001	0.05	1.05/ct	2.5/ct	2.8/ct	< 0.0001	< 0.0001	0.751
5	57.69%	59.17%	57.40%	0.541	0.855	0.629	2.6/ct	2.63/ct	2.63/ct	0.977	0.682	0.424
Average of 3–5	44.18%	61.14%	62.33%	< 0.0001	< 0.0001	0.559	1.98/ct	3.39/ct	3.2/ct	< 0.0001	< 0.0001	0.161
Interns
6	29.59%	65.09%	72.49%	< 0.0001	< 0.0001	0.948	1.4/ct	4.7/ct	4.43/ct	< 0.0001	< 0.0001	1.000
7	38.17%	64.79%	63.02%	0.013	0.002	0.307	2.83/ct	4.58/ct	4.63/ct	0.005	.0005	0.860
8	54.73%	72.19%	69.53%	< 0.01	< 0.0001	0.531	5.6/ct	6.68/ct	6.63/ct	0.030	0.035	0.310
Average of 6–8	40.83%	67.36%	68.34%	< 0.0001	< 0.0001	0.480	3.28/ct	5.31/ct	5.23/ct	< 0.0001	< 0.0001	0.263
Average of 1–8	44.27%	64.76%	66.86%	< 0.0001	< 0.0001	0.033	2.67/ct	4.09/ct	4.11/ct	< 0.0001	< 0.0001	0.259521

Notes: UR = unaided reading technique, CR = the concurrent-reader mode, SR = the second-reader mode, P1: UR versus CR, P2: UR versus SR, P3:CR versus SR.

**Table 4. Characteristic of nodules in the unaided reading, the concurrent-reader and second-reader modes of CAD on low-dose CT images.**
		Attenuation			Size range				Location
		Solid	Subsolid	Calcified	≤ 4	(4, 6)	[6, 8]	> 8	Subpleural	Nonsubpleural
Interns	UR	41.87%	27.93%	50.00%	37.40%	55.56%	78.79%	100%	41.85%	40.16%
	CR	67.50%	63.96%	71.67%	65.24%	74.07%	96.97%	100%	61.65%	71.06%
	SR	67.26%	63.96%	91.67%	66.01%	77.78%	100%	100%	65.66%	70.08%
	P1	< 0.0001	< 0.0001	0.017	0	0.012	0.034	1	< 0.0001	< 0.0001
	P2	< 0.0001	< 0.0001	< 0.0001	0	0.001	0.008	1	< 0.0001	< 0.0001
	P3	0.952	1	0.001	0.605	0.527	0.317	1	0.048	0.628
Residents	UR	43.89%	37.84%	60.00%	41.25%	51.85%	81.81%	100%	40.55%	46.57%
	CR	60.74%	54.05%	80.00%	58.64%	72.22%	90.90%	100%	51.74%	67.32%
	SR	61.45%	53.15%	91.67%	59.08%	81.48%	100%	100%	56.14%	66.34%
	P1	< 0.0001	0.001	0.003	< 0.0001	0.028	0.18	1	0.001	< 0.0001
	P2	< 0.0001	0.008	< 0.0001	< 0.0001	< 0.0001	0.014	1	< 0.0001	< 0.0001
	P3	0.712	0.813	0.008	0.936	0.096	0.083	1	0.095	0.984
Experienced radiologists	UR	48.58%	48.65%	57.50%	45.87%	63.89%	86.36%	100%	52.61%	46.81%
	CR	65.66%	66.22%	75.00%	65.02%	66.67%	81.81%	100%	57.09%	72.30%
	SR	69.75%	77.03%	85.00%	69.14%	86.11%	95.45%	100%	69.55%	72.68%
	P1	< 0.0001	0.049	0.09	< 0.0001	0.705	0.655	1	0.416	< 0.0001
	P2	< 0.0001	0.001	0.005	< 0.0001	0.005	0.157	1	0.001	< 0.0001
	P3	0.041	0.021	0.285	0.041	0.008	0.083	1	< 0.0001	0.383
All	UR	44.31%	36.82%	55.63%	40.97%	56.25%	81.81%	100%	44.35%	44.21%
	CR	64.50%	60.81%	75.63%	62.71%	71.53%	90.91%	100%	57.05%	69.76%
	SR	65.70%	63.18%	90%	64.19%	81.25%	98.86%	100%	63.06%	69.33%
	P1	< 0.0001	< 0.0001	< 0.0001	< 0.0001	0.001	0.059	1	< 0.0001	< 0.0001
	P2	< 0.0001	< 0.0001	< 0.0001	< 0.0001	< 0.0001	< 0.0001	1	< 0.0001	< 0.0001
	P3	0.183	0.311	< 0.0001	0.15	0.006	0.008	1	< 0.0001	0.881

Notes: UR = rmunaided reading technique, CR = the concurrent-reader mode, SR = the second-reader mode, P1: UR versus CR, P2: UR versus SR, P3:CR versus SR.

The observer-averaged JAFROC FOM was 0.58 (95% confidence interval: 0.53, 0.64) for the CR mode and 0.48 (95% confidence interval: 0.41, 0.54) for the unaided reading technique, yielding a significant difference (P < 0.0001) (Table 4). The average FOM value in the SR mode was 0.61 (95% confidence interval: 0.55, 0.66),indicating comparability between CR and SR modes (P = 0.259). The alternative FROC curves for all modes are shown in Fig. 2. For experienced radiologists, the FOM was significantly better when using CAD in the CR mode than in the unaided mode (0.61 versus 0.49, respectively; P = 0.0003) and not significantly different from using the SR mode (0.63, P = 0.53); the same trend applies to both residents (0.58 versus 0.48 versus 0.58, respectively) and interns (0.59 versus 0.44 versus 0.60, respectively). Detailed results are provided in Table 5.

Fig. 2. Receiver operating characteristic curves representing detection performances of 8 radiologists in the unaided reading (red line), concurrent-reader CAD (green line), and second-reader CAD (purple line). The area under the receiver operating characteristic curve (AUC) of 0.48 in the unaided reading, 0.58 in concurrent-reader CAD, and 0.61 in second-reader CAD.

**Table 5. Results of nodule detection: figure-of-merit value.**
Groups	UR	CR	SR	P1	P2	P3
Interns	0.41	0.61	0.63	< 0.0001	< 0.0001	0.4987
	0.42	0.58	0.57	< 0.0001	0.0001	0.752
	0.49	0.58	0.60	0.0085	0.003	0.7191
Average	0.44	0.59	0.60	< 0.0001	< 0.0001	0.675
Residents	0.45	0.59	0.56	0.0002	0.0033	0.355
	0.46	0.55	0.63	0.0517	0.0002	0.0564
	0.56	0.58	0.55	0.5509	0.5865	0.2556
Average	0.49	0.58	0.58	0.0016	0.0009	0.8675
Experienced radiologists	0.50	0.58	0.61	0.0176	0.0015	0.3868
	0.48	0.61	0.67	0.001	< 0.0001	0.1507
Average	0.49	0.61	0.63	0.0003	< 0.0001	0.5291
Average of all	0.48	0.58	0.61	< 0.0001	< 0.0001	0.259

Notes: UR = unaided reading technique, CR = the concurrent-reader mode, SR = the second-reader mode, P1: UR versus CR, P2: UR versus SR, P3:CR versus SR.

Reading times

The reading times of the 8 observers in unaided reading, CR and SR reading modes are shown in Table E4. Regardless of the experience level, reading times were significantly shorter in the CR mode(average, 165 ± 133s) than unaided reading(average, 235 ± 162s,P < 0.0001)and SR modes (average, 294 ± 153; P < 0.0001).

4. Discussion

LDCT decreases cumulative radiation exposure while maintaining good diagnostic quality for pulmonary nodules.³⁹ However, the noise interference during LDCT makes the detection of lesions more challenging,⁷ especially smaller pulmonary nodules. The present study aimed to assess the value of a DL-based CAD system in the detection of pulmonary nodules in patients with extra-thoracic malignancies. The study was conducted in SR and CR modes and evaluated SDCT and LDCT images, providing a comprehensive evaluation in a realistic clinical environment. The results showed that the CAD system improved the detection sensitivity on SD and LDCT images across all readers, especially beneficial for interns on reviewing LDCT images. We also addressed the relationship of the diagnostic sensitivity with varied dose and experience levels, specifically the SR mode allowed for comparable sensitivities regardless of the dose and experience level and the CR mode significantly decreased reading times without impacting the sensitivity.

Previous studies have shown that the SR mode increases the diagnostic sensitivity compared to the unaided reading.^{14,21,22,40,41,42} Our results confirmed and further expanded this finding by showing that, in comparison with the unaided reading mode, DL-based CAD improved the sensitivity particularly on nodules with a diameter less than 8mm, a clinically important size that related to malignancy.⁴³ The larger the pulmonary nodules, especially those nodules ≥ 5mm, the higher the probability of pulmonary metastasis; moreover, the larger nodules have more distinct morphological features.⁴⁴ But smaller pulmonary nodules also need more attention so as to detect small pulmonary metastases earlier. While the use of the SR mode on SDCT images is not a new concept, the performance comparison between SD and LDCT images in the SR mode has not been fully evaluated. It is acknowledged that LDCT images contain increased noise and comprised image quality that potentially results in an increased risk of missing pulmonary nodules. Our results confirmed this finding and showed reduced detection sensitivity in LDCT images than that in SDCT images. Whereas, the SR mode yielded comparable sensitivities between SD and LDCT images. The results suggested that SR improved diagnostic performance on LDCT images may potentially address the concern regarding the noise level in LDCT images.

Additionally, the present study demonstrated that the experience level-dependent detection sensitivity of pulmonary nodules was observed among varied reading modes. For experienced radiologists and residents who were bearing higher detection efficiency than interns in the unaided reading technique, the SR mode allowed identified sensitivities across all experience levels at either dose setting. This finding aligned well with the previous study that CAD-assisted diagnosis improves the ability of less-experienced readers, interns particularly in the detection of pulmonary nodules.³⁸ The consistency detection efficiency across all experience and dose levels adopts great significance in real clinical practice.

The prolonged reading time in the SR mode draws our attention by employing the CR mode in the second experimental design. Our results found that the CR mode indeed had decreased reading time than SR and unaided modes while preserving the similar sensitivity and FOM value (based on JAFROC analysis) of the SR mode. Moreover, the sensitivity was found to be independent of experience levels on LDCT images, which is also true to FP rates. According to Beyer et al.,²³ the CR mode had decreased reading times while leading to no improvement in the sensitivity compared to the unaided reading technique. In contrast, Matsumoto et al.²⁴ and Foti et al.⁴⁵ reported that the CR mode did not necessarily correlate with reduced reading time and Foti et al. found that the sensitivity in the CR mode has no significant improvement compared to the unaided reading technique. It shall be noted that authors in studies above^23,24,45 extracted the reading time from the SR mode and used it as the one in the unaided mode, whereas our study performed each experiment separately, the recorded time representing a more realistic reading condition. Shortened interpretation time (while maintaining an equally high sensitivity) may be important in real clinical practice that a large number of cases need read daily by radiologists.

This study indeed has several limitations. Firstly, the reference standard of pulmonary nodules was based on the consensus of radiologists rather than histological evidence, a limitation common to CAD studies. However, biopsying every nodule may not be appropriate clinically, especially for smaller pulmonary nodules, because of the difficulty in obtaining pathological specimens. In addition, it may have exposed patients to unnecessary risks. Further clinical and radiographic follow-up is needed to determine whether these nodules are metastatic pulmonary nodules. Secondly, 90% of nodules studied in the context were found to have a diameter less than 4mm, which lead to lower TP rates and higher FP rates. A more uniform distribution of the nodule size would be appropriate in our further study. Thirdly, the size of reader numbers is small. Studies with larger cohorts of readers are needed to substantiate our results. Lastly, CAD operating systems are not uniform; therefore, our results may not be applicable to other CAD systems.

In conclusion, this study aimed to assess the value of a CAD system based on DL in the detection of pulmonary nodules. We have demonstrated that the deep learning-based CAD system holds a great potential in both SR and CR modes that can help readers especially less experienced radiologists detecting pulmonary nodules on LDCT images. The evaluation of clinical scenarios in this research is closer to the actual clinical application. Results revealed that the deep learning-based CAD system is expected to be an effective method to improve the efficiency and accuracy of radiologists in detecting suspected metastatic pulmonary nodules. In the future, we will study the predictive value of radiomics combined with AI based on DL in lung cancer treatment response, screen benefit cases, and reduce unnecessary waste of medical resources. For example, investigating the performance of radiomics signatures extracted from pretherapy CT images may be important for preoperative predicting PD-1 immunotherapy response.

Conflicts of Interest

The authors declare that there are no conflicts of interest relevant to this paper.

Acknowledgments

The authors express our gratitude to Li Tan and Chenzhe Li for consultation on statistical analysis; Yula Hu, Yueying Pan, Jiahuang, Xinping Zhang, Lisi Liu, Haidan Zhu, Zhaoxia Yang, Jinhan Qiao, Wangzhen Zhang, Chenyong Sun, for reading the images. Qiongjie Hu and Shaofang Wang contributed equally to this work. This study was supported by a grant from the Key Project of Science and Technology Committee of Wuhan, China (Grant No. 2018060401011326).

References

1. L. L. L. Vassallo, A. Traverso, M. Agnello et al., “A cloud-based computer-aided detection system improves identification of lung nodules on computed tomography scans of patients with extra-thoracic malignancies,” Eur. Radiol. 29(1), 144–152 (2019). Crossref, Web of Science, Google Scholar
2. M. J. Wallace, S. Krishnamurthy, L. D. Broemeling et al., “CT-guided percutaneous fine-needle aspiration biopsy of small (< = 1-cm) pulmonary lesions,” Radiology 225(3), 823–828 (2002). Crossref, Web of Science, Google Scholar
3. K. Mori, S. Katayama, A. Okuyama et al., “Evaluation of helical scan CT in the detection of lung nodules of pulmonary metastasis,” Nihon Igaku Hoshasen Gakkai zasshi. Nippon acta radiologica, 53(7), 812–819 (1993). Google Scholar
4. M. M. Hanamiya, T. Aoki, Y. Yamashita et al., “Frequency and significance of pulmonary nodules on thin-section CT in patients with extrapulmonary malignant neoplasms,” Eur. J. Radiol. 81(1), 152–157 (2012). Crossref, Web of Science, Google Scholar
5. N. N. Muangman, N. Maitreesorrasan, K. Totanarungroj, “Comparison of low dose and standard dose MDCT in detection of metastatic pulmonary nodules,” J. Med. Assoc. Thailand 94(2), 215–223 (2011). Google Scholar
6. K. K. Hornbech, J. Ravn, D. A. Steinbruchel, “Outcome after pulmonary metastasectomy analysis of 5 years consecutive surgical resections 2002–2006,” J. Thoracic Oncol. 6(10), 1733–1740 (2011). Crossref, Web of Science, Google Scholar
7. Y. Zhao, G. H. De Bock, R. Vliegenthart et al., “Performance of computer-aided detection of pulmonary nodules in low-dose CT: Comparison with double reading by nodule volume,” Eur. Radiol. 22(10), 2076–2084 (2012). Crossref, Web of Science, Google Scholar
8. A. Hosny, C. Parmar, J. Quackenbush et al., “Artificial intelligence in radiology,” Nat. Rev. Cancer 18(8), 500–510 (2018). Crossref, Web of Science, Google Scholar
9. M. Annarumma, S. J. Withey, R. J. Bakewell et al., “Automated triaging of adult chest radiographs with deep artificial neural networks (vol 291, pg 196, 2019),” Radiology 291(1), 272 (2019). Crossref, Web of Science, Google Scholar
10. R. J. Mcdonald, K. M. Schwartz, L. J. Eckel et al., “The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload,” Acad. Radiol. 22(9), 1191–1198 (2015). Crossref, Web of Science, Google Scholar
11. W. L. Bi, A. Hosny, M. B. Schabath et al., “Artificial intelligence in cancer imaging: Clinical challenges and applications,” Ca-a Cancer J. Clin. 69(2), 127–157 (2019). Crossref, Web of Science, Google Scholar
12. K. N. Jeon, J. M. Goo, C. H. Lee et al., “Computer-aided nodule detection and volumetry to reduce variability between radiologists in the interpretation of lung nodules at low-dose screening computed tomography,” Investig. Radiol. 47(8), 457–461 (2012). Crossref, Web of Science, Google Scholar
13. M. M. Silva, C. M. Schaefer-Prokop, C. Jacobs et al., “Detection of subsolid nodules in lung cancer screening complementary sensitivity of visual reading and computer-aided diagnosis,” Investig. Radiol. 53(8), 441–449 (2018). Crossref, Web of Science, Google Scholar
14. M. Liang, W. Tang, D. M. Xu et al., “Low-dose CT screening for lung cancer: Computer-aided detection of missed lung cancers,” Radiology 281(1), 279–288 (2016). Crossref, Web of Science, Google Scholar
15. E. A. Takahashi, C. W. Koo, D. B. White et al., “Prospective pilot evaluation of radiologists and computer-aided pulmonary nodule detection on ultra-low-dose CT with tin filtration,” J. Thoracic Imag. 33(6), 396–401 (2018). Crossref, Web of Science, Google Scholar
16. Y. Yoshida, T. Sakane, J. Isogai et al., “Computer-assisted detection of metastatic lung tumors on computed tomography,” Asian Cardiovasc. Thorac. Ann. 27(3), 199–207 (2019). Crossref, Google Scholar
17. E.-A. Park, J. M. Goo, J. W. Lee et al., “Efficacy of computer-aided detection system and thin-slab maximum intensity projection technique in the detection of pulmonary nodules in patients with resected metastases,” Invest. Radiol. 44(2), 105–113 (2009). Crossref, Web of Science, Google Scholar
18. A. Schramm, D. Wormanns, G. Leschber et al., “Reliability of a computer-aided detection system in detecting lung metastases compared to manual palpation during surgery,” Interactive Cardiovasc. Thoracic Surg. 12(1), 20–23 (2011). Crossref, Web of Science, Google Scholar
19. M. Miyake, G. Iinuma, S. A. Taylor et al., “Comparative performance of a primary-reader and second-reader paradigm of computer-aided detection for CT colonography in a low-prevalence screening population,” Jpn. J. Radiol. 31(5), 310–319 (2013). Crossref, Web of Science, Google Scholar
20. H. Yoshida, A. H. Dachman, “Computer-aided diagnosis for CT colonography,” Seminars Ultrasound Ct Mri. 25(5), 419–431 (2004). Crossref, Web of Science, Google Scholar
21. M. Yanagawa, O. Honda, S. Yoshida et al., “Commercially available computer-aided detection system for pulmonary nodules on thin-section images using 64 detectors-row CT: Preliminary study of 48 cases,” Acad. Radiol. 16(8), 924–933 (2009). Crossref, Web of Science, Google Scholar
22. G. Iussich, L. Correale, C. Senore et al., “Computer-aided detection for computed tomographic colonography screening a prospective comparison of a double-reading paradigm with first-reader computer-aided detection against second-reader computer-aided detection,” Invest. Radiol. 49(3), 173–182 (2014). Crossref, Web of Science, Google Scholar
23. F. Beyer, L. Zierott, E. M. Fallenberg et al., “Comparison of sensitivity and reading time for the use of computer-aided detection (CAD) of pulmonary nodules at MDCT as concurrent or second reader,” Eur. Radiol. 17(11), 2941–2947 (2007). Crossref, Web of Science, Google Scholar
24. S. Matsumoto, Y. Ohno, T. Aoki et al., “Computer-aided detection of lung nodules on multidetector CT in concurrent-reader and second-reader modes: A comparative study,” Eur. J. Radiol. 82(8), 1332–1337 (2013). Crossref, Web of Science, Google Scholar
25. G. Valeri, S. Cegna, A. Mari et al., “Evaluating the appropriateness of dosimetric indices in body CT,” Radiol. Med. 120(5), 466–473 (2015). Crossref, Web of Science, Google Scholar
26. M. Gabusi, L. Riccardi, C. Aliberti et al., “Radiation dose in chest CT: Assessment of size-specific dose estimates based on water-equivalent correction,” Phys. Med. Eur. J. Med. Phys. 32(2), 393–397 (2016). Web of Science, Google Scholar
27. S. J. Van Riel, C. I. Sanchez, A. A. Bankier et al., “Observer variability for classification of pulmonary nodules on low-dose CT images and its effect on nodule management,” Radiology 277 (3), 863–871 (2015). Crossref, Web of Science, Google Scholar
28. D. P. Naidich, A. A. Bankier, H. Macmahon et al., “Recommendations for the management of subsolid pulmonary nodules detected at CT: A statement from the Fleischner society,” Radiology 266(1), 304–317 (2013). Crossref, Web of Science, Google Scholar
29. D. Manos, J. M. Seely, J. Taylor et al., “The lung reporting and data system (LU-RADS): A proposal for computed tomography screening,” Can. Assoc. Radiol. J. 65(2), 121–134 (2014). Crossref, Web of Science, Google Scholar
30. K. Liu, Q. Li, J. Ma et al., “Evaluating a fully automated pulmonary nodule detection approach and its impact on radiologist performance,” Radiol. Artif. Intell. 1, e180084 (2019). Crossref, Web of Science, Google Scholar
31. T. Kozuka, Y. Matsukubo, T. Kadoba et al., “Efficiency of a computer-aided diagnosis (CAD) system with deep learning in detection of pulmonary nodules on 1-mm-thick images of computed tomography,” Jpn. J. Radiol. 38(11), 1052–1061 (2020). Crossref, Web of Science, Google Scholar
32. K. Li, K. Liu, Y. Zhong et al., “Assessing the predictive accuracy of lung cancer, metastases, and benign lesions using an artificial intelligence-driven computer aided diagnosis system,” Quant. Imag. Med. Surg. 11(8), 3629–3642 (2021). Crossref, Web of Science, Google Scholar
33. H. Wu, W. Tang, C. Wu et al., “Highly robust prediction of lung nodule malignancy by deep learning model: A multiracial, multinational study,” medRxiv: 2020.11.24.20237354 (2020). Google Scholar
34. P. Prakash, M. K. Kalra, J. B. Ackman et al., “Diffuse lung disease: CT of the chest with adaptive statistical iterative reconstruction technique,” Radiology 256(1), 261–269 (2010). Crossref, Web of Science, Google Scholar
35. J. Vikgren, S. Zachrisson, A. Svalkvist et al., “Comparison of chest tomosynthesis and chest radiography for detection of pulmonary nodules: Human observer study of clinical cases,” Radiology 249(3), 1034–1041 (2008). Crossref, Web of Science, Google Scholar
36. J. T. Dobbins3rd, H. P. Mcadams, J. M. Sabol et al., “Multi-institutional evaluation of digital tomosynthesis, dual-energy radiography, and conventional chest radiography for the detection and management of pulmonary nodules,” Radiology 282(1), 236–250 (2017). Crossref, Web of Science, Google Scholar
37. D. P. Chakraborty, K. S. Berbaum, “Observer studies involving detection and localization: Modeling, analysis, and validation,” Med. Phys. 31(8), 2313–2330 (2004). Crossref, Web of Science, Google Scholar
38. Y. Yamada, E. Shiomi, M. Hashimoto et al., “Value of a computer-aided detection system based on chest tomosynthesis imaging for the detection of pulmonary nodules,” Radiology 287(1), 333–339 (2018). Crossref, Web of Science, Google Scholar
39. C. Rampinelli, D. Origgi, M. Bellomi, “Low-dose CT: Technique, reading methods and image interpretation,” Cancer Imag. 12(3), 548–556 (2013). Crossref, Web of Science, Google Scholar
40. G. D. Rubin, J. K. Lyo, D. S. Paik et al., “Pulmonary nodules on multi-detector row CT scans: Performance comparison of radiologists and computer-aided detection,” Radiology 234(1), 274–283 (2005). Crossref, Web of Science, Google Scholar
41. M. Das, G. Muehlenbruch, A. H. Mahnken et al., “Small pulmonary nodules: Effect of two computer-aided detection systems on radiologist performance,” Radiology 241(2), 564–571 (2006). Crossref, Web of Science, Google Scholar
42. X. Wang, J. Guo, D. Gu et al., “Tracking knowledge evolution, hotspots and future directions of emerging technologies in cancers research: A bibliometrics review,” J. Cancer 10(12), 2643–2653 (2019). Crossref, Web of Science, Google Scholar
43. J. Bueno, L. Landeras, J. H. Chung, “Updated Fleischner society guidelines for managing incidental pulmonary nodules: Common questions and challenging scenarios,” Radiographics 38(5), 1337 (2018). Crossref, Web of Science, Google Scholar
44. Q. Yang, Y. Wang, X. Ban et al., “Prediction of pulmonary metastasis in pulmonary nodules (ae¡currency¿10mm) detected in patients with primary extrapulmonary malignancy at thin-section staging CT,” Radiol. Med. 122(11), 837–849 (2017). Crossref, Web of Science, Google Scholar
45. G. Foti, N. Faccioli, M. D’onofrio et al., “Evaluation of a method of computer-aided detection (CAD) of pulmonary nodules at computed tomography,” Radiol. Med. 115(6), 950–961 (2010). Crossref, Web of Science, Google Scholar

Vol. 15, No. 02

Supplemental Materials

Metrics

Downloaded 877 times

History

Received 9 September 2021

Accepted 29 November 2021

Published: 18 February 2022

Information

This is an Open Access article. It is distributed under the terms of the Creative Commons Attribution 4.0 (CC-BY) License. Further distribution of this work is permitted, provided the original work is properly cited.

Keywords

PDF download