Performance of deep neural network-based artificial intelligence method in diabetic retinopathy screening: a systematic review and meta-analysis of diagnostic test accuracy

in European Journal of Endocrinology
View More View Less
  • 1 Eight-year Program of Clinical Medicine, Peking Union Medical College Hospital (PUMCH), Chinese Academe of Medical Sciences & Peking Union Medical College (CAMS & PUMC), Beijing, China
  • 2 Medical Research Center, PUMCH, CAMS & PUMC, Beijing, China
  • 3 Department of Endocrinology, Key Laboratory of Endocrinology of National Health Commission, PUMCH, CAMS & PUMC, Beijing, China
  • 4 School of Software Engineering, Beijing University of Technology, Beijing, China
  • 5 Research Institute of Information and Technology, Tsinghua University, Beijing, China

Correspondence should be addressed to S Chen or H Pan; Email: cs0083@126.com or panhui20111111@163.com

*(S Wang, Y Zhang and S Lei contributed equally to this work)

Restricted access

Objective

Automatic diabetic retinopathy screening system based on neural networks has been used to detect diabetic retinopathy (DR). However, there is no quantitative synthesis of performance of these methods. We aimed to estimate the sensitivity and specificity of neural networks in DR grading.

Methods

Medline, Embase, IEEE Xplore, and Cochrane Library were searched up to 23 July 2019. Studies that evaluated performance of neural networks in detection of moderate or worse DR or diabetic macular edema using retinal fundus images with ophthalmologists’ judgment as reference standard were included. Two reviewers extracted data independently. Risk of bias of eligible studies was assessed using QUDAS-2 tool.

Results

Twenty-four studies involving 235 235 subjects were included. Quantitative random-effects meta-analysis using the Rutter and Gatsonis hierarchical summary receiver operating characteristics (HSROC) model revealed a pooled sensitivity of 91.9% (95% CI: 89.6% to 94.3%) and specificity of 91.3% (95% CI: 89.0% to 93.5%). Subgroup analyses and meta-regression did not provide any statistically significant findings for the heterogeneous diagnostic accuracy in studies with different image resolutions, sample sizes of training sets, architecture of convolutional neural networks, or diagnostic criteria.

Conclusions

State-of-the-art neural networks could effectively detect clinical significant DR. To further improve diagnostic accuracy of neural networks, researchers might need to develop new algorithms rather than simply enlarge sample sizes of training sets or optimize image quality.

Supplementary Materials

    • Supplementary Figure 1
    • Table S1 Search strategy (Medline)
    • Table S2 Characteristics of included studies
    • Table S3 Methodological quality assessment of included studies using QUADAS-2

 

     European Society of Endocrinology

Sept 2018 onwards Past Year Past 30 Days
Abstract Views 2030 2030 360
Full Text Views 81 81 10
PDF Downloads 74 74 11
  • 1

    Cheung N, Mitchell P, Wong TY. Diabetic retinopathy. Lancet 2010 376 124136. (https://doi.org/10.1016/S0140-6736(09)62124-3)

  • 2

    Leasher JL, Bourne RR, Flaxman SR, Jonas JB, Keeffe J, Naidoo K, Pesudovs K, Price H, White RA & Wong TY Global estimates on the number of people blind or visually impaired by diabetic retinopathy: a meta-analysis from 1990 to 2010. Diabetes Care 2016 39 16431649. (https://doi.org/10.2337/dc15-2171)

    • Search Google Scholar
    • Export Citation
  • 3

    Wild S, Roglic G, Green A, Sicree R, King H. Global prevalence of diabetes: estimates for the year 2000 and projections for 2030. Diabetes Care 2004 27 10471053. (https://doi.org/10.2337/diacare.27.5.1047)

    • Search Google Scholar
    • Export Citation
  • 4

    Tozer K, Woodward MA, Newman-Casey PA. Telemedicine and diabetic retinopathy: review of published screening programs. Journal of Endocrinology and Diabetes 2015 2. (https://doi.org/10.15226/2374-6890/2/4/00131)

    • Search Google Scholar
    • Export Citation
  • 5

    Antonetti DA, Klein R, Gardner TW. Diabetic retinopathy. New England Journal of Medicine 2012 366 12271239. (https://doi.org/10.1056/NEJMra1005073)

    • Search Google Scholar
    • Export Citation
  • 6

    Murthy KR, Murthy PR, Kapur A, Owens DR. Mobile diabetes eye care: experience in developing countries. Diabetes Research and Clinical Practice 2012 97 343349. (https://doi.org/10.1016/j.diabres.2012.04.025)

    • Search Google Scholar
    • Export Citation
  • 7

    Mookiah MRK, Acharya UR, Chua CK, Lim CM, Ng EYK, Laude A. Computer-aided diagnosis of diabetic retinopathy: a review. Computers in Biology and Medicine 2013 43 21362155. (https://doi.org/10.1016/j.compbiomed.2013.10.007)

    • Search Google Scholar
    • Export Citation
  • 8

    Hinton GE, Salakhutdinov RR. Reducing the dimensionality of data with neural networks. Science 2006 313 504507. (https://doi.org/10.1126/science.1127647)

    • Search Google Scholar
    • Export Citation
  • 9

    Li B, Li HK. Automated analysis of diabetic retinopathy images: principles, recent developments, and emerging trends. Current Diabetes Reports 2013 13 453459. (https://doi.org/10.1007/s11892-013-0393-9)

    • Search Google Scholar
    • Export Citation
  • 10

    Ting DS, Cheung GC, Wong TY. Diabetic retinopathy: global prevalence, major risk factors, screening practices and public health challenges: a review. Clinical and Experimental Ophthalmology 2016 44 260277. (https://doi.org/10.1111/ceo.12696)

    • Search Google Scholar
    • Export Citation
  • 11

    A S, S S. Unravelling diabetic retinopathy through image processing, neural networks, and fuzzy logic: a review. Asian Journal of Pharmaceutical and Clinical Research 2017 10 3237. (https://doi.org/10.22159/ajpcr.2017.v10i4.17023)

    • Search Google Scholar
    • Export Citation
  • 12

    Halevy A, Norvig P, Pereira F. The unreasonable effectiveness of data. IEEE Intelligent Systems 2009 24 812. (https://doi.org/10.1109/MIS.2009.36)

    • Search Google Scholar
    • Export Citation
  • 13

    Photocoagulation for diabetic macular edema: Early Treatment Diabetic Retinopathy Study Report Number 1 Early Treatment Diabetic Retinopathy Study Research Group. Archives of Ophthalmology 1985 103 17961806.

    • Search Google Scholar
    • Export Citation
  • 14

    Quellec G, Charrière K, Boudi Y, Cochener B, Lamard M. Deep image mining for diabetic retinopathy screening. Medical Image Analysis 2017 39 178193. (https://doi.org/10.1016/j.media.2017.04.012)

    • Search Google Scholar
    • Export Citation
  • 15

    Voets M, Mollersen K, Bongo LA. Reproduction study using public data of: development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. PLoS One 2019 14 e0217541. (https://doi.org/10.1371/journal.pone.0217541)

    • Search Google Scholar
    • Export Citation
  • 16

    Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T & Cuadros J Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 2016 316 24022410. (https://doi.org/10.1001/jama.2016.17216)

    • Search Google Scholar
    • Export Citation
  • 17

    van der Heijden AA, Abramoff MD, Verbraak F, van Hecke MV, Liem A, Nijpels G. Validation of automated screening for referable diabetic retinopathy with the IDx-DR device in the Hoorn Diabetes Care System. Acta Ophthalmologica 2018 96 6368. (https://doi.org/10.1111/aos.13613)

    • Search Google Scholar
    • Export Citation
  • 18

    Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM & QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine 2011 155 529536. (https://doi.org/10.7326/0003-4819-155-8-201110180-00009)

    • Search Google Scholar
    • Export Citation
  • 19

    Deeks JJ, Bossuyt PM, Gatsonis C. Handbook for DTA Reviews, Cochrane Collaboration, London. 2011.

  • 20

    Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Statistics in Medicine 1993 12 12931316. (https://doi.org/10.1002/sim.4780121403)

    • Search Google Scholar
    • Export Citation
  • 21

    de la Torre J, Valls A, Puig D. A deep learning interpretable classifier for diabetic retinopathy disease grading. Neurocomputing 2019 In press. (https://doi.org/10.1016/j.neucom.2018.07.102)

    • Search Google Scholar
    • Export Citation
  • 22

    Gulshan V, Rajan RP, Widner K, Wu D, Wubbels P, Rhodes T, Whitehouse K, Coram M, Corrado G & Ramasamy K Performance of a deep-learning algorithm vs manual grading for detecting diabetic retinopathy in India. JAMA Ophthalmology 2019 137 987-993. (https://doi.org/10.1001/jamaophthalmol.2019.2004)

    • Search Google Scholar
    • Export Citation
  • 23

    Ramachandran N, Hong SC, Sime MJ, Wilson GA. Diabetic retinopathy screening using deep neural network. Clinical and Experimental Ophthalmology 2018 46 412416. (https://doi.org/10.1111/ceo.13056)

    • Search Google Scholar
    • Export Citation
  • 24

    Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, Niemeijer M. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Investigative Ophthalmology and Visual Science 2016 57 52005206. (https://doi.org/10.1167/iovs.16-19964)

    • Search Google Scholar
    • Export Citation
  • 25

    Al-Jarrah MA, Shatnawi H. Non-proliferative diabetic retinopathy symptoms detection and classification using neural network. Journal of Medical Engineering and Technology 2017 41 498505. (https://doi.org/10.1080/03091902.2017.1358772)

    • Search Google Scholar
    • Export Citation
  • 26

    Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, Peng L, Webster DR. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 2018 125 12641272. (https://doi.org/10.1016/j.ophtha.2018.01.034)

    • Search Google Scholar
    • Export Citation
  • 27

    Kwasigroch A, Jarzembinski B & Grochowski M (eds). Deep CNN based decision support system for detection and assessing the stage of diabetic retinopathy. In 2018 International Interdisciplinary PhD Workshop (IIPhDW), 9–12 May 2018, 2018. (https://doi.org/10.1109/IIPHDW.2018.8388337)

    • Search Google Scholar
    • Export Citation
  • 28

    Lam C, Yi D, Guo M, Lindsey T. Automated detection of diabetic retinopathy using deep learning. AMIA Joint Summits on Translational Science 2018 2017 147155.

    • Search Google Scholar
    • Export Citation
  • 29

    Bhaskaranand M, Ramachandra C, Bhat S, Cuadros J, Nittala MG, Sadda S, Solanki K. The value of automated diabetic retinopathy screening with the EyeArt system: a study of more than 100,000 consecutive encounters from people with diabetes. Diabetes Technology and Therapeutics 2019 21 635643.(https://doi.org/10.1089/dia.2019.0164)

    • Search Google Scholar
    • Export Citation
  • 30

    Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digital Medicine 2018 1 39. (https://doi.org/10.1038/s41746-018-0040-6)

    • Search Google Scholar
    • Export Citation
  • 31

    Kanagasingam Y, Xiao D, Vignarajan J, Preetham A, Tay-Kearney ML, Mehrotra A. Evaluation of artificial intelligence-based grading of diabetic retinopathy in primary care. JAMA Network Open 2018 1 e182665. (https://doi.org/10.1001/jamanetworkopen.2018.2665)

    • Search Google Scholar
    • Export Citation
  • 32

    Sahlsten J, Jaskari J, Kivinen J, Turunen L, Jaanio E, Hietala K, Kaski K. Deep learning fundus image analysis for diabetic retinopathy and macular edema grading. Scientific Reports 2019 9 10750. (https://doi.org/10.1038/s41598-019-47181-w)

    • Search Google Scholar
    • Export Citation
  • 33

    Ting DSW, Cheung CY, Lim G, Tan GSW, Quang ND, Gan A, Hamzah H, Garcia-Franco R, San Yeo IY & Lee SY Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 2017 318 22112223. (https://doi.org/10.1001/jama.2017.18152)

    • Search Google Scholar
    • Export Citation
  • 34

    Verbraak FD, Abramoff MD, Bausch GCF, Klaver C, Nijpels G, Schlingemann RO, van der Heijden AA. Diagnostic accuracy of a device for the automated detection of diabetic retinopathy in a primary care setting. Diabetes Care 2019 42 651656. (https://doi.org/10.2337/dc18-0148)

    • Search Google Scholar
    • Export Citation
  • 35

    Yang WH, Zheng B, Wu MN, Zhu SJ, Fei FQ, Weng M, Zhang X, Lu PR. An evaluation system of fundus photograph-based intelligent diagnostic technology for diabetic retinopathy and applicability for research. Diabetes Therapy 2019 10 18111822. (https://doi.org/10.1007/s13300-019-0652-0)

    • Search Google Scholar
    • Export Citation
  • 36

    Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, Tan GSW, Schmetterer L, Keane PA, Wong TY. Artificial intelligence and deep learning in ophthalmology. British Journal of Ophthalmology 2018 103 167175. (https://doi.org/10.1136/bjophthalmol-2018-313173)

    • Search Google Scholar
    • Export Citation
  • 37

    Rajpara SM, Botello AP, Townend J, Ormerod AD. Systematic review of dermoscopy and digital dermoscopy/artificial intelligence for the diagnosis of melanoma. British Journal of Dermatology 2009 161 591604. (https://doi.org/10.1111/j.1365-2133.2009.09093.x)

    • Search Google Scholar
    • Export Citation
  • 38

    Dorrius MD, Jansen-van der Weide MC, van Ooijen PM, Pijnappel RM, Oudkerk M. Computer-aided detection in breast MRI: a systematic review and meta-analysis. European Radiology 2011 21 16001608. (https://doi.org/10.1007/s00330-011-2091-9)

    • Search Google Scholar
    • Export Citation
  • 39

    Noble M, Bruening W, Uhl S, Schoelles K. Computer-aided detection mammography for breast cancer screening: systematic review and meta-analysis. Archives of Gynecology and Obstetrics 2009 279 881890. (https://doi.org/10.1007/s00404-008-0841-y)

    • Search Google Scholar
    • Export Citation
  • 40

    He J, Baxter SL, Xu J, Xu J, Zhou X, Zhang K. The practical implementation of artificial intelligence technologies in medicine. Nature Medicine 2019 25 3036. (https://doi.org/10.1038/s41591-018-0307-0)

    • Search Google Scholar
    • Export Citation
  • 41

    Raman B, Bursell ES, Wilson M, Zamora G, Benche I, Nemeth SC, Soliz P. The effects of spatial resolution on an automated diabetic retinopathy screening system’s performance in detecting microaneurysms for diabetic retinopathy. Eds Long R, Antani S, Lee DJ, Nutter B & Zhang M, pp. 128133, 2004. Proceedings 17th IEEE Symposiumon Computer-Based Medical Systems; 2004 25-25 June 2004.

    • Search Google Scholar
    • Export Citation
  • 42

    Thapa D, Raahemifar K, Bobier WR, Lakshminarayanan V. Comparison of super-resolution algorithms applied to retinal images. Journal of Biomedical Optics 2014 19 056002. (https://doi.org/10.1117/1.JBO.19.5.056002)

    • Search Google Scholar
    • Export Citation
  • 43

    Andreu Y, Lopez-Centelles J, Mollineda RA & Garcia-Sevilla P. IEEE. Analysis of the Effect of Image Resolution on Automatic Face Gender Classification. In 2014 22nd International Conference on Pattern Recognition, pp. 273278, 2014. (https://doi.org/10.1109/ICPR.2014.56)

    • Search Google Scholar
    • Export Citation
  • 44

    Pauli TW, Gangaputra S, Hubbard LD, Thayer DW, Chandler CS, Peng Q, Narkar A, Ferrier NJ, Danis RP. Effect of image compression and resolution on retinal vascular caliber. Investigative Ophthalmology and Visual Science 2012 53 51175123. (https://doi.org/10.1167/iovs.12-9643)

    • Search Google Scholar
    • Export Citation
  • 45

    Sim DA, Keane PA, Tufail A, Egan CA, Aiello LP, Silva PS. Automated retinal image analysis for diabetic retinopathy in telemedicine. Current Diabetes Reports 2015 15 14. (https://doi.org/10.1007/s11892-015-0577-6)

    • Search Google Scholar
    • Export Citation
  • 46

    Panwar N, Huang P, Lee J, Keane PA, Chuan TS, Richhariya A, Teoh S, Lim TH, Agrawal R. Fundus Photography in the 21st Century – a review of recent technological advances and their implications for worldwide healthcare. Telemedicine Journal and e-Health 2016 22 198208. (https://doi.org/10.1089/tmj.2015.0068)

    • Search Google Scholar
    • Export Citation
  • 47

    Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J. Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Transactions on Medical Imaging 2016 35 12991312. (https://doi.org/10.1109/TMI.2016.2535302)

    • Search Google Scholar
    • Export Citation
  • 48

    Wu L, Fernandez-Loaiza P, Sauma J, Hernandez-Bogantes E, Masis M. Classification of diabetic retinopathy and diabetic macular edema. World Journal of Diabetes 2013 4 290294. (https://doi.org/10.4239/wjd.v4.i6.290)

    • Search Google Scholar
    • Export Citation
  • 49

    Xie Q, Hov E, Luong M-T, Le QV. Self-training with Noisy Student improves ImageNet classification, 2019. (available at: arXiv:1911.04252)

  • 50

    Browning DJ, Fraser CM, Clark S. The relationship of macular thickness to clinically graded diabetic retinopathy severity in eyes without clinically detected diabetic macular edema. Ophthalmology 2008 115 533539.e2. (https://doi.org/10.1016/j.ophtha.2007.06.042)

    • Search Google Scholar
    • Export Citation
  • 51

    Browning DJ, Stewart MW, Lee C. Diabetic macular edema: evidence-based management. Indian Journal of Ophthalmology 2018 66 17361750. (https://doi.org/10.4103/ijo.IJO_1240_18)

    • Search Google Scholar
    • Export Citation
  • 52

    Hassan B, Hassan T, Li B, Ahmed R, Hassan O. Deep ensemble learning based objective grading of macular edema by extracting clinically significant findings from fused retinal imaging modalities. Sensors 2019 19 E2970. (https://doi.org/10.3390/s19132970)

    • Search Google Scholar
    • Export Citation
  • 53

    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015 521 436444. (https://doi.org/10.1038/nature14539)

  • 54

    Richardson M, Garner P, Donegan S. Interpretation of subgroup analyses in systematic reviews: a tutorial. Clinical Epidemiology and Global Health 2019 7 192198. (https://doi.org/10.1016/j.cegh.2018.05.005)

    • Search Google Scholar
    • Export Citation
  • 55

    Niu YH, Gu L, Lu F, Lv FF, Wang ZJ, Sato I, Zhang ZJ, Xiao YY, Dai XZ, Cheng TT. Pathological Evidence Exploration in Deep Retinal Image Diagnosis, pp. 10931101. Palo Alto: Association Advancement Artificial Intelligence, 2019.

    • Search Google Scholar
    • Export Citation