CNNs vs. Transformers: Performance and Robustness in Endoscopic Image Analysis

Carolus H. J. Kusters, Tim G. W. Boers, Tim J. M. Jaspers, Jelmer B. Jukema, Martijn R. Jong, Kiki N. Fockens, Albert J. de Groof, Jacques J. Bergman, Fons van der Sommen, Peter H. N. de With

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

1 Citation (Scopus)


In endoscopy, imaging conditions are often challenging due to organ movement, user dependence, fluctuations in video quality and real-time processing, which pose requirements on the performance, robustness and complexity of computer-based analysis techniques. This paper poses the question whether Transformer-based architectures, which are capable to directly capture global contextual information, can handle the aforementioned endoscopic conditions and even outperform the established Convolutional Neural Networks (CNNs) for this task. To this end, we evaluate and compare clinically relevant performance and robustness of CNNs and Transformers for neoplasia detection in Barrett’s esophagus. We have selected several top performing CNN and Transformers on endoscopic benchmarks, which we have trained and validated on a total of 10,208 images (2,079 patients), and tested on a total of 4,661 images (743 patients), divided over a high-quality test set and three different robustness test sets. Our results show that Transformers generally perform better on classification and segmentation for the high-quality challenging test set, and show on-par or increased robustness to various clinically relevant input data variations, while requiring comparable model complexity. This robustness against challenging video-related conditions and equipment variations over the hospitals is an essential trait for adoption in clinical practice. The code is made publicly available at:
Original languageEnglish
Title of host publicationApplications of Medical Artificial Intelligence - 2nd International Workshop, AMAI 2023, Held in Conjunction with MICCAI 2023, Proceedings
EditorsShandong Wu, Behrouz Shabestari, Lei Xing
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages11
Volume14313 LNCS
ISBN (Print)9783031470752
Publication statusPublished - 2024
Event2nd International Workshop on Applications of Medical Artificial Intelligence, AMAI 2023 - Vancouver, Canada
Duration: 8 Oct 20238 Oct 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14313 LNCS


Conference2nd International Workshop on Applications of Medical Artificial Intelligence, AMAI 2023


  • Barrett’s Esophagus
  • CNN
  • Robustness
  • Transformers

Cite this