Robust OCTA Vessel Segmentation for Early Detection of Neurodegenerative Disorders Using Multi-Scale CNN and Transformer Networks

Abstract

Early detection of vascular neurodegenerative diseases relies on precise segmentation of retinal vessels in Optical Coherence Tomography Angiography (OCTA) images. However, complex vascular structures, low-contrast micro-vessels, and overlapping vessel types pose significant challenges to conventional segmentation methods. This study proposes a deep learning architecture named CGOctaNet that combines convolutional neural networks (CNNs) with a transformer-based global context modeling framework for robust OCTA vessel segmentation. The model employs a U-Net-like encoder–decoder structure integrated with three modules: (i) a Shallow Feature Extraction Module (SFEM) to preserve vessel boundaries; (ii) a multi-scale convolution encoder for local geometric feature learning; and (iii) a Transformer bottleneck that captures long-range dependencies and inter-vessel relationships for enhanced structural consistency. The Transformer bottleneck enables awareness of global context by learning the spatial relationships between remote areas of the vessels, thereby complementing the small receptive field of CNNs and enhancing segmentation continuity in the presence of noise. This hybrid design achieves better generalization and fine-grained segmentation accuracy than CNN-only models in the past. Experiments on OCTA-500 and ROSE benchmark datasets demonstrate that CGOctaNet outperforms state-of-the-art methods, achieving Dice scores of 93.50%, 90.25%, and 89.50% on OCTA-3M, OCTA-6M, and ROSE datasets, respectively. The improvements are attributed to effective integration of local and global contextual cues, adaptive attention refinement, and balanced optimization through hybrid Dice-Cross Entropy loss.