EffNetEye: A Multimodal Fusion Model for Multiclass Classification of Retinal Diseases

Abstract

The Eye is the most sensitive human organ; if affected by any disease, it can hinder the individual’s quality of life. Some Retinal diseases, such as Epiretinal Membrane (ERM), Age-related Macular Degeneration (AMD), glaucoma, and Retinal Vein Occlusion (RVO), are major contributors to eyesight loss. Timely detection of such diseases is essential for successful treatment. A novel model called ‘EffNetEye has been proposed for the classification and diagnosis of retinal diseases by combining two modalities, fundus and OCT. The suggested model provides a simple but effective dual-modality feature-level fusion approach. It uses two EfficientNetB0 backbone networks to extract features from each modality and classifies four retinal diseases: AMD, ERM, RVO, and Normal. None of the studies on multimodal approaches included ERM disease, which distinguishes this model from the existing multimodal approaches. A total of 5,484 image datasets were constructed from three publicly available datasets: OIA-ODIR, RFMid, and OCTDL. Different preprocessing steps are applied to each modality image to address the domain differences among the three datasets. OCT images were preprocessed with Wiener filtering to reduce speckle noise and to improve local contrast in fundus images; the CLAHE technique was applied. Additionally, data augmentation was applied to address class imbalance in the dataset. The model was trained on a combined training and validation dataset and evaluated using 5-fold stratified cross-validation to ensure consistency and eliminate bias. The use of Grad-CAM demonstrated the model’s ability to highlight clinically relevant features in both fundus and OCT scans during prediction. Finally, the model was tested on an independent test set, which showed strong classification performance, achieving an accuracy of 94.2% and a high AUC of 99.99%.