Attention-Enhanced Multi-Modal Deep Learning for Explainable Pox and Skin Lesion Classification

Abstract

The identification of Monkeypox remains difficult because its symptoms duplicate the symptoms of various skin conditions and current automatic diagnostic systems show interpretability problems. The research introduces a deep learning system which combines attention-based convolutional networks with transparent artificial intelligence methods to achieve multi-skin lesion classification. The system used a multi-modal dataset which included both lesion images and structured patient metadata that contained age, gender, geographic region, and lesion location information to identify six different categories which included Chickenpox, Measles, Monkeypox, Normal, Smallpox, and Unknown. The research studied multiple deep learning framework, which included ConvNeXt-Tiny, EfficientNetV2-S, RegNetY, VGG16, Inception-based networks, a hybrid MobileNet–LSTM model, and a Deep Belief Network as baselines. The best results for ConvNeXt-Tiny were achieved when the model used Convolutional Block Attention Module (CBAM) with AdamW training at 1×10⁻⁴ learning rate. The research team used fixed train–test splits for their initial experiments, which resulted in almost perfect classification results while the 5-fold cross-validation process proved the model's strength by achieving 99% accuracy, precision, recall, and F1-score across all classes. The stacking-based ensemble classifier used patient metadata to enhance its ability to predict structured clinical attributes. The Grad-CAM method highlighted key lesion areas, enhancing medical professionals' understanding of results. The web-based system, developed with Flask and MySQL, outperformed baseline models in accuracy, reliability, and transparency. It provides real-time image diagnostics and disease predictions, offering a reliable, interpretable tool for automated Monkeypox detection.