Integrating Advanced CNN And BERT Models For Enhanced Visual Question Answering With Attention Mechanisms

Abstract

This research advances the field of Visual Question Answering (VQA) by integrating cutting-edge Convolutional Neural Network (CNN) models, including Vision Transformer (VIT), with attention mechanisms for superior image feature extraction. For text-based features, we employ BERT models optimized for medical applications, combined with Multimodal Compact Bilinear (MCB) pooling and various concatenation techniques. The objective of our research is to analyze the effectiveness of various Convolutional Neural Network (CNN) structures and the influence of attention processes on the performance of Visual Question Answering (VQA) systems. The results of our study indicate that the inclusion of attention layers greatly improves the model's capacity to handle intricate visual and textual information. The combination of the Vision Transformer (VIT) and BERT yielded the maximum performance, with an accuracy of 91.3% and an F1 score of 90.4% for closed-ended questions, and an accuracy of 89.4% and an F1 score of 88.5% for open-ended questions. Ablation studies confirmed the critical role of attention mechanisms and BERT embeddings in achieving high performance. Learning curves indicated effective prevention of overfitting, and confusion matrices highlighted specific areas for improvement, enhancing overall model reliability. Precision-recall curves illustrated a balanced performance, crucial for minimizing errors in medical applications. This research underscores the potential of integrating advanced deep learning techniques to refine VQA systems, leading to more accurate and contextually relevant answers, particularly in the medical domain.