Hybrid Deep Learning and Random Forest Framework for Multi-Class Skin Cancer Classification Using DenseNet121, InceptionV3, and U-Net Segmentation

Abstract

Skin cancer is among the most frequently diagnosed cancers worldwide, and early, accurate identification of malignant lesions is critical for improving patient survival and treatment outcomes. Manual dermoscopic evaluation is time-consuming, requires specialist expertise, and is subject to inter-observer variability, which motivates the development of automated computer-aided diagnosis (CAD) systems. This paper proposes a hybrid classification framework for seven-class skin lesion diagnosis that integrates classical image pre-processing (boundary localization, cropping and resizing, and normalization), U-Net-based lesion segmentation, dual convolutional neural network (CNN) feature extraction using DenseNet121 and InceptionV3, and a Random Forest ensemble classifier trained on the fused deep feature representation. The framework is designed around the HAM10000 dermatoscopic image dataset, spanning seven diagnostic categories: actinic keratoses (AKIEC), basal cell carcinoma (BCC), benign keratosis-like lesions (BKL), dermatofibroma (DF), melanoma (MEL), melanocytic nevi (NV), and vascular lesions (VASC). The proposed pipeline is evaluated using accuracy, precision, recall, F-measure, and receiver operating characteristic area-under-curve (ROC-AUC), and is compared against standalone DenseNet121 and InceptionV3 baselines as well as an end-to-end deep learning classification head trained on the same fused features. Results indicate that combining deep CNN feature extraction with an ensemble Random Forest classifier improves overall classification robustness relative to single-backbone deep learning models, particularly under the class imbalance that characterizes dermatoscopic datasets. The paper further discusses the architectural rationale, implementation details, and limitations of the proposed system, and positions the findings relative to comparable studies reported in the literature.