Explainable Deep Learning Approach for Covert Narcissism Detection with Cross-Domain Generalization

Abstract

Deep learning classifiers for covert narcissism detection have begun to reach usable in-domain accuracy, but two questions remain largely untested: do their explanations genuinely reflect the model's reasoning, and do accuracy, explanations, and fairness hold up outside the training data? This paper answers both questions for a MentalBERT-based detector trained on a 5,945-sentence taxonomy-grounded corpus. The detector reached 94.87% macro-F1 on a random split, 84.23% on held-out categories, and 92.20% under a length-matched control, all well above a length-only baseline, showing it learns the construct rather than a shortcut. Three explanation methods, SHAP, LIME, and Integrated Gradients, agreed moderately with each other (Jaccard@10 = 0.42-0.58) and stayed stable under paraphrasing (0.68), so the explanations look plausible. A stricter, causal test told a different story: removing the words these methods flagged as important changed the model's confidence no more than removing random words did (SHAP: 0.287 vs. 0.629 random; Integrated Gradients: 0.302 vs. 0.586 random). Plausible explanations, in other words, were not faithful ones. Testing the detector on two labelled corpora (Jigsaw, Davidson) showed it confusing general hostility with covert narcissism (53.0% accuracy, 17.4% precision at 96.1% recall on Jigsaw). On unlabelled real-world text (MBTI forum posts, Reddit/PANDORA), it flagged 66% and 81% of unrelated content as narcissistic, against a 50.1% in-domain rate, a shortcut traced to the corpus's fully LLM-generated writing style. Four fixes were tested: adding real-world negative examples during training eliminated most false positives (66%→0%; 81%→10.5%) without hurting accuracy; further pretraining on real text barely helped; confidence recalibration improved honesty but not correctness; and an anomaly detector caught some, not all, failures. The lesson: strong accuracy and convincing explanations do not prove a model is trustworthy, and fixing the training data mattered far more than adjusting the model around it.