Evaluating Explainable Artificial Intelligence (XAI) in the Network Security Domain A Case Study

Abstract

The rapid expansion of Internet of Things infrastructure has fundamentally altered the network security threat landscape, creating environments in which the volume, velocity, and diversity of malicious traffic exceed the capacity of rule-based detection systems and human analyst teams alike. Machine learning offers a powerful alternative, but the models that achieve the highest detection rates — deep neural networks and large ensemble architectures — are precisely those whose internal reasoning is most opaque, creating a critical gap between predictive capability and operational interpretability that undermines the trust and accountability requirements of real security operations. This thesis investigates whether explainable artificial intelligence can bridge that gap in an IoT network intrusion detection context. A comparative experimental framework was constructed in which three architecturally distinct models — a Deep Neural Network, a Bagging ensemble based on Random Forest, and a two-level Stacking ensemble — were trained and evaluated on the CIC IoT-DIAD 2024 dataset, comprising 27,713,532 labelled network flow records described by 36 traffic-derived features. Each model was assessed not only on standard detection metrics but on the quality, consistency, and operational interpretability of the SHAP attributions it produced, using architecture-appropriate explainer classes: TreeExplainer for Bagging, DeepExplainer for DNN, and PermutationExplainer for Stacking. All three architectures exceeded 97% accuracy and 97% F1-score on the balanced held-out test set. The Bagging ensemble achieved the highest precision (99.76%) and the lowest false positive count (188), while the Stacking ensemble achieved the highest recall (96.71%) and the lowest false negative count (2,622). SHAP analysis identified three features — SYN Flag Count, Bwd Packet Length Max, and Flow IAT Min — that appeared consistently in the top 10 rankings of all three architectures, providing a cross-model consensus that is both statistically robust and operationally interpretable in terms of known IoT attack mechanisms. Spearman rank correlation analysis revealed that attribution consistency is architecture-dependent: the tree-based Bagging and Stacking models produce strongly correlated rankings (ρ = 0.7521), while the DNN's attribution profile diverges significantly from both ensembles (ρ ≈ 0.12–0.20). The Bagging ensemble is identified as the recommended architecture for the evaluated scenario, optimising across detection reliability, explanation exactness, computational feasibility, and cross-model attribution consistency simultaneously. The study demonstrates that XAI-enhanced intrusion detection is both technically viable and operationally actionable for IoT security environments, and introduces cross-model rank correlation analysis as a practical framework for assessing explanation robustness in multi-model security deployments.