Large-Scale Intelligence Microservices For Robust Anomaly Detection In Real-Time Streaming Data Systems Using Deep Learning

Abstract

Cloud-native microservice architectures have grown at an unprecedented rate, imposing new and greater requirements on reliable and scalable frameworks that can analyze streaming data at very high velocities and rates and detect anomalies. This study proposes a thorough survey of large-scale intelligence microservice pipelines, incorporating cutting-edge deep learning models such as Long Short-Term Memory (LSTM) networks, transformer-based architectures, and Variational Autoencoders (VAEs) for robust anomaly detection in real-time distributed systems. Four benchmark datasets, MSCert-2023, HDFS-Log, Yahoo S5, and GCP-Trace-2022, with a total of more than 18.5 million records, were used for the evaluation. The proposed microservice-orchestrated pipeline on the MSCert-2023 dataset also outperformed the conventional baseline methods by a large margin, yielding an F1 score of 95.2% and AUC-ROC of 0.986 using the Transformer model, with an end-to-end inference latency as low as 41 ms. Experimental results demonstrate that the proposed microservice-oriented pipeline significantly outperforms conventional baseline methods in terms of performance, suggesting a deployable framework for enterprise-grade anomaly detection in distributed data streams. The results pave the way for an emerging new paradigm in which modular, containerized artificial intelligence services can be combined to solve the reliability issues in cloud computing systems.