Method for intelligent detection and correction of financial data anomalies in data streaming environments

Abstract

As financial institutions adapt to the new challenges of high velocity and volume transaction data processing for security and operational integrity, rule-based and batch anomaly detection systems are often generating too many false positives, lack flexibility to keep up with the changing nature of fraud and don't automate the fraud remediation process. Current detection methods (both temporal modelling, or embedding) were developed with a batch-processing paradigm, are ill-equipped for the multimodal nature of normal behaviour in the streaming world, or do not take advantage of the latest generation of cloud data platforms designed for low-latency, end-to-end pipelines. In this paper, we seek to tackle these challenges, and propose a new framework MS-VAE-C: a multi-scale temporal encoder with adaptive fusion and self-attention, to jointly detect and correct financial anomalies in streaming scenarios; a variational latent space with Gaussian mixture priors, that models various normal transaction patterns; and a hybrid anomaly scoring mechanism by combining reconstruction error and distributional deviation, which is then handled by the intelligent correct module that suggests auditable remediation without compromising data integrity, natively implemented on the Snowflake platform - Snow pipe for ingestion, Snowpark for model execution, and Dynamic Tables for real-time serving - ensuring seamless scalability with minimal data movement. A set of extensive experiments on large-scale financial graphs and real-world financial transactions streams shows that the proposed approach outperforms the state-of-the-art baselines with an average of 7-12% in F1 score, while also having low end-to-end detection latency of less than 200 milliseconds. Overall, these results show that MS-VAE-C is an efficient and feasible approach and can be used to improve the capability of financial anomaly management in the real world and its feasibility in cloud-native environments.