Temporal Anomaly Detection: Calibrating the Surprise

Authors

  • Eyal Gutflaish Ben-Gurion University
  • Aryeh Kontorovich Ben-Gurion University
  • Sivan Sabato Ben-Gurion Universtiy of the Negev
  • Ofer Biller IBM
  • Oded Sofer IBM

DOI:

https://doi.org/10.1609/aaai.v33i01.33013755

Abstract

We propose a hybrid approach to temporal anomaly detection in access data of users to databases — or more generally, any kind of subject-object co-occurrence data. We consider a high-dimensional setting that also requires fast computation at test time. Our methodology identifies anomalies based on a single stationary model, instead of requiring a full temporal one, which would be prohibitive in this setting. We learn a low-rank stationary model from the training data, and then fit a regression model for predicting the expected likelihood score of normal access patterns in the future. The disparity between the predicted likelihood score and the observed one is used to assess the “surprise” at test time. This approach enables calibration of the anomaly score, so that time-varying normal behavior patterns are not considered anomalous. We provide a detailed description of the algorithm, including a convergence analysis, and report encouraging empirical results. One of the data sets that we tested is new for the public domain. It consists of two months’ worth of database access records from a live system. This data set and our code are publicly available at https://github.com/eyalgut/TLR anomaly detection.git.

Downloads

Published

2019-07-17

How to Cite

Gutflaish, E., Kontorovich, A., Sabato, S., Biller, O., & Sofer, O. (2019). Temporal Anomaly Detection: Calibrating the Surprise. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 3755-3762. https://doi.org/10.1609/aaai.v33i01.33013755

Issue

Section

AAAI Technical Track: Machine Learning