Temporal multi-view inconsistency detection for network traffic analysis

Abstract

In this paper, we investigate the problem of identifying in- consistent hosts in large-scale enterprise networks by mining multiple views of temporal data collected from the network- s. The time-varying behavior of hosts is typically consistent across multiple views, and thus hosts that exhibit inconsis- tent behavior are possible anomalous points to be further investigated. To achieve this goal, we develop an effective approach that extracts common patterns hidden in multi- ple views and detects inconsistency by measuring the de- viation from these common patterns. Spe cifically, we first apply various anomaly detectors on the raw data and for- m a three-way tensor (host, time, detector) for each view. We then develop a joint probabilistic tensor factorization method to derive the latent tensor subspace, which cap- tures common time-varying behavior across views. Based on the extracted tensor subspace, an inconsistency score is calculated for each host that measures the deviation from common behavior. We demonstrate the effectiveness of the proposed approach on two enterprise-wide network-based anomaly detection tasks. An enterprise network consists of multiple hosts (servers, desktops, laptops) and each host sends/receives a time-varying number of bytes across net- work protocols (e.g.,TCP, UDP, ICMP) or send URL re- quests to DNS under various categories. The inconsistent behavior of a host is often a leading indicator of potential issues (e.g., instability, malicious behavior, or hardware mal- function). We perform experiments on real-world data col- lected from IBM enterprise networks, and demonstrate that the proposed method can find hosts with inconsistent be- havior that are important to cybersecurity applications.