Anomaly Detection – Unsupervised Approach

From data mining to intrusion detection: how anomaly helps to identify informational security risk.

Prepare and prevent, they say. In this paper our Data Science Group describes informational security risk identification by detecting anomalies, i.e. deviations from the typical pattern of network activity.

As a rule, the problem of detecting anomalies is mostly encountered in the context of different fields of application, including intrusion detection, fraud detection, failure detection, monitoring of system status, event detection in sensor networks, and ecosystem disorder indicators.


The urgent need to solve the problem of anomaly detection lies in the fact that any deviation from a general picture showing the current state of the system may carry important information about the existing issue. Ignoring deviations may lead to undesirable outcomes: for example, an unusual blackout on an X-ray image may serve as evidence of cancer. Prepare and prevent, they say.

In today’s data driven world, Information Security attracts special attention when it comes to pattern changes detection. The continuous development and complexity of information processing automation presupposes the decisive role of security in information technology. 2015 was especially rich in cyber-attacks with companies such as T-Mobile, Kaspersky, and Anthem having had their security compromised and all sorts of personal information about users exposed.

Having a close look at the informational environment of any organization, it’s not hard to see that applying traditional safety measures is a rather prohibitive approach, and often ineffective. Since there is a whole range of possible scenarios in a user’s workflow within the security system, basic rules face multiple exceptions, reducing preventive protection and making the regular analysis of inner threats identification more complicated. Detecting external attacks is also becoming more and more problematic since attackers are aware of typical intrusion detection means and apply covert agents for the attack. Here are main types of the network security breach:


These processes may be spotted, for instance, due to the increased activity of certain ports, new unusual services, changes in a user’s work with network resources, etc.

One possible solution to this problem is the development of systems that identify unusual user network behavior, based on analysis of network activity logs. Using data mining techniques, these systems reveal indicative behavior patterns and draw conclusions about behavior that differs from what’s considered conventional. The systems may though be self-adaptive, minimizing human involvement in configuring the system. Without taking into account an organization’s specifics, such systems are of particular interest to specialists in the field of machine learning and data mining.

In this paper our Data Science Group (DSG) describes informational security risk identification by detecting deviations from the typical pattern of network activity.

download pdf

other stories

Software development tips, opinions and latest industry news.