Anomaly detection

The algorithms LogX uses, the anomaly search function with its inputs and outputs, the engine-driven data stream anomalies, and how a monitor with a risk level feeds the engines.

What you will do: run anomaly detection on any aggregated series from Search, read the result, and schedule it as a monitor.

Anomaly detection runs in three places: the Analytics and LogX engines score every log continuously (data stream anomalies, logging activity, errors), the anomaly function applies the same algorithms to a series you build in a query, and monitors with a risk level hand their results to the engines. The algorithms are the same; what differs is who chooses the series.

Algorithms

Algorithm	Model
Naive Sigma / Olympic Sigma	a seasonal model: the next point is a smoothed average over the previous n periods
Moving Average Sigma	a series where each value is replaced by the mean of itself and its neighbours; weighted moving average and naive forecasting are special cases
Regression Sigma, Polynomial Regression Sigma, Multiple Linear Regression Sigma	the relationship between x and y with one or more variables
Simple Smoothing Sigma (single, double, triple exponential smoothing)	smoothed series with trend and seasonality; the engine picks the best fit automatically

The `anomaly` function

The function expects a set of values: a complex search that counts, groups and buckets by interval. It returns only the points that fall outside normal behaviour.

error | count | group by ext.log as Source Log | interval 5 minutes | anomaly mergealgorithms:true | group by Source Log | order by anomaly change desc
* | count | group by ext.server as Source | interval 5 minutes | anomaly mergealgorithms:true | group by Source | max anomaly value | group by Source | interval 5 minutes
* in log.access | count | group by URL | interval 5 minutes | anomaly mergealgorithms:true | group by URL | max anomaly value | group by URL | interval 5 minutes

Output column	Meaning
Anomaly Name	the algorithm that detected it (one row per algorithm unless `mergealgorithms:true` unites them)
Anomaly Time	when it was detected
Anomaly Description	the change in values that led to the conclusion
Confidence	the certainty based on the algorithms used
Anomaly Change	the difference between the normal state and the anomalous value
Anomaly Value	the value at the time of the anomaly

Engine-driven detection

Analytics scores statistical problems (high or low logging activity, a source that stops logging, an IP hammering one URL) on every analysed log, every five minutes, with a risk from 1 to 10. Read them in the Analytics console or with analyticstream.
LogX adds data stream statistics and anomalies and errors anomalies to its App dashboards.
Since build 7.8050, monitor queries that carry a risk are added to the Analytics and LogX engines automatically; the engines then watch the monitor's series as well.
The same detection can be turned inward on the platform's own audit and system logs to watch privileged activity.

Steps: from query to monitor

Build the series in Search: a count, a group by on the dimension that identifies the thing (host, URL, user), an interval short enough to catch the change.
Append | anomaly mergealgorithms:true and run over a long enough time range to include normal days.
Keep the dimensions whose anomalies you would act on; add | where anomaly change > <n> or | order by anomaly change desc | first 10 to cut noise.
Save the search, then Save Monitor (name, schedule, email recipients); open the monitor under Monitors to set its risk level and the other alert channels — see Monitoring.
Watch the first alerts against the events; lower the risk or widen the interval when a daily cycle is reported as an anomaly.

What it costs

Anomaly detection is not metered per rule. The LogX models (Observability Detect, Cyber Detect, Service Intelligence) consume XMC (XPLG Model Credits) per GB of the LogX Base tier; see Models and credits.

Next: Correlation

Source: XPLG wiki, pages 1602191861, 1602846618, 1594131648, 1594133717, 1602191858, 1594133710, 3161751578, 1594131316, 1594131703, 1594130366 (https://xpolog.atlassian.net/wiki/spaces/XPOL/pages/1602191861).

Search documentation