Anomaly Detection
The Anomaly Detection processor uses the three-sigma rule and allows you to define the number of standard deviations considered normal for a data-set. If the data is within the designated standard deviations of the mean, it is not considered to be anomalous.
- Anomaly detection by 3-Sigma Rule is a conventional heuristic used for an approximately normalized data-set.
- A rolling window of values are "observed", and their average and standard deviations are calculated.
- You can define the number of standard deviations that would be considered "normal", hence everything beyond that would be anomalous data.
- In case of an anomaly, the standard deviation window is expanded ever so slightly, so that it can adjust if this anomalous data becomes seasonal.
- This calculation is done because in live data, it is sometimes not preferable for one large anomaly to drastically change the moving average and moving standard deviations.
- With the control chart mode, this calculation can be completely bypassed, if all you need to check is whether the value is within upper/lower limit or not.
- If control chart mode is enabled, there will be no modifications done to the moving window before calculation.
- Currently if you have very small deviations (close to 0), it is difficult to distinguish between anomalous data, so it would be better to have a larger window size for those kinds of data.
- Expected Output Fields: timestamp, current value, moving average, moving standard deviation, upper limit, lower limit, total anomalies, and anomaly field replaces current value, if detected.
Parameters | Details |
|---|---|
Control Chart Mode? | This parameter enables or disables the control chart mode, which bypasses the calculation for adjusting the standard deviation window and allows for simple upper/lower limit checks. |
Deviations | This parameter allows you to define the number of standard deviations considered normal for a data-set. |
Window Size | This represents the window in which the calculations are performed, measured in seconds. It determines the number of data points included in the rolling window for observing average and standard deviation. |
Pass Through Value | This parameter allows you to specify how to handle anomalous data, such as replacing the current value with an anomaly field if detected. |
Note: When creating an analytics flow with Anomaly Detection processor, refer the Use the Anomaly Detection Function guide for more details.