references/agent-builder-companion.md
# Agent Builder companion (optional, non-universal)
The universal skill body uses Elasticsearch and Kibana HTTP APIs only. Prebuilt **Kibana Agent Builder** tools (`ad_*`),
workflows (`ad_wf_*`), and registration scripts are **not** part of the universal contract.
Teams running Kibana Agent Builder may optionally deploy custom ES|QL tools and workflows that wrap the same underlying
indices (`.ml-anomalies-*`, `.ml-config`, `.ml-notifications-*`) and ML APIs documented in this skill. Those assets are
maintained separately from this skill package and are not required for investigation, explanation, or troubleshooting
via the `elastic` CLI.
When Agent Builder tools are available in a runtime, they can accelerate repetitive RCA queries — but the expert
judgment (rank by `influencer_score`, diagnose `memory_status = hard_limit`, lifecycle sequencing) remains identical.
references/anomaly-detection-functions.md
# Elastic ML Anomaly Detection Functions
Functions marked with `*` support **high/low one-sided variants** (e.g., `high_count`, `low_mean`) to detect anomalies
in only one direction.
---
## Count Functions
Analyze the **occurrence rate** of events or documents over time.
| Function | Description |
| ------------------- | --------------------------------------------------------------------- |
| `count` \* | Number of documents in a bucket |
| `non_zero_count` \* | Like `count` but ignores zero-count buckets — use for **sparse data** |
| `distinct_count` \* | Cardinality (uniqueness) of values for a specific field |
### `count` vs `non_zero_count`
| Scenario | Use |
| ------------------------------------------------------------- | ----------------------------------------------------- |
| Events arrive in every bucket (e.g., web traffic, heartbeats) | `count` — zero buckets are meaningful (outage signal) |
| Events arrive intermittently (e.g., batch jobs, error logs) | `non_zero_count` — zeros are expected, not anomalous |
| Detecting a **drop to zero** as an outage | `count` with `low_count` variant |
| Detecting **bursts** in normally sparse traffic | `non_zero_count` with `high_non_zero_count` variant |
**Example:** An intrusion detection log index gets events only when triggered. Using `count` means the model learns that
zero-event buckets are normal overnight — making it impossible to distinguish a genuine quiet night from a monitoring
gap. Use `non_zero_count` so the model only learns from buckets that had events, and flags when event volumes spike
unexpectedly.
### `distinct_count`
Counts the number of unique values for a field within each bucket. Useful for detecting credential stuffing (unusually
high distinct usernames), data exfiltration (high distinct destination IPs), or DGA activity (high distinct DNS query
names).
---
## Metric Functions
Operate on **numerical fields** within the data.
| Function | Description |
| ------------------------- | -------------------------------------------------------------- |
| `min` / `max` | Minimum or maximum value in a bucket |
| `mean` \* | Average value |
| `median` \* | Median value |
| `sum` / `non_null_sum` \* | Total sum of a field; `non_null_sum` is for **sparse data** |
| `varp` \* | Variance / volatility of a metric |
| `metric` | Shorthand that applies `min`, `max`, and `mean` simultaneously |
### Choosing the right metric function
| Goal | Function |
| ---------------------------------------------- | ---------------------------------- |
| Detect **average** latency spike | `mean` or `high_mean` |
| Detect **worst-case** latency (tail) | `max` |
| Detect **sustained volume** drop | `low_sum` |
| Detect **unusual volatility** (erratic metric) | `high_varp` |
| Detect both high and low deviations | `mean` (bidirectional, default) |
| Detect only spikes, not drops | `high_mean` |
| Monitor noisy metrics with outliers | `median` (more robust than `mean`) |
### `sum` vs `non_null_sum`
Use `non_null_sum` when the field is frequently absent from documents (sparse). Like `non_zero_count`, it skips empty
buckets so the model learns only from active periods.
### `metric` shorthand
Creates three detectors in one: `min`, `max`, and `mean` on the same field. Convenient for a quick initial setup, but
produces three anomaly records per detection event. Prefer explicit functions once you know which direction matters.
---
## Advanced & Specialized Functions
Handle complex analysis types such as rarity or geographic data.
| Function | Description |
| ----------------- | -------------------------------------------------------------------------------- |
| `rare` | Identifies values that occur at **low frequency** compared to the dataset |
| `freq_rare` | Finds population members that **cause rare values to occur frequently** |
| `info_content` \* | Entropy of text strings — useful for detecting **encrypted/obfuscated commands** |
| `lat_long` | Detects unusual **geographic locations** from latitude/longitude coordinates |
| `time_of_day` | Detects behavioral changes relative to **time of day** |
| `time_of_week` | Detects behavioral changes relative to **day of week** |
---
### `rare` vs `freq_rare`
These two are often confused but answer different questions:
| Function | Question answered | Requires `over_field` |
| ----------- | ----------------------------------------------------------- | --------------------- |
| `rare` | "Which values of the `by_field` are unusual globally?" | No |
| `freq_rare` | "Which entities (over_field) frequently cause rare values?" | Yes |
**`rare` example:** Detect rare `process.name` values across all hosts. If `svchost.exe` with unusual arguments appears
only once in 30 days, `rare` flags it. The focus is the rarity of the _value_.
**`freq_rare` example:** Detect which _users_ (`over_field`) frequently trigger rare process executions (`by_field`).
Most users run rare processes occasionally, but a user running rare processes consistently is a lateral movement signal.
The focus is the entity's behavior pattern.
**Practical guidance:**
- Use `rare` for hunting unknown unknowns — values that shouldn't exist at all.
- Use `freq_rare` for insider threat and lateral movement scenarios — who is repeatedly doing unusual things.
- `rare` generates high false-positive rates in noisy environments; use custom rules to suppress known-good rare values.
- `freq_rare` requires an `over_field` (population) — without it, use `rare`.
### Types of rare analysis
Translate business goals to the correct `rare` detector configuration:
| Goal | Example | Detector config |
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Find infrequent values for a field | Detect hosts occurring infrequently | `rare` `by_field_name: host` |
| Find infrequent values compared to peers | Detect hosts visited by few users vs. other hosts; detect users visiting rare hosts | `rare` `by_field_name: host` `over_field_name: user` |
| Find infrequent values segmented by another field | Per location, detect infrequently seen hosts | `rare` `by_field_name: host` `partition_field_name: location` |
| Find infrequent values by another field compared to peers, segmented | Per location, detect hosts visited by few users vs. peers; detect users visiting rare hosts in a location | `rare` `by_field_name: host` `over_field_name: user` `partition_field_name: location` |
---
### `info_content`
Measures the **Shannon entropy** of text strings. High entropy = high randomness = potentially encoded, encrypted, or
machine-generated content.
| Entropy range | Interpretation | Example |
| ----------------- | ------------------------------------ | -------------------------------------------------- |
| Low (predictable) | Normal human-readable text | `GET /api/users HTTP/1.1` |
| Medium | Mixed structured/variable content | Log messages with variable IDs |
| High (random) | Encoded, encrypted, or DGA-generated | `aGVsbG8gd29ybGQ=` (base64), `zxq7v9abc.com` (DGA) |
**Use cases:**
- **DNS query names** with `by_field_name: "dns.question.name"` — DGA malware generates high-entropy domain names (e.g.,
`xk9p2mnqabcdef.ru`).
- **User-agent strings** — malware C2 frameworks often use randomized or encoded user agents.
- **URL paths / query strings** — webshell commands embedded in request parameters.
- **Command-line arguments** — base64-encoded PowerShell payloads.
**`high_info_content`** (one-sided): only alerts on unusually high entropy, which is almost always the right choice for
security use cases.
---
### `time_of_day` vs `time_of_week`
Both detect _when_ something happens relative to established patterns, not _how much_.
| Function | Granularity | Best for |
| -------------- | ------------------ | -------------------------------------------------------- |
| `time_of_day` | Hour/minute of day | Detecting off-hours access (3am login for a 9-5 user) |
| `time_of_week` | Day of week | Detecting weekend/holiday activity (batch job on Sunday) |
**`time_of_day` example:** A database admin who always connects between 08:00–18:00 on weekdays. `time_of_day` learns
this pattern. A connection at 02:30 scores anomalous — even if the volume of activity is normal.
**`time_of_week` example:** A data pipeline that runs Monday–Friday. `time_of_week` flags it running on Saturday.
`time_of_day` would not catch this (the time of day, e.g., 08:00, may be normal).
**Choosing between them:**
- Use `time_of_day` when the anomaly is "wrong hour of the day."
- Use `time_of_week` when the anomaly is "wrong day of the week."
- Use both together (two detectors) for comprehensive temporal coverage.
- Neither function cares about _count_ or _metric values_ — use `count`/`mean` detectors in the same job for
volume-based detection.
---
## One-Sided Variants
Functions marked with `*` support `high_` and `low_` prefixes:
| Variant | Detects |
| ------------------------ | ------------------------------------------------------ |
| `high_<function>` | Only values significantly **above** the expected range |
| `low_<function>` | Only values significantly **below** the expected range |
| `<function>` (no prefix) | Both directions (bidirectional) |
**When to use one-sided:**
- `high_mean(response_time)` — only alert on latency spikes, not drops (a faster response is never a problem).
- `low_count(login_events)` — only alert on unusually low login volume (could indicate authentication system failure).
- `high_distinct_count(destination.ip)` — only alert on abnormally high unique destination IPs (exfiltration signal).
Bidirectional variants (`mean`, `count`) generate alerts for both directions, which can produce noise when only one
direction is operationally relevant.
references/investigation-queries.md
# Investigation query reference
`POST /.ml-anomalies-*/_search` query patterns for Investigate mode. Replace `{job_id}`, `{start}`, `{end}`, and entity
values with the incident context.
## Bucket timeline (scope incident)
```json
{
"size": 100,
"sort": [{ "timestamp": "desc" }],
"query": {
"bool": {
"filter": [
{ "term": { "result_type": "bucket" } },
{ "term": { "job_id": "{job_id}" } },
{ "range": { "timestamp": { "gte": "{start}", "lte": "{end}" } } },
{ "range": { "anomaly_score": { "gte": 25 } } }
]
}
}
}
```
For cross-job scope, replace the single `job_id` term with `terms: { job_id: ["job-a", "job-b"] }` or omit it to scan
all jobs (add a time range).
## Influencers (entity attribution — RCA)
**Critical:** sort by `influencer_score` descending. The top row is the primary suspect entity.
```json
{
"size": 20,
"sort": [{ "influencer_score": "desc" }],
"query": {
"bool": {
"filter": [
{ "term": { "result_type": "influencer" } },
{ "term": { "job_id": "{job_id}" } },
{ "range": { "timestamp": { "gte": "{start}", "lte": "{end}" } } },
{ "range": { "influencer_score": { "gte": 25 } } }
]
}
}
}
```
## Records (drill-down)
```json
{
"size": 50,
"sort": [{ "record_score": "desc" }],
"query": {
"bool": {
"filter": [
{ "term": { "result_type": "record" } },
{ "term": { "job_id": "{job_id}" } },
{ "range": { "timestamp": { "gte": "{start}", "lte": "{end}" } } },
{ "range": { "record_score": { "gte": 25 } } }
],
"must": [{ "term": { "partition_field_value": "{entity}" } }]
}
}
}
```
Adjust entity filter field to `by_field_value` or `over_field_value` as configured on the job.
## Cross-job entity match
Find all jobs where a specific entity value appears in influencer results:
```json
{
"size": 0,
"query": {
"bool": {
"filter": [
{ "term": { "result_type": "influencer" } },
{ "term": { "influencer_field_value": "{entity}" } },
{ "range": { "timestamp": { "gte": "{start}", "lte": "{end}" } } }
]
}
},
"aggs": {
"by_job": {
"terms": { "field": "job_id", "size": 20 },
"aggs": { "max_score": { "max": { "field": "influencer_score" } } }
}
}
}
```
Entities appearing in 2+ jobs (`by_job.buckets.length ≥ 2`) are prime root-cause suspects.
## Model plot (bounds check)
Requires `model_plot_config.enabled` on the job.
```json
{
"size": 100,
"sort": [{ "timestamp": "asc" }],
"query": {
"bool": {
"filter": [
{ "term": { "result_type": "model_plot" } },
{ "term": { "job_id": "{job_id}" } },
{ "range": { "timestamp": { "gte": "{start}", "lte": "{end}" } } }
]
}
}
}
```
## Category definitions (log categorization jobs)
```json
{
"size": 50,
"query": {
"bool": {
"filter": [{ "term": { "result_type": "category_definition" } }, { "term": { "job_id": "{job_id}" } }]
}
}
}
```
## Related job discovery
No single API returns "related jobs." Compare outputs of:
- `GET /_ml/datafeeds/datafeed-{job_id}` — shared `indices` imply shared infrastructure.
- `GET /_ml/anomaly_detectors/{job_id}` — shared `by_field_name` / `partition_field_name` imply shared entity
dimensions.
List all jobs with `GET /_ml/anomaly_detectors` and filter by matching index patterns or entity fields.
references/job-creation-recipes.md
# Job creation recipes
Worked examples for creating Elastic ML anomaly detection jobs and datafeeds via ML REST APIs.
> For the high-level process, see the **Manage** section of the parent `SKILL.md`. For detector function selection, see
> [anomaly-detection-functions.md](anomaly-detection-functions.md).
---
## API call sequence
```text
PUT /_ml/anomaly_detectors/<job_id> # 1. Define job + detectors
PUT /_ml/datafeeds/datafeed-<job_id> # 2. Define datafeed (source + query)
POST /_ml/anomaly_detectors/<job_id>/_open # 3a. Open job
POST /_ml/datafeeds/datafeed-<job_id>/_start # 3b. Start datafeed
POST /.ml-anomalies-*/_search # 4. Read results (Serverless-safe)
```
To stop: `POST /_ml/datafeeds/datafeed-<job_id>/_stop` → `POST /_ml/anomaly_detectors/<job_id>/_close`.
For **batch analysis on historical data**, pass `start` and `end` to the datafeed start call:
```json
POST _ml/datafeeds/datafeed-revenue_over_users_api/_start
{ "start": "2024-01-01T00:00:00Z", "end": "2024-03-01T00:00:00Z" }
```
---
## Key `analysis_config` fields
| Field | Description |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `bucket_span` | Analysis interval (e.g. `15m`, `1h`). Align with data granularity and detection window. |
| `detectors[].function` | Analysis function (`high_sum`, `rare`, `mean`, etc). See [anomaly-detection-functions.md](anomaly-detection-functions.md). |
| `detectors[].field_name` | Numeric field to analyze. |
| `detectors[].over_field_name` | Population analysis — each entity compared to its peers in the same bucket. |
| `detectors[].by_field_name` | Per-entity modeling — each entity compared to its own history. |
| `detectors[].partition_field_name` | Fully independent sub-model per entity with its own score normalization. |
| `influencers` | Fields to track as anomaly contributors (shown as `influencer_score`). |
| `data_description.time_field` | Timestamp field for time series ordering (e.g. `@timestamp`, `order_date`). |
## Key datafeed fields
| Field | Description |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `indices` | Array of index patterns containing source data. |
| `query` | Elasticsearch DSL to filter source documents. Defaults to `match_all`. |
| `query_delay` | How far behind real time the datafeed queries. Set to P95 ingest latency + buffer (default `60s`–`120s`). Too low → missing docs. |
| `scroll_size` | Documents fetched per scroll request. Default `1000`. |
| `frequency` | How often the datafeed polls. Defaults to `min(query_delay, bucket_span / 2)`. |
---
## Recipe 1 — `rare` detector: rare usernames
**User query:** "Create an ML job to detect rare usernames in login events across logs-\*"
Verify `user.name` (keyword) and `@timestamp` (date) exist via `GET /logs-*/_mapping`. Then
`PUT /_ml/anomaly_detectors/rare-login-usernames` → `PUT /_ml/datafeeds/datafeed-rare-login-usernames` →
`POST /_ml/anomaly_detectors/rare-login-usernames/_open` → `POST /_ml/datafeeds/datafeed-rare-login-usernames/_start`.
**Job body:**
```json
{
"description": "Detects rare values of user.name during login activity",
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{ "function": "rare", "by_field_name": "user.name", "detector_description": "Rare user.name values" }
],
"influencers": ["user.name", "source.ip"]
},
"data_description": { "time_field": "@timestamp" }
}
```
**Datafeed body:**
```json
{ "job_id": "rare-login-usernames", "indices": ["logs-*"], "query": { "match_all": {} } }
```
---
## Recipe 2 — `high_mean` detector: DNS exfiltration with datafeed filter
**User query:** "detect hosts with unusually high DNS query volume per domain, only for external DNS traffic on port 53"
Verify `dns.question.count` (numeric), `host.name` (keyword), `dns.question.name` (keyword), `@timestamp` (date).
**Job body:**
```json
{
"description": "Detects unusually high DNS query volume per host, partitioned by queried domain",
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"function": "high_mean",
"field_name": "dns.question.count",
"over_field_name": "host.name",
"partition_field_name": "dns.question.name",
"detector_description": "High DNS query volume per host per domain"
}
],
"influencers": ["host.name", "dns.question.name", "source.ip"]
},
"data_description": { "time_field": "@timestamp" }
}
```
**Datafeed body:**
```json
{
"job_id": "high-dns-query-volume-per-host",
"indices": ["logs-*"],
"query": {
"bool": { "filter": [{ "term": { "network.transport": "udp" } }, { "term": { "destination.port": 53 } }] }
}
}
```
---
## Recipe 3 — `high_sum` detector: large downloads with time range
**User query:** "detect users downloading unusually large amounts of data for sshd and sftp processes in the last 30
days"
Verify `destination.bytes` (numeric), `user.name` (keyword), `process.name` (keyword), `@timestamp` (date).
**Job body:**
```json
{
"description": "Detects unusually high total bytes downloaded per user for specific processes",
"analysis_config": {
"bucket_span": "1h",
"detectors": [
{
"function": "high_sum",
"field_name": "destination.bytes",
"by_field_name": "user.name",
"detector_description": "High total bytes downloaded per user"
}
],
"influencers": ["user.name", "process.name", "source.ip"]
},
"data_description": { "time_field": "@timestamp" }
}
```
**Datafeed body:**
```json
{
"job_id": "high-download-volume-per-user",
"indices": ["logs-*"],
"query": {
"bool": {
"filter": [
{ "terms": { "process.name": ["sshd", "sftp"] } },
{ "range": { "@timestamp": { "gte": "now-30d", "lte": "now" } } }
]
}
}
}
```
---
## Retrieving results
Use `POST /.ml-anomalies-*/_search` with `result_type` filters (Serverless-compatible):
| Result type | `result_type` value | Description |
| ----------- | ------------------- | ------------------------------------------------------------------- |
| Buckets | `bucket` | Aggregate anomaly score per time bucket |
| Records | `record` | Individual anomaly records with `actual`, `typical`, `record_score` |
| Influencers | `influencer` | Entity contribution scores |
Forecasts: `POST /_ml/anomaly_detectors/<job_id>/_forecast` with `duration` (e.g. `"10d"`). **Not supported** for
population analysis jobs (`over_field_name` set).
references/observability-anomaly-expert.md
# Observability / SRE framing — Elastic ML anomaly detection
**Role:** Treat Elasticsearch ML anomaly detection as a reliability signal for SRE and platform work: degradation,
incident scope, and capacity decisions. Combine the **Investigate**, **Explain**, and **Troubleshoot** modes of the
parent skill, biased toward reliability interpretation.
---
## Reliability-first interpretation
Interpret anomalies through three reliability lenses:
1. **Incident detection** — is this active degradation? What is the scope?
2. **Change attribution** — tie signal to deployments, config changes, dependencies when possible.
3. **Capacity signals** — separate acute incidents from resource-exhaustion trajectories.
## Signal → reliability mapping
| Anomaly pattern | Reliability interpretation | Action |
| -------------------------------------------------------------- | -------------------------------------------------- | ------------------------------------ |
| Latency spike + error rate spike (same service, same time) | Service degradation in progress | Incident response |
| Throughput drop (`actual << typical` with `count`/`low_count`) | Service unavailable or upstream dependency failure | Check dependencies, circuit breakers |
| Cross-service entity anomalies with temporal chain | Cascading failure / blast propagation | Identify blast radius, isolate |
| Memory/CPU creep (`multi_bucket_impact ≥ 3`) | Resource exhaustion trajectory | Capacity intervention before OOM |
| Anomaly onset matches deployment timestamp | Deployment regression | Rollback candidate |
| Single service anomaly, no related job co-firing | Isolated issue, contained | Service-level investigation |
| Anomaly during known maintenance window | Expected — suppress via calendar event | Add calendar event via ML API |
## SRE investigation protocol
### Phase 1 — Incident scoping (Investigate mode)
1. `GET /_ml/anomaly_detectors` — identify observability jobs (latency, error rate, throughput, saturation).
2. `POST /.ml-anomalies-*/_search` (`result_type: bucket`) — establish incident start time and breadth.
3. Cross-job influencer aggregation — co-firing metrics on the same entity = the degraded service.
### Phase 2 — Root cause attribution (Investigate mode)
1. Compare datafeed indices across jobs — find all jobs monitoring the same infrastructure layer.
2. Search influencers/records for co-firing entity values across related jobs.
3. Sort record timestamps — leading metric (first anomaly) = root cause; lagging = symptoms.
4. Examine which behavioral dimensions are anomalous (latency? saturation? error rate? throughput drop?).
### Phase 3 — Evidence and context (Investigate mode)
1. `GET /_ml/datafeeds/datafeed-{job_id}` → source index → search raw metrics/logs for the suspect service/host.
2. `POST /.ml-anomalies-*/_search` (`result_type: influencer`) — rank hosts/pods/instances by `influencer_score`.
3. Profile the suspect entity across all related jobs.
### Phase 4 — Deployment regression check (Explain mode)
When incident onset aligns with a recent deployment:
1. Compare `initial_record_score` vs `record_score` — confirm whether a score drop reflects renormalization instead of
real recovery.
2. Query `result_type: model_plot` — confirm the anomaly sits outside expected bounds instead of being a model artifact.
3. Search source metrics before and after the deployment timestamp.
### Phase 5 — Capacity planning (Explain + Troubleshoot modes)
When `multi_bucket_impact ≥ 3` on resource metrics:
1. `GET /_ml/anomaly_detectors/{job_id}/_stats` — check `memory_status` and field cardinality counts.
2. If approaching `hard_limit`, raise `model_memory_limit` or split jobs **before** results stop.
3. Do not treat a stopped datafeed as the root cause when `memory_status = hard_limit`.
## Rules
- **Name the entity with highest `influencer_score`** when scoping an incident — "checkout latency spiked" is not
enough; identify which host/service instance drove it.
- **`multi_bucket_impact ≥ 3` on saturation metrics** → capacity intervention, not just alert acknowledgment.
- **Cross-job co-firing on the same service entity** → systemic degradation; isolate blast radius before deep-diving one
metric.
references/protocols/investigation.md
# Investigation Protocol (14 Steps)
Canonical workflow for root cause analysis of Elastic ML anomaly detection events.
> For a worked example, see [../worked-example.md](../worked-example.md). Query templates:
> [../investigation-queries.md](../investigation-queries.md).
---
## When to Use This Protocol
- Starting from a single alert and need to determine the root cause
- Multiple jobs are co-firing and you need to find the common denominator
- Asked "what broke?", "which entity caused this?", or "why is service X slow?"
For **score explanation questions** (why is my score low/high?), see [../score-reference.md](../score-reference.md)
instead.
---
## Three-Layer Job Discovery
Before beginning analysis, identify all related jobs using these signals in priority order:
1. **Shared datafeed index patterns** (strongest) — `GET /_ml/datafeeds/datafeed-{job_id}` → compare `indices` across
jobs via `GET /_ml/anomaly_detectors`.
2. **Shared entity field names** (config signal) — `GET /_ml/anomaly_detectors/{job_id}` → compare `by_field_name`,
`over_field_name`, `partition_field_name`.
3. **Shared entity values in results** (active incident) — `POST /.ml-anomalies-*/_search` with
`result_type: influencer` for co-firing entity values across jobs.
---
## The 14 Steps
### Phase 1: Discovery
**Step 1 — Discover** Call `GET /_ml/anomaly_detectors` to list jobs. Call `GET /_ml/anomaly_detectors/{job_id}` for
detector functions, entity fields, and `bucket_span`. Always start here when jobs are unknown.
**Step 2 — Find related jobs** Compare datafeed `indices` and entity field names across jobs. Jobs sharing a source
index or entity dimension monitor the same system from different angles.
**Step 3 — Scope** Call `POST /.ml-anomalies-*/_search` with `result_type: bucket`, a time range, and minimum
`anomaly_score`. Identify the incident window and count of affected jobs.
---
### Phase 2: Entity Attribution
**Step 4 — Expand from alert** Extract entity values from the alert (`partition_field_value`, `by_field_value`,
`over_field_value`). Search influencers across related jobs for those values.
**Step 5 — Multi-job entities** Aggregate influencers by entity value across jobs. Entities anomalous in **2+ jobs**
simultaneously are the strongest root cause signal — prime suspects. Single-job entities are likely downstream victims.
> Resource faults (CPU, memory, disk) affect multiple metrics → multi-job. Network faults (packet loss) affect latency
> but not resource metrics → single-job.
**Step 6 — Fingerprint** Query records across the related job group. Understand which system aspects are anomalous: CPU?
Latency? Error rate? Memory? The combination of anomalous detectors characterizes the fault type.
---
### Phase 3: Deep Analysis
**Step 7 — Drill down per job** Call `POST /.ml-anomalies-*/_search` with `result_type: record` and an exact job ID to
examine a specific job's anomalies without cross-job noise.
**Step 8 — Attribute (critical)** Call `POST /.ml-anomalies-*/_search` with `result_type: influencer`, sort by
**`influencer_score` descending**, and use a low minimum score (25). The entity with the highest `influencer_score` is
the primary suspect — not the bucket `anomaly_score` alone.
**Step 9 — Profile** Query all record and influencer results for the suspect entity across jobs, sorted by timestamp, to
build a complete dossier.
**Step 10 — Characterize** Examine `multi_bucket_impact` in results:
- `≥ 3` → sustained behavioral shift (system change), not a transient spike
- `0–2` → isolated event (one-off anomaly)
---
### Phase 4: Root Cause Confirmation
**Step 11 — Cascade** Sort record timestamps across jobs for the suspect entity. The **earliest anomaly** points toward
the root cause. Reconstruct chronology: which metric became anomalous first?
**Step 12 — Evidence** Get source indices from `GET /_ml/datafeeds/datafeed-{job_id}`, then search source data for the
suspect entity and time window. Raw source documents show the actual values at ingestion.
**Step 13 — Log categories** _(only when `by_field_name == "mlcategory"`)_ Query `result_type: category_definition` for
the job. Compare category terms and examples between baseline and anomaly windows. Cross-reference changed entities with
influencers from related jobs.
---
### Phase 5: Synthesis
**Step 14 — Synthesize** Present findings as a structured RCA report:
| Section | Content |
| ------------------------ | ------------------------------------------------------------- |
| **Root cause entity** | Entity with highest `influencer_score` and multi-job presence |
| **Affected systems** | Which jobs/metrics were impacted |
| **Temporal progression** | Which metric became anomalous first (from Step 11) |
| **Fault type** | Resource / Network / Application / Pipeline |
| **Severity** | `record_score` range, `multi_bucket_impact`, duration |
| **Recommended actions** | Remediation steps |
---
## Key Decision Rules
- **Rank by `influencer_score` for entity attribution** — bucket `anomaly_score` is aggregate severity, not cause.
- **Low scores across many jobs** > one high score — composite cross-job signal often indicates systemic root cause.
- **`actual << typical` with count/low_count** → absence/outage, not just a numerically low value.
- **Entities in 2+ jobs** → prime suspects (resource fault or systemic failure).
- **Entities in only 1 job** → likely downstream victims or surface-level effects.
- **Earliest anomaly chronology** → the earliest metric to become anomalous is closest to the root cause.
references/README.md
# kibana-anomaly-detection references
Universal skill references for Elastic ML anomaly detection — investigation, score explanation, troubleshooting, and job
configuration. All analytical workflows use Elasticsearch ML REST APIs and `POST /.ml-anomalies-*/_search`.
| File | Purpose |
| ------------------------------------------------------------------ | ------------------------------------------------------------ |
| [score-reference.md](score-reference.md) | Score field definitions, bands, renormalization, explanation |
| [anomaly-detection-functions.md](anomaly-detection-functions.md) | Detector function selection guide |
| [protocols/investigation.md](protocols/investigation.md) | 14-step RCA protocol |
| [investigation-queries.md](investigation-queries.md) | Search query templates for Investigate mode |
| [troubleshooting-reference.md](troubleshooting-reference.md) | Memory, datafeed, and lifecycle troubleshooting |
| [worked-example.md](worked-example.md) | End-to-end investigation walkthrough |
| [job-creation-recipes.md](job-creation-recipes.md) | Job + datafeed JSON recipes for Manage mode |
| [security-anomaly-expert.md](security-anomaly-expert.md) | Threat-first framing (MITRE mappings, attack-chain protocol) |
| [observability-anomaly-expert.md](observability-anomaly-expert.md) | SRE / reliability framing |
| [agent-builder-companion.md](agent-builder-companion.md) | Optional non-universal Agent Builder deployment note |
For job create/open/start lifecycle details, see also the `elasticsearch-anomaly-detection` skill in this repository.
references/score-reference.md
# Anomaly Score Reference
Canonical definitions for all score and impact fields in Elastic ML anomaly detection.
---
## Score Types
| Field | Scope | Range | Description |
| ----------------------- | --------------------- | ----- | -------------------------------------------------------------------------------------- |
| `record_score` | Single anomaly record | 0–100 | Current normalized severity. May change over time as the model sees more extreme data. |
| `initial_record_score` | Single anomaly record | 0–100 | Score at detection time — never changes. Use for alerting on fresh anomalies. |
| `anomaly_score` | Bucket (time window) | 0–100 | Aggregate severity across all detectors in a bucket. |
| `initial_anomaly_score` | Bucket | 0–100 | Bucket score at detection time — never changes. |
| `influencer_score` | Entity × bucket | 0–100 | How anomalous a specific entity (host, user, service) is within that bucket. |
---
## `record_score` Severity Bands
| Band | Range | Interpretation |
| ------------- | ----- | ---------------------------------------------------------- |
| Critical | > 75 | High-confidence anomaly; warrants immediate investigation |
| Warning | 50–75 | Notable deviation; triage and correlate with other signals |
| Minor | 25–50 | Potentially interesting; aggregate with cross-job signals |
| Informational | < 25 | Weak signal; useful for context, not standalone action |
> **Cross-job composite signal**: Low scores (25–50) across many jobs simultaneously are often more significant than a
> single high score. Five jobs each scoring 30 = composite signal 150, pointing to a systemic root cause.
---
## `multi_bucket_impact`
Scale from -5 to +5 indicating whether the anomaly spans multiple consecutive time buckets.
| Value | Meaning |
| -------- | ---------------------------------------------------- |
| 0 | One-off event, no sustained pattern |
| 1–2 | Mild persistence across a few buckets |
| ≥ 3 | Genuine behavioral shift — not a transient spike |
| Negative | Anomaly is suppressed by surrounding normal behavior |
Values ≥ 3 strongly suggest a real system change (e.g., a resource exhaustion event that persists) rather than a
momentary blip.
---
## `initial_record_score` vs `record_score`
Elasticsearch continuously renormalizes scores relative to the most extreme anomaly ever seen by the job. A score of 90
today may become 60 if a more extreme event appears later — by design, so the "worst ever" event always scores near 100.
**When to use each:**
| Use case | Field |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| Alerting on newly detected anomalies | `initial_record_score` — captures severity at detection time |
| Ranking historical anomalies by current importance | `record_score` — reflects how bad this was relative to all history |
| Detecting renormalization (model calibrated away an anomaly) | Compare: if `initial_record_score >> record_score`, the model saw worse events later |
**Quantify drift:** `score_drift = initial_record_score - record_score`
- Large positive drift = renormalized away (model calibrated)
- Small drift = score is stable and genuine
---
## `anomaly_score_explanation` Components
When available, this field explains the factors that contributed to the final score.
| Component | Effect on score | What it means |
| -------------------------------- | --------------- | ------------------------------------------------------------------- |
| `anomaly_length` | ↑ increases | More consecutive anomalous buckets — sustained deviation |
| `single_bucket_impact` | ↑ increases | Lower statistical probability → more surprising → higher impact |
| `multi_bucket_impact` | ↑ increases | Contribution from sustained pattern across multiple buckets |
| `anomaly_characteristics_impact` | ↑ increases | Mean shift (value moved) vs. variance change (volatility increased) |
| `high_variance_penalty` | ↓ decreases | Historically noisy data; wide confidence bounds absorb the spike |
| `incomplete_bucket_penalty` | ↓ decreases | Bucket has less data than expected (ingest lag, sparse events) |
---
## Absence Anomalies
When `actual << typical` with `count`, `low_count`, `low_mean`, or `low_sum` functions, a low or zero value indicates a
real-world absence — not just a numerically low observation:
- Zero `count` when traffic is normally constant → pipeline stopped, service unavailable
- `low_mean(response_time)` → requests completing too fast (cache hit storm, bypassed processing)
- Very low `sum(bytes_sent)` → network partition or data source failure
**Key insight:** A `record_score` of 80 with `actual = 0` and `typical = 5000` is an outage signal, not just a low
number.
---
## Why a Score Is Unexpectedly Low
1. **`high_variance_penalty`** — Metric is historically noisy; wide model bounds absorb the spike.
2. **Renormalization** — A more extreme anomaly appeared later, pushing this score down.
3. **Insufficient training** — Model needs ≥ 3 weeks for weekly seasonality, ≥ 2 full cycles for any period.
4. **`bucket_span` too large** — Long span smooths short-duration spikes; use smaller span for high-frequency detection.
5. **Detector function mismatch** — `mean` vs `high_mean`, `count` vs `high_count`. Wrong function = missed direction.
6. **`incomplete_bucket_penalty`** — Ingest latency or sparse events reduced bucket data volume.
7. **`custom_rules`** — A detector filter may be suppressing or conditioning the anomaly.
## Why a Score Is Unexpectedly High
1. **Insufficient training history** — Early training: moderate deviations flag as extreme.
2. **High-cardinality split** — Too few data points per entity per bucket → unreliable probabilities.
3. **`use_null: true`** — Missing entities produce "null" anomalies that may not be operationally meaningful.
---
## See Also
- [anomaly-detection-functions.md](anomaly-detection-functions.md) — Function selection guide
- [protocols/investigation.md](protocols/investigation.md) — 14-step investigation workflow
- [investigation-queries.md](investigation-queries.md) — Search query templates
- [worked-example.md](worked-example.md) — End-to-end investigation walkthrough
references/security-anomaly-expert.md
# Security framing — Elastic ML anomaly detection
**Role:** Treat Elasticsearch ML anomaly detection as a behavioral threat-detection surface. Assume unusual behavior is
suspicious until benign intent is proven. Combine the **Investigate**, **Explain**, and **Troubleshoot** modes of the
parent skill, biased toward attack-first interpretation.
---
## Threat-first interpretation
Treat operational monitoring as benign-first; treat security anomalies as attack-first. Then:
1. Map behavioral deviations to known attack patterns.
2. Reconstruct attacker chains from cross-job signals.
3. Separate attacker behavior from benign operational noise.
4. Classify threats with MITRE ATT&CK context.
## Signal mapping
| Anomaly pattern | Threat hypothesis | MITRE tactic |
| -------------------------------------------------------------- | ----------------------------------------------- | ------------------------------- |
| Unusual auth failures for a user/host | Brute force, credential stuffing | Credential Access (TA0006) |
| `actual << typical` with `low_count` on auth/process | Service stop, log clearing, defense evasion | Defense Evasion (TA0005) |
| New/rare entity (first-seen IP, user, process) | Initial access, new implant, new C2 | Initial Access (TA0001) |
| Entity anomalous in multiple jobs simultaneously | Active compromise, lateral movement in progress | Lateral Movement (TA0008) |
| Unusual data volume (bytes_out spike) | Data exfiltration | Exfiltration (TA0010) |
| Rare process execution (high influencer_score on process name) | Malware execution, living-off-the-land | Execution (TA0002) |
| Auth success following prior auth failures | Successful credential compromise | Credential Access → Persistence |
| Privilege escalation patterns (sudo, admin role changes) | Admin abuse, shadow IT, misconfiguration | Privilege Escalation (TA0004) |
| Regular low-volume network spikes (beaconing) | C2 communication | Command & Control (TA0011) |
## Investigation questions
For each anomalous entity, determine:
1. **Known vs first-seen entity** — treat first-seen entities as higher risk.
2. **Blast radius** — count how many jobs or systems co-fire.
3. **Temporal chain** — treat auth failure → auth success → lateral movement as a compromise chain hypothesis.
4. **Source evidence** — treat raw logs as the ground truth.
5. **MITRE mapping** — map the pattern to the closest tactic and technique.
## Investigation protocol
### Phase 1 — Triage (Investigate mode)
1. `GET /_ml/anomaly_detectors` — identify security-relevant jobs (auth, network, process, DNS, endpoint).
2. `POST /.ml-anomalies-*/_search` (`result_type: bucket`) — establish incident time window.
3. Cross-job influencer aggregation — multi-job entities in security = active threat actors.
### Phase 2 — Entity attribution (Investigate mode)
1. Search influencers across jobs for the alert entity — expand from single alert to full activity chain.
2. `POST /.ml-anomalies-*/_search` (`result_type: influencer`, low score threshold) — surface all associated entities;
**rank by `influencer_score`**.
3. Profile the suspect entity across all jobs and field types, sorted by timestamp.
### Phase 3 — Attack chain reconstruction (Investigate mode)
1. Sort record timestamps across jobs — reconstruct chronological order. First anomaly = entry point hypothesis.
2. Count affected jobs/entities — determine lateral spread.
3. Examine which behavioral dimensions are anomalous (auth? process? network? data volume?).
### Phase 4 — Evidence (Investigate mode)
1. `GET /_ml/datafeeds/datafeed-{job_id}` → source index → search raw logs for the suspect entity and time window.
2. For categorization jobs, query `result_type: category_definition` and compare baseline vs anomaly log patterns.
### Phase 5 — Score sanity (Explain mode)
When scores seem inconsistent with threat severity:
1. Compare `initial_record_score` vs `record_score` — renormalization may lower historical alert scores.
2. Query `result_type: model_plot` when enabled — confirm anomaly sits outside expected bounds.
3. Check `GET /_ml/anomaly_detectors/{job_id}/_stats` for `memory_status = hard_limit` — corrupted models produce
unreliable scores; fix via Troubleshoot mode before concluding false negative.
## Rules
- **High `influencer_score` on a process or user in a security job** warrants immediate deep-dive — do not dismiss
because bucket `anomaly_score` is moderate.
- **Multi-job entity co-firing in security context** → assume active compromise until ruled out.
- **Absence anomalies on auth/process indices** → investigate defense evasion (log clearing, service stop) before
dismissing as benign maintenance.
references/troubleshooting-reference.md
# Troubleshooting reference
Lifecycle and health diagnostics via ML REST APIs and system indices. Apply **memory diagnosis before query_delay
tuning**.
## Memory status fields
From `GET /_ml/anomaly_detectors/{job_id}/_stats` → `model_size_stats`:
| Field | Meaning |
| ----------------------------------- | ------------------------------------------------------- |
| `model_bytes` | Current memory used |
| `peak_model_bytes` | High-water mark since job opened |
| `model_bytes_memory_limit` | Configured `model_memory_limit` |
| `memory_status` | `ok` / `soft_limit` (pruning) / `hard_limit` (critical) |
| `total_by_field_count > 100k` | `by_field` cardinality too high — dominant driver |
| `total_partition_field_count > 10k` | Partition explosion |
| `total_category_count > 10k` | Too many distinct log patterns |
### hard_limit diagnosis
When `memory_status` is `hard_limit` and `model_bytes == model_bytes_memory_limit`:
- The model hit its configured ceiling and **stops learning new entities**.
- Results degrade or stop; the datafeed may show `stopped` as a downstream symptom.
- **Restarting the datafeed alone does not fix hard_limit.**
Remediation (in order of preference):
1. **Raise `model_memory_limit`** via `POST /_ml/anomaly_detectors/{job_id}/_update` with
`{"analysis_limits": {"model_memory_limit": "<new_limit>"}}` — requires close/reopen lifecycle.
2. **Reduce cardinality** — fewer partition/by/over values, split into multiple jobs, or narrow the datafeed query.
3. **Estimate sizing** — `POST /_ml/anomaly_detectors/_estimate_model_memory` with representative cardinality hints.
Prefer estimate API over heuristics like `peak_model_bytes * 1.3` — heuristics ignore influencer and categorizer memory.
## Config change lifecycle
Required sequence for memory limit or datafeed timing changes:
1. `POST /_ml/datafeeds/datafeed-{job_id}/_stop`
2. `POST /_ml/anomaly_detectors/{job_id}/_close`
3. Update job and/or datafeed config
4. `POST /_ml/anomaly_detectors/{job_id}/_open`
5. `POST /_ml/datafeeds/datafeed-{job_id}/_start`
Preview after changes: `POST /_ml/datafeeds/datafeed-{job_id}/_preview`.
Recover corrupted model periods: `POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert`.
## Missing documents / query timing
Inspect via `GET /_ml/datafeeds/datafeed-{job_id}`:
| Field | Role |
| --------------------------- | --------------------------------------------------------------- |
| `query_delay` | How far behind real time the datafeed queries |
| `delayed_data_check_config` | How aggressively late data is checked |
| `frequency` | Poll interval (defaults to `min(query_delay, bucket_span / 2)`) |
Search delayed-data annotations:
```json
POST /.ml-annotations-*/_search
{
"query": {
"bool": {
"filter": [
{ "term": { "job_id": "{job_id}" } },
{ "term": { "event": "delayed_data" } }
]
}
}
}
```
Set `query_delay` to **P95 ingest latency + buffer** (typically `60s`–`120s`). Too small → missing docs; too large →
slower alerts. **Fix hard_limit before tuning query_delay** — memory corruption causes false missing-doc alarms.
## Job messages
Search notification index for errors and warnings:
```json
POST /.ml-notifications-*/_search
{
"size": 20,
"sort": [{ "timestamp": "desc" }],
"query": {
"bool": {
"filter": [{ "term": { "job_id": "{job_id}" } } ]
}
}
}
```
## Datafeed state
`GET /_ml/datafeeds/datafeed-{job_id}/_stats` — check `state` (`started` / `stopped`), `timing_stats`, and
`running_state`. A stopped datafeed after hard_limit is expected until memory is fixed and the lifecycle sequence
completes.
references/worked-example.md
# Worked Investigation Example
End-to-end walkthrough of a multi-job anomaly investigation using the [14-step protocol](protocols/investigation.md).
---
## Scenario
**Alert received:** `rcaeval-ob-cpu` — `partition_field_value: frontend` — `record_score: 72` — `2024-03-15T14:30:00Z`
The on-call engineer receives this alert and needs to determine: Is `frontend` the root cause, or a victim of something
upstream?
---
## Phase 1: Discovery
### Step 1 — Discover available jobs
Call `GET /_ml/anomaly_detectors`.
**Result summary:**
```text
job_count: 6
job_ids: [rcaeval-ob-cpu, rcaeval-ob-latency, rcaeval-ob-memory,
rcaeval-ob-errors, rcaeval-nw-throughput, rcaeval-logs-app]
functions: [mean, high_mean, count, rare]
partition_fields: [service]
bucket_spans: [5m]
```
**Interpretation:** 6 jobs all partitioned by `service`. All likely monitor the same system from different angles.
---
### Step 2 — Find related jobs
Call `GET /_ml/datafeeds/datafeed-rcaeval-ob-cpu` → index `rcaeval-re1-ob`. Compare indices across all datafeeds.
**Result:**
```text
Jobs sharing index rcaeval-re1-ob:
rcaeval-ob-cpu, rcaeval-ob-latency, rcaeval-ob-memory, rcaeval-ob-errors
Jobs sharing index rcaeval-re1-nw:
rcaeval-nw-throughput
```
All five observability jobs share `partition_field_name: service`. The `rcaeval-logs-app` job uses `mlcategory` (log
categorization), not `service`. The 5 observability jobs are our related group.
---
### Step 3 — Scope the incident
Call `POST /.ml-anomalies-*/_search` with `result_type: bucket`, `job_id` wildcard `rcaeval-*`, score ≥ 25, window
13:00–16:00.
**Result:**
```text
timestamp max_score job_count composite_score jobs
2024-03-15T14:00:00Z 48 2 76 [rcaeval-ob-cpu, rcaeval-ob-latency]
2024-03-15T14:15:00Z 61 3 138 [rcaeval-ob-cpu, rcaeval-ob-latency, rcaeval-ob-memory]
2024-03-15T14:30:00Z 78 4 198 [rcaeval-ob-cpu, rcaeval-ob-latency, rcaeval-ob-memory, rcaeval-ob-errors]
2024-03-15T14:45:00Z 72 4 201 [rcaeval-ob-cpu, rcaeval-ob-latency, rcaeval-ob-memory, rcaeval-ob-errors]
2024-03-15T15:00:00Z 45 3 112 [rcaeval-ob-latency, rcaeval-ob-memory, rcaeval-ob-errors]
```
**Interpretation:** Peak at 14:30 with 4 jobs co-firing (composite score 198). Incident started ~14:00, peak 14:30,
declining by 15:00. CPU was first to fire (14:00).
---
## Phase 2: Entity Attribution
### Step 4 — Expand from alert
Search influencers/records for `partition_field_value: frontend` across related jobs (13:30–15:30):
```text
job_id max_score anomaly_count first_anomaly functions
rcaeval-ob-cpu 78 8 2024-03-15T14:00:00Z [high_mean]
rcaeval-ob-latency 74 7 2024-03-15T14:05:00Z [high_mean]
rcaeval-ob-memory 61 5 2024-03-15T14:15:00Z [high_mean]
rcaeval-ob-errors 52 4 2024-03-15T14:20:00Z [high_count]
```
**Interpretation:** `frontend` is anomalous in 4 jobs. CPU was first (14:00), then latency (14:05), then memory (14:15),
then errors (14:20). Cascading pattern suggests CPU is upstream.
---
### Step 5 — Multi-job entities
Aggregate influencers by `partition_field_value` across jobs (14:00–15:00, min score 25):
```text
partition_field_value job_count max_score functions
frontend 4 78 [high_mean, high_count]
backend 1 31 [high_mean]
```
**Interpretation:** `frontend` is the only entity anomalous in 4+ jobs — confirmed prime suspect. `backend` appears in
only 1 job at score 31 — likely incidental.
---
### Step 6 — Fingerprint
Query records across `rcaeval-ob-*` jobs (14:00–15:00):
```text
job_id function field_name max_score count
rcaeval-ob-cpu high_mean system.cpu.percent 78 8
rcaeval-ob-latency high_mean http.response_time 74 7
rcaeval-ob-memory high_mean system.memory.used 61 5
rcaeval-ob-errors high_count http.error_count 52 4
```
**Interpretation:** Classic resource exhaustion pattern: CPU spike → latency increase → memory pressure → errors.
---
## Phase 3: Deep Analysis
### Steps 7–10 — Drill down, attribute, profile, characterize
Query records for `rcaeval-ob-cpu` with `partition_field_value: frontend`, score ≥ 50:
```text
record_score: 78
actual: 94.7 (% CPU)
typical: 31.2
multi_bucket_impact: 4
initial_record_score: 79
```
`multi_bucket_impact: 4` → sustained shift across 4+ buckets, not a transient spike. `initial ≈ current` → no
renormalization; score is stable and genuine.
Query influencers for the 14:30 bucket — if multiple hosts appear, rank by **`influencer_score`** to identify the
specific instance driving the service-level anomaly.
---
## Phase 4: Root Cause Confirmation
### Step 11 — Cascade
Sort records by timestamp across jobs:
```text
timestamp job_id record_score function field_name
2024-03-15T14:00:00Z rcaeval-ob-cpu 78 high_mean cpu.percent → FIRST
2024-03-15T14:05:00Z rcaeval-ob-latency 74 high_mean response_time
2024-03-15T14:15:00Z rcaeval-ob-memory 61 high_mean memory.used
2024-03-15T14:20:00Z rcaeval-ob-errors 52 high_count error_count
```
**Interpretation:** CPU anomaly at 14:00 is the earliest — this is the root cause signal.
### Step 12 — Evidence
`GET /_ml/datafeeds/datafeed-rcaeval-ob-cpu` → source index `rcaeval-re1-ob`.
Search source index for `service: frontend`, window 13:50–14:10:
```json
{"service": "frontend", "system.cpu.percent": 96.2, "event": "metricset", "@timestamp": "2024-03-15T14:00:34Z"}
{"service": "frontend", "system.cpu.percent": 93.8, "event": "metricset", "@timestamp": "2024-03-15T14:01:05Z"}
{"service": "frontend", "system.cpu.percent": 94.1, "event": "metricset", "@timestamp": "2024-03-15T14:01:41Z"}
```
**Interpretation:** CPU is genuinely at ~95% on `frontend` starting at 14:00, confirming the ML anomaly matches reality.
---
## Step 14 — RCA Report
**Root cause:** `frontend` service experienced CPU exhaustion starting at 2024-03-15T14:00Z.
**Evidence:**
- CPU rose from baseline 31% to 94–97% at 14:00 (score: 78, `multi_bucket_impact: 4` — sustained shift)
- CPU anomaly preceded all other anomalies by 5–20 minutes
- `frontend` was the only entity anomalous in 4 jobs simultaneously — no other service implicated
**Cascading impact:**
1. **14:00** — CPU saturates (94%+)
2. **14:05** — HTTP latency climbs (CPU-bound request processing)
3. **14:15** — Memory rises (queued/retried requests consuming heap)
4. **14:20** — Error rate spikes (timeouts from downstream callers)
**Fault type:** Resource exhaustion — CPU-bound processing on `frontend` pod(s)
**Recommended actions:**
1. Check `frontend` deployment for runaway process or CPU-hungry code path (profiling)
2. Review recent deploys to `frontend` in the 30 min before 14:00
3. Horizontal scale or vertical CPU limit increase as immediate mitigation
4. Add CPU throttling alert at 80% to catch before next saturation
**Severity:** Score 78, `multi_bucket_impact: 4`, duration ~1 hour — **high severity**
---
## See Also
- [protocols/investigation.md](protocols/investigation.md) — Full 14-step protocol
- [investigation-queries.md](investigation-queries.md) — Search query templates
- [score-reference.md](score-reference.md) — Score field definitions and severity bands
SKILL.md
---
name: kibana-anomaly-detection
description: >
Elastic ML anomaly detection — investigation/RCA, score explanation, job lifecycle
troubleshooting, and job operations. Use when answering "what broke?"/"which entity?"/RCA,
"why is score high/low?"/renormalization, "datafeed stopped"/"memory limit"/hard_limit,
or configuring ML anomaly detection jobs. Reads results from `.ml-anomalies-*` and
job state from ML REST APIs.
metadata:
author: elastic
version: 0.3.0
universal: true
compatibility: Elasticsearch 8.x–9.x or Elastic Cloud Serverless with ML anomaly detection;
Kibana 8.x–9.x for saved-object context only
---
# Elastic ML Anomaly Detection
Expert process for ML anomaly detection: attribute incidents to entities, explain scores and model behavior, diagnose
job lifecycle failures, and manage jobs. Read anomaly **results** from `POST /.ml-anomalies-*/_search` (Serverless-safe)
and **job/datafeed state** from ML REST APIs. When the user embeds fixture evidence (influencer rows, job stats) in the
prompt, apply the judgment below directly — do not re-fetch fields already supplied.
<!-- begin-partial: preamble -->
## Environment Configuration
This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.
This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.
<!-- end-partial: preamble -->
## Mode selector
| User intent | Mode |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| "What broke?" / RCA / cross-job / blast radius / influencers / log categories | **Investigate** |
| "Why score high/low?" / renormalization / model bounds / forecasts | **Explain** |
| Missing docs / memory limit / datafeed stopped / lifecycle / calendars | **Troubleshoot** |
| Create a job / configure a datafeed / start analysis / retrieve results | **Manage** |
| Security framing (attack chains, MITRE, exfil) | Investigate + [references/security-anomaly-expert.md](references/security-anomaly-expert.md) |
| Observability/SRE framing (degradation, capacity, deployment regression) | Investigate + [references/observability-anomaly-expert.md](references/observability-anomaly-expert.md) |
When a question spans modes: **Investigate → Explain → Troubleshoot**. Finish one mode before blending logic.
> **Serverless note:** Legacy `/_ml/anomaly_detectors/{job_id}/results/*` endpoints return HTTP 410 in Serverless.
> Always query `.ml-anomalies-*` via `POST /.ml-anomalies-*/_search` with `result_type` filters.
## Score quick reference
- `record_score` bands: **>75** critical · **50–75** warning · **25–50** minor · **<25** informational
- `multi_bucket_impact ≥ 3` → sustained shift (not a transient spike)
- `initial_record_score >> record_score` → renormalization (model saw worse anomalies later)
- `actual << typical` with `count`/`low_count`/`low_mean` → absence/outage, not just a low value
- Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity
> Full score definitions, renormalization mechanics, and `anomaly_score_explanation` components:
> [references/score-reference.md](references/score-reference.md).
## Core concepts
Treat `.ml-anomalies-*` as layered result types via `result_type` in search queries:
| `result_type` | Scope | Key fields |
| --------------------- | --------------- | ---------------------------------------------------------------------------------------- |
| `bucket` | Time window | `anomaly_score`, `initial_anomaly_score`, `timestamp` |
| `record` | Detector row | `record_score`, `initial_record_score`, `actual`, `typical`, `anomaly_score_explanation` |
| `influencer` | Entity × bucket | `influencer_field_name`, `influencer_field_value`, **`influencer_score`** |
| `model_plot` | Bounds | `model_lower`, `model_upper`, `actual` |
| `category_definition` | Log patterns | `category_id`, `terms`, `regex`, `examples` |
Read scores this way:
- `anomaly_score` / `record_score` = **current normalized** values (move as the model sees new extremes).
- `initial_anomaly_score` / `initial_record_score` = **immutable snapshots** from detection time.
- **`influencer_score` ranks entity responsibility within a bucket** — the highest score is the primary suspect, not the
bucket-level `anomaly_score` alone.
- Map entities via `partition_field_value` / `by_field_value` / `over_field_value`.
- Read `multi_bucket_impact` (-5 to +5) to separate single-bucket spikes from sustained trends.
---
## Mode: Investigate — RCA
**When:** "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.
### Process
1. **Discover jobs.** Call `GET /_ml/anomaly_detectors` when the job ID is unknown. Call
`GET /_ml/anomaly_detectors/{job_id}` and `GET /_ml/datafeeds/datafeed-{job_id}` to learn source indices, entity
fields (`by_field_name`, `over_field_name`, `partition_field_name`), and `bucket_span`. The decision: identify the
related job group — jobs sharing a datafeed index or entity field monitor the same system from different angles.
2. **Scope the incident window.** Call `POST /.ml-anomalies-*/_search` with `result_type: bucket`, a time range, and
optional minimum `anomaly_score`. The decision: fix the incident start/end and count how many jobs co-fire in that
window. Low scores across many jobs simultaneously often indicate a systemic root cause.
3. **Attribute to entities (critical for RCA).** For the anomalous bucket timestamp, call
`POST /.ml-anomalies-*/_search` with `result_type: influencer`, the job ID(s), and the bucket time range. Sort by
**`influencer_score` descending**. The decision: name the entity with the **highest `influencer_score`** as the
likely cause — it ranks how unusual each entity is in that bucket. Do not restate only the bucket `anomaly_score`
without attributing responsibility. Recommend drilling into that entity's records next.
4. **Cross-job confirmation.** Re-query influencers (or bucket records) across related job IDs for the same entity
values and time window. Entities anomalous in **2+ jobs** are prime suspects (resource fault or systemic failure);
single-job entities are often downstream victims. See
[references/protocols/investigation.md](references/protocols/investigation.md).
5. **Drill into records.** Call `POST /.ml-anomalies-*/_search` with `result_type: record`, exact job ID, entity filters
(`partition_field_value`, `by_field_value`), and low minimum `record_score` (25 or lower). Read
`multi_bucket_impact ≥ 3` as sustained behavioral shift. Read `actual` vs `typical` for fault class (spike vs
absence/outage).
6. **Confirm with source evidence.** Call `POST /{index}/_search` on the datafeed source index for the suspect entity
and time window. Raw source documents are ground truth — never close an RCA without them.
7. **Synthesize.** Report: **root cause entity · affected jobs · temporal progression · fault class · severity ·
recommended actions**. Worked walkthrough: [references/worked-example.md](references/worked-example.md). Query
templates: [references/investigation-queries.md](references/investigation-queries.md).
### Rules
1. **Rank by `influencer_score`, not `anomaly_score`, for "which entity?"** — bucket score is aggregate; influencer
score attributes cause.
2. **Multi-job entities are prime suspects; single-job entities are usually victims.**
3. **Earliest anomaly timestamp wins** — reconstruct chronology from record timestamps across jobs.
4. **`multi_bucket_impact ≥ 3` = sustained behavioral shift**, weight higher than transient spikes.
5. **Use low score thresholds (25 or lower) for influencer/record queries** — high thresholds miss correlated entities.
6. **Never close an RCA without source evidence** from the datafeed index.
---
## Mode: Explain — Score / model behavior
**When:** "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?".
### Process
1. **Decide fetch vs interpret.** If the user supplies a record with `record_score`, `initial_record_score`, `actual`,
and `typical`, interpret directly. Otherwise load config with `GET /_ml/anomaly_detectors/{job_id}` and records with
`POST /.ml-anomalies-*/_search` (`result_type: record`).
2. **Always show both `initial_record_score` and `record_score`.** The gap is the renormalization story. Large positive
drift (`initial_record_score >> record_score`) means a later, more extreme anomaly rescale this record downward —
expected healthy behavior, not a broken model.
3. **Classify the pattern before speculating.**
| Pattern | Interpretation |
| ------------------------------------------------------------ | ----------------------------------------------------------------- |
| `initial_record_score >> record_score` | Renormalization — explain before suggesting config changes |
| `actual << typical` with `low_count`/`count`/`low_mean` | Absence/outage anomaly — investigate the outage, not score tuning |
| `high_variance_penalty: true` in `anomaly_score_explanation` | Noisy metric — wide bounds absorbed the spike |
| `incomplete_bucket_penalty: true` | Ingest lag or sparse bucket — score legitimately reduced |
Only cite `anomaly_score_explanation` factors **present** in the record.
4. **Quantify renormalization (optional).** Re-query records sorted by `timestamp`; compute
`score_drift = initial_record_score - record_score` and flag large drift.
5. **Add visual context when needed.** If `model_plot_config.enabled`, query `result_type: model_plot` and compare
`actual` to `model_lower`/`model_upper`. For categorization jobs, query `result_type: category_definition`.
6. **Check job health when scores look wrong persistently.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` —
`model_size_stats.memory_status` of `hard_limit` corrupts learning and can invalidate scores. Escalate to
Troubleshoot mode.
### `anomaly_score_explanation` components
| Component | Effect | What it means |
| -------------------------------- | ------- | ------------------------------------------------------------ |
| `anomaly_length` | ↑ score | More consecutive anomalous buckets |
| `single_bucket_impact` | ↑ score | Lower probability → higher impact |
| `multi_bucket_impact` | ↑ score | Sustained pattern contribution |
| `anomaly_characteristics_impact` | ↑ score | Mean shift vs. variance change |
| `high_variance_penalty` | ↓ score | Noisy data → wide bounds → anomaly less surprising |
| `incomplete_bucket_penalty` | ↓ score | Bucket has less data than expected (ingest lag, sparse data) |
### Rules
1. **Explain renormalization before diagnosing config** — score drift is the most common "score dropped" cause.
2. **`actual << typical` with count/low_count is an absence anomaly** — distinguish outages from value spikes.
3. **Weekly seasonality needs ≥3 weeks of training data** — flag young jobs as the cause.
4. **Detector function direction matters** — see
[references/anomaly-detection-functions.md](references/anomaly-detection-functions.md).
---
## Mode: Troubleshoot — Job lifecycle
**When:** "missing documents", "datafeed stopped", **`hard_limit`**, "results look wrong", lifecycle changes.
### Process
1. **Load job and datafeed state.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` and
`GET /_ml/datafeeds/datafeed-{job_id}/_stats`. Read `state`, `data_counts`, **`model_size_stats`**, and datafeed
`state`. If the user embeds stats JSON, diagnose from `memory_status` and datafeed state directly.
2. **Diagnose memory status first (critical).** Inspect `model_size_stats`:
| Field | Meaning |
| -------------------------- | ----------------------------------------------------------- |
| `memory_status` | `ok` / `soft_limit` (pruning) / **`hard_limit` (critical)** |
| `model_bytes` | Current memory used |
| `model_bytes_memory_limit` | Configured `model_memory_limit` |
When **`memory_status` is `hard_limit`** and `model_bytes` equals `model_bytes_memory_limit`, the model hit its
memory ceiling — it stops learning new entities and results degrade or stop. A stopped datafeed is often a
**symptom**, not the root cause. **Do not recommend only restarting the datafeed** — that alone does not clear a hard
limit.
3. **Remediate hard_limit.** The fix is to **raise `model_memory_limit`** (via job update) **and/or reduce model size**
by lowering cardinality (fewer partition/by/over field values, split into multiple jobs). Raising the limit requires
the lifecycle sequence below (stop datafeed → close job → update → open → start). Optionally call
`POST /_ml/anomaly_detectors/_estimate_model_memory` to size the new limit from source cardinality.
4. **Diagnose missing documents / query timing.** After memory is healthy, inspect datafeed `query_delay` and
`delayed_data_check_config` via `GET /_ml/datafeeds/datafeed-{job_id}`. Search `.ml-annotations-*` for delayed-data
events. Set `query_delay` to P95 ingest latency + buffer (default `60s`–`120s`).
5. **Read job messages.** Search `.ml-notifications-*` for the job ID when errors are unclear.
6. **Recover corrupted model state.** Call `POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert`
to revert to a known-good snapshot when the model was corrupted during hard_limit.
### Lifecycle for config changes (memory limit, query_delay)
Apply in order — skipping steps causes rejected updates:
1. `POST /_ml/datafeeds/datafeed-{job_id}/_stop`
2. `POST /_ml/anomaly_detectors/{job_id}/_close`
3. `POST /_ml/anomaly_detectors/{job_id}/_update` (memory limit) and/or `POST /_ml/datafeeds/datafeed-{job_id}/_update`
(query_delay)
4. `POST /_ml/anomaly_detectors/{job_id}/_open`
5. `POST /_ml/datafeeds/datafeed-{job_id}/_start`
Preview changes with `POST /_ml/datafeeds/datafeed-{job_id}/_preview` before restarting.
> **`hard_limit` corrupts model state** and causes downstream missing-doc false alarms. **Fix memory before fixing
> `query_delay`.** Full troubleshooting detail:
> [references/troubleshooting-reference.md](references/troubleshooting-reference.md).
### Rules
1. **Ground lifecycle diagnosis in `memory_status`** — not generic "restart it" advice.
2. **Fix memory before `query_delay`** — hard_limit invalidates downstream diagnostics.
3. **Stop datafeed → close job → update → open → start** for any memory or datafeed config change.
4. **Do not delete the job** as first remediation for hard_limit — raise limit and/or reduce cardinality.
---
## Mode: Manage — Create / configure jobs
**When:** "set up a job", "create an ML detector", "monitor X over time".
For the full create/open/start lifecycle, prefer the `elasticsearch-anomaly-detection` skill. This mode summarizes the
sequence and detector selection:
1. **Verify target index.** Call `GET /{index}/_mapping` — confirm time field and detector fields exist.
2. **Create job.** Call `PUT /_ml/anomaly_detectors/{job_id}` with `analysis_config` (detectors, `bucket_span`,
influencers) and `data_description.time_field`.
3. **Create datafeed.** Call `PUT /_ml/datafeeds/datafeed-{job_id}` with `indices`, `query`, and `query_delay`.
4. **Open and start.** Call `POST /_ml/anomaly_detectors/{job_id}/_open`, then
`POST /_ml/datafeeds/datafeed-{job_id}/_start`.
5. **Confirm.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` and `GET /_ml/datafeeds/datafeed-{job_id}/_stats`.
Choose detector functions from user intent — see
[references/anomaly-detection-functions.md](references/anomaly-detection-functions.md). Worked JSON bodies:
[references/job-creation-recipes.md](references/job-creation-recipes.md).
### Rules
1. **Create job before datafeed.** Open job before starting datafeed.
2. **`query_delay` = P95 ingest latency + buffer** (60s–120s safe default).
3. **`by_field_name` vs `over_field_name`:** `by` compares entity to its own history; `over` compares to peer group.
4. **Forecasts require non-population jobs** — jobs with `over_field_name` cannot be forecasted.
---
## Examples
**RCA:** "Something caused a spike in checkout latency — which entity?" → Query influencers for the bucket → **web-07**
has highest `influencer_score` (91.5) vs 22.0 and 8.4 → name web-07 as likely cause → recommend drilling into its
records — do not answer with only bucket `anomaly_score` 88.
**Score drop:** "Score went from 90 to 55 — did the model change?" → Compare `initial_record_score` vs `record_score` →
explain renormalization if drift is large.
**Memory limit:** "Job shows `hard_limit` and datafeed stopped." → Diagnose
`model_size_stats.memory_status = hard_limit` → raise `model_memory_limit` via close/update/open lifecycle and/or reduce
cardinality — **not** "just restart the datafeed".
**New job:** "Detect unusual error rates per host." → `high_count` with `by_field_name: host.keyword` →
create/open/start sequence.
---
## Guidelines
1. **Pick a mode first.** Don't blend RCA logic with score-explanation logic in one response.
2. **For "which entity?" rank `influencer_score`**, not bucket `anomaly_score`.
3. **For lifecycle failures read `memory_status`** before recommending datafeed restarts.
4. **Show `initial_record_score` alongside `record_score`** — the gap tells the renormalization story.
5. **Fix memory before `query_delay`.** Hard_limit invalidates downstream diagnostics.
6. **Confirm RCAs with source evidence** from the datafeed index.
## Operations
| HTTP API (shorthand) | `elastic` CLI command |
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `GET /{index}/_mapping` | `elastic es indices get-mapping --index '<index>'` |
| `POST /{index}/_search` | `elastic es search --index '<index>' --input-file '<search-body.json>'` |
| `GET /_ml/anomaly_detectors` | `elastic es ml get-jobs` |
| `GET /_ml/anomaly_detectors/{job_id}` | `elastic es ml get-jobs --job-id '<job_id>'` |
| `GET /_ml/anomaly_detectors/{job_id}/_stats` | `elastic es ml get-job-stats --job-id '<job_id>'` |
| `GET /_ml/datafeeds/datafeed-{job_id}` | `elastic es ml get-datafeeds --datafeed-id 'datafeed-<job_id>'` |
| `GET /_ml/datafeeds/datafeed-{job_id}/_stats` | `elastic es ml get-datafeed-stats --datafeed-id 'datafeed-<job_id>'` |
| `POST /.ml-anomalies-*/_search` | `elastic es search --index '.ml-anomalies-*' --input-file '<search-body.json>'` |
| `POST /.ml-annotations-*/_search` | `elastic es search --index '.ml-annotations-*' --input-file '<search-body.json>'` |
| `POST /.ml-notifications-*/_search` | `elastic es search --index '.ml-notifications-*' --input-file '<search-body.json>'` |
| `POST /_ml/anomaly_detectors/_estimate_model_memory` | `elastic es ml estimate-model-memory --analysis-config '<json>'` |
| `PUT /_ml/anomaly_detectors/{job_id}` | `elastic es ml put-job --job-id '<job_id>' --input-file '<job-body.json>'` |
| `PUT /_ml/datafeeds/datafeed-{job_id}` | `elastic es ml put-datafeed --datafeed-id 'datafeed-<job_id>' --input-file '<datafeed-body.json>'` |
| `POST /_ml/anomaly_detectors/{job_id}/_open` | `elastic es ml open-job --job-id '<job_id>'` |
| `POST /_ml/anomaly_detectors/{job_id}/_close` | `elastic es ml close-job --job-id '<job_id>'` |
| `POST /_ml/anomaly_detectors/{job_id}/_update` | `elastic es ml update-job --job-id '<job_id>' --analysis-limits '<json>'` |
| `POST /_ml/datafeeds/datafeed-{job_id}/_update` | `elastic es ml update-datafeed --datafeed-id 'datafeed-<job_id>' --input-file '<update-body.json>'` |
| `POST /_ml/datafeeds/datafeed-{job_id}/_start` | `elastic es ml start-datafeed --datafeed-id 'datafeed-<job_id>'` |
| `POST /_ml/datafeeds/datafeed-{job_id}/_stop` | `elastic es ml stop-datafeed --datafeed-id 'datafeed-<job_id>'` |
| `POST /_ml/datafeeds/datafeed-{job_id}/_preview` | `elastic es ml preview-datafeed --datafeed-id 'datafeed-<job_id>'` |
| `POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert` | `elastic es ml revert-model-snapshot --job-id '<job_id>' --snapshot-id '<snapshot_id>'` |
Search body shapes for each `result_type` and troubleshooting queries are documented in
[references/investigation-queries.md](references/investigation-queries.md) and
[references/troubleshooting-reference.md](references/troubleshooting-reference.md).