references/apm-signals.md
# APM Signals
ES|QL patterns for the trace- and metric-derived signals used in triage: throughput, latency, error rate, dependency
health, subpopulation correlation, ML anomalies, and infrastructure saturation. Every query here was executed against
Elasticsearch 9.6.0 with live OpenTelemetry data. Run them with `POST /_query`.
## Scoping rules
- Filter by `service.name` and a bounded `@timestamp` range on every query.
- Add `kind == "Server"` when measuring a service's own throughput and latency. Without it, client spans emitted by the
same service are counted as inbound traffic and both the rate and the percentiles are wrong.
- `duration` on OTel spans is in **nanoseconds**. Divide by `1000000.0` for milliseconds.
- Prefer `event.outcome == "failure"` to identify failed spans. `status.code == "Error"` marks the same spans but is
`null` on successful ones, so aggregating it requires a `CASE`.
- Cap every query with `LIMIT`. Choose a bucket size that yields roughly 20-50 buckets over the window.
## Throughput, latency, and error rate over time
```esql
FROM traces-*.otel-*
| WHERE service.name == "frontend" AND kind == "Server"
AND @timestamp >= NOW() - 1 hour
| STATS requests = COUNT(*),
failures = COUNT(*) WHERE event.outcome == "failure",
p95_ms = ROUND(PERCENTILE(duration, 95) / 1000000.0, 2)
BY bucket = BUCKET(@timestamp, 5 minute)
| EVAL error_rate = ROUND(COALESCE(failures, 0)::double / requests, 4)
| SORT bucket DESC
| LIMIT 500
```
`COALESCE` matters: `COUNT(*) WHERE ...` returns `null`, not `0`, when nothing matches, and a `null` error rate reads as
missing data rather than as zero failures.
## Current window against the prior window
A single window of absolute numbers does not support a verdict. Split one query into two comparable windows:
```esql
FROM traces-*.otel-*
| WHERE service.name == "frontend" AND kind == "Server"
AND @timestamp >= NOW() - 2 hours
| EVAL window = CASE(@timestamp >= NOW() - 1 hour, "current", "previous")
| STATS requests = COUNT(*),
avg_ms = ROUND(AVG(duration) / 1000000.0, 2),
p95_ms = ROUND(PERCENTILE(duration, 95) / 1000000.0, 2),
p99_ms = ROUND(PERCENTILE(duration, 99) / 1000000.0, 2),
failures = COUNT(*) WHERE event.outcome == "failure"
BY window
| EVAL error_rate = ROUND(COALESCE(failures, 0)::double / requests, 4)
| KEEP window, requests, avg_ms, p95_ms, p99_ms, error_rate
```
Reading it:
- **p99 up, p50/avg flat** — a tail problem. A subset of requests is affected; go to subpopulation correlation.
- **All percentiles up together** — a whole-service change: a deploy, a saturated resource, or a slow dependency on the
hot path.
- **Requests down sharply, error rate flat** — the callers stopped calling. The change is upstream, not here.
- **Requests up and latency up** — load-driven. Check saturation before blaming code.
## Error rate by route
Localize errors to a route before going deeper:
```esql
FROM traces-*.otel-*
| WHERE service.name == "checkout" AND kind == "Server"
AND @timestamp >= NOW() - 1 hour
| STATS requests = COUNT(*),
failures = COUNT(*) WHERE event.outcome == "failure"
BY span.name
| EVAL error_rate = ROUND(COALESCE(failures, 0)::double / requests, 4)
| WHERE requests >= 20
| SORT error_rate DESC
| LIMIT 20
```
`transaction.name` is available on transaction documents and is often coarser (for example the HTTP method); `span.name`
carries the operation. Group by whichever is populated for the service at hand.
## Dependency health
Downstream call volume, latency, and failure rate come from the 1-minute dependency rollup:
```esql
FROM metrics-service_destination.1m.otel-*
| WHERE service.name == "frontend" AND @timestamp >= NOW() - 1 hour
| STATS calls = SUM(span.destination.service.response_time.count),
total_us = SUM(span.destination.service.response_time.sum.us),
failed = SUM(span.destination.service.response_time.count) WHERE event.outcome == "failure"
BY span.destination.service.resource
| EVAL avg_ms = ROUND(total_us / calls / 1000.0, 2),
failure_rate = ROUND(COALESCE(failed, 0)::double / calls, 4)
| KEEP span.destination.service.resource, calls, avg_ms, failure_rate
| SORT calls DESC
| LIMIT 20
```
**Zero rows is a finding, not a pass.** A service with no rows in `metrics-service_destination.1m.otel-*` is not
APM-instrumented for dependencies. Report insufficient dependency data and give the verdict from the signals that do
exist. Never translate an empty dependency result into "upstreams are healthy".
For a service-map-style view of which services call which, aggregate the destination rollup across services:
```esql
FROM metrics-service_destination.1m.otel-*
| WHERE @timestamp >= NOW() - 1 hour
| STATS calls = SUM(span.destination.service.response_time.count)
BY service.name, span.destination.service.resource
| SORT calls DESC
| LIMIT 50
```
## Subpopulation correlation
Replaces the correlations technique that previously required a bespoke script. When only part of the traffic is slow or
failing, the question is: **which attribute value is over-represented in the affected set relative to the population as
a whole?**
Compute the overall rate and the per-attribute rate in one query, using `FORK` so both come back from a single call.
`FORK` is GA on Serverless; on Stack it is preview in 9.1-9.3 and GA from 9.4, and it does not parse below 9.1. Where it
is unavailable, run the two branches as two queries and compute the lift yourself — `FORK` saves a round trip here,
nothing more:
```esql
FROM traces-*.otel-*
| WHERE service.name == "frontend" AND kind == "Server"
AND @timestamp >= NOW() - 3 hours
| EVAL affected = CASE(duration > 100000000, 1, 0)
| FORK (STATS total = COUNT(*), affected = SUM(affected) | EVAL scope = "overall", attribute_value = "*")
(STATS total = COUNT(*), affected = SUM(affected) BY attribute_value = transaction.name | EVAL scope = "by-attribute")
| EVAL rate = ROUND(affected::double / total, 4)
| KEEP scope, attribute_value, total, affected, rate
| SORT rate DESC
| LIMIT 20
```
The `SORT` and `LIMIT` here apply to both branches combined, so on a service with more than twenty distinct
`transaction.name` values the `overall` row competes with the per-attribute rows and can be pushed out of the result —
leaving you the subpopulations with no baseline to compare them against. Raise the `LIMIT` above the expected
cardinality, or read the baseline from the separate error-rate query instead. This is the same combined-limit behaviour
that silently drops whole branches in [the log funnel](log-investigation.md#the-one-call-funnel-query); it is worth
knowing wherever `FORK` is followed by a limit.
Swap the `affected` expression for the symptom under investigation:
- Failures: `EVAL affected = CASE(event.outcome == "failure", 1, 0)`
- Slow requests: `EVAL affected = CASE(duration > 100000000, 1, 0)` — pick the nanosecond threshold from the p95 of the
healthy window, not from a round number.
Repeat the query once per candidate attribute, changing only the `BY attribute_value = ...` clause. Candidates worth
testing, in the order they usually pay off:
`service.version` · `k8s.pod.name` · `host.name` · `k8s.deployment.name` · `container.id` · `cloud.region` ·
`cloud.availability_zone` · `service.environment` · `span.name` · `transaction.name` · `http.response.status_code`
### What makes an attribute correlated
An attribute value is correlated when all three hold:
1. **Lift.** Its rate is meaningfully higher than the overall rate — roughly 2x or more. Small differences on a busy
service are noise.
2. **Volume.** It has enough events to be stable. Discard groups below about 20 events; a single failure out of three
requests is a 33% rate and means nothing.
3. **Concentration.** It accounts for a substantial share of the total affected events. An attribute with a 90% failure
rate that explains 4 of 500 failures is a curiosity, not the cause.
An attribute that shows high lift **and** covers most of the affected events is the localization. On live data, frontend
server spans grouped by route gave a 3.8% slow rate for `POST` against a 0.9% overall rate — a 4x lift covering half the
slow requests, which localizes the problem to write paths.
Two traps:
- **Cardinality artifacts.** Attributes near-unique per request (trace ID, user ID, session ID) always look correlated.
Only test attributes shared by many events.
- **Proxy attributes.** If a single pod is running the only instance of a bad version, `k8s.pod.name` and
`service.version` both light up. Prefer the explanation with a mechanism — a version rollout beats a pod name — and
check whether the attribute values move together.
When no attribute shows lift, the degradation is uniform across the population. That is itself a result: report that the
problem is service-wide and look at dependencies or infrastructure instead of hunting for a slice.
## ML anomalies
Anomaly detection describes deviation from a learned baseline, not from a target, so it corroborates and time-bounds a
finding rather than setting the verdict.
1. `GET /_ml/anomaly_detectors` — find jobs whose configuration references the service.
2. `GET /_ml/anomaly_detectors/_stats` — confirm the job state is `opened` and the datafeed is running. A stopped job
produces no records, which is not the same as no anomaly. Report the job as unavailable in that case.
3. `GET /_ml/anomaly_detectors/{id}/results/records` — read scored records. Treat `record_score` at or above 75 as
significant, 50-75 as worth corroborating, below 50 as background.
Use the anomaly's start time to narrow the trace and log windows in the rest of the triage.
## Infrastructure saturation
Read the resource attributes off the service's own spans first, so infrastructure is scoped to the instances actually
serving traffic:
```esql
FROM traces-*.otel-*
| WHERE service.name == "cart" AND @timestamp >= NOW() - 1 hour
| STATS spans = COUNT(*) BY k8s.pod.name, k8s.namespace.name, host.name
| SORT spans DESC
| LIMIT 20
```
That result decides which branch to take, and the branch matters because the two paths share no field names. Pod and
namespace attributes mean the service is Kubernetes-hosted and the kubeletstats fields apply. A populated `host.name`
with no pod attributes means the service runs on a VM or a bare host, where every `k8s.*` field is empty — read the host
metrics instead. Do not conclude that a service is unsaturated because the Kubernetes query returned nothing.
### Kubernetes-hosted services
Check limit utilization per container for the pods that serve the traffic:
```esql
FROM metrics-kubeletstatsreceiver.otel-*
| WHERE @timestamp >= NOW() - 1 hour AND k8s.namespace.name == "otel-demo"
| STATS cpu_limit_pct = ROUND(MAX(k8s.container.cpu_limit_utilization) * 100, 1),
mem_limit_pct = ROUND(MAX(k8s.container.memory_limit_utilization) * 100, 1)
BY k8s.pod.name, k8s.container.name
| SORT mem_limit_pct DESC
| LIMIT 20
```
Memory limit utilization approaching 100% precedes OOM kills; sustained CPU limit utilization at 100% means throttling,
which shows up in APM as latency without any error-rate change. Container restarts and OOM events are also visible in
`logs-k8seventsreceiver.otel-*` when the Kubernetes events receiver is deployed — absence of that data stream means the
receiver is not installed, not that no restarts happened.
The container-level fields are the measure to use, and the pod-level pair `k8s.pod.cpu_limit_utilization` /
`k8s.pod.memory_limit_utilization` is not a substitute for them. See
[Container-level against pod-level limit utilization](#container-level-against-pod-level-limit-utilization) below for
what separates them and why the container fields are the default. **observability-k8s-investigation** applies the same
rule.
`k8s.container.cpu_limit_utilization` and `k8s.container.memory_limit_utilization` are only populated when the container
declares the corresponding limit. A `null` result means no limit is set, not that the container is idle.
**These two fields fail in two different ways, and only one of them returns `null`.** Where the receiver emits the
metric, an undeclared limit gives `null` per the paragraph above. Where the metric is not enabled at all, the field is
absent from the mapping and the query above **fails with HTTP 400 `Unknown column`** rather than returning `null`. On a
Stack 9.4.4 cluster `k8s.container.memory_limit_utilization` was absent from `metrics-kubeletstatsreceiver.otel-*`
entirely while `k8s.container.cpu_limit_utilization` was present, so the query above returned
`Unknown column [k8s.container.memory_limit_utilization], did you mean [k8s.container.cpu_limit_utilization]?` — and
both fields were present on a Serverless comparator, so this is a collector-configuration difference rather than a
flavour one. Run `GET /_field_caps` on the two field names before building the query, and drop whichever is unmapped
rather than reading the error as missing data.
Two fall-backs, in order of preference:
- `k8s.pod.cpu.node.utilization` and `k8s.pod.memory.node.utilization` express consumption as a fraction of node
capacity and are emitted regardless of whether limits are declared. They answer "is this pod a heavy tenant of its
node?" rather than "is this pod at its own ceiling." Both were populated on the Stack cluster above and returned
sensible values where the container-level fields were unusable.
- `container.cpu.usage` and `container.memory.usage` give absolute consumption, which is only interpretable against a
baseline for the same workload. `container.memory.usage` is itself absent on some collector builds — the Stack cluster
above carried `container.memory.working_set` instead — so confirm which of the two exists before using it.
Both fall-backs sit on the pod-level documents, not the container-level ones, so query them grouped by `k8s.pod.name`
rather than by `k8s.container.name`.
Saturation is the mechanism, not the verdict. When the finding is a restart loop, an OOM kill to confirm, node pressure,
an admission failure, or a stuck rollout, stop here and hand off to the **observability-k8s-investigation** skill, which
owns workload, node, and control-plane diagnosis.
#### Container-level against pod-level limit utilization
Both field families exist in the kubeletstats receiver and they are not interchangeable. The receiver emits them on
**separate documents in the same data stream**: pod-level fields appear on documents that carry no `k8s.container.name`,
and container-level fields appear only on documents that do. A `STATS ... BY k8s.container.name` therefore returns
`null` for every pod-level field, and the reverse holds too. Measured over one hour on a live 9.6.0 cluster: of 42,240
documents without a container name, 360 carried `k8s.pod.cpu_limit_utilization` and none carried
`k8s.container.cpu_limit_utilization`; of 18,240 documents with a container name, 900 carried the container field and
none carried the pod field.
Availability differs as well. Container-level utilization is emitted for each container that declares the limit, while
the pod-level aggregate requires **every** container in the pod to declare it. Across two live clusters over three
hours, no pod carried the pod-level field without also carrying the container-level one, while 27 pods carried the
container-level field with the pod-level field absent — every one of them a multi-container pod in which only some
containers declared limits.
| Cluster | Container level only | Both levels | Neither | Pod level only |
| -------------------------- | -------------------- | ----------- | ------- | -------------- |
| forge-factory (Serverless) | 18 | 7 | 44 | 0 |
| k8s-demo (Serverless) | 9 | 6 | 40 | 0 |
| Stack 9.4.4, CPU | 0 | 3 | 19 | 0 |
| Stack 9.4.4, memory | 0 | 0 | 18 | 4 |
**The "pod level only: 0" column does not generalize.** On the Stack cluster the container-level memory field was not
mapped at all, so four pods carried pod-level memory limit utilization with no container-level counterpart — the case
both Serverless clusters showed zero of. Where the container-level field exists it remains the right default, because a
limit is enforced per container and the container form is emitted per container that declares one. But "available
strictly more often" is a property of those two Serverless deployments, not a rule: check which of the two families is
mapped before choosing, and be prepared for the pod-level field to be the only one available. Use the pod-level fields
when the question really is about the pod as a whole — total pod consumption against the sum of its containers' limits —
or when the container-level field is absent, and in either case say which level the number came from.
### Host-based services
For a service on a VM or bare host with the OTel hostmetrics receiver collecting, saturation comes from
`metrics-hostmetricsreceiver.otel-*`. CPU utilization there is reported per state, so busy CPU is derived from the
`idle` state rather than read directly:
```esql
FROM metrics-hostmetricsreceiver.otel-*
| WHERE host.name == "prod-app-01" AND @timestamp >= NOW() - 1 hour
| STATS cpu_idle = AVG(system.cpu.utilization) WHERE state == "idle",
mem_used_pct = ROUND(MAX(system.memory.utilization) * 100, 1) WHERE state == "used",
load_1m = ROUND(MAX(`system.cpu.load_average.1m`), 2),
cores = MAX(system.cpu.logical.count)
BY host.name
| EVAL cpu_busy_pct = ROUND((1 - cpu_idle) * 100, 1)
| KEEP host.name, cpu_busy_pct, mem_used_pct, load_1m, cores
| SORT cpu_busy_pct DESC
| LIMIT 20
```
Two syntax points that cause silent wrong answers. `system.cpu.utilization` and `system.memory.utilization` are both
broken out by a `state` dimension, so an unfiltered `AVG` averages across `idle`, `user`, `system`, `wait`, and `steal`
and means nothing. And `system.cpu.load_average.1m` needs backticks in ES|QL, because the `1m` segment starts with a
digit; without them the query fails to parse.
This query is executed and confirmed on Stack 9.4.4: it returned `cpu_busy_pct` 4.4, `mem_used_pct` 22.2, `load_1m` 1.4
and `cores` 8 for a live host, so the `1 - idle` derivation, the per-aggregate `WHERE` inside `STATS`, and the
backticked load-average field all behave as written on Stack.
Reading it: a host has no enforced ceiling the way a container does, so the pressure signal is `cpu_busy_pct` near 100
sustained, `load_1m` above `cores`, or `mem_used_pct` high enough that the kernel is reclaiming. Load average above core
count with moderate busy CPU means processes are queueing on something other than CPU — usually disk or a lock.
For a full disk, `system.filesystem.utilization` lives in the same data stream **only when the collector's `filesystem`
scraper is enabled**, which is not the default in every distribution. On the Stack 9.4.4 cluster above, `_field_caps`
for `system.filesystem.*` over `metrics-hostmetricsreceiver.otel-*` returned nothing at all while every `system.cpu.*`,
`system.memory.*`, `system.disk.*` and `system.network.*` field was present — so naming it fails the query with
`Unknown column` rather than returning no rows. Where it is absent, `system.disk.io_time` and
`system.disk.pending_operations` are usually collected and answer the I/O-saturation question, though not the capacity
one.
When the host is monitored by the Elastic Agent system integration rather than the hostmetrics receiver, the equivalents
are `system.cpu.total.norm.pct` in `metrics-system.cpu-*` and `system.memory.actual.used.pct` in
`metrics-system.memory-*`. Both are already normalized fractions of capacity, so the `1 - idle` derivation is not
needed. These two names remain documentation-derived: the Stack cluster used to validate the rest of this file had no
System integration installed — `GET /_resolve/index/metrics-system.*` returned zero indices and zero data streams, and
both field names were absent from `_field_caps` across all of `metrics-*` — so they have not been executed against data.
Note also that the two collection paths do **not** share field names, so a query written for one returns
`Unknown column` against the other rather than empty results. Confirm which path exists with
`GET /_resolve/index/metrics-system.*,metrics-hostmetricsreceiver.*` before building on either.
## Aggregate rollups
`metrics-service_summary.1m.otel-*` is the cheapest way to enumerate which services reported telemetry in a window,
which distinguishes "healthy" from "not reporting":
```esql
FROM metrics-service_summary.1m.otel-*
| WHERE @timestamp >= NOW() - 1 hour
| STATS docs = COUNT(*) BY service.name
| SORT docs DESC
| LIMIT 50
```
`metrics-service_transaction.1m.otel-*` and `metrics-transaction.1m.otel-*` hold pre-aggregated transaction latency as
histogram and summary fields (`transaction.duration.histogram`, `transaction.duration.summary`). They are cheaper than
raw spans over long windows; raw `traces-*.otel-*` remains the right source when the investigation needs per-request
attributes for correlation.
For OTel application metrics, the `TS` (time series) command produces more efficient queries than `FROM`. It is GA on
Serverless; on Stack it is preview in 9.2 and GA in 9.4, so on anything below 9.4 use `FROM` with `BUCKET` over the same
data stream. `TS` also rejects `COUNT(*)` — count a specific field instead.
references/log-investigation.md
# Log Investigation
The log funnel: how to get from a raw log stream to the handful of messages that explain a degradation. Logs explain a
verdict; they never set one. Every query here is ES|QL run with `POST /_query`. Do not use Query DSL, and do not use the
ES|QL `KQL` search function — write predicates natively.
## The funnel workflow
**You must iterate.** Do not stop after one query. Keep excluding noise with `NOT` until **fewer than 20 log patterns**
(distinct message categories) remain. **Always keep the full filter when iterating:** concatenate new exclusions onto
the previous predicate; do not zoom out or drop earlier exclusions.
1. **Round 1 — broad.** Run a query with only the scope filter (for example `service.name == "cart"`) and the time
range. Get total count, histogram, sample logs, and message categorization (common and rare patterns) in one call.
2. **Inspect.** Look at the **histogram** (when spikes or drops occur), the **sample messages**, and the **categorized
patterns**. If the histogram shows a sharp spike at a specific time, narrow the time range around that spike for the
next round. Count how many distinct patterns remain and identify the high-volume noise to exclude.
3. **Round 2 — exclude noise.** Add `NOT ... LIKE` clauses for the dominant noise patterns. Re-run with the **full**
predicate — all previous exclusions plus the new ones.
4. **Repeat.** Keep adding exclusions and re-running with the full predicate. Do **not** stop after one or two rounds.
Continue until **fewer than 20 log patterns remain**. The remaining set is small enough to interpret as the
interesting bits: errors, anomalies, root cause.
5. **Pivot (optional).** Once the funnel isolates a specific entity (`container.id`, `k8s.pod.name`, `host.name`), run
one more query focused on that entity to see its dying words and surrounding context.
6. **Step back (if needed).** If the funnel does not reveal the cause, view logs in context around the key document
(preceding and following it in time), or pivot to a different entity and start a fresh funnel.
If you stop before reaching fewer than 20 log patterns, you will report noise instead of the actual failures. Each
intermediate result exists only to decide the next call; only the final narrowed result belongs in context and in the
summary.
## Query conventions
| Parameter | Type | Description |
| --------- | ------ | ----------------------------------------------------------------------- |
| `start` | string | Start of the time range (Elasticsearch date math, for example `now-1h`) |
| `end` | string | End of the time range (for example `now`) |
| `limit` | number | Maximum log samples to return (10-20 by default; cap at 500) |
| `groupBy` | string | Optional field to group the histogram by (for example `service.name`) |
Narrowing is expressed as an **ES|QL predicate** in a `WHERE` clause, not as a separate filter string. Combine scope and
exclusions with `AND`:
```esql
WHERE service.name == "checkout" AND severity_number >= 17
WHERE NOT message LIKE "*GET /health*"
WHERE error.exception.message IS NOT NULL AND NOT message LIKE "*Known benign warning*"
```
The first predicate deliberately does not say `log.level == "error"`; see
[the `log.level` caveat](#two-caveats-that-decide-whether-a-funnel-works) for why that particular comparison returns
nothing on most OTel data.
Use flat OTel field paths (`k8s.pod.name`, `k8s.namespace.name`). Observability index templates alias ECS names onto
OTel documents, so `kubernetes.pod.name` and `service.environment` also resolve; when a field can be absent in the
deployment at hand, confirm it before building a funnel on it.
### Null-safety when excluding
`NOT msg LIKE "*noise*"` evaluates to `null` — and therefore drops the row — when `msg` itself is `null`. On real OTel
data a large share of log records have no `message` at all, so a naive exclusion silently deletes most of the stream.
Guard it:
```esql
| EVAL msg = COALESCE(body.text, message, error.exception.message)
| WHERE msg IS NOT NULL AND NOT msg LIKE "*ValkeyCartStore*"
```
Or keep the unmatched rows explicitly with `WHERE msg IS NULL OR NOT msg LIKE "*...*"` when the records without a
message body still matter.
## Context minimization
Keep the context window small. In the sample branch of the query, **`KEEP` only a subset of fields**; do not return full
documents by default. A small summary (10 documents with `KEEP`) stays under roughly 1000 tokens; a single full JSON
document can exceed 4000.
**Recommended `KEEP` list for sample logs:** `@timestamp`, `message`, `service.name`, `k8s.container.name`,
`k8s.node.name`, `k8s.namespace.name`, `k8s.pod.name`, `host.name`, `container.id`, `agent.name`, `trace.id`.
**Limit samples:** default to 10-20 logs per query. Cap at 500; do not fetch thousands in one call. Each funnel step
only decides the next call — only the final narrowed result is worth keeping in context and summarizing.
## Message field fallback order
When building a single message value for display or categorization, use the first non-empty of:
1. `body.text` (OTel)
2. `message`
3. `error.message`
4. `event.original`
5. `exception.message`
6. `error.exception.message`
7. `attributes.exception.message` (OTel)
Express it as `COALESCE(...)` over the fields that exist in the deployment. `COALESCE` on a field absent from the
mapping fails the query, so confirm the fields first when working against unfamiliar data.
**Do not assume the whole list is available.** On a Stack 9.4.4 cluster carrying EDOT OTel logs, `_field_caps` over
`logs-*.otel-*` found `body.text`, `message`, `exception.message`, `error.exception.message` and
`attributes.exception.message`, but `error.message` and `event.original` were **absent from the mapping entirely** —
naming either in a `COALESCE` or a `WHERE` fails the whole query with `Unknown column`. `error.message` is an ECS field
that arrives via the ECS-to-OTel aliasing on integration-sourced logs; it is not created by the OTel index templates
alone. Run `GET logs-*/_field_caps?fields=body.text,message,error.message,error.exception.message,exception.message` and
build the `COALESCE` from what comes back.
## The one-call funnel query
Always return, in a single request: a time-series histogram, the total count, a small sample of logs, and message
categorization (common and rare patterns). The histogram is the primary signal — it shows when spikes or drops occur and
guides the next filter. `FORK` computes all five branches in one query.
```esql
FROM logs-*.otel-* METADATA _id, _index
| WHERE @timestamp >= NOW() - 1 hour AND service.name == "cart"
| FORK (STATS count = COUNT(*) BY bucket = BUCKET(@timestamp, 1 minute) | SORT bucket)
(STATS total = COUNT(*))
(SORT @timestamp DESC | LIMIT 10 | KEEP _id, _index, @timestamp, message, service.name, k8s.pod.name)
(LIMIT 10000 | STATS pattern_count = COUNT(*) BY pattern = CATEGORIZE(message) | SORT pattern_count DESC | LIMIT 20)
(LIMIT 10000 | STATS pattern_count = COUNT(*) BY pattern = CATEGORIZE(message) | SORT pattern_count ASC | LIMIT 20)
| LIMIT 500
```
**The trailing `LIMIT` applies across all branches combined, and when it truncates it drops whole branches silently.**
`FORK` concatenates the branches in order, so the outer limit is spent on `fork1` first. With a 1-minute bucket over a
one-hour window the branches are 60 + 1 + 10 + 20 + 20 = 111 rows, which already exceeds the `LIMIT 100` this query
carried until now; a 30-second bucket makes it 120 + 1 + 10 + 20 + 20. Measured on Stack 9.4.4 over a one-hour window
with a 30-second bucket, the two limits return:
| Outer `LIMIT` | Rows | `fork1` | `fork2` | `fork3` | `fork4` | `fork5` |
| ------------- | ---- | ------- | ------- | ------- | ------- | ------- |
| 100 | 100 | 99 | 1 | 0 | 0 | 0 |
| 500 | 169 | 120 | 1 | 10 | 19 | 19 |
At `LIMIT 100` the samples and both categorization branches are **absent entirely** — no error, no partial rows, no
indication that three of five branches were discarded. An agent reading that result concludes the logs have no message
patterns when they have nineteen. This is not version- or flavour-specific: it reproduces identically on Serverless
9.6.0 and Stack 9.4.4. Keep the outer limit above the sum of the branch limits, and if you shrink the bucket size, raise
it again.
### Before you run this: two availability gates
This one query depends on the two newest features in the skill, and both fail in ways that are easy to misread as "the
service has no logs".
- **`FORK`** is GA on Serverless. On Stack it is preview in 9.1-9.3 and GA from 9.4, and it does not parse at all on 8.x
or 9.0. Below 9.1, run the five branches as five separate queries against the same `WHERE` clause and combine them
yourself — you lose the single-round-trip property, not the workflow. On Stack 9.1-9.3 the command parses but adds an
implicit `LIMIT 1000` to each branch, so the `LIMIT 10000` on the categorization branches is silently capped and the
pattern counts under-report.
- **`CATEGORIZE`** is GA on Serverless, and on Stack it is preview in 9.0 and GA from 9.1 — but it **requires a Platinum
licence** on Stack at every version. This is not a version check: a 9.6 cluster on Basic or Gold fails it, and the
error names the licence. When it is unavailable, drop the two categorization branches and group by a truncated message
prefix instead:
```esql
| STATS n = COUNT(*) BY pattern = LEFT(msg, 60)
| SORT n DESC
| LIMIT 20
```
The prefix form is coarser: it splits one logical pattern into several when the variable part appears early in the
message. On live OTel data it still surfaced the same dominant noise patterns that `CATEGORIZE` found, which is enough
to drive the exclusion loop. Say in the answer that categorization was approximate.
Check `GET /` first. `build_flavor: "serverless"` means both are available; otherwise read `version.number` and, on
Stack, confirm the licence before using `CATEGORIZE`. If either is unavailable, say which one and that the funnel ran in
degraded form. Never let a parse or licence error become "log data is unavailable".
**Fork interpretation.** The response carries a `_fork` column identifying each branch:
| Branch | Contents | How to use it |
| --------- | ------------------------------------------------------- | ---------------------------------------------------------- |
| **fork1** | Trend — count per time bucket | Spot spikes and drops; narrow the time range around them |
| **fork2** | Total count, single row | See how much noise remains after each round |
| **fork3** | Sample logs | Decide which exclusions to add next |
| **fork4** | Common patterns — top 20 by count, from up to 10k logs | Add exclusions for the dominant noise |
| **fork5** | Rare patterns — bottom 20 by count, from up to 10k logs | Find the needles: the one-off exception, the first failure |
Count distinct patterns across fork4 and fork5 and **continue iterating until fewer than 20 patterns remain**.
Adjust the index pattern (`logs-*.otel-*`, `logs-*`), the time range, and the bucket size (`30s`, `1m`, `5m`, `1h`) to
the investigation. `logs-*.otel-*` covers EDOT/OTel ingest only: logs shipped by Filebeat or an Elastic Agent
integration land in `filebeat-*` and `logs-*-*`, and wired streams in `logs.*` — when the service's ingest path is
unknown, start wide with `logs-*,filebeat-*` and narrow from what returns. Aim for roughly 20-50 buckets over the
window: a 1-hour window suits a `1m` or `2m` bucket.
## Excluding noise
Add exclusions to the same `WHERE` clause and re-run the whole `FORK` query with the accumulated predicate:
```esql
FROM logs-*.otel-*
| WHERE @timestamp >= NOW() - 1 hour AND service.name == "cart"
| EVAL msg = COALESCE(body.text, message)
| WHERE msg IS NOT NULL
AND NOT msg LIKE "*ValkeyCartStore*"
AND NOT msg LIKE "*called with userId*"
| STATS n = COUNT(*) BY pattern = CATEGORIZE(msg)
| SORT n DESC
| LIMIT 20
```
`LIKE` uses `*` and `?` wildcards on keyword fields. For regular-expression exclusions use `RLIKE`. For full-text
matching on analyzed fields, `MATCH` is available — but prefer `LIKE` on the message field for funnel work, because it
is literal and predictable.
Every round keeps every earlier exclusion. Dropping one and re-adding another later re-admits noise you already ruled
out and makes the pattern count meaningless.
## Histogram grouped by a dimension
Break the trend down by a second dimension to see which entity drives a spike:
```esql
FROM logs-*.otel-*
| WHERE @timestamp >= NOW() - 1 hour AND k8s.namespace.name == "otel-demo"
| STATS count = COUNT(*) BY bucket = BUCKET(@timestamp, 1 minute), service.name
| SORT count DESC
| LIMIT 200
```
Keep the number of group values bounded — take the top N by count rather than every value — or the result explodes.
## Two caveats that decide whether a funnel works
**`log.level` is unreliable, and `log.level == "error"` is worse than unreliable — it is a silent empty.** Many logs
have missing or incorrect level metadata: everything logged as `info`, or the level present only in the message text. On
live OTel data roughly three-quarters of log records carried no level at all, and the records that did have one
disagreed on case and spelling.
On Stack 9.4.4, `log.level` is mapped as a `keyword` and mirrors `severity_text` value for value, so the field exists
and the query is valid — it just matches nothing. Measured over the full retention of one cluster:
| `log.level` value | Records |
| ----------------- | --------- |
| _(empty)_ | 1,081,170 |
| `Information` | 208,272 |
| `SEVERE` | 3,035 |
| `Warning` | 553 |
| `Normal` | 280 |
| `INFO` | 15 |
There is no `error` value anywhere, so `WHERE log.level == "error"` returns zero rows with no error on a cluster holding
3,035 error-severity records — it reports a healthy service. The lowercase ECS vocabulary (`error`, `warn`, `info`) is
not what the OTel SDKs emit; each language SDK writes its own `severity_text` (`SEVERE` from Java, `Information` from
.NET, `Normal` from the Kubernetes events receiver), and the ECS-to-OTel aliasing copies that string through unchanged
rather than normalizing it.
When you need a severity predicate, use the numeric `severity_number`, which **is** normalized by the OTel
specification: `>= 17` is error and above, `>= 13` warning and above, `>= 9` info and above. On the cluster above it
lined up exactly — `9` for all three of `INFO`, `Normal` and `Information`, `13` for `Warning`, `17` for `SEVERE` — and
`WHERE severity_number >= 17` returned the 3,035 records that the `log.level` comparison missed. Note that
`severity_number` is null on the same records that have no `severity_text`, so it fixes the vocabulary problem but not
the coverage one: a severity predicate of any kind still only sees the quarter of records that carry a level at all.
Otherwise treat `log.level` and `severity_text` as hints only, and funnel by message content or by the structured error
fields instead. If you must use the text form, enumerate the values that exist first with
`STATS COUNT(*) BY severity_text` rather than assuming a vocabulary.
**Bare keyword searches for "error" are flawed.** Searching for words like `error` or `fail` matches harmless mentions:
"no error", "error code 0", stack traces that merely reference the word, and healthy retry messages. They also miss
failures that never use the word. Scope by service or entity and iterate with exclusions on real message patterns rather
than trusting a single keyword.
## Finding actual failures
Prefer structured error fields over keyword matching:
```esql
FROM logs-*.otel-*
| WHERE @timestamp >= NOW() - 1 hour AND service.name == "payment"
| WHERE error.exception.message IS NOT NULL OR exception.type IS NOT NULL
| EVAL msg = COALESCE(error.exception.message, exception.message, body.text, message)
| KEEP @timestamp, service.name, error.exception.type, msg, trace.id
| SORT @timestamp DESC
| LIMIT 20
```
Zero rows here means no structured exceptions were recorded — the application might not be logging them, or might be
writing them into the message body. It does not mean nothing failed. Fall back to the funnel.
## Pivoting to a single trace
When a failing trace is in hand from the APM signals, pull every log that shares its ID. This is the single most direct
way to explain one failure:
```esql
FROM logs-*.otel-*
| WHERE @timestamp >= NOW() - 1 hour AND trace.id == "5158cfc84aa0e3d4a16365c81c21bf0e"
| EVAL msg = COALESCE(body.text, message)
| KEEP @timestamp, service.name, k8s.pod.name, msg
| SORT @timestamp ASC
| LIMIT 100
```
`trace.id` is populated on OTel log records that were emitted inside an instrumented request, and it spans services — so
one query returns the whole request path, in order, across every service that touched it.
## Resource metadata field fallbacks
For display or grouping, use the first field that exists in the deployment:
| Resource | Try in order |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| Service | `service.name` |
| Container | `k8s.container.name` → `kubernetes.container.name` → `container.name` |
| Host/Node | `k8s.node.name` → `kubernetes.node.name` → `host.name` |
| Cluster | `k8s.cluster.name` → `orchestrator.cluster.name` |
| Namespace | `k8s.namespace.name` → `kubernetes.namespace` |
| Pod | `k8s.pod.name` → `kubernetes.pod.name` |
| Workload | `k8s.deployment.name` → `k8s.replicaset.name` → `k8s.statefulset.name` → `k8s.daemonset.name` → `k8s.job.name` → `k8s.cronjob.name` |
| ECS field | OTel equivalent |
| --------------------- | ------------------------ |
| `message` | `body.text` |
| `log.level` | `severity_text` |
| `trace.id` | `trace_id` |
| `span.id` | `span_id` |
| `service.environment` | `deployment.environment` |
## Related documentation
- [ES|QL FORK command](https://www.elastic.co/docs/reference/query-languages/esql/commands/fork) — branch limits,
default `LIMIT` behavior, preview status
- [ES|QL CATEGORIZE function](https://www.elastic.co/docs/reference/query-languages/esql/functions-operators/grouping-functions/categorize)
— license requirement and grouping constraints
- [Use the ES|QL REST API](https://www.elastic.co/docs/reference/query-languages/esql/esql-rest) — `POST /_query`, async
queries, response formats
references/slo-and-alerts.md
# Reading SLO and Alert State
How to read SLO status and active alerting rules during triage. This is about **reading** state under time pressure —
for authoring SLO definitions, burn-rate rules, and alert thresholds, use the **observability-service-reliability**
skill.
## Why these rank first
An SLO is the only signal that encodes an agreed definition of "good" for a service. Every other signal describes a
change; the SLO says whether the change matters. When an SLO covers the symptom, it decides the verdict and the rest of
the triage exists to explain it. When no SLO covers the symptom, active alerting rules are the next best proxy for an
agreed threshold, because someone chose that threshold deliberately.
## Reading SLO state
List SLOs with `GET kbn:/api/observability/slos` and read the individual definition with
`GET kbn:/api/observability/slos/{id}`. Match SLOs to the service through the indicator's filter — SLOs are not tagged
with `service.name` in a uniform way, so read the indicator params rather than assuming a naming convention.
| Field | Meaning | How to read it |
| --------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------- |
| `summary.status` | `HEALTHY`, `DEGRADING`, `VIOLATED`, or `NO_DATA` | `VIOLATED` is the verdict. `NO_DATA` is missing telemetry, **not** health |
| `summary.sliValue` | Current SLI over the SLO's own time window | Compare against `objective.target`; a negative value means no data |
| `objective.target` | The target, as a fraction (`0.995` = 99.5%) | The line the SLI must stay above |
| `summary.errorBudget.remaining` | Fraction of budget left (`1` = untouched, `0` = spent) | Below 0 means the budget is overspent and the SLO is violated |
| `summary.errorBudget.consumed` | Fraction of budget already used | Read alongside how far into the window you are |
| `summary.errorBudget.isEstimated` | Whether the budget is extrapolated | True for occurrences-based SLOs early in a window; treat the number as provisional |
| `timeWindow` | Rolling window length, or calendar-aligned | Governs how far back the SLI is computed |
### Burn rate
Burn rate is how fast the error budget is being consumed relative to the rate that would exactly exhaust it over the
full window. A burn rate of 1 spends the budget precisely at the end of the window; 10 spends it in a tenth of the time.
- **Burn rate < 1, status healthy** — sustainable. Nothing to escalate.
- **Burn rate 1-2, status degrading** — the budget will run out if this continues. Report as degraded with a trend.
- **Burn rate > 2** — the budget is being consumed materially faster than planned. Report as unhealthy on trajectory
even while `summary.status` still reads healthy; that field reflects consumption to date, not the current rate.
- **Burn rate very high over a short window** — a sharp incident. Anchor the trace and log queries to that window.
Long and short burn-rate windows disagree by design: the short window catches fast incidents, the long window catches
slow bleeds. When they disagree, the short window is describing now and the long window is describing the last day.
### Common SLO reading errors
- **`NO_DATA` reported as healthy.** It means the SLI query returned nothing. That is missing telemetry, and it hides
outages rather than proving their absence.
- **SLO window versus query window.** An SLO evaluated over 30 days can be violated while the last 15 minutes look
perfect. Do not use a short ES|QL window to contradict a violated SLO — the SLO's window is longer, and it is right.
- **Assuming an SLO exists.** If no SLO covers the service, say so explicitly and move down the signal hierarchy. Do not
narrate the absence more than once.
- The SLO API's `sli.kql.custom` indicator takes a KQL string. That is the API's contract, not an exception to the rule
that data queries in this skill are written in ES|QL.
## Reading active alerting rules
**Determine active alert state from the Alerting API, not from indices.** Call `GET kbn:/api/alerting/rules/_find` with:
```text
per_page=100&filter=alert.attributes.enabled:true
```
Page with `page=2`, `page=3` and so on while `total` exceeds what you have received, then do all remaining narrowing on
the response in memory.
**Do not query `.alerts*` indices to determine whether an alert is currently active.** Those indices hold alert
documents whose lifecycle state can lag or be interpreted incorrectly; the Alerting API response is the source of truth.
### Why the call is deliberately unnarrowed
Server-side narrowing on this endpoint drops exactly the rules triage needs. Measured against a live Kibana project
(9.6.0) holding two enabled rules — `[Kubernetes OTel] Pod CrashLoopBackOff` (a `.es-query` rule tagged `kubernetes`,
`pod-health`, `errors`) and `[Kubernetes OTel] Availability — fast burn` (a `slo.rules.burnRate` rule tagged `k8s-otel`,
`demo`), neither carrying `params.serviceName`:
| Query | Rules returned |
| -------------------------------------------------------------- | -------------- |
| `per_page=100` (no filter) | 2 of 2 |
| `per_page=100&filter=alert.attributes.enabled:true` | 2 of 2 |
| `search=apm&search_fields=tags` | 0 of 2 |
| `filter=alert.attributes.executionStatus.status:active` | 0 of 2 |
| `search=apm&search_fields=tags` + the `executionStatus` filter | 0 of 2 |
| `filter=alert.attributes.params.serviceName:cart` | 0 of 2 |
Three separate reasons, each sufficient on its own:
- **`search_fields=tags` filters on a user convention.** `tags` is a free-text array the rule author chooses. Nothing
requires an observability rule to be tagged `apm`, and neither of these two is.
- **`executionStatus.status:active` means "firing", not "enabled".** Kibana sets the status to `ok` when the last run
produced zero alert instances and `active` when it produced one or more — the assignment is
`alertIds.length === 0 ? 'ok' : 'active'` in `rule_execution_status.ts`. The full enum is `ok`, `active`, `error`,
`pending`, `unknown`, `warning`. Filtering on `active` therefore returns only rules that are firing right now and
hides every healthy rule, so it cannot answer "what covers this service".
- **`params` is not filterable.** The `filter` parameter is documented as KQL over saved-object attributes; rule
`params` are stored but not mapped for query, so `alert.attributes.params.serviceName:<name>` matches nothing even
when a rule has that exact value. Service matching has to happen client-side regardless.
Narrowing by `alertTypeId` or `consumer` fails the same way for a different reason: the CrashLoopBackOff rule above is a
generic `.es-query` stack rule watching Kubernetes OTel data, so an observability-rule-type allowlist returns 1 of 2.
### The all-services rule trap
When checking a single service, evaluate **both**:
1. Rules whose `params.serviceName` matches the target service, and
2. Rules where **`params.serviceName` is absent** — these are all-services rules and they apply to the target service
too.
Filtering only on a matching `params.serviceName` silently drops the environment-wide latency and error-rate rules,
which are exactly the ones most likely to be firing during a broad incident. Treat either kind as applicable to the
service before declaring health. This is the requirement that the tag-and-status narrowing above makes unsatisfiable:
both rules in the measured project are all-services rules, and every narrowed query returned none of them.
Fetch a rule's full definition with `GET kbn:/api/alerting/rule/{id}` when its thresholds or params are needed.
### Separating coverage from firing
The two questions have one answer set. Fetch the enabled rules once, then partition in memory:
- **Coverage** — every fetched rule whose `params.serviceName` matches the service or is absent.
- **Firing** — the subset of those whose `execution_status.status` is `active`.
- **Blind spots** — the subset whose status is `error`, plus any rule with `mute_all: true`.
Do not issue a second, `active`-filtered call for the firing question. The status is already on every rule in the
response, and a second narrowed call reintroduces the possibility of the two answers disagreeing.
### Reading a rule
| Field | What it tells you |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rule_type_id` | Which signal the rule watches (APM latency, APM error rate, custom threshold, SLO burn rate) |
| `params.serviceName` | The scoped service, or absent for an all-services rule |
| `params.environment` | The scoped environment; `ENVIRONMENT_ALL` means every environment |
| `params.threshold` | The value that triggers it — needed to judge whether firing means much |
| `params.windowSize`/`windowUnit` | The evaluation window; a 1-minute window fires on transients |
| `execution_status.status` | One of `ok`, `active`, `error`, `pending`, `unknown`, `warning`. `active` means the last run produced alerts; `ok` means it ran and produced none; `error` means the rule is not evaluating at all |
| `mute_all` / `muted_alert_ids` | Whether the rule is muted — a muted rule that would be firing is a finding |
A rule in `execution_status.status: error` is **not** a healthy rule. It is a blind spot: report it as a signal that is
unavailable rather than as an absence of alerts.
## Putting the two together
- **SLO violated and rules firing** — unhealthy, high confidence. Use the rule's threshold and window to time-bound the
rest of the investigation.
- **SLO violated, no rules firing** — unhealthy on the SLO. The absence of alerts means alerting coverage is thinner
than the SLO, which is a recommendation to make, not evidence of health.
- **Rules firing, no SLO** — the rules are the verdict. Confirm the threshold is meaningful before escalating: a latency
rule at a threshold well below normal operating latency fires constantly and means nothing.
- **Neither exists** — say so once and triage on golden signals, comparing against the prior window. A verdict from raw
metrics alone is legitimate; it carries lower confidence than one anchored to an agreed target.
SKILL.md
---
name: observability-sre-triage
description: >
Triage a degraded or suspect service end to end: read SLO status and burn rate,
check active alerting rules and ML anomalies, measure throughput, latency, and error
rate, assess dependency health and infrastructure saturation, and funnel logs down
to the failures that explain it. Use when someone asks whether a service is healthy,
why it is slow or erroring, what is in its logs, or which attribute distinguishes
the requests that are failing. Also use when someone asks for the query behind any
of those signals — throughput, latency percentiles, error rate, dependency health,
or log volume — over APM/OTel traces, metrics, or logs.
compatibility: >
Requires the `elastic` CLI (>= 0.2) with Elasticsearch and Kibana contexts on the
same cluster. Base floor is Elasticsearch 8.11+ or Serverless. Three ES|QL features
need more, each with a fallback at its point of use: `FORK` (Stack GA 9.4), `CATEGORIZE`
(Stack GA 9.1, Platinum licence) and `TS` (Stack GA 9.4); all are GA on Serverless.
Reads APM/OTel traces, metrics and logs, the Kibana SLO and Alerting APIs, and the
Elasticsearch ML APIs. Read-only.
metadata:
author: elastic
version: 0.5.1
universal: true
---
# SRE Service Triage
Decide whether a service is healthy, degraded, or unhealthy, and say why. Triage is a hierarchy, not a checklist: SLOs
and alerts define whether the service is failing its contract, trace-derived golden signals describe how it is failing,
dependencies and infrastructure explain where the failure comes from, and logs supply the sentence you put in the
incident channel. Work down the hierarchy until the evidence supports a verdict, then stop.
For authoring and tuning SLO definitions, burn-rate rules, and alert thresholds, use the
**observability-service-reliability** skill. This skill only reads that state. For Kubernetes workload, node, or
control-plane diagnosis — restart loops, OOM kill confirmation, node pressure, admission rejections, stuck rollouts —
hand off to the **observability-k8s-investigation** skill. This skill checks whether a Kubernetes-hosted service is
saturated; it does not diagnose why the pod or the node behind it is failing.
<!-- begin-partial: preamble -->
## Environment Configuration
This skill executes Elasticsearch operations through the `elastic` CLI. If the
[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.
This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
directly.
<!-- end-partial: preamble -->
### Analysis without cluster access
The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the
evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason
from that evidence and deliver the conclusion.
When you genuinely do need data the user has not provided, still say what you would check and how — name the specific
query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is
useful without a cluster; one that only asks for setup is not.
Everything here is expressed in ES|QL (`POST /_query`) or the Kibana Observability APIs. Do not use Query DSL, and do
not use the ES|QL `KQL` search function — express predicates natively (`WHERE service.name == "checkout"`).
## Jobs to be done
- Answer "is service X healthy?" with a verdict and the evidence behind it
- Answer "why is service X slow / erroring / quiet?" by localizing the change to the service, a dependency, or its
infrastructure
- Read SLO status, burn rate, and remaining error budget during an incident
- Determine which alerting rules currently apply to a service, including all-services rules
- Funnel a noisy log stream down to the failures that explain the degradation
- Identify which attribute (version, host, pod, region, route) distinguishes the failing or slow subpopulation
- Distinguish a healthy service from a service with no telemetry
## Output discipline
Applies to every response produced under this skill.
- **Commit to the best-supported conclusion.** When the evidence points one way, say so. Do not downgrade confidence to
sound cautious — hedging on unambiguous evidence is a defect, not humility.
- **Commit to a verdict**: healthy, degraded, or unhealthy, followed by the reason. A triage answer that does not name
one of the three has not done the job.
- **State confidence once**, in the conclusion. Do not restate it per bullet.
- **Do not speculate past the evidence.** If the telemetry did not show a cause, it does not go in the answer. Name what
is unknown and stop. Never offer a mechanism ("probably a GC pause", "likely a noisy neighbor") that no signal
measured.
- **Report absence as absence.** Zero rows means the data is missing or not collected; it never means the underlying
condition is healthy. "No dependency metrics" is not "dependencies are fine".
- **Do not pad.** No restating the question, no narrating which queries were run unless the result mattered, no
summarizing the summary.
- **End on the finding.** No trailing offers such as "want me to dig deeper?". Actionable follow-ups belong in a
recommendations list, phrased as recommendations, not as questions.
## Signal hierarchy
Signals disagree constantly. This ordering decides which one wins.
| Rank | Signal | Authority |
| ---- | ------------------------------------- | ------------------------------------------------------------------------------------------- |
| 1 | **SLO status and burn rate** | Authoritative when SLOs exist. They encode the agreed definition of "good" for this service |
| 2 | **Active alerting rules** | Authoritative when no SLO covers the symptom. Sourced from the Alerting API |
| 3 | **Error rate, latency, throughput** | Describes the degradation. Decisive only when nothing above it exists |
| 4 | **Dependency health** | Locates the cause upstream or downstream; does not by itself set the verdict |
| 5 | **ML anomalies** | Deviation from learned baseline, not from a target. Corroborates and time-bounds |
| 6 | **Infrastructure (CPU, memory, OOM)** | Explains a mechanism. A saturated pod with healthy golden signals is a risk, not an outage |
| 7 | **Logs** | Explain, never decide. Log volume is not health |
Conflict rules:
- **SLO healthy, latency elevated** → degraded but within error budget. The verdict follows the SLO; report the trend as
a risk with the burn rate.
- **SLO violated, current-window metrics look fine** → trust the SLO and check its window. SLOs are evaluated over hours
or days; a 15-minute ES|QL window can look clean while the budget is already spent.
- **Alerts firing, no SLO defined** → the alerts are the verdict. Resolve each rule's `params` to confirm it actually
targets this service before attributing it.
- **Logs noisy, golden signals flat** → not degraded. High log volume without an error-rate or latency change is a
logging-configuration finding, not a health finding.
- **Throughput collapsed, error rate flat** → the caller stopped calling. Look upstream before blaming this service.
- **Any query returns zero rows** → missing data. Say which signal is unavailable and lower the scope of the verdict
accordingly; never convert silence into health.
## Routing: symptom to first signal
| Presenting symptom | Pull first | Reference |
| --------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------- |
| "Is X healthy?" / unclear | SLO status, then active rules, then golden signals | [slo-and-alerts.md](references/slo-and-alerts.md) |
| "X is slow" | Latency percentiles versus the prior period, then dependency latency | [apm-signals.md](references/apm-signals.md) |
| "X is erroring" / 5xx | Error rate by route, then failed-transaction correlation | [apm-signals.md](references/apm-signals.md) |
| "X is down" / no traffic | Throughput, then confirm the service still ingests at all | [apm-signals.md](references/apm-signals.md) |
| "Only some requests are bad" | Subpopulation correlation over candidate attributes | [apm-signals.md](references/apm-signals.md) |
| "An alert fired" / "the SLO is burning" | Rule `params` and SLO burn rate, then the metric the rule watches | [slo-and-alerts.md](references/slo-and-alerts.md) |
| "What is in the logs?" / noisy logs | The log funnel — iterate with `NOT` exclusions | [log-investigation.md](references/log-investigation.md) |
| Suspected OOM, throttling, restarts | Container CPU and memory limit utilization | [apm-signals.md](references/apm-signals.md) |
| "Is it saturated?" on a non-K8s host | Host CPU, memory, and load average from the hostmetrics receiver | [apm-signals.md](references/apm-signals.md) |
| "Which downstream is hurting X?" | Per-destination call volume, latency, and failure rate | [apm-signals.md](references/apm-signals.md) |
## Data sources
OTel-native data streams, verified against Elasticsearch 9.6.0:
| Data | Index pattern |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Traces (spans, transactions) | `traces-*.otel-*`; classic Elastic APM agent ingest also lands in `traces*apm*` |
| Logs | `logs-*.otel-*` |
| Raw metrics | `metrics-*.otel-*`; classic APM agent ingest also lands in `metrics*apm*` |
| Service inventory (1m rollup) | `metrics-service_summary.1m.otel-*` |
| Transaction rollups (1m) | `metrics-service_transaction.1m.otel-*`, `metrics-transaction.1m.otel-*` |
| Dependency rollups (1m) | `metrics-service_destination.1m.otel-*` |
| Kubernetes | `metrics-kubeletstatsreceiver.otel-*`, `metrics-k8sclusterreceiver.otel-*`, `logs-k8seventsreceiver.otel-*` |
| Host (VM, bare metal) | `metrics-hostmetricsreceiver.otel-*`; the Elastic Agent system integration lands in `metrics-system.*` |
`service.name` is populated on traces, metrics, and logs, so it is the join key across all three. Use flat OTel field
paths in ES|QL (`k8s.pod.name`, not `resource.attributes.k8s.pod.name`). When analyzing OTel application metrics, the
ES|QL `TS` (time series) command gives more efficient metric queries. It is GA on Serverless; on Stack it is preview in
9.2 and GA in 9.4, so below 9.4 use `FROM` with `BUCKET` instead. `TS` also rejects `COUNT(*)` — count a field instead.
The recipes in this skill and its references are written against the OTel-native streams above. A service instrumented
with the classic Elastic APM agent ships to `traces-apm*` and `metrics-apm*` under different field names
(`transaction.duration.us`, `event.outcome`), so these recipes return no rows for it. An empty result on a service that
is otherwise clearly alive is therefore a scope boundary, not evidence of an outage: check which index family the
service actually writes (`GET /_cat/indices`) and report the ingest path rather than concluding from silence.
## ES|QL feature availability
Three features this skill uses are newer than its 8.11 base floor. Check `GET /` before relying on them:
`build_flavor: "serverless"` means all three are available; otherwise compare `version.number` against the Stack column.
Never report "no data" when the real answer is that the query did not run — say which feature was unavailable and use
the fallback.
| Feature | Serverless | Stack | Licence | Used by | Fallback |
| ------------ | ---------- | ------------------------ | ------------ | ------------------------------------------------ | ------------------------------------------------------------------------- |
| `FORK` | GA | preview 9.1-9.3, GA 9.4+ | any | The log funnel, and the subpopulation comparison | Run each branch as a separate query and combine the results yourself |
| `CATEGORIZE` | GA | preview 9.0, GA 9.1 | **Platinum** | Message categorization inside the log funnel | Group by a truncated message prefix, or funnel on structured error fields |
| `TS` | GA | preview 9.2, GA 9.4 | any | OTel application metric queries | `FROM` with `BUCKET` over the same data stream |
The Platinum requirement on `CATEGORIZE` is not a version check. A 9.6 Stack cluster on a Basic or Gold licence fails it
exactly as an 8.11 cluster fails `FORK`, and the error names the licence rather than the syntax. On Serverless the
function is GA with no separate licence gate.
## Process: triage a degraded service
1. **Fix the service and the window.** Resolve the service name and the time range from the request. Use the user's time
range — do not silently assume the last hour when the complaint is historical. If no range is given, use the last
hour and say so. Confirm the service actually exists in telemetry with a `COUNT(*) BY service.name` over
`traces-*.otel-*` via `POST /_query`; if the name does not appear, resolve the ambiguity before querying further.
Decision: which service and window every later query is scoped to. Data: distinct `service.name` values in range.
2. **Read SLO status and burn rate.** List SLOs with `GET kbn:/api/observability/slos` and fetch the ones bound to this
service with `GET kbn:/api/observability/slos/{id}`. Read status, current SLI, burn rate, and remaining error budget.
Decision: does an agreed contract exist, and is it being violated? If yes, the verdict is already determined and the
remaining steps only explain it. If no SLO covers this service, say so once and fall through to step 3.
3. **Determine which alerting rules apply to this service, and which of them are firing.** Call
`GET kbn:/api/alerting/rules/_find` with `per_page=100&filter=alert.attributes.enabled:true`, paging with `page` if
`total` exceeds what you received. Then filter the response **client-side**. **Do not query `.alerts*` indices to
determine active state** — the Alerting API response is the source of truth. Fetch a rule's full definition with
`GET kbn:/api/alerting/rule/{id}` when its `params` are needed.
**Do not narrow this call server-side.** The `_find` `filter` parameter is KQL over saved-object _attributes_, and
`params` is not among them — `filter=alert.attributes.params.serviceName:<name>` returns zero rules on a cluster that
has them. Narrowing by `search=apm&search_fields=tags`, by `alertTypeId`, or by `consumer` is worse: it drops rules
on a naming convention or a rule-type allowlist, and the rules it drops are disproportionately the all-services ones.
See [references/slo-and-alerts.md](references/slo-and-alerts.md) for the measured failure.
From the fetched set, evaluate **both** rules whose `params.serviceName` matches the service **and** rules where
`params.serviceName` is absent, because the latter are all-services rules that apply to it too. Read
`execution_status.status` on each: `active` means the rule's last run produced alerts, `ok` means it ran and produced
none, and `error` means it is not evaluating at all — a blind spot, not a pass.
Decision: what covers this service, and is any of it currently firing? Data: rule `params.serviceName`, rule type,
and execution status.
4. **Check ML anomalies, if any jobs exist.** List jobs with `GET /_ml/anomaly_detectors` and confirm they are running
with `GET /_ml/anomaly_detectors/_stats` — a stopped job produces no anomalies, which is not the same as no anomaly.
Pull scored records with `GET /_ml/anomaly_detectors/{id}/results/records`.
Decision: did latency, throughput, or error rate deviate from its learned baseline, and when? Use the anomaly window
to narrow steps 5 and 6.
5. **Measure the golden signals.** Run ES|QL over `traces-*.otel-*` for throughput, latency (avg, p95, p99), and error
rate, bucketed over the window and compared against the immediately preceding window of equal length. See
[references/apm-signals.md](references/apm-signals.md).
Decision: is the service actually changed relative to itself, and in which dimension? Data: request count, latency
percentiles, and failure ratio for the current and prior windows.
6. **Localize: dependencies, then subpopulation, then infrastructure.**
- **Dependencies** — aggregate `metrics-service_destination.1m.otel-*` by `span.destination.service.resource` for
call volume, average latency, and failure rate. If this query returns zero rows for the service, the service is
**not APM-instrumented for dependencies**; report insufficient dependency data and do not claim upstreams are
healthy.
- **Subpopulation** — when only part of the traffic is bad, compare the failure or slow rate per candidate attribute
against the overall rate to find which attribute is over-represented. See
[references/apm-signals.md](references/apm-signals.md).
- **Infrastructure** — read the resource attributes on the service's spans (`k8s.pod.name`, `container.id`,
`host.name`) first, then branch on what they contain. Pod and namespace attributes mean the service is
Kubernetes-hosted: check `k8s.container.cpu_limit_utilization` and `k8s.container.memory_limit_utilization` in
`metrics-kubeletstatsreceiver.otel-*`. A `host.name` with no pod attributes means the service runs on a VM or bare
host, where every `k8s.*` field is empty: check `system.cpu.utilization`, `system.memory.utilization`, and
`system.cpu.load_average.1m` in `metrics-hostmetricsreceiver.otel-*` instead. OOM kills, CPU throttling, and host
saturation degrade APM health directly. See [references/apm-signals.md](references/apm-signals.md).
- **Recent change** — a deploy is the most common cause of a step change. Search deploy annotations for the service
with `GET kbn:/api/apm/services/{serviceName}/annotation/search` over the incident window, and compare the failure
or latency rate by `service.version` in the subpopulation query. An annotation inside the onset window is a strong
correlation; confirm it plausibly explains the symptom before attributing.
Decision: is the cause inside this service, in something it calls, in one slice of its instances, under it, or in a
change that landed?
When the Kubernetes branch shows saturation, restarts, or an OOM kill, the mechanism is established and the remaining
diagnosis — why the pod is being killed, whether the node is under pressure, whether a rollout is stuck — belongs to
the **observability-k8s-investigation** skill. Hand off rather than continuing here.
7. **Explain with logs.** Scope logs by `service.name`, or by `trace.id` when a specific failing trace is in hand, and
run the funnel until the remaining set is small enough to read. See
[references/log-investigation.md](references/log-investigation.md). Logs confirm and articulate the cause; they do
not overturn steps 2 and 3.
8. **State the verdict.** Healthy, degraded, or unhealthy, with the reason and one statement of confidence, followed by
recommendations. Name any signal that was unavailable.
## Examples
**"Is checkout healthy?"** — resolve the window, read its SLOs, then the active rules including all-services rules, then
throughput, latency percentiles, and error rate over `traces-*.otel-*` against the prior window. If the availability SLO
is at 99.2% against a 99.5% target with a burn rate above 1, the verdict is unhealthy on SLO violation, and the golden
signals are the explanation, not the verdict.
**"Why is the frontend slow?"** — compare p95 and p99 for the current window against the previous window of equal
length. If service-level latency rose while per-destination latency in `metrics-service_destination.1m.otel-*` is flat,
the added time is inside the service; if one destination's average response time rose in step with it, the dependency is
the cause and the frontend is a victim.
**"Only some checkout requests fail"** — run the subpopulation comparison: failure rate grouped by `service.version`,
`k8s.pod.name`, `host.name`, and `cloud.region` alongside the overall failure rate. An attribute value whose failure
rate is several times the overall rate, on a volume large enough to matter, is the correlated attribute. On live data,
grouping frontend server spans by route showed a 3.8% slow rate for `POST` against a 0.9% overall rate — a 4x lift that
localizes the problem to write paths.
**"The cart service logs look bad"** — run the funnel over `logs-*.otel-*` scoped to `service.name == "cart"`: get
trend, total, samples, and message categorization in one `FORK`, then add `NOT ... LIKE` exclusions for each dominant
pattern and re-run with the full accumulated filter until fewer than 20 patterns remain. High log volume alone is not a
health verdict — check the golden signals before calling the service degraded.
**"Is the payment service's upstream healthy?"** — query `metrics-service_destination.1m.otel-*` for it. Zero rows means
the service does not emit dependency metrics. Report that dependency data is unavailable for this service and give the
verdict from the signals that do exist; do not report the upstreams as healthy.
**"An alert fired on api-gateway"** — fetch the enabled rules with no server-side narrowing, then match in memory on
`params.serviceName == "api-gateway"` **and** on rules with no `params.serviceName`, reading `execution_status.status`
to see which are firing. Read the firing rule's threshold from `GET kbn:/api/alerting/rule/{id}`, then query the same
metric over the same window in ES|QL to confirm the rule is describing a real change rather than a threshold that is set
too tight.
## Guidelines
- Work the signal hierarchy in order and stop when the evidence supports a verdict. Do not run every query in this
document on every request.
- Anchor to SLO status and burn rate when SLOs exist. When they do not, fall back to alerts, ML anomalies, throughput,
latency, error rate, dependencies, infrastructure, and logs — and say that no SLO covers the service.
- Use the Alerting API for active-alert state. **Never** query `.alerts*` indices for it. Always evaluate both
service-scoped rules and rules with no `params.serviceName`.
- Fetch alerting rules unnarrowed and filter client-side. `_find` cannot filter on `params`, tag search drops rules that
do not follow a naming convention, and `executionStatus.status:active` returns only rules that are firing right now —
each of those silently hides the all-services rules the bullet above requires.
- Always use the user's time range. Compare every metric against the immediately preceding window of equal length —
absolute numbers without a baseline do not support a verdict.
- Zero rows is missing data. Say which signal is unavailable rather than treating silence as a pass.
- Scope every query by `service.name` and a bounded `@timestamp` range, and cap output with `LIMIT`. Prefer coarse
buckets when only a trend is needed.
- Prefer `event.outcome == "failure"` for failed spans; `status.code == "Error"` is equivalent on OTel traces but is
null on successes, so it cannot be counted directly.
- Filter server-side traffic with `kind == "Server"` when measuring a service's own throughput and latency, so client
spans do not double-count.
- Treat `log.level` and `severity_text` as hints, never as filters you rely on. On real OTel data most log records carry
no level at all and those that do disagree on case and vocabulary (`INFO`, `Information`, `SEVERE`, `Normal`). In
particular **never write `log.level == "error"`** — the lowercase ECS vocabulary is not what the OTel SDKs emit, so it
returns zero rows with no error even on a service that is logging errors, and reports the service healthy. Use the
normalized numeric `severity_number >= 17` if you need a severity predicate at all.
- Logs explain; they do not decide. Never issue a verdict whose only support is log content.
- Do not invent field names. If a field might not exist in this deployment, confirm the data stream exists with
`GET /_resolve/index/{pattern}` before building on it.
- Establish where the service runs before checking saturation. Kubernetes and host telemetry share no field names, so a
Kubernetes query against a VM-hosted service returns zero rows and says nothing about whether it is saturated.
- Pass `--drop-null-columns` on `POST /_query` when a result is mostly empty columns. Infrastructure metrics are sparse
by nature — limit utilization is absent wherever no limit is declared — and the flag collapses the noise while listing
the suppressed column names under `all_columns`, so nothing is hidden.
## Operations
| HTTP API (shorthand) | `elastic` CLI command |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `GET /` | `elastic es info` |
| `POST /_query` | `elastic es esql query --format tsv --query '<esql>'` |
| `GET /_resolve/index/{pattern}` | `elastic es indices resolve-index --name '<pattern>'` |
| `GET /_ml/anomaly_detectors` | `elastic es ml get-jobs` |
| `GET /_ml/anomaly_detectors/_stats` | `elastic es ml get-job-stats` |
| `GET /_ml/anomaly_detectors/{id}/results/records` | `elastic es ml get-records --job-id '<id>'` |
| `GET kbn:/api/observability/slos` | `elastic kb slo find-slos-op --space-id '<space>' --kql-query '<kql>'` |
| `GET kbn:/api/observability/slos/{id}` | `elastic kb slo get-slo-op --space-id '<space>' --slo-id '<id>'` |
| `GET kbn:/api/alerting/rules/_find` | `elastic kb alerting get-alerting-rules-find --filter '<filter>'` |
| `GET kbn:/api/alerting/rule/{id}` | `elastic kb alerting get-alerting-rule-id --id '<id>'` |
| `GET kbn:/api/apm/services/{serviceName}/annotation/search` | `elastic kb apm-annotations get-annotation --service-name '<service>' --environment '<env>' --start '<iso8601>' --end '<iso8601>'` |
The SLO find command takes a KQL query string because that is the API's contract; it is not an exception to the ES|QL
rule for data queries.
The annotation search route rejects a request that omits `environment`, so pass `ENVIRONMENT_ALL` when the service's
environment is not known. Only the search direction is in scope: this skill is read-only, so the companion
create-annotation operation is deliberately not bound.