evals/evals.json
{
"skill_name": "signoz-explaining-alerts",
"evals": [
{
"id": 0,
"eval_name": "fuzzy-match-audit",
"prompt": "Explain my checkout error rate alert. Is it configured correctly?",
"expected_output": "Agent paginates signoz_list_alert_rules to find candidates (multiple 'checkout-svc Error Rate > 3%' rules + eval-1 alerts exist on staging). Surfaces ambiguity and either asks user or picks closest with a clear assumption note. Then signoz_get_alert + cursor-paginated signoz_get_alert_history (first call timeRange=7d, state=firing, order=desc), leads with an audit verdict, explains the A*100/B query, threshold, evaluation, notifications, and labels, and includes the complete fire-frequency summary.",
"expectations": [
"The first signoz_get_alert_history call uses timeRange=7d, state=firing, and order=desc",
"Each history continuation uses data.nextCursor as cursor plus the resolved absolute start/end and preserves state, filter, and order",
"History pagination never uses offset or page fullness as the continuation signal",
"The fire-frequency summary counts distinct overallStateChanged transitions into overallState=firing, not raw history rows"
],
"files": []
},
{
"id": 1,
"eval_name": "formula-having-decode",
"prompt": "What does alert 019de287-a765-7c80-93e7-f2f0f2f5f927 do? Specifically explain how the count guard works and what conditions trigger a fire.",
"expected_output": "Agent calls signoz_get_alert directly with the rule ID. Decodes the 3-query structure: A=p99(duration_nano) WITH having: count()>50, B=count() (disabled), F1=A pass-through. Explains that the having clause is the count guard — it suppresses any 1-minute bucket where span count <= 50, so the threshold evaluator sees no data for those buckets and cannot fire. Translates op=above + matchType=on_average + target=1s into plain language. Pulls paginated firing history for 7d in descending order.",
"files": []
},
{
"id": 2,
"eval_name": "anomaly-threshold-explain",
"prompt": "Help me understand alert rule 019de287-6752-7629-83e4-375919d56dd8 — I'm confused about how anomaly thresholds work versus regular ones.",
"expected_output": "Agent fetches the rule, identifies ruleType=anomaly_rule and algorithm=standard, and explicitly contrasts anomaly vs threshold semantics: target=3 means 3 standard deviations from the learned baseline, NOT 3 raw req/s. Explains daily seasonality and effective routing through top-level preferredChannels. Pulls paginated firing history for 7d in descending order and includes a rule-incident fire-frequency summary.",
"expectations": [
"The explanation treats top-level preferredChannels as the anomaly rule's direct notification routing because anomaly rules have no threshold channels",
"The fire-frequency summary counts rule-wide overallStateChanged transitions, not per-fingerprint rows"
],
"files": []
},
{
"id": 3,
"eval_name": "url-parse-panic-logs",
"prompt": "What does this alarm fire on? https://us2.staging.signoz.cloud/alerts/edit?ruleId=019dad90-1754-770f-8dae-d28a897167e3",
"expected_output": "Agent extracts ruleId from the URL query string, calls signoz_get_alert, identifies as LOGS_BASED_ALERT for 'Payments service panic logs'. Decodes the body filter (panic / fatal / etc keywords), groupBy, threshold target=0/at_least_once (any panic log fires it). Pulls paginated firing history for 7d in descending order. Explains in plain language without dumping the JSON.",
"files": []
},
{
"id": 4,
"eval_name": "multi-severity-audit-kafka",
"prompt": "Audit alert 019dad90-1c73-715e-b040-79901928deb2 (kafka consumer lag) — anything I should change?",
"expected_output": "Agent fetches the rule, decodes the multi-severity threshold spec (warning at lag >= 50, critical at >= 200), explains the tiered routing. Performs Step 5 audit: flags missing labels.severity (rule has no severity label even though thresholds have severity tiers), considers groupBy cardinality, evaluates threshold calibration, suggests adding severity label, runbook annotations, alertOnAbsent if relevant. Pulls paginated firing history for 7d in descending order.",
"expectations": [
"For a v2 threshold rule with usePolicy=false, the explanation derives each tier's routing only from that threshold's channels array",
"The explanation never uses top-level preferredChannels as a fallback for an empty v2 threshold tier; it identifies such a tier as unrouted or misconfigured"
],
"files": []
},
{
"id": 5,
"eval_name": "grouped-history-rule-incidents",
"prompt": "Using the attached offline SigNoz responses, explain the grouped checkout latency alert and tell me how often it fired in the lookback.",
"expected_output": "Agent uses the fixture without querying a live tenant. It reports one rule-wide firing incident, not three fires: two fingerprint rows represent the same overall transition and the later checkout row is a series transition while the rule was already firing. It may additionally report three series-level firing transitions if clearly labeled as such.",
"expectations": [
"Output says the rule entered firing once in the lookback",
"Output does not report three rule fires by counting raw firing rows",
"If series activity is mentioned, it distinguishes fingerprint or label-group transitions from rule-wide incidents"
],
"files": ["evals/files/grouped-alert-history.json"]
}
]
}
evals/files/grouped-alert-history.json
{
"mode": "offline_mcp_fixture",
"instructions": "Use these responses to explain the rule and compute fire frequency offline. Do not query a live tenant. History was requested with timeRange=7d, state=firing, order=desc.",
"responses": [
{
"tool": "signoz_get_alert",
"request": {"id": "019f0000-0000-7000-8000-000000000101"},
"response": {
"data": {
"id": "019f0000-0000-7000-8000-000000000101",
"alert": "Checkout latency by service",
"alertType": "TRACES_BASED_ALERT",
"ruleType": "threshold_rule",
"condition": {
"compositeQuery": {"queryType": "builder", "queries": []},
"thresholds": {"kind": "basic", "spec": [{"name": "critical", "target": 2, "targetUnit": "s", "op": "above", "matchType": "at_least_once", "channels": ["pagerduty-oncall"]}]}
},
"evaluation": {"kind": "rolling", "spec": {"evalWindow": "5m", "frequency": "1m"}},
"notificationSettings": {"groupBy": ["service.name"], "usePolicy": false},
"labels": {"severity": "critical"}
}
}
},
{
"tool": "signoz_get_alert_history",
"request": {"id": "019f0000-0000-7000-8000-000000000101", "timeRange": "7d", "state": "firing", "order": "desc"},
"response": {
"data": {
"items": [
{"overallState": "firing", "overallStateChanged": false, "state": "firing", "stateChanged": true, "unixMilli": 1750000300000, "labels": [{"key": {"name": "service.name"}, "value": "checkout"}], "fingerprint": 101, "value": 2.4},
{"overallState": "firing", "overallStateChanged": true, "state": "firing", "stateChanged": true, "unixMilli": 1750000000000, "labels": [{"key": {"name": "service.name"}, "value": "payments"}], "fingerprint": 202, "value": 2.8},
{"overallState": "firing", "overallStateChanged": true, "state": "firing", "stateChanged": true, "unixMilli": 1750000000000, "labels": [{"key": {"name": "service.name"}, "value": "checkout"}], "fingerprint": 101, "value": 2.3}
],
"total": 3
},
"note": "hasMore: false"
}
}
]
}
SKILL.md
---
name: signoz-explaining-alerts
description: >
Describe what an existing SigNoz alert rule does in plain language:
the signal it watches, the threshold and evaluation behavior, the
notification routing, and a one-line fire-frequency summary so the user
knows whether the alert has been active. Make sure to use this skill
whenever the user asks "what does this alert do", "explain alert X",
"walk me through this rule", "how does my [Y] alert work", "is this
alert configured correctly", or otherwise asks for an interpretation
of an existing alert's configuration. Static explanation only; for
diagnosing a specific firing incident, use `signoz-investigating-alerts`.
argument-hint: <alert name or rule id>
---
# Alert Explain
Decode an existing SigNoz alert's configuration into a plain-language
explanation. The skill is read-only and stays focused on the rule
itself: what it watches, when it fires, where it notifies. A single
line of fire-frequency data is included to ground the explanation, but
this skill does **not** investigate any specific fire; that is
`signoz-investigating-alerts`'s job.
## Prerequisites
This skill calls SigNoz MCP server tools (`signoz_get_alert`,
`signoz_list_alert_rules`, `signoz_get_alert_history`). Before running
the workflow, confirm the `signoz_*` tools are available. If they are
not, run `signoz-mcp-setup` first to initialize or repair the MCP connection.
Do not guess at alert configuration from the rule name alone.
## When to use
Use this skill when the user wants to:
- Understand or interpret an existing alert rule.
- Confirm what signal an alert watches and at what threshold.
- Audit whether an alert is reasonably configured.
- Translate raw alert JSON into operational language.
Do NOT use when the user wants to:
- Create a new alert → `signoz-creating-alerts`.
- Diagnose why an alert fired or correlate signals around a fire window
→ `signoz-investigating-alerts`.
- Modify an existing alert → call `signoz_update_alert` directly.
## Required inputs
| Input | Required | Source if missing |
|---|---|---|
| Alert identifier (rule ID or name) | yes | `$ARGUMENTS`, recent context, or fuzzy match |
If the input is missing or ambiguous, this skill is **best-effort** (not
strict; read-only operations are cheap to recover from):
1. Call `signoz_list_alert_rules`, paginate through every page, and find
the closest name match.
2. State the interpretation in the response:
"Interpreting your request as alert 'High Error Rate - Checkout' (id 42).
If you meant a different one, tell me the name or id."
3. Proceed with the explanation. The user can correct after.
## Workflow
### Step 1: Resolve the alert
If the user provided a rule ID (UUID or legacy numeric ID), skip to Step 2.
Otherwise:
1. Call `signoz_list_alert_rules` and **paginate every page**:
`pagination.hasMore` is true until the full list is walked.
2. Match by name (case-insensitive substring). If multiple match,
present the candidates and ask which one (interactive) or pick the
closest and flag the assumption (autonomous).
### Step 2: Fetch the full configuration
Call `signoz_get_alert` with the rule id. This is **mandatory**: the
list response does not include the full condition / thresholds /
notification settings, and explanations based on the name alone are
guesses.
### Step 3: Pull a one-line fire-frequency summary
First call `signoz_get_alert_history` with `timeRange: "7d"`,
`state: "firing"`, `order: "desc"`, and a large limit. Continue only when
`data.nextCursor` exists (the completeness note also reports `hasMore: true`):
pass that value as `cursor`, replace `timeRange` with the note's resolved
absolute `start` and `end`, and preserve the same state, filter, and order.
Stop when `nextCursor` is absent / the note reports `hasMore: false`. Never use
`offset` or infer completeness from page fullness.
History `state` accepts `inactive`, `pending`, `recovering`, `firing`, `nodata`,
or `disabled`. For "resolved" / "recovered", use `state: "inactive"`;
`resolved` is not a value; `recovering` is a transient keep-firing state, not resolution.
Derive a single rule-level line from the complete set. History rows are emitted
per label-group `fingerprint`, so never equate row count with alert-fire count.
Count distinct rule transitions where `overallStateChanged: true` and
`overallState: "firing"` (deduplicate rows from the same transition timestamp):
> Fired N times in the last 7d (last fire: <relative-time>).
If the user asks about a particular service/host/group instead, partition by
`fingerprint` and count that series' rows where `stateChanged: true` and
`state: "firing"`; include its labels and call the result a **series fire**, not
a rule-wide fire. A grouped rule can have many series transitions while the
overall rule remains continuously firing.
If the alert never fired in the window, say so explicitly:
"Has not fired in the last 7d." If the alert is disabled, mention that
separately; disabled rules can still have earlier fires in the lookback.
This single line grounds the explanation. Do **not** drill into specific
fires here; that's `signoz-investigating-alerts`.
### Step 4: Build the explanation
The single most useful thing for the user is a tight summary. Lead
with a **TL;DR that directly answers the question they asked**, not a
generic alert summary. The TL;DR is the only thing some users will
read; burying their answer under a fixed template forces them to
scroll for what they wanted in the first place.
Match the TL;DR shape to the user's question:
- **"What does this alert do?" / "Explain X"**: describe what fires:
> **TL;DR**: Fires when `<condition>` for `<scope>`, notifies
> `<channel>`. `<fire-frequency line>`.
- **"Is it configured correctly?" / "Audit this" / "Anything I should
change?"**: lead with the **verdict and the top 1–3 changes**, not
the description of what fires:
> **TL;DR**: Mostly well-configured, but recommend: (1) add
> `alertOnAbsent` (currently a crashed service stays silent); (2)
> fix annotation template `{{$topic}}` → `{{$labels.topic}}` (won't
> interpolate); (3) split critical to PagerDuty (both tiers
> currently route to Slack). `<fire-frequency line>`.
- **"How does X work?" / "Explain the count guard"**: answer the
mechanism in 1–2 sentences before any framing:
> **TL;DR**: The count guard is a `having: count() > 50` clause on
> query A: any 1-minute bucket with ≤50 spans is dropped before
> evaluation, so low-traffic minutes can't fire the alert.
- **"What's the threshold?" / focused config question**: state the
exact thing they asked about:
> **TL;DR**: Threshold is **3 standard deviations** (z-score), not
> a raw rate value. Daily seasonality means the model compares
> each hour against historical norms for that hour.
Always include the fire-frequency line and `disabled` status if
non-default; those ground every kind of TL;DR. But put the answer to
the user's specific question first.
After the TL;DR, write the explanation in prose, organized into the
four sections below. **Skip any section that has nothing meaningful to
add**: empty severity labels, default notification settings, vanilla
annotations don't deserve a header. Short and skimmable beats
perfunctorily complete; the user is not reading a checklist.
**1. What it watches**: one short paragraph. Combine signal type
(metrics / logs / traces / exceptions), what the query measures, and
scope. Translate the query to operational language; for formulas, name
each sub-query (A, B, …) and state what F1 (or whichever
`selectedQueryName` triggers) computes, e.g. "F1 = A × 100 / B → error
percentage". Decode filter operators (`=` equals, `!=` not equals,
`IN` / `NOT IN`, `LIKE` / `ILIKE`, `CONTAINS`, `REGEXP`, `EXISTS` /
`NOT EXISTS`); enumerate `IN` / `NOT IN` value lists so the user can
verify them. Name each `groupBy` dimension and its practical effect
("fires separately per service" for `service.name`).
For **anomaly rules** (`ruleType: anomaly_rule`), explicitly state that
the threshold is in **standard deviations from the learned pattern, not
the raw value**; this is the most common point of confusion. Include
`algorithm` (`standard`, which is z-score based), `seasonality` (hourly /
daily / weekly), and how
lower/higher targets shift sensitivity (lower → more noise, higher →
only extreme deviations).
**2. When it fires**: one paragraph covering threshold + timing.
Decode the threshold spec into plain English using these mappings:
- `op` codes: `1` above, `2` below, `3` equal, `4` not equal, `5`
above-or-equal, `6` below-or-equal, `7` outside-bounds
(`outside_bounds` in payloads).
- `matchType` codes: `1` at_least_once (any point in window), `2`
all_the_times (entire window), `3` on_average (window average), `4`
in_total (window sum), `5` last (most recent point).
State each threshold tier's `name`, `target`, `targetUnit`, and
attached channels. **Always state the threshold in `targetUnit`, not
the native query unit** (e.g. "fires when p99 exceeds 500 ms", not
"…exceeds 500 000 000 ns"). Note `recoveryTarget` if set (hysteresis);
if absent, mention flap risk when the value hovers near the boundary.
Describe timing as "checks every `<frequency>` over the last
`<evalWindow>`", and mention that with `at_least_once` a single-point
breach triggers, while `all_the_times` requires the full window.
**3. Where it notifies**: decode effective routing, not just fields in
isolation. If `notificationSettings.usePolicy` is true, say that org policy
matches the rule/static/dynamic labels and that threshold channels are ignored.
Otherwise, a v2 `threshold_rule` or `promql_rule` routes only through each
threshold tier's `channels`. If a tier has no channels, call it unrouted or
misconfigured; never substitute top-level `preferredChannels` as a per-tier
fallback. A v1 `anomaly_rule` has no thresholds or notification settings, so
its direct routing comes from `preferredChannels`. Treat that field as routing
only when the fetched rule schema defines it, not as a cross-schema fallback.
Resolve names from the fully paginated `signoz_list_notification_channels`
result if needed. Also explain `notificationSettings.groupBy` (bundling) and
`renotify` (interval + states). Skip this section only when routing is already
unambiguous in the TL;DR.
**4. Notable concerns**: flag *only* what's non-default and worth the
user's attention. Don't list every absent field; focus on the
high-impact ones:
- **`alertOnAbsent` missing** when the signal is critical: silent data
loss (crashed service, broken instrumentation) won't trigger the
alert. Always call this out for production-tier rules.
- **`alertOnAbsent: true` but `nodata` not in `renotify.alertStates`**:
the absent-data fire pages once and then goes silent (easy to miss).
- **Template variable bugs**: `{{$topic}}` won't interpolate; the
correct form is `{{$labels.topic}}`. Dots in label keys become
underscores (`service.name` → `{{$labels.service_name}}`).
- **Multiple severity tiers but `labels.severity` missing on the rule**:
breaks label-based routing policies. Common gap.
- **All tiers route to the same channel**: defeats the point of
graduated thresholds.
- **High-cardinality `groupBy`** (e.g. `pod.name` × `partition`) →
notification-storm risk during cluster-wide events.
- **Annotation/description text contradicts `matchType`** (e.g.
description says "for over 5 minutes" but `matchType=at_least_once`
fires on first breach within the window).
- **Alert name doesn't match the filter target** (e.g. name says
"checkout" but filter targets `payments`); call this out.
If none of these apply, omit the section rather than pad it.
If the user asked only "what does this alert do", stop here. The audit
(Step 5) is for "is it configured correctly" / "audit this" /
"anything I should change" requests.
### Step 5: Assess the configuration (only if asked)
The user may ask "is this alert reasonable" alongside the explanation.
Only assess when asked or when the request implies it (audit, review,
"is this configured correctly"). Keep assessment grounded in what's
actually in the config:
- **Threshold calibration**: appropriate for the signal? Consider
service criticality and traffic.
- **matchType fit**: `at_least_once` is sensitive (catches transients);
`all_the_times` is conservative; `on_average` smooths noise.
- **Window vs frequency**: short window + `at_least_once` can be noisy.
Long window can delay detection.
- **Multi-severity**: alerts with both warning and critical thresholds
enable graduated response. Single-severity alerts miss this.
- **Notification routing**: critical → high-urgency channels (PagerDuty);
warning → low-urgency (Slack).
- **Missing runbook / description**: if `annotations` are empty or
default, suggest adding context.
- **Absent-data monitoring**: for critical signals, recommend
`alertOnAbsent: true` if it isn't set.
- **GroupBy cardinality**: high-cardinality groupBy fields can produce
many independent alert series; flag potential notification storms.
- **Filter completeness**: for `IN` / `NOT IN` filters with explicit
value lists, flag values that look out of place or missing values
that seem expected.
- **Fire frequency vs threshold**: if Step 3 shows the alert fires
many times a day (>10/day in the 7d window), the threshold is likely
too tight; if it never fires and the user is asking because they
expected it to, the threshold may be too loose or the query may be
wrong.
### Step 6: Offer next steps
Surface up to 3 follow-up intents based on what the explanation
revealed: things like investigating a recent fire, running the
underlying query to see current values, adjusting a threshold, or
creating a related alert for a coverage gap. Use your judgment; do
not pad to 3.
Skip follow-ups entirely when the user is purely inspecting ("what
does this alert do?") and signals no further intent. Offering no
follow-ups is better than offering wrong ones.
## Guardrails
- **Fetch before explaining.** Always call `signoz_get_alert`. Do not
base explanations on the rule name or list response alone.
- **Always pull fire history.** The one-line frequency summary grounds the
explanation, including for disabled alerts: disabled rules can still have
earlier fires in the lookback.
- **Decode, don't dump.** Translate `op`, `matchType`, filter
expressions, and query JSON into operational language. Show raw JSON
only if the user asks.
- **Decode every sub-query.** For formula alerts, explain each builder
query and the formula. Don't gloss over a formula as "error rate";
show the user what A, B, and F1 each compute.
- **Anchor to the actual config.** Don't speculate based on the alert
name. If the name says "checkout" but the filter targets `payments`,
call that out.
- **No live queries by default.** Do not run the alert's query unless
the user asks. Explain is about understanding the rule, not
inspecting current data.
- **Paginate alert listing.** Before concluding an alert does not
exist, walk every page of `signoz_list_alert_rules`.
- **SigNoz evaluation model.** SigNoz uses `evalWindow` + `frequency`
+ `matchType`, not Prometheus-style `for` durations. Don't describe
the evaluation model in Prometheus terms.
- **Stay in scope.** Investigation of a specific fire window belongs
to `signoz-investigating-alerts`. Modifications belong to
`signoz_update_alert`.
## Examples
**User:** "Explain my checkout error rate alert"
**Agent:**
1. `signoz_list_alert_rules` (paginated) → finds
"High Error Rate - Checkout" id 42.
2. `signoz_get_alert id=42` → traces formula (A errored spans / B
total spans × 100), single critical threshold at 5%, `op=1`,
`matchType=1`, channel `pagerduty-oncall`.
3. `signoz_get_alert_history id=42 timeRange=7d state=firing order=desc`
→ fired 3 times, last fire 2h ago.
4. Replies:
> **TL;DR**: Fires when checkout error rate (errored spans / total
> spans) exceeds 5% at any point in the last 5 min, pages
> `pagerduty-oncall`. Fired 3 times in the last 7d (last 2h ago).
>
> **What it watches**: traces from `service.name = 'checkout'`.
> Query A counts spans with `has_error = true`, query B counts all
> spans, F1 = A × 100 / B is the error percentage; the alert
> triggers on F1.
>
> **When it fires**: `op=above`, `matchType=at_least_once`, target
> 5%. Checks every 1 min over the last 5 min; a single elevated
> minute fires it.
>
> **Notable**: no `recoveryTarget` (could flap near 5%); no
> `alertOnAbsent` (a checkout outage that drops traffic to zero
> won't trigger this rule).
5. Offers: "Want me to investigate the fire from 2h ago?"
---
**User:** "Explain alert rule 17 - is it well configured?"
**Agent:**
1. `signoz_get_alert id=17` → metrics, `system.cpu.utilization`,
two thresholds (warning 80%, critical 90%), `matchType=3` on_average,
filter `deployment.environment.name = 'production'`, groupBy
`host.name`.
2. `signoz_get_alert_history id=17` → has not fired in last 7d.
3. The user asked "is it well configured?"; the TL;DR leads with the
**verdict**, not a generic description. Then explanation + full
audit:
> **TL;DR**: Mostly well-configured (dual-threshold + `on_average`
> is the right call for CPU), but two gaps to fix: (1) no
> `alertOnAbsent` (a crashed agent silently stops reporting and
> this rule won't catch it); (2) annotations are default (a
> runbook URL would help on-call). Has not fired in the last 7d
> (could mean production is healthy or thresholds too loose).
>
> **What it watches**: `system.cpu.utilization` on
> `deployment.environment.name = 'production'`, grouped by
> `host.name` (fires per host).
>
> **When it fires**: `op=above`, `matchType=on_average`, two tiers
> (warning 80%, critical 90%) over a 5-min window. Short spikes
> don't fire; smoothing is correct for CPU.
>
> **Notable concerns**: same as TL;DR fixes above; nothing else
> non-default.
4. Offers next steps.
---
**User:** "Help me understand this anomaly detection alert" (alert id 88)
**Agent:**
1. `signoz_get_alert id=88` → `ruleType: anomaly_rule`,
`algorithm=standard` (z-score based), `seasonality=daily`, target 3, metric
`http.server.request.duration`, scope `service.name = 'api-gateway'`.
2. History: fired 1 time in last 7d.
3. Replies:
> **TL;DR**: Fires when api-gateway request latency deviates by more
> than **3 standard deviations** (not raw latency, not a fixed value)
> from its learned daily pattern. Fired once in the last 7d.
>
> **What it watches**: `http.server.request.duration` for
> `service.name = 'api-gateway'`, evaluated as a Z-score anomaly
> with **daily seasonality**: the model learns the typical pattern
> for each hour of day, so peak-hour latency won't false-trigger if
> it matches the historical norm for that hour.
>
> **When it fires**: when |Z-score| > 3, i.e. the value is more
> than 3 standard deviations away from the expected pattern. Lower
> target → more sensitive (more noise); higher → only extreme
> deviations. The threshold is **not** in seconds or milliseconds.
4. Offers to investigate the recent fire.