references/field-catalog.md
# Log & Trace Field Catalog
The complete, exact field catalog the KubeSense MCP server accepts. Source of truth:
`kubeapi/observability/fieldcatalog/{logs,traces}.go`.
**Type the "Label" column.** The "Storage column" is what the backend uses internally; it
is **rejected** at every input slot with a message naming the correct label.
Input slots this applies to: `where`, `required_fields`, `group_by_fields`, `fields`,
`sort_by`.
---
## Logs — 16 fields
| Label (type this) | Storage column | Type | Enum values | Notes |
|---|---|---|---|---|
| `type` | `level` | string | `ERROR` `WARN` `INFO` `DEBUG` `TRACE` `FATAL` `PANIC` | **UPPERCASE** |
| `instance` | `pod_name` | string | | pod |
| `container` | `container_name` | string | | |
| `node` | `host` | string | | **not `node_name`** |
| `workload` | `workload` | string | | app name — logs have no `service` |
| `namespace` | `namespace` | string | | |
| `cluster` | `cluster` | string | | |
| `body` | `body` | string | | message text — **cannot be a group-by** |
| `format` | `format` | string | `json` `klog` `nginx` | lowercase |
| `source` | `source` | string | | |
| `region` | `region` | string | | |
| `app_version` | `app_version` | string | | |
| `env_type` | `env_type` | string | | |
| `timestamp` | `timestamp` | string | | hidden from discovery, still accepted |
| `pattern_id` | `pattern_id` | string | | hidden; group by this to cluster similar messages |
| `body_length` | `body_length` | **float** | | hidden; the only numeric log field |
**13 advertised by discovery** (sorted): `app_version`, `body`, `cluster`, `container`,
`env_type`, `format`, `instance`, `namespace`, `node`, `region`, `source`, `type`,
`workload`.
**Hidden but accepted**: `timestamp`, `pattern_id`, `body_length`. Discovery filters them
out, validation does not — a documented asymmetry you can rely on.
**The four renames**: `type`→`level`, `instance`→`pod_name`, `container`→`container_name`,
`node`→`host`. Everything else is identity.
---
## Traces — 35 fields
| Label (type this) | Storage column | Type | Enum values | Notes |
|---|---|---|---|---|
| `service` | `app_service` | string | | **preferred app identity** |
| `workload` | `workload` | string | | K8s topology only |
| `instance` | `pod_name` | string | | pod |
| `container` | `container_name` | string | | |
| `node_name` | `node_name` | string | | **not `node`** |
| `namespace` | `namespace` | string | | |
| `cluster` | `cluster` | string | | |
| `status` | `status` | string | `error` `ok` | **lowercase** |
| `status_code` | `return_code` | **string** | `200` `400` `401` `403` `404` `500` `502` `503` `504` | string — no range comparisons |
| `method` | `subtype` | string | `GET` `POST` `PUT` `DELETE` `PATCH` | HTTP method |
| `resource` | `clustered_resource` | string | | endpoint |
| `role` | `kind` | string | `server` `client` | span role |
| `protocol` | `protocol_type` | string | `HTTP` `gRPC` `TCP` `MongoDB` `Redis` `MySQL` `PostgreSQL` | |
| `source` | `source` | string | `eBPF` `OTel` | |
| `request_type` | `is_external` | **bool** | | no grouping |
| `duration` | `duration` | **float** | | hidden from discovery, accepted. **ms in, ns out** |
| `trace_id` | `trace_id` | string | | no grouping; sets free-search |
| `span_id` | `span_id` | string | | no grouping; sets free-search |
| `issue_id` | `issue_id` | string | | no grouping; join key to `trace_issues` |
| `reason` | `issue_reason` | string | | sets free-search |
| `operation_name` | `operation_name` | string | | |
| `server` | `server` | string | | |
| `client` | `client` | string | | |
| `server_namespace` | `server_namespace` | string | | |
| `client_namespace` | `client_namespace` | string | | |
| `partner_cluster` | `partner_cluster` | string | | |
| `region` | `region` | string | | |
| `app_version` | `app_version` | string | | |
| `env_type` | `env_type` | string | | |
| `domain` | `customer_identifier` | string | | tenant / customer |
| `timestamp` | `start_timestamp` | string | | hidden; accepted |
| `primary_namespace` | `perspective_namespace` | string | | hidden; accepted |
| `primary_workload` | `perspective_workload` | string | | hidden; accepted |
| `associate_namespace` | `partner_namespace` | string | | hidden; accepted |
| `associate_workload` | `partner_workload` | string | | hidden; accepted |
**29 advertised by discovery** (sorted): `app_version`, `client`, `client_namespace`,
`cluster`, `container`, `domain`, `env_type`, `instance`, `issue_id`, `method`,
`namespace`, `node_name`, `operation_name`, `partner_cluster`, `protocol`, `reason`,
`region`, `request_type`, `resource`, `role`, `server`, `server_namespace`, `service`,
`source`, `span_id`, `status`, `status_code`, `trace_id`, `workload`.
**Hidden but accepted**: `timestamp`, `duration`, `primary_*`, `associate_*`.
**The renames**: `service`→`app_service`, `instance`→`pod_name`,
`container`→`container_name`, `method`→`subtype`, `status_code`→`return_code`,
`resource`→`clustered_resource`, `role`→`kind`, `protocol`→`protocol_type`,
`request_type`→`is_external`, `domain`→`customer_identifier`, `reason`→`issue_reason`,
`timestamp`→`start_timestamp`, `primary_*`→`perspective_*`, `associate_*`→`partner_*`.
---
## Cross-Signal Comparison
The traps, side by side:
| Concept | Logs | Traces | Same? |
|---|---|---|---|
| severity / outcome | `type` = `ERROR` | `status` = `error` | **No — name and casing** |
| node | `node` (→ `host`) | `node_name` | **No — label and column** |
| service identity | *does not exist* | `service` | **Traces only** |
| message text | `body` | *does not exist* | **Logs only** |
| latency | *does not exist* | `duration` | **Traces only** |
| pod | `instance` | `instance` | Yes |
| container | `container` | `container` | Yes |
| namespace / workload / cluster | same | same | Yes |
| region / app_version / env_type | same | same | Yes |
| timestamp | `timestamp` (→ `timestamp`) | `timestamp` (→ `start_timestamp`) | Label yes, column no |
| source | `source`, no enums | `source` = `eBPF`/`OTel` | Label yes, enums traces-only |
Confirmed failures from carrying names across signals:
```
service = checkout on logs → unknown field "service" for signal=logs
node = node-1 on traces → unknown field "node" for signal=traces
pod = my-pod on either → unknown field "pod" (it is "instance")
observation_point = x on traces → unknown field "observation_point"
duration_ms > 500 on traces → unknown field "duration_ms"
```
`observation_point` is worth calling out: the `search-traces` tool description suggests
pinning it for exhaustive pagination, but it is not in the catalog and is rejected.
---
## Error Messages
**Storage name supplied:**
```
field "pod_name" is a storage column; use the catalog label "instance" instead
(call get-trace-or-log-fields to see all labels)
```
Act on the suggestion — do not retry the same name.
**Unknown field:**
```
unknown field "service" for signal=logs; call get-trace-or-log-fields to discover
valid fields (attributes carry an @ prefix)
```
Errors are prefixed by the slot that failed: `where clause: …`,
`required_fields[0]: …`, `group_by_fields[0]: …`, `fields[0]: …`, `sort_by: …`.
---
## Attributes
Dynamic attributes are **not in the catalog** — they are discovered per-window by scanning
storage, so they differ between deployments and windows. Neither signal's catalog contains
a single attribute entry.
| Where | Syntax | Works? |
|---|---|---|
| `where`, logs | `@user.id = abc-123` | ✅ |
| `where`, traces | `@db.system = postgresql` | ❌ **rejected** as unknown field |
| `group_by_fields` / `fields` / `required_fields` / `sort_by`, both signals | `{"field": "db.system", "is_attribute": true}` | ✅ |
Discovery reports the key **bare** in the `field` column with `is_attribute=true`; the `@`
appears only in the `example` cell.
**Log attribute operators**: `=` `!=` `IN` `<` `>` `<=` `>=` `LIKE` `ILIKE`. Unquoted
numerics are coerced, so `@latency > 100` compares numerically.
**Trace attributes** are stored as parallel `attribute_names[]`/`attribute_values[]` arrays
and are always string-typed. The parser is supposed to remap `=`/`IN`/`LIKE` to
`ARRAY_MAP*` operators, but it clears the attribute marker in the process, so the field
then fails column validation — hence the WHERE rejection above. Group-by is unaffected.
`exist(@field)` is broken on both signals: it emits an `EXIST` operator carrying a value,
which the validator rejects (`operator "EXIST" takes no values (got 1)`). `NOT_EXIST` has
no syntax at all. There is no working existence check.
---
## Free-Search Routing
Touching any of these fields routes the query to the **raw table** instead of the
pre-aggregated rollups — slower, but the only way to see message text or per-span detail:
- logs: `body`, `trace_id`, `span_id`, any attribute
- traces: `duration`, `trace_id`, `span_id`, `reason`, any attribute
It is computed automatically and reported as `is_free_search` in the output header.
Column-only filters stay on the fast rollup path. This is also why `duration` filters make
a trace query noticeably slower.
---
## Operator Availability vs Advertised
Discovery's `operators` column reflects an internal catalog that is **broader than the
WHERE parser**.
**Parseable in a WHERE string:** `=` `!=` `<` `>` `<=` `>=` `LIKE` `ILIKE` `SUBSTR_ILIKE`
`IN`
**Advertised but with no WHERE syntax** — these fail to parse: `HAS_TOKEN`, `HAS_ALL`,
`HAS_ANY`, `NOT_HAS_ANY`, `LIKE_AND`, `ILIKE_AND`, `ILIKE_LOG`, `ILIKE_LOG_AND`,
`SUBSTR_ILIKE_AND`, `IS_IP_ADDRESS`, `NOT_IS_IP_ADDRESS`, `EXIST`, `NOT_EXIST`, and the
whole `ARRAY_MAP*` family.
Other parser constraints:
- Bare values allow only `[A-Za-z0-9_.-]`. Anything with `/`, `:`, `%`, or a space must be
**double-quoted**: `resource = "/api/v1"`, `timestamp > "2026-07-30T10:00:00Z"`.
- `NOT IN` is not a leaf operator — write `NOT (field IN (...))`.
- Trace `IN` values stay strings: `status_code IN (500, 502)` → `"500"`, `"502"`.
- `duration` is the only trace field whose value is coerced to a number (and scaled ×1e6
from ms to ns). Every other trace value stays a string.
references/multi-query.md
# Multi-Query and Formulas (`analyze-telemetry`)
Runs several queries across logs, traces, and metrics in **one** request, and composes them
with formulas. This is the only way to get a ratio, error rate, or percentage — no single
`analyze-*` call can divide one series by another.
## When to Use
- Any derived value: error rate, success rate, a share of total, a ratio
- Correlating two datasources in one call (e.g. log errors against request volume)
- Replacing several `analyze-*` round-trips
For one aggregation over one datasource, use `analyze-logs` / `analyze-traces` /
`analyze-metrics` — they are simpler and return the same shape.
## Structure
```json
{
"from_time": "2026-07-30T10:00:00Z",
"to_time": "2026-07-30T11:00:00Z",
"query_type": "range",
"queries": {
"A": { "selectedMode": "traces", "where": "status = error", "value_operation": "row_count" },
"B": { "selectedMode": "traces", "value_operation": "row_count" },
"C": { "query_type": "formula", "expression": "(A/B)*100" }
}
}
```
Top-level `from_time`, `to_time`, `query_type`, and `queries` are all required. The
top-level `query_type` is `range` or `instant` and applies to every sub-query.
Query keys are single-letter labels: `A`, `B`, `C`, …
## Discriminators — get these exactly right
| Sub-query kind | Discriminator |
|---|---|
| logs | `"selectedMode": "logs"` |
| traces | `"selectedMode": "traces"` |
| metrics | `"selectedMode": "metrics"` |
| formula | `"query_type": "formula"` (or `"queryType": "formula"`) |
> [!WARNING]
> **A formula is identified by `query_type`, not `selectedMode`.** And there is no
> `datasource` field — the tool's own description shows
> `{"datasource": "formula", "expression": "(A/B)*100"}`, which is **wrong**; `datasource`
> is never read.
>
> **An omitted `selectedMode` silently defaults to `metrics`.** A logs sub-query missing it
> is parsed as a metrics query and returns nothing useful. Always set it explicitly.
## Sub-Query Shapes
### Logs / traces
Same fields as `analyze-logs` / `analyze-traces`. `selectedMode` and `value_operation` are
required.
```json
{
"selectedMode": "logs",
"where": "type = ERROR",
"group_by_fields": [ { "field": "workload", "is_attribute": false } ],
"value_operation": "row_count",
"clusters": [],
"sort_direction": "DESC"
}
```
Field names follow the normal catalog-label contract — `type` for log severity, `service`
for trace identity. See [field-catalog.md](./field-catalog.md).
### Metrics
`selectedMode` and `promql` both required.
```json
{ "selectedMode": "metrics", "promql": "sum(rate(container_cpu_usage_seconds_total{container!=''}[5m])) by (namespace)" }
```
### Formula
`query_type` and `expression` required.
```json
{ "query_type": "formula", "expression": "(A/B)*100" }
```
Expressions reference other labels in the **same** request. Arithmetic: `+ - * /` and
parentheses.
## Matching Group-By Across Queries
> [!IMPORTANT]
> Queries combined in a formula must use the **same `group_by_fields`**. The formula engine
> matches series by their labels — mismatched grouping produces empty or nonsensical
> results, not an error.
If `A` groups by `workload` and `B` doesn't group at all, `A/B` cannot pair the series.
Either group both, or group neither.
## Examples
### Trace error rate per workload
```json
{
"from_time": "2026-07-30T10:00:00Z",
"to_time": "2026-07-30T11:00:00Z",
"query_type": "range",
"queries": {
"A": {
"selectedMode": "traces",
"where": "status = error",
"group_by_fields": [ { "field": "workload", "is_attribute": false } ],
"value_operation": "row_count"
},
"B": {
"selectedMode": "traces",
"group_by_fields": [ { "field": "workload", "is_attribute": false } ],
"value_operation": "row_count"
},
"C": { "query_type": "formula", "expression": "(A/B)*100" }
}
}
```
### Log error rate as a share of all logs
```json
{
"from_time": "2026-07-30T10:00:00Z",
"to_time": "2026-07-30T11:00:00Z",
"query_type": "instant",
"queries": {
"A": { "selectedMode": "logs", "where": "type = ERROR", "value_operation": "row_count" },
"B": { "selectedMode": "logs", "value_operation": "row_count" },
"C": { "query_type": "formula", "expression": "(A/B)*100" }
}
}
```
### 5xx share of HTTP requests
```json
{
"from_time": "2026-07-30T10:00:00Z",
"to_time": "2026-07-30T11:00:00Z",
"query_type": "range",
"queries": {
"A": {
"selectedMode": "traces",
"where": "protocol = HTTP AND status_code LIKE \"5%\"",
"group_by_fields": [ { "field": "service", "is_attribute": false } ],
"value_operation": "row_count"
},
"B": {
"selectedMode": "traces",
"where": "protocol = HTTP",
"group_by_fields": [ { "field": "service", "is_attribute": false } ],
"value_operation": "row_count"
},
"C": { "query_type": "formula", "expression": "(A/B)*100" }
}
}
```
### Log errors against request volume (cross-signal)
```json
{
"from_time": "2026-07-30T10:00:00Z",
"to_time": "2026-07-30T11:00:00Z",
"query_type": "range",
"queries": {
"A": {
"selectedMode": "logs",
"where": "type = ERROR",
"group_by_fields": [ { "field": "workload", "is_attribute": false } ],
"value_operation": "row_count"
},
"B": {
"selectedMode": "traces",
"group_by_fields": [ { "field": "workload", "is_attribute": false } ],
"value_operation": "row_count"
},
"C": { "query_type": "formula", "expression": "A/B" }
}
}
```
Both sides group by `workload` — the one identity field that means the same thing on both
signals. Grouping logs by `workload` and traces by `service` would not pair.
### Latency alongside throughput (no formula)
Formulas are optional; use `analyze-telemetry` purely to save round-trips:
```json
{
"from_time": "2026-07-30T10:00:00Z",
"to_time": "2026-07-30T11:00:00Z",
"query_type": "range",
"queries": {
"A": {
"selectedMode": "traces",
"group_by_fields": [ { "field": "service", "is_attribute": false } ],
"fields": [ { "field": "duration", "is_attribute": false } ],
"value_operation": "p95"
},
"B": {
"selectedMode": "traces",
"group_by_fields": [ { "field": "service", "is_attribute": false } ],
"value_operation": "row_count"
}
}
}
```
`A`'s values are in **nanoseconds** (aggregated trace `duration`).
## Output
Columnar JSON with one `results` entry per label you supplied — including the formula's.
For `range` queries there is additionally a `series_meta` keyed by the same labels.
Each entry holds `series` (an array of `{labels, ...}`) plus `bucket_duration_seconds`, and
optionally `promql`, `resource`, or `error_message`.
- `range`: a shared `timestamps` array (Unix seconds, emitted once) with each series
carrying a positionally aligned `values` array; `null` = no data in that bucket.
- `instant`: each series carries one scalar `value`.
A sub-query that fails reports an `error_message` on its own entry rather than failing the
whole call — check for it before trusting a formula that depends on it.
## RBAC
`analyze-telemetry` resolves permissions for **every module its sub-queries touch** —
`logs`, `traces`, and `infrastructure` for metrics. A mixed request needs access to all of
them; the error names the module that failed.
## Rules
1. `selectedMode` on every logs/traces/metrics sub-query — omitting it silently defaults to
`metrics`.
2. Formulas use `query_type: "formula"`, never `selectedMode`. There is no `datasource`
field.
3. Formula operands need **matching `group_by_fields`**, or the series cannot pair.
4. `value_operation` is required on logs/traces sub-queries, and it is `row_count`, not
`count`.
5. One top-level `query_type` (`range` or `instant`) governs every sub-query.
6. Check each entry for `error_message` before reporting a formula result.
7. Use `analyze-*` for a single aggregation; reach for this tool when you need a ratio or
want to save round-trips.
SKILL.md
---
name: kubesense-mcp
description: The KubeSense MCP tool layer — connection and auth, the full 29-tool inventory, tool selection, the discovery-first rule, the catalog-label field contract shared by every logs/traces query, WHERE syntax, multi-datasource formula queries, and how to read the TSV/columnar output formats. Read this when a tool returns a field-name or WHERE error.
metadata:
version: "2.0.0"
author: kubesense
repository: https://github.com/kubesense-ai/kubesense-mcp-skills
tags: kubesense,mcp,observability,tools,where-clause,discovery,field-catalog,auth
---
# KubeSense MCP
The tool layer every KubeSense query skill sits on. This skill owns the **shared
contract** — field names, WHERE syntax, output formats — so the per-signal skills don't
each have to redefine it.
## Routing
| The user is asking… | Read |
|---|---|
| logs, error messages, grep | [kubesense-logs](../kubesense-logs/SKILL.md) |
| latency, p99, requests, spans, which hop broke | [kubesense-traces](../kubesense-traces/SKILL.md) |
| CPU, memory, disk, PromQL | [kubesense-metrics](../kubesense-metrics/SKILL.md) |
| what's running, pod restarts, what changed | [kubesense-infra](../kubesense-infra/SKILL.md) |
| alerts — list, investigate, create | [kubesense-alerts](../kubesense-alerts/SKILL.md) |
## Connection
Served by kubeapi at **`/mcp`** over Streamable HTTP, mounted outside the `/api` group.
Same credentials as the REST API.
```bash
# API key — recommended for agents, does not expire
claude mcp add --scope user --transport http kubesense \
https://<your-kubesense-host>/mcp \
--header "x-api-key: <your-api-key>"
# JWT access token — expires
claude mcp add --scope user --transport http kubesense \
https://<your-kubesense-host>/mcp \
--header "Authorization: Bearer <access-token>"
```
Liveness: `GET /mcp_health`. To confirm *who* you are authenticated as, call
`get-current-user` — no arguments, cheap, resolved from the request credentials.
**RBAC** is resolved per tool call against the modules `logs`, `traces`,
`infrastructure` (metrics + nodes/pods), and `alerts`. A permission error names the
module. `analyze-telemetry` resolves every module its sub-queries touch.
Set `MCP_LOG_LEVEL=debug` server-side to log per-call argument payloads without raising
the global log level.
## Tools (29)
**Discovery — call before querying**
| Tool | Returns |
|---|---|
| `get-trace-or-log-fields` | Field catalog for logs or traces in a window |
| `get-available-metrics` | Metric names |
| `get-metric-labels` | Label names on one metric |
**Query**
| Tool | Returns |
|---|---|
| `search-logs` | Raw log rows (default 10) |
| `search-traces` | Raw span rows (default 10) |
| `analyze-logs` | Aggregated log series |
| `analyze-traces` | Aggregated span series |
| `analyze-metrics` | PromQL result |
| `analyze-telemetry` | Multiple queries + formulas in one call |
| `get-distributed-trace` | Full span tree for one `trace_id` |
**Inventory** — see [kubesense-infra](../kubesense-infra/SKILL.md)
`list-clusters` · `list-nodes` · `get-node-detail` · `list-pods` · `get-pod-detail` ·
`list-workloads` · `get-workload-detail` · `list-issues` · `get-infra-issues` ·
`get-recent-changes`
**Alerts** — see [kubesense-alerts](../kubesense-alerts/SKILL.md)
`list-alerts` (external Alertmanager) · `list-active-alerts` (native store, firing) ·
`list-alert-rules` (rule definitions) · `get-alert-details` · `get-alert-history` ·
`list-notification-channels` · `find-investigation-for-alert`
**Identity**
`get-current-user` — returns `username`, `name`, `email`, `role`, `auth_type`. Note
`username` and `email` are frequently **different**; ownership fields such as
`list-alert-rules`' `created_by` match on **username**.
**Validation — checks a document, stores nothing**
`validate-alert-json` · `validate-dashboard-json`
Both take a `document` (a JSON object, or a string containing one) and report every
schema problem at once as `path` / `rule` / `message`, where `path` is a JSON Pointer to
the value to fix. Read-only and free to call repeatedly. Run whatever you assembled by
hand through the matching one before creating it or handing it over.
**Write — mutates state**
`create-alert` · `create-dashboard`
Both carry write annotations and expect user approval. State exactly what you are about
to create before calling. Each validates against the same schema as the tool above and
refuses with the same findings, so validating first turns a refusal into a fix.
## Discovery-First Rule
**Never guess a name.** Not a field, metric, cluster, or label.
```
logs / traces: get-trace-or-log-fields → search-* / analyze-*
metrics: get-available-metrics → get-metric-labels → analyze-metrics
clusters: list-clusters → anything with a "clusters" filter
```
Two reasons this matters more than usual here:
1. Field names are **rejected** with an error naming the correct label — good, but only
if you read it rather than retrying.
2. A wrong **cluster** name or **metric** name is *not* an error — it returns an empty
result indistinguishable from "the value is zero".
Pass the **same time window** to `get-trace-or-log-fields` that you will use for the
query: attribute keys are window-scoped and differ across windows.
## The Field-Name Contract
Logs/traces inputs — `where`, `required_fields`, `group_by_fields`, `fields`, `sort_by` —
accept **catalog labels only**. Storage column names are rejected:
```
field "pod_name" is a storage column; use the catalog label "instance" instead
(call get-trace-or-log-fields to see all labels)
unknown field "service" for signal=logs; call get-trace-or-log-fields to discover
valid fields (attributes carry an @ prefix)
```
| Concept | Logs | Traces |
|---|---|---|
| severity / outcome | `type` (`ERROR` — **UPPER**) | `status` (`error` — **lower**) |
| pod | `instance` | `instance` |
| container | `container` | `container` |
| **node** | **`node`** | **`node_name`** |
| **service identity** | **not available** — use `workload` | **`service`** |
| workload | `workload` | `workload` |
| HTTP method | — | `method` |
| HTTP status | — | `status_code` |
| endpoint | — | `resource` |
| span role | — | `role` |
| protocol | — | `protocol` |
| message text | `body` | not available |
| latency | — | `duration` |
The full per-signal tables, including hidden-but-accepted fields and every storage
mapping, are in **[references/field-catalog.md](./references/field-catalog.md)**.
> [!WARNING]
> The two signals disagree on `node`/`node_name` and on severity name *and* casing, and
> `service` exists only on traces. Carrying a field name from one signal to the other is
> the most common failure.
Read enum casing from the **`enum_values`** column of discovery output, never the
`example` column — the example generator silently returns blanks for several enum fields
(logs `type`; traces `method`, `protocol`, `status_code`).
## WHERE Syntax
```
type = ERROR AND namespace = production
status = error AND role = server
namespace IN (production, staging)
NOT (env_type = dev)
body ILIKE "%timeout%"
duration > 500
@user.id = abc-123
```
**Operators the parser accepts:** `=` `!=` `<` `>` `<=` `>=` `LIKE` `ILIKE`
`SUBSTR_ILIKE` `IN` — combined with `AND` / `OR` / `NOT` and parentheses.
> [!WARNING]
> Discovery's `operators` column **over-advertises**. `HAS_TOKEN`, `HAS_ALL`, `HAS_ANY`,
> `LIKE_AND`, `ILIKE_LOG`, `IS_IP_ADDRESS`, `EXIST` appear in the legend but have **no
> WHERE-string syntax**. `body HAS_TOKEN oom` is a parse error. `exist(@field)` is broken
> on both signals (it emits a value-bearing `EXIST` the validator rejects) — there is no
> working existence check.
**Quoting.** Bare values may contain only letters, digits, `_`, `.`, `-`. Anything with a
slash, colon, `%`, or space must be double-quoted:
```
resource = "/api/v1/checkout" ✓
timestamp > "2026-07-30T10:00:00Z" ✓
resource = /api/v1/checkout ✗ parse error
```
**`NOT IN` is not a leaf operator.** Write `NOT (namespace IN (kube-system, default))`.
**Attributes** carry `@` in WHERE — **but only on logs.** On traces, `@attr` filters are
rejected as unknown fields (a server-side defect; the tool descriptions claim otherwise).
Trace attributes still work in `group_by_fields` / `fields` / `required_fields` /
`sort_by` via `{"field": "db.system", "is_attribute": true}` — no `@` there, for either
signal.
**`duration` is millisecond-in, nanosecond-out** — see
[kubesense-traces](../kubesense-traces/SKILL.md).
## Choosing a Tool
| Question | Tool |
|---|---|
| "show me recent errors" | `search-logs` |
| "how many errors in the last hour" | `analyze-logs`, `row_count` |
| "p99 latency for checkout" | `analyze-traces`, `p99` over `duration` |
| "why did this request fail" | `get-distributed-trace` |
| "CPU usage" | `get-available-metrics` → `analyze-metrics` |
| "error rate as a percentage" | `analyze-telemetry` with a formula |
| "what's running / what's broken / what changed" | inventory tools — one call for what would otherwise be several queries |
**Aggregate with `analyze-*`, don't paginate `search-*`.** Reading raw rows to compute a
count is slower, costs far more tokens, and — because paged scans are ordered
hour-bucket-first across clusters — is not even guaranteed complete.
`query_type` on every `analyze-*`: `range` = time series (trends), `instant` = one value
per series (totals, current state, top-N).
Time ranges are RFC3339 (`2026-07-30T10:00:00Z`). Start narrow (15–30 min) and widen.
Discovery tools default to the last hour; inventory tools too.
## Multi-Datasource Queries and Formulas
`analyze-telemetry` runs several queries in one call and composes them. This is the only
way to get a ratio, error rate, or percentage.
```json
{
"from_time": "2026-07-30T10:00:00Z",
"to_time": "2026-07-30T11:00:00Z",
"query_type": "instant",
"queries": {
"A": { "selectedMode": "traces", "value_operation": "row_count" },
"B": { "selectedMode": "traces", "where": "status = error", "value_operation": "row_count" },
"C": { "query_type": "formula", "expression": "(B/A)*100" }
}
}
```
Queries are keyed by single-letter labels. Formula entries use
`"query_type": "formula"` with an `expression` referencing other labels. See
**[references/multi-query.md](./references/multi-query.md)** for cross-signal examples.
## Reading the Output
Three shapes, all designed to save tokens:
**TSV** (`search-*`, all inventory and alert list tools) — a `# key=value` metadata line,
then a header row, then one row per record. Empty cell = field absent. A `## name` line
starts a labelled sub-table (e.g. `## services`, `## silences`).
**Columnar JSON** (all `analyze-*`) — `results` keyed by query label:
- `range`: one shared `timestamps` array (Unix seconds, emitted **once**), and each series
carries a bare `values` array aligned positionally — `values[i]` is the value at
`timestamps[i]`, `null` = no data in that bucket.
- `instant`: no `timestamps`; each series carries a single scalar `value`.
- `total_series` / `truncated` report how many non-zero series existed **before** the
top-N cap.
**Record with sub-tables** (`get-*-detail`) — `field<TAB>value` lines for scalars, `## name`
TSV blocks for nested arrays, compact JSON for maps. Empty fields omitted.
Timestamps are always RFC3339 UTC at millisecond precision — exactly the format
`from_time`/`to_time` accept, so a cell can be copied straight back as a cursor.
## Pagination
`search-logs` / `search-traces` have **no `page` or `offset`**. You page by narrowing
`to_time`; the window is half-open so the cursor row isn't repeated.
Take the **minimum** timestamp from the page (`timestamp` for logs, `start_timestamp` for
traces) and pass it verbatim as the next `to_time`. Copy the millisecond precision exactly.
Inventory and alert tools *do* paginate conventionally, with `page`/`page_size` (default
50) or `limit`/`offset`.
See the per-signal skills for the ordering caveats that make exhaustive raw scans
unreliable — and prefer aggregation whenever the answer is a number.
## Rules
1. Discover before querying — fields, metrics, clusters. Never guess.
2. Pass the same window to `get-trace-or-log-fields` that you'll query.
3. Catalog labels only. On a field error, read the suggested label; don't retry the same
name.
4. Never carry a field name across signals (`node` vs `node_name`; `type` vs `status`;
`service` is traces-only).
5. Enum casing from `enum_values`, not `example`.
6. WHERE operators: `= != < > <= >= LIKE ILIKE SUBSTR_ILIKE IN` only. Quote values with
`/`, `:`, `%`, or spaces. `NOT (x IN (...))`, never `x NOT IN (...)`.
7. `value_operation` is `row_count` — never `count`.
8. `analyze-*` for numbers, `search-*` only to read records.
9. `analyze-telemetry` + formula for ratios and percentages.
10. An empty result is not zero — verify names and widen the window before reporting a
value.
11. `create-alert` mutates state: say what you're creating, and call
`list-notification-channels` first or it will notify nobody.