references/ai-center-queries.md
# AI Center — GenAI span querying (DataPrime), schema & playbooks
This is the telemetry half of the AI Center skill: how to read GenAI interactions and
compute AI metrics from **spans**, using the cx CLI. Run every query with:
```bash
cx spans '<DataPrime query>' --start now-1d --end now
```
`cx spans` defaults to the last 1 hour when you don't pass `--start`; widen it (e.g.
`--start now-1d`) when the question needs more history and the user hasn't specified a window.
Use `-o json` for machine-readable output. For general DataPrime/spans syntax see the
`cx-telemetry-querying` skill and [dataprime-reference.md](dataprime-reference.md) /
[spans-querying.md](spans-querying.md). For **configuration** (inventory, evaluations,
policies, coverage, pricing) use the `cx ai-center` commands documented in the parent
`SKILL.md` — that data is not in spans.
> **Shell quoting:** DataPrime uses single quotes for string literals, so wrap the whole
> query in double quotes and escape the `$` sigils (`\$d`, `\$l`, `\$m`) so the shell doesn't
> expand them — or put the query in a file and pipe it in. Examples below show the raw
> DataPrime; quote as needed for your shell.
---
## Golden rule: read the interactions, not just the verdicts
Each interaction's **GenAI spans** carry the **conversation** plus model, tokens, cost, latency,
errors, and tool calls. Two conventions store the conversation — current
(`gen_ai.input.messages` / `gen_ai.output.messages`) and older indexed (`gen_ai.prompt.<n>` /
`gen_ai.completion.<n>`); see **Reading conversations (content questions)** for each one's exact
shape and how to read it. Evaluations, when configured, add **optional** verdict tags on the GenAI
span; guardrails are **optional** too and live on **separate** guardrail spans
(`otel.library.name == 'cx_guardrails.client'`). **Match the data to the
question:** for **content** questions (quality, hallucination, sentiment, frustration,
satisfaction, topics) read the conversation and reason about it; for everything else use the
relevant tags/aggregations. For plain dialogue read the **`user`**/**`assistant`** turns (via the **Reading conversations
(content questions)** queries below), but the **`system`**/**`tool`** turns matter too when you're
debugging an agent or sub-agent (the tool calls it made and the task it was given). The system
prompt lives in the `gen_ai.system_instructions` tag **or** the `role:"system"` turn: the first
`role:"system"` message inside `gen_ai.input.messages` (current) / `gen_ai.prompt.0.content` when
`gen_ai.prompt.0.role == "system"` (indexed). Check both places.
**Cap the volume first — sample, don't read them all.** When the answer comes from **reading the
conversations** (satisfaction, sentiment, topics, quality, summaries, "read the last day and tell
me if X"), never run the heavy query over *all* of an app's interactions — cap to **≤150** (100
default; 50–150 when each unit is large) and read only that sample. Report how many matched vs.
how many you read (e.g. "312 matched; I read 100") and offer to narrow. Cite the `traceID` of any
interaction you reference.
**Do NOT sample in two cases:**
1. **An exact metric** — `count`/`distinct_count`, sums (cost, tokens), error/issue rates,
percentiles, "how many…". Compute over *all* matching data (still scoped by a tight time window
+ indexed filters); a sample gives a wrong number.
2. **A specific target / needle** — a given `traceID`, one user's session, "the last 5 traces", or
"does **any** interaction do X". Filter **directly** to the target; sampling could miss it.
**Trace-level reads (one interaction at a time — sentiment, topics, per-session content) → two
steps.** *Step 1* — get the capped distinct trace IDs (cheap, IDs only):
```dataprime
source spans
| filter (tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))
| filter tags['gen_ai.input.messages'] != null || tags['gen_ai.output.messages'] != null || tags['gen_ai.prompt.0.role'] != null
| distinct $d.traceID
| limit 100
```
*Step 2* — paste those IDs and run the requested (heavy) query over **only** those traces:
```dataprime
source spans
| filter ['TRACE_1','TRACE_2', … ].arrayContains($d.traceID)
| … the requested query …
```
**Span-level reads** (per span, no per-interaction grouping): skip the ID round-trip — add
`| limit <N>` (≤150) right after the GenAI spans filter. Keep the GenAI filter and add the
question's own scope (time window, app, user/model) on the Step-1 query; Step 2 inherits it via the
trace IDs. The same two-step cap applies to the indexed convention.
**Scope before you scan:** start with a narrow time window (widen only if needed), anchor on
indexed `$l`/`$m` fields before matching/reading `$d.*` payload, and discover an unknown field
cheaply (a `limit 5` sample) instead of guessing with a full scan.
Use `cx ai-center applications list` to get the exact `applicationName`/`subsystemName` pairs.
The app filter in the queries below is **optional**: keep it to scope to one app, **drop it** for
org-wide questions, or **group by** `$l.applicationName, $l.subsystemName` to compare.
**To read the actual user+assistant transcript** — both conventions, system prompt and tool
traffic excluded — use the queries under **Reading conversations (content questions)** below.
Don't hand-select the raw `input.messages` blob for content questions: it carries the system
prompt and tool traffic and bloats context.
---
## How AI Center data is stored
**Granularity.** An **AI span** = one AI operation (an LLM call, embeddings, retrieval/agent/
tool/workflow step, or a Guardrails SDK invocation). An **interaction** = the multiple spans
under one `traceID` — one exchange plus the surrounding spans and operations. Counts (AI spans,
errors, guardrail actions) are **span-level**; the only trace-level rate is Issue Rate (Q9). For
a deduplicated interaction count use `distinct_count(traceID)` and say so. For very
high-cardinality counts, use `approx_count_distinct(traceID)` to avoid memory blowups when an
exact number isn't needed.
**Find GenAI spans — the mandatory filter on every AI Center query.** `source spans` also
holds ordinary APM traffic; omitting this filter computes AI metrics over non-AI data and
returns wrong answers.
```dataprime
| filter (tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))
```
To also include guardrail spans (for AI-span counts and issue/guardrail rates),
add `|| tags['otel.library.name']:string == 'cx_guardrails.client'`. Guardrail spans are marked
by `tags['otel.library.name'] == 'cx_guardrails.client'`; from there, filter by whatever the
question needs — `tags['guardrails.triggered'] == 'true'` for **triggered** guardrails, `== 'false'`
for ones that **passed**, or the per-policy `gen_ai.{prompt|response}.guardrails.{policy}.triggered`
tags for a specific guardrail.
> An **AI application** is a GenAI `($l.applicationName, $l.subsystemName)` **pair** — the apps
> returned by `cx ai-center applications list`, not every `applicationName` in spans. **AI error
> rate** = `otel.status_code == 'ERROR'` on GenAI spans (not generic HTTP 5xx). When unsure
> which apps are AI apps, get them from `cx ai-center applications list`.
**Scoping to an application.** A name may live in either `applicationName` or `subsystemName`;
if a filter returns nothing, try the other field, or use `cx search-fields "<name>" --dataset spans`
to find which field holds it. Time = `$m.timestamp` (`roundTime($m.timestamp, <interval>ms)`
for series); duration = `$m.duration` (µs).
**End-user identity:**
```dataprime
| create user from firstNonNull(tags['enduser.id'], tags['user.id'], tags['gen_ai.request.user'], tags['traceloop.association.properties.user_id'], tags['langsmith.metadata.user_id'])
```
**Message formats.** `gen_ai.input.messages` / `gen_ai.output.messages` are JSON arrays of
`{role, parts:[{type, content}]}` (part `type` = `text`, `tool_call`, `tool_call_response`,
`reasoning`, `blob`/`file`/`uri`). (The older indexed convention stores the same
roles as flat per-message tags — `gen_ai.prompt.<n>.{role,content}` /
`gen_ai.completion.<n>.{role,content}`, no `parts` array — see Reading conversations below.)
### Reading conversations (content questions)
**Two conventions carry the conversation — handle both** (pick by which tags exist: use the
**Current** query when `gen_ai.input.messages` is present, the **Indexed** query when
`gen_ai.prompt.0.role` is present):
- **Current:** `gen_ai.input.messages` and `gen_ai.output.messages` are each **one tag holding a
JSON array of `{role, parts:[{type, content}]}` messages** — `input.messages` = the full input
history (all prior turns), `output.messages` = the model's response turn(s).
- **Older indexed:** **one tag per message**, numbered from 0 — `gen_ai.prompt.<n>.role` /
`gen_ai.prompt.<n>.content` for each **input** message and `gen_ai.completion.<n>.role` /
`gen_ai.completion.<n>.content` for the **output** message(s). `gen_ai.prompt.0` is the first
message (usually the system prompt); the conversation is spread across `prompt.0, prompt.1, …`
and `completion.0, …`. An assistant turn that calls tools carries
`gen_ai.prompt.<n>.tool_calls.*` (function name/arguments/id) and **no** `.content`; a tool
**result** comes back as `role:"tool"` with its output in `.content`.
**Current convention — general conversation read.** Parse the messages JSON with `jsonobject()`,
keep `user`/`assistant` roles (drops `system` + `tool`) and `type:"text"` parts (drops `tool_call`
/ `tool_call_response`), and collect one transcript per span. **Run this over the capped trace IDs**
(see *Cap the volume first* — this is the Step-2 heavy read, not a full scan): prepend
`| filter ['ID1','ID2', … ].arrayContains($d.traceID)` after `source spans`. Scope by adding
`| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'` after `source spans`; a
span's `input.messages` accumulates the prior turns, so the fullest span of a conversation holds the
whole history:
```dataprime
source spans
| filter $d.tags['gen_ai.input.messages'] != null || $d.tags['gen_ai.output.messages'] != null
| create input_json from concat('{"messages":', firstNonNull($d.tags['gen_ai.input.messages'], '[]'), '}')
| create output_json from concat('{"messages":', firstNonNull($d.tags['gen_ai.output.messages'], '[]'), '}')
| extract $d.input_json into input_parsed using jsonobject()
| extract $d.output_json into output_parsed using jsonobject()
| create all_messages from arrayConcat($d.input_parsed.messages, $d.output_parsed.messages)
| explode $d.all_messages into $d.message original preserve
| filter ['user','assistant'].arrayContains($d.message.role)
| explode $d.message.parts into $d.part original preserve
| filter $d.part.type == 'text' && trim(firstNonNull($d.part.content, '')) != ''
| create line from concat($d.message.role, ': ', $d.part.content)
| groupby $d.traceID as trace_id, $d.spanID as span_id aggregate collect($d.line) as lines
| create conversation_text from arrayJoin($d.lines, '\n')
| choose trace_id, span_id, conversation_text
```
This keeps `type:"text"` parts only. Some SDKs embed model reasoning (`<thinking>…</thinking>`) or
serialize a tool call into a text part (e.g. `ResponseFunctionToolCall(…)`) — those are inside the
content, so they can appear in the transcript; treat them as noise when judging.
Aggregate instead of read (e.g. top user questions): keep only `$d.message.role == 'user'` before
the parts explode, then `| groupby $d.part.content aggregate count() as times | orderby times desc`.
**Indexed convention — general conversation read.** Run this over the capped IDs too (see *Cap the
volume first* — prepend the same `['ID1', … ].arrayContains($d.traceID)` filter). The messages are
separate numbered tags (`gen_ai.prompt.<n>.*`), so there's no JSON string to parse and DataPrime can't enumerate the
numbered keys directly. Instead serialize the whole span with `$d:string` and `multi_regexp` every
`role`/`content` pair at once (any message count, no hardcoded range); group by index to re-pair
role with content, then keep `user`/`assistant` with non-empty content (drops `system`, `role:"tool"`
results, and tool-call turns which have no `.content`):
```dataprime
source spans
| filter $d.tags['gen_ai.prompt.0.role'] != null
| create doc_json from $d:string
| extract doc_json into matches using multi_regexp(e=/"gen_ai\.(prompt|completion)\.(\d+)\.(role|content)":"((?:[^"\\]|\\.)*)"/)
| explode matches into m original preserve
| extract m into p using regexp(e=/"gen_ai\.(?<grp>prompt|completion)\.(?<idx>\d+)\.(?<kind>role|content)":"(?<value>(?:[^"\\]|\\.)*)"/)
| groupby $d.traceID as trace_id, $d.spanID as span_id, p.grp:string as grp, p.idx:number as idx
aggregate any_value(if(p.kind == 'role', p.value:string, null)) as role,
any_value(if(p.kind == 'content', p.value:string, null)) as content
| filter (role == 'user' || role == 'assistant') && content != null && trim(content) != ''
| create line from concat(if(grp == 'prompt', '0', '1'), ':', padLeft(idx:string, 5, '0'), ':::', role, ': ', content)
| groupby trace_id, span_id aggregate arrayJoin(arraySort(collect(line)), '\n') as conversation_text
| redact conversation_text matching /[01]:\d{5}:::/ to ''
| choose trace_id, span_id, conversation_text
```
The `grp`/`idx` sort prefix orders prompt-before-completion and ascending index (`collect` doesn't
preserve order); `redact` strips it. Same content-level caveat as the current convention
(`<thinking>…` / tool-call-as-text can appear). Scope with
`| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'`.
**Span attributes (GenAI spans).** Span kind — `gen_ai.operation.name`: `chat` (also
`generate_content`, `text_completion`), `embeddings`, `retrieval` (RAG), `create_agent`,
`invoke_agent`, `invoke_workflow`, `execute_tool`.
| Attribute | Meaning |
| --- | --- |
| `gen_ai.provider.name` (or deprecated `gen_ai.system`) | Provider |
| `otel.status_code` (`ERROR`) / `error.type` | Failure status / error class |
| `otel.library.name` | `cx_guardrails.client` marks a guardrail span |
| `gen_ai.agent.{id,name,description,version}` / `gen_ai.workflow.name` | Agent / workflow identity |
| `gen_ai.request.model` / `gen_ai.response.model` / `gen_ai.response.id` | Requested / actual model; completion id |
| `gen_ai.conversation.id` | Conversation/session/thread id |
| `enduser.id` / `user.id` (+ fallbacks above) | End user |
| `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` / cache token fields | Token usage |
| `gen_ai.prompt_price` / `gen_ai.response_price` / `gen_ai.read_cache_price` / `gen_ai.write_cache_price` | Cost (USD) |
| `gen_ai.response.finish_reasons` | Stop reason (`~ 'length'` = truncated; `~ 'tool_'` = tool call — values seen: `tool_call`/`tool_calls`/`tool_use`) |
| `gen_ai.input.messages` / `gen_ai.output.messages` (current) or `gen_ai.prompt.<n>` / `gen_ai.completion.<n>` (indexed) | Conversation transcript — read via the **Reading conversations (content questions)** queries, don't grep raw |
| `gen_ai.tool.{name,call.arguments,call.result}` / `gen_ai.tool.definitions` | Tools executed / advertised |
| `gen_ai.{target}.evaluations.{type}.{score,label}`, `…evaluations.custom.{0..9}.{…}`, `guardrails.triggered` | Eval/guardrail results (see below) |
**Cost (USD):** total = `firstNonNull(gen_ai.prompt_price,0)+firstNonNull(gen_ai.response_price,0)`;
cache: `gen_ai.read_cache_price`/`gen_ai.write_cache_price`. Cache token accounting differs by
provider: OpenAI-style `cache_read` ⊂ `input_tokens`; Anthropic-style input/cache_read/
cache_creation are disjoint. Custom per-model pricing is team-wide, new data only — read it via
`cx ai-center model-pricing get` (price tags already reflect it).
---
## Evaluations, Guardrails & Policies
A **policy** is a configurable check. The same policy can act as an **evaluation** (post-hoc
scoring/flagging) and/or a **guardrail** (real-time block). UI evaluation categories:
Hallucinations, Security, Toxicity, Topics, User experience, Compliance, plus Custom (typed
Quality or Security).
**Evaluations** run on spans and write results back as tags. The target is **`prompt` or
`response`** (the backend supports only these two — ignore the unused `conversation` enum value):
- Built-in: `gen_ai.{prompt|response}.evaluations.{eval_type}.{score|label|version|details}`.
**`label == 'p1'` marks a flagged issue.**
- Custom: `gen_ai.{target}.evaluations.custom.{0..9}.{name,target,category,triggered,score,label}`
(`triggered` is the string `"true"`/`"false"`).
- Eval types — Hallucinations: `hallucination_context_adherence|context_relevance|completeness|correctness|task_adherence`, `sql_hallucination`;
Security: `prompt_injection`, `pii`, `sql_read_only|sql_load|sql_restricted_tables|sql_allowed_tables`;
Toxicity: `toxicity`, `sexism`; Topics: `restricted_topics`, `allowed_topics`, `competition`;
User experience: `language_mismatch`.
- **Issue class:** Security = `prompt_injection`, `pii`, `sql_*`; everything else = Quality.
**Guardrails** act in real time via the `cx-guardrails` SDK and **block** on violation. Each
invocation is a span (`otel.library.name == 'cx_guardrails.client'`) with
`tags['guardrails.triggered'] == 'true'` when it triggered; `$l.operationName` starts with
`guardrails.prompt`/`guardrails.response`. Prebuilt: Prompt Injection, PII, Toxicity — plus
**custom** guardrails.
**Configuration is not telemetry.** Which policies exist, which are enabled per app, the
inventory, guarded status (`guardrailsIntegrated`), coverage, and the team's **custom model
pricing** (`cx ai-center model-pricing get`) come from the backend — use the `cx ai-center`
commands, not span queries.
---
## DataPrime query library (Q1–Q15, runnable)
Q1–Q15 are the queries behind the AI Center UI widgets — here to **show the agent how each
metric is computed, not as mandatory copy-paste**. If the user asks something the UI already
answers, use the matching Q as-is; if it's close, take the relevant fields and adjust; if it's
new, compose your own using these as reference. Keep the GenAI filter, set the time range, and
scope to one app with `| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'`
(drop for org-wide).
**Q1 — Key insights (batched)** — Models Used, Time to Response, Token Usage, Estimated Cost,
Errors, Guardrail Actions, AI Spans, Unique Users:
```dataprime
source spans
| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'
| filter ((tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))) || tags['otel.library.name']:string == 'cx_guardrails.client'
| create cost from firstNonNull(tags['gen_ai.prompt_price']:number,0) + firstNonNull(tags['gen_ai.response_price']:number,0)
| create tokens from firstNonNull(tags['gen_ai.usage.input_tokens']:number,0) + firstNonNull(tags['gen_ai.usage.output_tokens']:number,0)
| create user from firstNonNull(tags['enduser.id'], tags['user.id'], tags['gen_ai.request.user'], tags['traceloop.association.properties.user_id'], tags['langsmith.metadata.user_id'])
| aggregate count() as ai_spans, avg(duration) as ttr_avg, sum(tokens) as token_usage,
sum(cost) as estimated_cost, count_if(tags['otel.status_code']:string == 'ERROR') as errors,
count_if(tags['guardrails.triggered']:string.toLowerCase() == 'true') as guardrail_actions,
distinct_count(user) as unique_users,
collect(firstNonNull(tags['gen_ai.request.model']:string, ''), distinct=true) as models_used
```
> Q1 is a **batched** query computing many metrics at once. When the user asks for **one**
> metric, run a minimal query with only that aggregation (and the `create` lines it needs). TTR only → `… | aggregate avg(duration) as ttr_avg`
> (or `percentile(0.95, duration)`; selector 0.5/0.75/0.9/0.95/0.99). Issue Rate = Q9.
**Q2 — Response time (avg + percentiles):**
```dataprime
source spans
| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'
| filter (tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))
| aggregate avg(duration) as avg, percentile(0.75, duration) as p75, percentile(0.90, duration) as p90, percentile(0.95, duration) as p95, percentile(0.99, duration) as p99
```
For a response-time **trend over time**, add `| groupby roundTime($m.timestamp, <interval>ms) as Time` before the aggregate and `| orderby Time` after.
**Q3 — Latency by model:** as Q2 but `groupby firstNonNull(tags['gen_ai.request.model']:string,'unknown') as model`, `orderby avg desc`.
**Q4 — Top slowest apps/spans:** groupby `$l.applicationName, $l.subsystemName` (apps) or `$l.operationName` (spans), `aggregate avg(duration) as avg | orderby avg desc | limit 5`.
**Q5 — Cost & tokens (totals):**
```dataprime
source spans
| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'
| filter (tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))
| create input_tokens from firstNonNull(tags['gen_ai.usage.input_tokens']:number,0)
| create output_tokens from firstNonNull(tags['gen_ai.usage.output_tokens']:number,0)
| create cache_read from firstNonNull(tags['gen_ai.usage.cache_read.input_tokens']:number, tags['gen_ai.usage.cache_read_input_tokens']:number, 0)
| create cache_creation from firstNonNull(tags['gen_ai.usage.cache_creation.input_tokens']:number, tags['gen_ai.usage.cache_creation_input_tokens']:number, 0)
| create cost from firstNonNull(tags['gen_ai.prompt_price']:number,0) + firstNonNull(tags['gen_ai.response_price']:number,0)
| aggregate sum(cost) as cost, sum(input_tokens) as inputTokens, sum(output_tokens) as outputTokens, sum(cache_read) as cacheReadTokens, sum(cache_creation) as cacheCreationTokens
```
Cache Hit Rate = `sum(cacheReadTokens)/sum(inputTokens)`. For a cost/tokens **trend over time**, add `| groupby roundTime($m.timestamp, <interval>ms) as Time` before the aggregate and `| orderby Time` after.
**Q6 — Cost by model:** as Q5 but `groupby firstNonNull(tags['gen_ai.request.model']:string,'unknown') as model | orderby cost desc`.
**Q7 — Most expensive apps / high-spending users:** as Q5 but `groupby $l.applicationName,$l.subsystemName` (apps) or the `user` expr, `aggregate sum(cost) as cost, sum(input_tokens+output_tokens) as tokens | orderby cost desc | limit 5`.
**Q8 — Errors (count) / top errored:**
```dataprime
source spans
| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'
| filter (tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))
| aggregate count() as total, count_if(tags['otel.status_code']:string == 'ERROR') as errors
```
For an errors **trend over time**, add `| groupby roundTime($m.timestamp, <interval>ms) as Time` before the aggregate and `| orderby Time` after. Top errored apps/spans: `groupby $l.applicationName,$l.subsystemName` (or `$l.operationName`) `aggregate count_if(tags['otel.status_code']:string=='ERROR') as errors | orderby errors desc | limit 5`.
**Q9 — Issue rate (trace-level).** An issue on a target = any built-in eval `label == 'p1'`
OR any custom eval `custom.{0..9}` with `triggered == 'true'` OR a guardrail that triggered on
that target. Repeat the `{eval}` term per enabled eval type and the `custom.{n}` term per
configured index (for AI-SPM use only security types):
```dataprime
source spans
| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'
| filter ((tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))) || tags['otel.library.name']:string == 'cx_guardrails.client'
| create prompt_issue from (tags['gen_ai.prompt.evaluations.{eval}.label']:string == 'p1' || tags['gen_ai.prompt.evaluations.custom.0.triggered']:string.toLowerCase() == 'true' || ($l.operationName.startsWith('guardrails.prompt') && tags['guardrails.triggered']:string.toLowerCase() == 'true'))
| create response_issue from (tags['gen_ai.response.evaluations.{eval}.label']:string == 'p1' || tags['gen_ai.response.evaluations.custom.0.triggered']:string.toLowerCase() == 'true' || ($l.operationName.startsWith('guardrails.response') && tags['guardrails.triggered']:string.toLowerCase() == 'true'))
| groupby traceID aggregate max(if(prompt_issue,1,0)) as p, max(if(response_issue,1,0)) as r
| aggregate count() as traces, sum(p) as prompt_issue_traces, sum(r) as response_issue_traces
| create prompt_issue_rate from prompt_issue_traces*100.0/traces
| create response_issue_rate from response_issue_traces*100.0/traces
```
Issue Distribution / Top Apps With Issues = the same detection grouped by eval **category**
(Security = `prompt_injection`/`pii`/`sql_*`; else Quality) or by `$l.applicationName`.
**Q10 — Tool calls:**
```dataprime
source spans
| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'
| filter (tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))
| create is_tool from (tags['gen_ai.operation.name']:string == 'execute_tool' || tags['gen_ai.response.finish_reasons']:string ~ 'tool_')
| filter is_tool
| extract tags['gen_ai.output.messages']:string into msg_tool using regexp(e=/"type"\s*:\s*"tool_call".*?"name"\s*:\s*"(?<name>[^"]+)"/)
| create tool from firstNonNull(tags['gen_ai.tool.name']:string, msg_tool.name, 'unknown')
| groupby tool aggregate count() as uses | orderby uses desc
```
Calls-per-interaction: `groupby traceID aggregate count_if(is_tool) as n | groupby n aggregate count() as cnt`.
Tool usage %: `aggregate distinct_count(traceID) as total, distinct_count_if(is_tool, traceID) as with_tools`.
**Q11 — Read interactions / AI Explorer:** use the **Reading conversations (content questions)**
queries (current-convention `jsonobject`, indexed-convention `$d:string` + `multi_regexp`) to get
the clean user+assistant transcript; add tokens/cost/user/duration tags alongside if needed.
**Q12 — Session walkthrough:** `filter traceID == '<traceID>'` then read ordered turns.
**Q13 — Who's using / sessions & user insights:**
```dataprime
source spans
| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'
| filter (tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))
| create user from firstNonNull(tags['enduser.id'], tags['user.id'], tags['gen_ai.request.user'], tags['traceloop.association.properties.user_id'], tags['langsmith.metadata.user_id'])
| filter user != null
| groupby user aggregate count() as interactions, distinct_count(tags['gen_ai.conversation.id']) as sessions,
sum(firstNonNull(tags['gen_ai.prompt_price']:number,0)+firstNonNull(tags['gen_ai.response_price']:number,0)) as cost
| orderby interactions desc | limit 5
```
High-Activity = orderby interactions; High-Spend = orderby cost; Risky = add a security-eval
`count_if(...label=='p1')` and orderby that.
**Q14 — Agentic workflow walkthrough:**
```dataprime
source spans
| filter traceID:string == '<traceID>'
| filter (tags['gen_ai.system']:string != null || tags['gen_ai.provider.name']:string != null || tags['gen_ai.operation.name']:string != null) && !(['cursor-agent','codex_cli_rs','codex-app-server','github-copilot','gemini-cli'].arrayContains($l.serviceName))
| choose $m.timestamp as ts, tags['gen_ai.operation.name']:string as step,
tags['gen_ai.agent.name']:string as agent, tags['gen_ai.workflow.name']:string as workflow,
tags['gen_ai.tool.name']:string as tool, tags['gen_ai.tool.call.arguments']:string as tool_args,
tags['gen_ai.tool.call.result']:string as tool_result, tags['gen_ai.request.model']:string as model,
tags['gen_ai.input.messages']:string as input_messages, tags['gen_ai.output.messages']:string as output_messages
| orderby ts asc | limit 200
```
**Q15 — Evaluation score distribution / trend** (score is a 0–1 float, separate from `p1`):
```dataprime
source spans
| filter $l.applicationName == '<APP>' && $l.subsystemName == '<SUB>'
| filter tags['gen_ai.response.evaluations.{eval}.score']:number != null
| create bucket from round(tags['gen_ai.response.evaluations.{eval}.score']:number, 1)
| groupby bucket aggregate count() as n | orderby bucket
```
Trend: `groupby roundTime($m.timestamp, <interval>ms) as Time aggregate avg(tags['gen_ai.response.evaluations.{eval}.score']:number) as avg_score`. Swap `response`→`prompt` for the other target.
---
## Phase 1 — interaction intelligence (read the content)
**Method for sentiment / satisfaction / frustration:**
1. Take a **capped sample** for the app (see *Cap the volume first* — ≤150, 100 default), don't read them all. State matched-vs-read.
2. **Read the conversations — don't keyword-search for emotion.** Use the **Reading conversations
(content questions)** queries (current or indexed convention) to get clean user+assistant
transcripts, group by session (`gen_ai.conversation.id`, fall back to `traceID`), and read each
session's turns in order.
3. **Judge from the USER's side.** Read user turns: satisfied (got their answer, "thanks",
no re-asks) vs not (re-asks/rephrasing, rising turns, negative wording, excessive `?`/`!`/
CAPS, "that's not what I asked", abandonment). Response quality / `finish_reason` / evals are
secondary — never the headline. Don't pivot a satisfaction question into "the app is broken".
**Answer shape:** summarize the split, then back it with a few **verbatim user-message quotes +
their `traceID`** (satisfied and dissatisfied) — an example is the user's own words, not a
description of the agent's reply. Don't dump raw ID lists.
- **Topic analysis** ("what do users ask about?") is an LLM-judgment task — read the **user turns of your capped sample**
(see *Cap the volume first*) and cluster the asks yourself; report themes + how many you read. Don't bucket with
keyword filters.
- **Counting a specific term** ("how often do users mention RUM?") is narrower — decide if they
want a count, list, or examples. Match **user turns** at the **word level** (a bare `~ 'rum'`
also hits *forum/premium*) — extract them first with the **Reading conversations (content questions)** queries — and
**never grep the whole `gen_ai.input.messages` blob (or the indexed `prompt.<n>` tags)**: they
carry the system prompt + tool definitions, so they match nearly everything. Report counts as a
floor.
- **Quality / hallucination:** optionally pre-filter `…evaluations.{name}.label=='p1'`, but
confirm by reading the messages.
- **Root cause:** `otel.status_code=='ERROR'` (+ `error.type`), then read messages + errored
child spans; watch truncation, tool failures, retrieval misses. **Pull the whole trace
(`filter $d.traceID == '<id>'`), not just the GenAI spans** — the cause often lives in a
sibling **non-GenAI** span (a DB call, HTTP request, retrieval step).
- **Agent behaving off-task / deviating from its instructions:** read the **system prompt**
(`gen_ai.system_instructions` and/or the `role:'system'` turn) alongside the user/assistant
turns, and judge whether the responses stayed within the task the system prompt defines.
Cite the `traceID`s where it drifted.
---
## Examples (question → approach)
> When a question names an app/agent, **run `cx ai-center applications list` first** to resolve
> the exact `applicationName`/`subsystemName` pair before scoping a query.
- *"Which of my agents / LLM apps do I have?"* → `cx ai-center applications list`.
- *"Average time to response for Financial Advisor last 24h?"* → list apps to get the exact
pair, then **Q1** (`ttr_avg`) scoped to it, `--start now-1d` (or **Q2** for the trend;
`percentile(0.95, duration)` for P95).
- *"Token spend by model this week?"* → **Q6**, `--start now-7d`.
- *"Which apps have the highest error rate?"* → **Q8** top variant, no app scope.
- *"Which region / agent is the most active?"* → GenAI filter, no app scope, `groupby
$l.applicationName, $l.subsystemName aggregate count() as spans | orderby spans desc`.
- *"Which agent deviates from its intended task?"* → read each agent's **system prompt** +
user/assistant turns and judge drift (see the *Agent behaving off-task* bullet in Phase 1);
cite traceIDs.
- *"Compare agent X now vs a week ago"* / *"compare agent X to agent Y"* → run the relevant Q
twice with different `--start`/`--end` windows, or grouped by `$l.applicationName,$l.subsystemName`,
and diff the results (e.g. cost, error rate, latency, or the system prompt via a trace from each).
- *"Are users frustrated in app X?"* → Phase 1 frustration playbook (**Q11** to read the
conversations; quote user turns + traceIDs).
- *"Which apps lack guardrails?"* → `cx ai-center applications list -o json | jq '[.[]|select(.guardrailsIntegrated==false)]'` (config).
- *"What policies are configured for app X?"* → `cx ai-center evaluations list --application <app> --subsystem <sub>`.
## Troubleshooting
- **No rows for an app** — the name may be in `subsystemName` not `applicationName`; try the
other field or `cx search-fields "<name>" --dataset spans`; widen the time range.
- **Empty `gen_ai.*.messages`** — the app isn't capturing content (opt-in); say so.
- **No eval/guardrail tags** — those policies aren't enabled; still answer content questions by
reading messages.
- **Guarded status / configured policies / inventory / coverage** — configuration; use the
`cx ai-center` commands, not spans.
- **AI Security Posture Score** — backend-computed; no CLI command or span query returns it.
Say so rather than trying to derive it.
- **AI App Discovery** (GitHub repo scan) — not reachable from the CLI (the scan service only
accepts user-session auth). Explain it exists; don't attempt to fetch it.
references/dataprime-reference.md
# DataPrime Query Language Reference
## Query Structure
A DataPrime query is a pipeline of commands separated by `|`. Each command transforms the output of the previous one:
```dataprime
filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors
```
### Source Handling
Every query targets a **source** (`logs`, `spans`, etc.). The source is set by whichever `cx` command you use. A full query with an explicit source looks like:
```dataprime
source <logs|spans> | filter ... | groupby ...
```
When running via a source-specific command (e.g. `cx logs`, `cx spans`), the source is injected automatically - omit it from the query. When running via `cx dataprime query`, use the `--source` flag or include `source` in the query itself.
The examples below focus on the DataPrime query language and omit the source and CLI command prefix.
### Comments
Comments are supported with `#` or `//`:
```dataprime
filter $m.severity == ERROR # only errors
| limit 10 // cap results
```
## Data Prefixes
All fields are accessed through three namespaces:
| Prefix | Description | Examples |
|--------|-------------|----------|
| `$m` | Metadata (system-managed) | `$m.timestamp`, `$m.severity`, `$m.duration` |
| `$l` | Labels (indexed key-value pairs) | `$l.applicationname`, `$l.subsystemname`, `$l.serviceName` |
| `$d` | User data (application payload) | `$d.message`, `$d.user_id`, `$d.traceID` |
`$d` is the default prefix and can sometimes be omitted, but being explicit avoids ambiguity.
## Data Types
| Type | Description | Example |
|------|-------------|---------|
| `string` | Text, enclosed in **single quotes** | `'some_text'` |
| `number` | Numeric value | `123`, `3.14` |
| `boolean` | True or false | `true`, `false` |
| `timestamp` | Date and time (nanoseconds since epoch) | `1714636800000000000` |
| `interval` | Time duration | `1h`, `1d`, `1w` |
| `array` | List of values | `[1, 2, 3]` |
| `object` | Key-value pairs | `{"name": "John"}` |
| `null` | Missing value or key | `null` |
## Commands
### Filtering and Selection
| Command | Description | Example |
|---------|-------------|---------|
| `filter` | Keep rows matching a condition | `filter $m.severity == ERROR` |
| `choose` | Select specific fields | `choose $m.timestamp, $d.message` |
| `limit` | Cap the number of results | `limit 10` |
| `wildfind` | Token match across the whole record (see note below) | `wildfind 'connection refused'` |
| `lucene` | Filter using Lucene syntax (`field:value`, field names relative to `$d`, combine with `AND`/`OR`/parens) | `lucene 'field:"value"'` |
> **Note on `wildfind`:** It is a standalone command, not a condition within `filter`. You cannot combine it with other filter expressions - use it as its own pipeline stage. `wildfind` with very short search terms (only a few characters) is much slower — prefer longer, more specific terms.
### Aggregation
| Command | Description | Example |
|---------|-------------|---------|
| `groupby` | Group rows and apply aggregations | `groupby $l.subsystemname aggregate count() as n` |
| `multigroupby` | Group by multiple field sets | `multigroupby a, b aggregate count()` |
| `count` | Count all rows | `count` |
| `countby` | Count rows grouped by a field | `countby $l.applicationname` |
| `distinct` | Return unique values of a field | `distinct $l.subsystemname` |
### Transformation
| Command | Description | Example |
|---------|-------------|---------|
| `create` | Add a computed field | `create latency_ms from $m.duration / 1000` |
| `orderby` | Sort results | `orderby $d.timestamp desc` |
| `extract` | Parse fields with regex or JSON | See [Text Extraction](#text-extraction) |
| `dedupeby` | Remove duplicates by a field (cost grows with number of distinct keys) | `dedupeby $m.templateid` |
## Operators
| Operator | Description | Example |
|----------|-------------|---------|
| `==` | Equals | `filter $m.severity == ERROR` |
| `!=` | Not equals | `filter $l.subsystemname != 'test'` |
| `>`, `<`, `>=`, `<=` | Comparison | `filter $d.response_time > 1000` |
| `~` | Case-insensitive token match (matches whole tokens, not arbitrary substrings) | `filter $d.message ~ 'timeout'` |
| `&&` | AND | `filter $m.severity == ERROR && $l.applicationname == 'api'` |
| `\|\|` | OR | `filter $m.severity == ERROR \|\| $m.severity == CRITICAL` |
| `!= null` | Field exists | `filter $d.some_field != null` |
> **`~` vs `contains()`:** `~` is a **case-insensitive, token-based** match — it matches whole tokens (words), so `~ 'timeout'` also matches `TIMEOUT` and `Timeout`. `contains()` is a **case-sensitive raw substring** match — `contains('time')` matches inside `timeout`, but only in that exact case. Use `~` for word/term search; use `contains()` when you need an exact-case partial-string match.
## Type Conversions
Cast fields inline with `:type`:
```dataprime
filter $d.http_error_code:number == 500
```
Supported types: `bool`, `number`, `string`, `timestamp`, `interval`, `array`, `object`
## Field Access
```dataprime
# Chained field names (dot notation)
filter $d.tags.user_context.email == 'test@example.com'
# Special characters require brackets
filter $d.http['status/code'] == 500
```
## Aggregation Functions
| Function | Description |
|----------|-------------|
| `count()` | Count rows |
| `sum($field)` | Sum values |
| `avg($field)` | Average |
| `min($field)` | Minimum |
| `max($field)` | Maximum |
| `percentile(0.95, $field)` | Percentile |
| `median($field)` | Median value |
| `stddev($field)` | Standard deviation |
| `variance($field)` | Variance |
| `distinct_count($field)` | Count unique values (exact) |
| `approx_count_distinct($field)` | Approximate count of unique values |
| `any_value($field)` | Random sample value |
| `collect($field)` | Collect values into an array |
> **`distinct_count` vs `approx_count_distinct`:** Exact `distinct_count` can be slow or run out of memory on high-cardinality fields. When an exact count isn't required, use `approx_count_distinct` instead.
Example - full CLI invocation:
```bash
cx dataprime query --source logs 'groupby $l.subsystemname aggregate count() as error_count, avg($d.response_time) as avg_response | orderby error_count desc'
```
## Utility Functions
### firstNonNull - Field Coalescing
Return the first non-null value from a list of fields. Useful when the same data may appear in different fields across log sources:
```dataprime
# Merge fields
create message from firstNonNull($d.error_message, $d.msg, $d.body)
# Use inside groupby
groupby firstNonNull($d.error_message, $d.msg) as message aggregate count() as n
```
### Template Sampling
Find top error patterns with a sample message for each:
```dataprime
filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5
```
## Time-Based Grouping
Use `roundTime()` to bucket timestamps:
```dataprime
# Group by hour
groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count
# Error rate over 15-minute intervals
filter $m.severity == ERROR | groupby roundTime($m.timestamp, 15m) as interval aggregate count() as errors
```
## Multi-Value Matching
Use `arrayContains` to match against a set of values:
```dataprime
# Match multiple subsystems
filter ['api', 'web', 'worker'].arrayContains($l.subsystemname)
# Match multiple severity levels
filter [ERROR, CRITICAL].arrayContains($m.severity)
```
## Text Extraction
### Regex Extraction
```dataprime
# Extract with unnamed capture group
extract $d.email into domain using regexp(e=/@(.*)/) | distinct $d.domain._0
# Named capture groups
extract $d.email into extracted using regexp(e=/(?<username>[a-zA-Z0-9._%+-]+)@(?<domain>.*)/) | choose $d.extracted.username, $d.extracted.domain
```
### JSON String Parsing
```dataprime
# Parse a JSON string field into an object for further querying
extract $d.json_payload into parsed using jsonobject() | filter $d.parsed.status == 'failed'
```
## Deduplication
```dataprime
# Remove duplicates by log template
dedupeby $m.templateid
# Keep one row per key (cost grows with the number of distinct keys)
dedupeby $d.session_id
```
> **Performance:** `dedupeby` cost scales with the number of **distinct keys** it tracks, so it gets slow and memory-heavy on high-cardinality or near-unique keys (e.g. a request id). Cardinality depends on your data and time window — a key like session or trace id can still be large. Keep `dedupeby` when you genuinely need one representative row per key; to cut cost, narrow the input first (tighter `filter`, smaller time window) rather than swapping in `filter`/`limit` (which does **not** keep one row per key) or an aggregation (which changes the row shape) unless that different output is acceptable. The `dedupeby <key> orderby ...` form (keep the latest row per key) is heavier still.
## Built-In Documentation
For the full list of commands and functions with detailed syntax:
```bash
cx dataprime list # List all commands and functions
cx dataprime list --filter commands # Commands only
cx dataprime list --filter functions --name time # Search functions by name
cx dataprime show filter # Detailed help for a specific command
cx dataprime show groupby
```
## Validating a DataPrime query
A query that looks right can still fail on a typoed field path, an invented function, or a malformed pipeline stage. Validate before trusting the output — a short-window run through the CLI is cheap and catches almost all of these:
```bash
cx logs '<pipeline>' --start now-15m --end now --limit 1
cx spans '<pipeline>' --start now-15m --end now --limit 1
```
`now-15m` is a good default; widen it only if 15 minutes is unlikely to exercise the pipeline. Per "Source Handling" above, omit any leading `source logs` / `source spans` — `cx logs` and `cx spans` inject the source themselves.
Check both the exit code and the output — some errors surface only in the output.
**Pass** = exit 0 and the output is rows or `[]` with no error or warning lines.
**Hard fail** — query is broken, fix it:
- non-zero exit
- `error from profile '...': API request failed` — HTTP error from the API
- `Compilation errors:` — parse error, unknown function, malformed expression
**Soft fail** (needs investigation):
- `keypath does not exist` — the query parsed, but no record in the window had the referenced field. This is ambiguous: the field name might be a typo, or it might be real but absent from records in this 15-minute slice. Confirm with `cx search-fields "<field hint>" --dataset logs` (or `--dataset spans`). If the field is real, the query is fine — try a wider window or accept the empty result. If it isn't, fix the field name.
On fail: re-discover fields with `cx search-fields`, look up command syntax with `cx dataprime show <command>`, fix, re-run.
SKILL.md
---
name: cx-ai-center
description: >
Use this skill for any question or action about the user's AI/GenAI applications or agents —
their behavior, prompts/responses, quality, hallucinations, guardrails, security, cost/tokens,
errors, evaluations/policies, model pricing, or configuration — including comparing or tracking
agents over time. It covers both analyzing AI telemetry (GenAI spans) and managing AI Center
config via the `cx ai-center` commands.
metadata:
version: "0.1.0"
---
# AI Center Skill
**This is the tool for anything about AI/GenAI applications** — both **questions** about their
behavior (prompts/responses, quality, hallucinations, guardrails, security, cost/tokens, errors,
latency — everything AI apps expose through their GenAI spans/tags) and **actions** to manage them
(applications, evaluations/policies, policy↔app links, model pricing). If a request touches an AI
application or its GenAI telemetry, use this skill.
Coralogix **AI Center** observes, evaluates, and guards GenAI/LLM applications. This skill
answers questions about AI apps from two sources:
- **Configuration** (this skill's `cx ai-center` commands): the AI application inventory,
configured evaluations/policies, coverage, custom evaluations, and model pricing — none of
which live in span telemetry.
- **Telemetry** (GenAI spans): what users asked, how the model answered, cost, tokens,
latency, errors, tool calls, and eval/guardrail verdicts — queried with **`cx spans '<DataPrime>'`**.
See [references/ai-center-queries.md](references/ai-center-queries.md) for the full,
runnable query library, span schema, and playbooks.
Match the source to the question: *"which apps lack guardrails"* → config
(`cx ai-center applications list`); *"what are users asking my chatbot"* → telemetry
(`cx spans '…'`, reading the conversation from the GenAI spans). Some questions need **both** —
e.g. *"is my chatbot's PII policy actually catching PII?"* joins config (is the policy enabled)
with telemetry (the PII verdicts + the messages).
---
## Destructive Operation Safety
All write operations (`create`, `update`, `delete`, `add`, `remove`, `set`)
require interactive confirmation. `ai-center` is a **risky** command, so writes are also
gated by `allow_risky_commands` in `~/.cx/config.toml`. To skip the prompt in scripts, pass
`--yes`.
**IMPORTANT: NEVER pass `--yes` without explicit user approval.** Before executing any write:
1. Describe the exact operation to the user (what will be created/modified/deleted/linked).
2. Wait for the user to confirm.
3. Only then execute with `--yes`.
Read operations (`list`, `get`, `coverage`, `list-for-application`, `model-pricing get`) do
not require confirmation and can be run freely.
### Read-Only Mode
Use `--read-only` (or `CX_READ_ONLY=1`) to block every write at the CLI level — safe for
exploration.
### Agent Mode
When running inside an AI agent (Claude Code, Cursor, Codex, …), cx detects it and — instead of
showing a confirmation prompt that would hang forever (no human is there to type y/n) — stops
immediately with an error telling you to get the user's approval, then re-run with `--yes`.
### No delete commands (by design)
The CLI intentionally exposes **no delete** for custom-evaluation policies, AI applications, or
model pricing — even though the AI v3 API has those delete endpoints, `cx ai-center` does not
surface them.
- **Custom-evaluation policy:** can't be deleted; to take it off an app, detach with
`custom-evaluations remove` (the policy object survives and can be re-attached).
- **Model pricing:** no delete command. It's **team-wide** (not per-app), so to change or clear
it, run `model-pricing set` with a new map (an empty map `{}` clears all overrides) — `set`
replaces the whole set.
---
## Golden rule
For **content** questions (quality, hallucination, sentiment, topics) read the actual
conversation and cite the `traceID` — don't rely on verdict tags alone. The transcript lives in
one of two conventions (`gen_ai.input.messages`/`output.messages`, or the older indexed
`gen_ai.prompt.<n>`/`completion.<n>` tags); read it with the **Reading conversations (content
questions)** queries in the library, which handle both and exclude the system prompt and tool
traffic. Full guidance:
[references/ai-center-queries.md](references/ai-center-queries.md).
---
## CLI Commands
**Show names to the user; use UUIDs only internally.** When presenting results, refer to apps
and evaluations by their human names (application/subsystem, evaluation name), not raw UUIDs.
The UUID is only needed to *call* a by-id or write command — resolve it yourself from the
matching `list` command (never guess or make the user paste a UUID).
### Applications (inventory + guarded status)
| Command | Purpose |
|---------|---------|
| `cx ai-center applications list` | List AI apps incl. `guardrailsIntegrated` (guarded) status |
| `cx ai-center applications list --evaluation-type <TYPE>` | Filter to apps using an eval type (repeatable) |
| `cx ai-center applications list --page-size <N> --page-offset <N>` | Paginate |
| `cx ai-center applications get <application-id>` | One application by UUID |
### Evaluations (configured policies on apps)
| Command | Purpose |
|---------|---------|
| `cx ai-center evaluations list` | All configured evaluations |
| `cx ai-center evaluations list --application <app> --subsystem <sub>` | Scope to one app (the pair) |
| `cx ai-center evaluations list --evaluation-type <TYPE>` | Filter by type — `<TYPE>` is the API enum (e.g. `PII`, `TOXICITY`, `PROMPT_INJECTION`; the keys from `coverage`), **not** the lowercase form |
| `cx ai-center evaluations get <evaluation-id>` | One evaluation by UUID |
| `cx ai-center evaluations create --from-file eval.json` | Create/enable an evaluation *(write)* |
| `cx ai-center evaluations update <evaluation-id> --from-file patch.json` | Partial update *(write)* |
| `cx ai-center evaluations delete <evaluation-id>` | Remove an evaluation from its app *(write)* |
### Custom evaluations (policies) & application links
| Command | Purpose |
|---------|---------|
| `cx ai-center custom-evaluations list` | All custom evaluation policies |
| `cx ai-center custom-evaluations list-for-application <application-id>` | Policies linked to one app |
| `cx ai-center custom-evaluations create --from-file policy.json` | Create a custom policy *(write)* |
| `cx ai-center custom-evaluations update <id> --from-file patch.json` | Partial update *(write)* |
| `cx ai-center custom-evaluations add <evaluation-id> <application-id>` | Attach a policy to an app *(write)* |
| `cx ai-center custom-evaluations remove <evaluation-id> <application-id>` | Detach (reversible) *(write)* |
> **By-id is prebuilt-only.** `evaluations get <id>` fetches a **prebuilt/configured** evaluation.
> Custom policies have **no** get-by-id — find one via `custom-evaluations list` /
> `list-for-application` and match by `id`/name.
### Coverage & model pricing
| Command | Purpose |
|---------|---------|
| `cx ai-center coverage` | Map of each evaluation type → number of apps using it (coverage / gap analysis) |
| `cx ai-center model-pricing get` | Team's custom per-model pricing overrides |
| `cx ai-center model-pricing set --from-file prices.json` | Set team pricing (team-wide, new data only) *(write)* |
The `--from-file` bodies for `evaluations` and `custom-evaluations` match the AI v3 API
shape verbatim; use `-` to read JSON from stdin. For `evaluations create`, `target` is
**required** and must be uppercase (`PROMPT` or `RESPONSE`); for `custom-evaluations create`,
`name`, `instructions`, and `policyType` are required. **Exception:** `model-pricing set` takes just
the raw `model→price` map — cx wraps it as `{"prices": …}` for you, so do **not** include the
outer `prices` envelope. Each model maps to a price object; all four fields are optional doubles
(USD per **one million** tokens), omit the ones that don't apply:
```json
{
"gpt-4o": {
"inputPricePerMillionTokens": 2.5,
"outputPricePerMillionTokens": 10,
"cacheReadPricePerMillionTokens": 1.25,
"cacheWritePricePerMillionTokens": 3.75
}
}
```
An empty map `{}` clears all overrides (set replaces the whole set — it's team-wide, new data only).
`model-pricing get` returns the wrapper `{ "pricing": { "id", "companyId", "prices": { … } } }` — the
per-model overrides live under `prices` (empty when none are set).
---
## Common workflows
### Inventory & guardrail gaps
```bash
# Which apps are NOT guarded?
cx ai-center applications list -o json | jq '[.[] | select(.guardrailsIntegrated==false)]'
```
### Enable a policy on an app (write — confirm first!)
```bash
# 1. Describe to the user; 2. get approval; 3. then:
cx ai-center evaluations create --from-file eval.json --yes
# eval.json: { "application": "...", "subsystem": "...", "target": "PROMPT", "config": { "<type>": {...} }, "isEnabled": true }
# `target` is REQUIRED and must be UPPERCASE — "PROMPT" or "RESPONSE" (the API rejects lowercase / a missing target).
```
### Read the actual conversations (telemetry, not config)
Use `cx spans` with the query library in
[references/ai-center-queries.md](references/ai-center-queries.md) — reading messages, cost,
latency, errors, tool calls, and per-user analysis.
---
## Key principles
- **Config vs. telemetry:** inventory / evaluations / policies / coverage / pricing → `cx ai-center`;
content / cost / latency / errors / verdicts → GenAI spans via `cx spans`. Don't answer one
from the other.
- **Confirm before writes.** Describe the operation, get approval, then run with `--yes`.
---
## Related Skills
- `cx-telemetry-querying` — general logs/spans/metrics/DataPrime querying (the engine behind
the `cx spans` queries used here).
- `cx-olly` — the conversational AI assistant (`cx olly ask`).