references/dataprime-reference.md
# DataPrime Query Language Reference
## Query Structure
A DataPrime query is a pipeline of commands separated by `|`. Each command transforms the output of the previous one:
```dataprime
filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors
```
### Source Handling
Every query targets a **source** (`logs`, `spans`, etc.). The source is set by whichever `cx` command you use. A full query with an explicit source looks like:
```dataprime
source <logs|spans> | filter ... | groupby ...
```
When running via a source-specific command (e.g. `cx logs`, `cx spans`), the source is injected automatically - omit it from the query. When running via `cx dataprime query`, use the `--source` flag or include `source` in the query itself.
The examples below focus on the DataPrime query language and omit the source and CLI command prefix.
### Comments
Comments are supported with `#` or `//`:
```dataprime
filter $m.severity == ERROR # only errors
| limit 10 // cap results
```
## Data Prefixes
All fields are accessed through three namespaces:
| Prefix | Description | Examples |
|--------|-------------|----------|
| `$m` | Metadata (system-managed) | `$m.timestamp`, `$m.severity`, `$m.duration` |
| `$l` | Labels (indexed key-value pairs) | `$l.applicationname`, `$l.subsystemname`, `$l.serviceName` |
| `$d` | User data (application payload) | `$d.message`, `$d.user_id`, `$d.traceID` |
`$d` is the default prefix and can sometimes be omitted, but being explicit avoids ambiguity.
## Data Types
| Type | Description | Example |
|------|-------------|---------|
| `string` | Text, enclosed in **single quotes** | `'some_text'` |
| `number` | Numeric value | `123`, `3.14` |
| `boolean` | True or false | `true`, `false` |
| `timestamp` | Date and time (nanoseconds since epoch) | `1714636800000000000` |
| `interval` | Time duration | `1h`, `1d`, `1w` |
| `array` | List of values | `[1, 2, 3]` |
| `object` | Key-value pairs | `{"name": "John"}` |
| `null` | Missing value or key | `null` |
## Commands
### Filtering and Selection
| Command | Description | Example |
|---------|-------------|---------|
| `filter` | Keep rows matching a condition | `filter $m.severity == ERROR` |
| `choose` | Select specific fields | `choose $m.timestamp, $d.message` |
| `limit` | Cap the number of results | `limit 10` |
| `wildfind` | Token match across the whole record (see note below) | `wildfind 'connection refused'` |
| `lucene` | Filter using Lucene syntax (`field:value`, field names relative to `$d`, combine with `AND`/`OR`/parens) | `lucene 'field:"value"'` |
> **Note on `wildfind`:** It is a standalone command, not a condition within `filter`. You cannot combine it with other filter expressions - use it as its own pipeline stage. `wildfind` with very short search terms (only a few characters) is much slower — prefer longer, more specific terms.
### Aggregation
| Command | Description | Example |
|---------|-------------|---------|
| `groupby` | Group rows and apply aggregations | `groupby $l.subsystemname aggregate count() as n` |
| `multigroupby` | Group by multiple field sets | `multigroupby a, b aggregate count()` |
| `count` | Count all rows | `count` |
| `countby` | Count rows grouped by a field | `countby $l.applicationname` |
| `distinct` | Return unique values of a field | `distinct $l.subsystemname` |
### Transformation
| Command | Description | Example |
|---------|-------------|---------|
| `create` | Add a computed field | `create latency_ms from $m.duration / 1000` |
| `orderby` | Sort results | `orderby $d.timestamp desc` |
| `extract` | Parse fields with regex or JSON | See [Text Extraction](#text-extraction) |
| `dedupeby` | Remove duplicates by a field (cost grows with number of distinct keys) | `dedupeby $m.templateid` |
## Operators
| Operator | Description | Example |
|----------|-------------|---------|
| `==` | Equals | `filter $m.severity == ERROR` |
| `!=` | Not equals | `filter $l.subsystemname != 'test'` |
| `>`, `<`, `>=`, `<=` | Comparison | `filter $d.response_time > 1000` |
| `~` | Case-insensitive token match (matches whole tokens, not arbitrary substrings) | `filter $d.message ~ 'timeout'` |
| `&&` | AND | `filter $m.severity == ERROR && $l.applicationname == 'api'` |
| `\|\|` | OR | `filter $m.severity == ERROR \|\| $m.severity == CRITICAL` |
| `!= null` | Field exists | `filter $d.some_field != null` |
> **`~` vs `contains()`:** `~` is a **case-insensitive, token-based** match — it matches whole tokens (words), so `~ 'timeout'` also matches `TIMEOUT` and `Timeout`. `contains()` is a **case-sensitive raw substring** match — `contains('time')` matches inside `timeout`, but only in that exact case. Use `~` for word/term search; use `contains()` when you need an exact-case partial-string match.
## Type Conversions
Cast fields inline with `:type`:
```dataprime
filter $d.http_error_code:number == 500
```
Supported types: `bool`, `number`, `string`, `timestamp`, `interval`, `array`, `object`
## Field Access
```dataprime
# Chained field names (dot notation)
filter $d.tags.user_context.email == 'test@example.com'
# Special characters require brackets
filter $d.http['status/code'] == 500
```
## Aggregation Functions
| Function | Description |
|----------|-------------|
| `count()` | Count rows |
| `sum($field)` | Sum values |
| `avg($field)` | Average |
| `min($field)` | Minimum |
| `max($field)` | Maximum |
| `percentile(0.95, $field)` | Percentile |
| `median($field)` | Median value |
| `stddev($field)` | Standard deviation |
| `variance($field)` | Variance |
| `distinct_count($field)` | Count unique values (exact) |
| `approx_count_distinct($field)` | Approximate count of unique values |
| `any_value($field)` | Random sample value |
| `collect($field)` | Collect values into an array |
> **`distinct_count` vs `approx_count_distinct`:** Exact `distinct_count` can be slow or run out of memory on high-cardinality fields. When an exact count isn't required, use `approx_count_distinct` instead.
Example - full CLI invocation:
```bash
cx dataprime query --source logs 'groupby $l.subsystemname aggregate count() as error_count, avg($d.response_time) as avg_response | orderby error_count desc'
```
## Utility Functions
### firstNonNull - Field Coalescing
Return the first non-null value from a list of fields. Useful when the same data may appear in different fields across log sources:
```dataprime
# Merge fields
create message from firstNonNull($d.error_message, $d.msg, $d.body)
# Use inside groupby
groupby firstNonNull($d.error_message, $d.msg) as message aggregate count() as n
```
### Template Sampling
Find top error patterns with a sample message for each:
```dataprime
filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5
```
## Time-Based Grouping
Use `roundTime()` to bucket timestamps:
```dataprime
# Group by hour
groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count
# Error rate over 15-minute intervals
filter $m.severity == ERROR | groupby roundTime($m.timestamp, 15m) as interval aggregate count() as errors
```
## Multi-Value Matching
Use `arrayContains` to match against a set of values:
```dataprime
# Match multiple subsystems
filter ['api', 'web', 'worker'].arrayContains($l.subsystemname)
# Match multiple severity levels
filter [ERROR, CRITICAL].arrayContains($m.severity)
```
## Text Extraction
### Regex Extraction
```dataprime
# Extract with unnamed capture group
extract $d.email into domain using regexp(e=/@(.*)/) | distinct $d.domain._0
# Named capture groups
extract $d.email into extracted using regexp(e=/(?<username>[a-zA-Z0-9._%+-]+)@(?<domain>.*)/) | choose $d.extracted.username, $d.extracted.domain
```
### JSON String Parsing
```dataprime
# Parse a JSON string field into an object for further querying
extract $d.json_payload into parsed using jsonobject() | filter $d.parsed.status == 'failed'
```
## Deduplication
```dataprime
# Remove duplicates by log template
dedupeby $m.templateid
# Keep one row per key (cost grows with the number of distinct keys)
dedupeby $d.session_id
```
> **Performance:** `dedupeby` cost scales with the number of **distinct keys** it tracks, so it gets slow and memory-heavy on high-cardinality or near-unique keys (e.g. a request id). Cardinality depends on your data and time window — a key like session or trace id can still be large. Keep `dedupeby` when you genuinely need one representative row per key; to cut cost, narrow the input first (tighter `filter`, smaller time window) rather than swapping in `filter`/`limit` (which does **not** keep one row per key) or an aggregation (which changes the row shape) unless that different output is acceptable. The `dedupeby <key> orderby ...` form (keep the latest row per key) is heavier still.
## Built-In Documentation
For the full list of commands and functions with detailed syntax:
```bash
cx dataprime list # List all commands and functions
cx dataprime list --filter commands # Commands only
cx dataprime list --filter functions --name time # Search functions by name
cx dataprime show filter # Detailed help for a specific command
cx dataprime show groupby
```
## Validating a DataPrime query
A query that looks right can still fail on a typoed field path, an invented function, or a malformed pipeline stage. Validate before trusting the output — a short-window run through the CLI is cheap and catches almost all of these:
```bash
cx logs '<pipeline>' --start now-15m --end now --limit 1
cx spans '<pipeline>' --start now-15m --end now --limit 1
```
`now-15m` is a good default; widen it only if 15 minutes is unlikely to exercise the pipeline. Per "Source Handling" above, omit any leading `source logs` / `source spans` — `cx logs` and `cx spans` inject the source themselves.
Check both the exit code and the output — some errors surface only in the output.
**Pass** = exit 0 and the output is rows or `[]` with no error or warning lines.
**Hard fail** — query is broken, fix it:
- non-zero exit
- `error from profile '...': API request failed` — HTTP error from the API
- `Compilation errors:` — parse error, unknown function, malformed expression
**Soft fail** (needs investigation):
- `keypath does not exist` — the query parsed, but no record in the window had the referenced field. This is ambiguous: the field name might be a typo, or it might be real but absent from records in this 15-minute slice. Confirm with `cx search-fields "<field hint>" --dataset logs` (or `--dataset spans`). If the field is real, the query is fine — try a wider window or accept the empty result. If it isn't, fix the field name.
On fail: re-discover fields with `cx search-fields`, look up command syntax with `cx dataprime show <command>`, fix, re-run.
references/deploy.md
# Phase 8: Deploy via `cx dashboards create`
Don't tell the user to paste JSON into the Coralogix UI - deploy the dashboard directly.
---
## 1. Pick a folder
List folders and suggest the best match:
```bash
cx dashboards folders list -o json
```
Rank the existing folders by relevance (service name, team, product area) and present the top matches with `AskQuestion`:
- "Folder X (id: `<id>`) - best match by name"
- "Folder Y (id: `<id>`)"
- "Root (no folder)"
- "None of these - I'll create a folder in the Coralogix UI first"
Default to "Root" if nothing fits.
---
## 2. If the user wants a new folder
Ask them for a folder name (and an optional parent folder id - omit for a top-level folder), then create it directly:
```bash
cx dashboards folders create --name "<Folder Name>"
# or, as a sub-folder of an existing one:
cx dashboards folders create --name "<Sub-folder>" --parent-id <parent-folder-id>
```
The command prints the new folder id. Use that id as `--folder` in step 3.
If folder creation fails (most common cause: API key missing the `team-dashboards:Update` permission), fall back to the Coralogix UI - **Dashboards → Folders → + New folder** - then rerun `cx dashboards folders list -o json` and proceed with the chosen id.
---
## 3. Save and deploy
1. Write the verified JSON to `/tmp/cx-dashboard-<slug>.json` (use the file-write tool; don't prescribe a specific shell idiom).
2. Deploy into the chosen folder (omit `--folder` for root):
```bash
cx dashboards create --from-file /tmp/cx-dashboard-<slug>.json --folder <folder-id>
```
The CLI generates the `requestId` envelope automatically and prints the created dashboard ID and name on success. Pipe into `-o json` or `-o toon` for structured output.
On failure: show the CLI error verbatim, return to Phase 5 (most common cause: a query that parses locally but the live API rejects), fix, and redeploy.
On success: continue to step 4 below — the workflow is **not finished** until the user has the link.
---
## 4. Share the link (final step — mandatory)
Don't stop at "dashboard created". The very last action is to give the user a clickable link to the dashboard.
Don't hand-build the URL. `cx dashboards create` / `cx dashboards replace` already print a `View in Coralogix: <url>` line to stderr on success once it can resolve a console link for the active profile - capture that line and reuse the URL verbatim.
If no `View in Coralogix:` line was printed (no console link could be resolved for the profile and no `console_url` override is configured - see `docs/configuration.md` § "Console links"), **omit the link entirely** — do not invent a URL. Use the second (no-link) template in `SKILL.md` § "Output format for the user", which drops the markdown link from the `Deployed` line *and* drops the standalone `Open it:` line so the user is never shown a broken URL.
Render the link as a markdown link using the dashboard **name** as the link text:
```
Dashboard: **[<Name>](<url from the View in Coralogix line>)**
```
Then emit the summary defined in the main `SKILL.md` § "Output format for the user".
---
## 5. Replace an existing dashboard
To update a dashboard that already exists (instead of creating a new one), use the replace workflow:
1. Get the current definition:
```bash
cx dashboards get <dashboard-id> -o json > dashboard.json
```
2. Edit `dashboard.json` (change widgets, queries, filters, etc.). The `id` field must remain intact.
3. Deploy the updated version:
```bash
cx dashboards replace --from-file dashboard.json --yes
```
This is a full replacement - the entire dashboard definition is overwritten. The `id` field in the JSON determines which dashboard is updated.
Use replace when:
- The user asks to update, modify, or iterate on an existing dashboard.
- You're refining a dashboard after Phase 5 verification found issues.
- The user exported a dashboard and wants to push changes back.
Use create (not replace) when:
- Building a new dashboard from scratch.
- Duplicating an existing dashboard (remove the `id` field first).
---
## 6. Idempotency note
Each `create` run generates a fresh top-level `id` (21-char nanoid), so re-running creates a *new* dashboard rather than overwriting an existing one. Use `replace` to update in place.
references/logs-querying.md
# Log Querying Reference
Query and analyze Coralogix logs using the `cx logs` command with DataPrime syntax.
> **DataPrime syntax:** See `dataprime-reference.md` for the full query language reference.
## Understanding Logs in Coralogix
Logs in Coralogix are **largely unstructured**. Every log entry has a small structured envelope - metadata and labels - but the actual application payload (`userData`) is free-form and varies entirely by application. There is no universal schema for `$d.*` fields.
This means:
- **Metadata (`$m.*`)** and **labels (`$l.*`)** are predictable - you can always filter on severity, timestamp, application name, and subsystem name without discovery.
- **User data (`$d.*`)** is not predictable - field names, nesting, and types depend on whatever the application chose to log. Always verify `$d` fields before assuming they exist.
---
## CLI Command
```bash
cx logs '<dataprime_query>'
```
The `source logs` prefix is automatically injected if the query doesn't already include a `source` command.
### Options
| Flag | Default | Description |
|------|---------|-------------|
| `--start` | `now-1h` | Start time (ISO 8601 or relative, e.g. `now-6h`) |
| `--end` | `now` | End time |
| `--limit` | `100` | Maximum number of results |
| `--tier` | `frequent` | Storage tier: `frequent` (hot/recent) or `archive` (cold/historical) |
| `-o, --output` | `text` | Output format: `text`, `json`, or `toon` |
---
## Log Data Model
### Standard Fields (Always Available)
| Field | Description |
|-------|-------------|
| `$m.timestamp` | Log timestamp |
| `$m.severity` | Severity level (see below) |
| `$m.templateid` | Log template identifier (groups structurally similar logs) |
| `$l.applicationname` | Application name - the highest-level label. All data in Coralogix is tagged with it. Meaning varies by customer (environment, team, region) but it always exists. |
| `$l.subsystemname` | Subsystem name - second highest-level label. All data is tagged with it. Typically maps to a service or component. |
| `$d.*` | User data - free-form, application-specific (see [Field Discovery](#field-discovery)) |
### Severity Values
Severity keywords are used **without quotes** in DataPrime:
`VERBOSE` | `DEBUG` | `INFO` | `WARNING` | `ERROR` | `CRITICAL`
```bash
cx logs 'filter $m.severity == ERROR'
cx logs 'filter [ERROR, CRITICAL].arrayContains($m.severity)'
```
---
## Essential Query Examples
```bash
# Filter by severity
cx logs 'filter $m.severity == ERROR'
# Text search in a known field
cx logs "filter \$d.message ~ 'timeout'"
# Filter by application and subsystem
cx logs "filter \$l.applicationname == 'api' && \$l.subsystemname == 'auth'"
# Aggregate errors by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'
# Wider time range and archive tier
cx logs "filter \$l.subsystemname == 'payments'" --tier archive --start now-7d
```
> **`~` (text match):** `~` is a **case-insensitive, token-based** match — `~ 'timeout'` also matches `TIMEOUT`/`Timeout`, and it matches whole tokens (words), not arbitrary substrings. For a **case-sensitive raw substring** match, use `contains()` instead. See the operator table in `dataprime-reference.md`.
### Wildfind Policy
`wildfind` runs the same **case-insensitive, token-based** match as `~` (whole tokens, not arbitrary substrings), but over the whole record instead of a single field — it matches tokens anywhere, in any field.
**Prefer a field-targeted `~`/`filter` when you know the field** — for **precision**: because `wildfind` matches tokens anywhere in the record, it can't be narrowed to a specific field and will match the term in fields you didn't intend.
**Performance:** `wildfind` with very short search terms (only a few characters) is much slower — prefer longer, more specific terms.
The **one exception**: when the user provides a specific, quoted error message or log string and you don't know which field contains it:
```bash
# User says: "Find logs with 'connection refused'"
cx logs "wildfind 'connection refused'"
```
In all other cases, use `filter` with known fields (`$m.severity`, `$l.subsystemname`, `$d.<field>`) or discover field names first with `cx search-fields`.
---
## Field Discovery
**Skip discovery when:**
- The query only uses standard fields (`$m.severity`, `$m.timestamp`, `$l.applicationname`, `$l.subsystemname`)
- The user explicitly names the fields they want (e.g., "filter by `$d.customer_id`")
- You're searching for a specific error message - use `wildfind` directly
- The fields have already been discovered earlier in the conversation
For customer-specific `$d.*` fields that need discovery, use one of these approaches:
### 1. Infer from Source Code (Preferred)
If you have access to the application's source code, examine logger calls, structured logging configs, and log format templates to identify field names directly.
### 2. Semantic Search
```bash
cx search-fields "customer identifier" --dataset logs
cx search-fields "http response code" --dataset logs
```
Returns DataPrime paths with similarity scores:
```
+------------------------+-----------------------------------+-----------+
| DataPrime path | Description | Similarity|
+------------------------+-----------------------------------+-----------+
| $d.customer_id | Unique customer identifier | 0.89 |
| $d.user.account_id | Customer account reference | 0.85 |
+------------------------+-----------------------------------+-----------+
```
### 3. Sample Query Inspection
```bash
cx logs "filter \$l.subsystemname == 'api'" --limit 5 -o json
```
Inspect the JSON output to see all available fields in the actual data.
---
## Investigation Workflow
### 1. Understand the Request
Identify:
- What type of logs are needed (errors, info, specific events)
- Time frame of interest
- Key entities (services, users, transactions)
### 2. Start with Standard Fields
For basic queries, use standard fields directly:
```bash
# Recent errors - no discovery needed
cx logs 'filter $m.severity == ERROR | limit 20'
# Errors in a specific subsystem
cx logs "filter \$m.severity == ERROR && \$l.subsystemname == 'payment-service'"
```
### 3. Build and Execute Query
Start simple, add complexity:
```bash
# Step 1: Check if data exists
cx logs "filter \$l.subsystemname == 'checkout'" --limit 10
# Step 2: Add filters
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR"
# Step 3: Add aggregation
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR | groupby \$d.error_type aggregate count() as occurrences"
```
### 4. Troubleshooting
If a query returns no results, change **one thing at a time**. Keep the query window as **narrow** as possible and widen it deliberately — start from the window you already have rather than jumping to a huge range:
1. **Relax filters**: remove the most restrictive condition
2. **Verify field names**: run a sample query with `-o json` to inspect the actual schema
3. **Extend the time range**: widen gradually from your current window (e.g. `now-1h` → `now-6h` → `now-24h`)
4. **Try archive tier**: for older data, add `--tier archive` and widen the window to cover the period you're after
---
## Common Query Patterns
### Error Investigation
```bash
# All errors in last hour
cx logs 'filter $m.severity == ERROR'
# Critical errors only
cx logs 'filter $m.severity == CRITICAL'
# Errors with text search
cx logs "filter \$m.severity == ERROR && \$d.message ~ 'database connection'"
```
### Aggregation by Service
```bash
# Error count by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'
# Error count by application and subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.applicationname, $l.subsystemname aggregate count() as errors'
```
### Time-Based Analysis
```bash
# Errors per hour
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count'
# Find error spikes in 5-minute windows
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 5m) as interval aggregate count() as count | orderby count desc | limit 10'
```
### Finding Unique Values
```bash
# List all subsystems with errors
cx logs 'filter $m.severity == ERROR | distinct $l.subsystemname'
# List unique error types
cx logs 'filter $m.severity == ERROR | distinct $d.error_type'
```
### Fetching Sample Logs by Template
Find top error patterns with sample messages:
```bash
cx logs 'filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5'
```
---
## Performance Tips
- Use `--limit` for exploratory queries
- Use `groupby` with aggregations instead of fetching all raw logs
- Filter by time first when dealing with large datasets
- Use specific filters (application, subsystem) to reduce scan scope
- For large result sets, use `--output toon` which spills to a temp file automatically:
```bash
cx logs 'filter $m.severity == ERROR' --start now-24h --limit 1000 -o toon
```
references/promql-guidelines.md
# PromQL Guidelines
## Core Principles
1. **Pick the right query type**
- **Instant queries** (`cx metrics query`) evaluate an expression at a single timestamp (now, or a given `--time`). Use when the question requires **one number or one vector** *as of* a moment - essentially any query that does not require results over different timeframes.
- **Range queries** (`cx metrics query-range`) evaluate the expression **repeatedly** across `[--start, --end]` at a given `--step`. Use for **time series** over a period (e.g., daily active users per day).
- Note: Range queries **evaluate the expression repeatedly** at each step. If `--step=1d`, `--start=now-1d`, `--end=now`, and the query is `max_over_time(metric[1d])`, the query evaluates at `now-1d` and `now` - two evaluations covering two days of data.
- Prefer instant queries over range queries for most questions, except when comparing different timeframes.
2. **Understand PromQL value types**
- **Instant vector** - set of series with 1 sample each at eval time
- **Range vector** - series with many samples over a window `[t-range, t]`
- **Scalar** - single number
- **String** - rare
- Functions like `*_over_time()` **require a range vector**. Aggregations like `sum/max/min/avg ... by(...)` **consume instant vectors**.
- **Important**: When using `*_over_time()` functions with range queries, be aware that the query also evaluates at the `--start` time and includes the window specified in the function.
- **Example**: If `max_over_time(metric[1d])` is used with `--start=now-1d`, `--end=now`, `--step=1d`, the query evaluates at `now-1d` and `now` - the result is the max over `[now-2d, now]`. This is a common mistake. If a user asks "What is the max of x between 2025-01-01 and 2025-01-07?" and `max_over_time(x[7d])` is used with `--start=2025-01-01`, `--end=2025-01-07`, `--step=1d`, the evaluation at `2025-01-01` includes `[2024-12-25, 2025-01-01]` - which is wrong. Use an instant query with `--time` to avoid this.
3. **Separation of concerns**
- Use `*_over_time()` for **temporal reductions** across a window (e.g., `max_over_time`, `avg_over_time`, `quantile_over_time`).
- Use `sum/max/min/avg by (...)` for **label-set aggregation** across series at the eval point.
- Chain them as needed (temporal reduction first, then label aggregation, or vice versa).
4. **Counters vs. gauges**
- **Counters** (monotonic, suffixed `_total`) → use `rate()`/`irate()` or `increase()` over a window.
- **Gauges** (current value) → use `avg_over_time`, `max_over_time`, etc., or plain `avg(...)` depending on intent.
5. **Suffix conventions**
- Canonical: `_total` (counter), `_bucket/_sum/_count` (histogram), `_sum/_count` (summary), `_created`.
- Non-standard: `_avg`, `_mean`, etc. Prefer computing averages via PromQL unless the exporter dictates otherwise.
---
## CLI Usage
### Instant Query
```bash
cx metrics query '<expr>'
cx metrics query '<expr>' --time 2024-01-01T12:00:00Z
cx metrics query '<expr>' --output json
```
**Example: absolute max over last 24h (single result)**
```bash
cx metrics query 'max by () (max_over_time(http_requests_in_flight[24h]))'
```
### Range Query
```bash
cx metrics query-range '<expr>' --start now-7d --end now --step 1d
```
**Example: absolute max per day over the last 7 days**
```bash
cx metrics query-range 'max by () (max_over_time(metric[1d]))' \
--start now-7d --end now --step 1d
```
**IMPORTANT**: Align `--step` with any window used in temporal reduction functions. If using `max_over_time(metric[1d])`, set `--step 1d`.
---
## PromQL Fundamentals
### Label Matching & Aggregation
- Matchers: `{label="v"}`, `{label!="v"}`, `{label=~"re.*"}`, `{label!~"re"}`
- Aggregate **by** labels to keep them; use **without** to drop them.
```promql
sum by (job) (rate(http_requests_total[5m]))
sum without (instance) (up)
```
### Temporal Reductions (range → instant)
```promql
max_over_time(cpu_usage[1h])
avg_over_time(node_memory_Active_bytes[30m])
quantile_over_time(0.99, queue_length[1h])
```
### Counters: Rates, Increases, Windows
Per-instance RPS:
```promql
rate(http_requests_total[5m])
```
Total RPS across fleet:
```promql
sum by () (rate(http_requests_total[5m]))
```
Events in last day (per user, then count actives):
```promql
count( sum by (user_id) (increase(api_call_count[24h])) > 0 )
```
### Histograms & Summaries
p95 from a histogram:
```promql
histogram_quantile(
0.95,
sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)
```
Average from summary parts:
```promql
sum(rate(req_duration_seconds_sum[5m]))
/
sum(rate(req_duration_seconds_count[5m]))
```
### Max over a Period
Correct - temporal reduction, then aggregation:
```promql
max by () (max_over_time(metric[4d]))
```
Per-label max:
```promql
max by (label) (max_over_time(metric[4d]))
```
Incorrect - `max()` cannot take a range vector:
```promql
max(metric[4d]) ← error
```
### Top-k / Ranking
```promql
topk(5, sum by (instance) (rate(http_requests_total[5m])))
```
---
## Common Tasks (ready to adapt)
1. **Absolute peak per instance over 7d, then pick the winner**
```promql
topk(1, max by (instance) (max_over_time(my_metric[7d])))
```
Run as instant query (no `--time` needed - defaults to now).
2. **Global CPU usage % (avg across cores & hosts)**
```promql
avg by () (
rate(process_cpu_seconds_total[5m])
) * 100
```
3. **Error rate (%) per route**
```promql
100 * sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
/ sum by (route) (rate(http_requests_total[5m]))
```
4. **Daily active users over a week (time series)**
Expression:
```promql
count(count by (user_id) (increase(api_call_count[1d]) > 0))
```
Run as range query with `--step 1d --start now-6d --end now`. (Starting from 6 days ago because `increase` looks back one full day from each evaluation point.)
---
## Performance & Safety Guidelines
- Prefer **short windows** for `rate()` (e.g., 1–5m) unless data is bursty or sparse.
- Avoid unbounded fan-out (e.g., joining massive label sets).
- Keep **cardinality** under control; aggregate early (`sum by (...)`) when only totals are needed.
- Use `clamp_max`/`clamp_min` to tame outliers when needed.
- For histograms, **always** aggregate buckets (`sum by (le, ...)`) before `histogram_quantile`.
- Be mindful of counter resets; `rate()`/`increase()` handle resets automatically.
---
## Frequent Gotchas (and fixes)
- **"Why am I getting a time series when I only want one number?"**
Use `cx metrics query` (instant) instead of `cx metrics query-range`.
- **"`max(metric[...])` errors."**
`max()` can't take a range vector. Use `max_over_time(metric[...])`, then aggregate with `max by () (...)`.
- **"`_over_time(metric[...]) by (label)` errors."**
`_over_time` aggregations cannot include a `by` clause. Use `max_over_time(metric[...])`, then `max by (label) (...)`.
- **"Avg looks wrong for counters."**
Counters need `rate()`/`increase()`, not `avg_over_time`.
- **"p95 from a summary?"**
Summaries expose quantiles directly via the `quantile` label. For histograms, use `histogram_quantile` on bucket rates.
- **"Results show empty label values."**
Add `{label!=""}` to the selector to filter out empty label values. Example: `max by (deployment) (rate(cpu_usage{deployment!=""}[5m]))`.
---
## Mini Cheat-Sheet
| Goal | PromQL |
|---|---|
| Rate of a counter | `rate(x_total[5m])` |
| Increase last 24h | `increase(x_total[24h])` |
| Avg of a gauge over 1h | `avg_over_time(x[1h])` |
| Max over 4d (absolute) | `max by () (max_over_time(x[4d]))` |
| Top 5 by RPS | `topk(5, sum by (instance) (rate(x_total[5m])))` |
| p95 latency (histogram) | `histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m])))` |
| Filter labels | `{env="prod", job=~"api\|web"}` |
| Drop a label in agg | `sum without (instance) (x)` |
| Time travel | `expr @ <unix_ts>` or `expr offset 1h` |
references/query-syntax.md
# Coralogix Dashboard Query Syntax - Dashboard-Specific Gotchas
This file documents **only** the non-obvious rules specific to authoring a Coralogix dashboard JSON - the things that silently break an imported dashboard even when the query itself is valid.
For the underlying query languages, use the sibling skills:
- **DataPrime** (filters, aggregations, operators, type conversions, `extract`, `roundTime`): `cx-dataprime` skill → `skills/cx-dataprime/references/dataprime-reference.md`.
- **PromQL** (counters vs gauges, histograms, label matching, temporal reductions, full function list): `cx-metrics-query` skill → `skills/cx-metrics-query/references/promql-guidelines.md`.
- **Inline help**: `cx dataprime list` and `cx dataprime show <command>`.
---
## 1. The `${__range}` dashboard variable
Coralogix injects the dashboard-level time range as `${__range}` - this is a dashboard-only variable, not a PromQL feature.
- Correct: `increase(foo_total[${__range}])`
- Wrong: `increase(foo_total[$__range])` - missing braces; Coralogix drops it
- Wrong: `increase(foo_total[5m])` - hard-codes a 5-minute window; the dashboard time picker has no effect
Use fixed ranges (`[5m]`, `[1h]`) only when the panel intentionally shows a rolling window independent of the dashboard time picker (rare; document it in the panel title).
### Verifying `${__range}` queries with the CLI
`cx metrics query` doesn't expand `${__range}` (it's a dashboard-side substitution). During Phase 5 verification, swap `${__range}` for the concrete token that matches the dashboard's `relativeTimeFrame` (e.g. `[48h]` for the default `"172800s"`). Restore `${__range}` in the JSON before deploy.
---
## 2. `promqlQueryType` - instant vs time-series widgets
Every metrics widget has a `promqlQueryType`:
- `PROM_QL_QUERY_TYPE_INSTANT` - single point in time. Required for `gauge`, `pieChart`, and `dataTable`.
- Omit (or default) - time series. Use for `lineChart`.
Leaving an instant widget in time-series mode makes the panel render a single average across the window instead of the intended single-point value, which breaks success-rate gauges and top-N tables.
---
## 3. DataPrime inside widget JSON
The DataPrime language itself is documented in the `cx-dataprime` skill. A few rules matter specifically inside dashboard widgets:
- DataPrime widget queries **must** start with `source logs` or `source spans` - Coralogix doesn't infer the source for DataPrime widgets the way it does for metrics. Example: `source logs | filter $m.severity == ERROR | agg count()`.
- The JSON key is `dataprimeQuery.text` (a string), not `dataprimeQuery.value`.
- `filter` / `groupby` / aggregate forms use the same syntax as the CLI - the `cx-dataprime` skill is authoritative.
- Severity enums are unquoted in DataPrime: `$m.severity == ERROR`, not `"ERROR"`.
### Widget-side gotchas that commonly appear in reviews
- `$d.message`, not `$m.text`.
- `contains` is a method on the field: `$d.message.contains('timeout')`, not `$d.message contains 'timeout'`.
- Application filter uses `$l.applicationname` (lowercase), not `$m.applicationName`.
If you're unsure, consult the `cx-dataprime` skill reference rather than guessing.
---
## 4. PromQL idioms that recur in dashboards
The full PromQL reference is in the `cx-metrics-query` skill. The patterns below show up in almost every dashboard and are the ones to copy-paste.
**Histogram average over the dashboard range**:
```
sum by (label) (rate(foo_latency_sum[${__range}]))
/
sum by (label) (rate(foo_latency_count[${__range}]))
```
**Histogram P95**:
```
histogram_quantile(0.95,
sum by (le, label) (rate(foo_latency_bucket[${__range}]))
)
```
**Counter increments over the range**:
```
sum by (account_id) (increase(foo_total[${__range}]))
```
**Success rate (%) with safe denominator** (wrap denominators in `clamp_min(..., 1)`):
```
100 *
sum(increase(foo_success_total[${__range}]))
/
clamp_min(
sum(increase(foo_success_total[${__range}]))
+ sum(increase(foo_failure_total[${__range}])),
1
)
```
**Propagating a label across metrics** (metric A has `account_id`; metric B has both `account_id` and `account_slug`):
```
sum by (account_id) (increase(foo_total[${__range}]))
* on (account_id) group_left(account_slug)
max by (account_id, account_slug) (
(max_over_time(bar_total{account_slug!=""}[365d] @ end()) * 0) + 1
)
```
---
## 5. Lucene (legacy logs query)
Only use if the user explicitly requests Lucene - prefer DataPrime.
- Severity: `coralogix.metadata.severity:ERROR`
- Application: `coralogix.metadata.applicationName:"my-service"`
- Field match: `message:"is stuck"`
---
## 6. Cross-references: how Olly and Coralogix consume this JSON
Queries authored by this skill are eventually consumed by the Coralogix dashboard-rendering layer and by the Olly agent that analyses dashboards. The rules below reflect how the JSON is actually treated - verified against the `cx-olly`, `olly`, and `olly-knowledge-base` repositories.
### Field precedence Olly enforces
Source: `olly/apps/api/src/api/agent/handlers/utils/dashboard_enrichment.py` (`_get_important_fields`) surfaces exactly two dashboard fields to the LLM as "important":
- **`relativeTimeFrame`** - Olly uses this as the default time range when the user doesn't pass `from`/`to`. Our Phase 5 CLI verification therefore has to run with the same `relativeTimeFrame` - don't verify a 48h dashboard with a `[5m]` window.
- **`filters`** - Olly injects the dashboard-level `filters` array into every DataPrime and PromQL query before sending it. `DASHBOARD_FOCUSED_PROMPT` in `olly/apps/api/src/api/agent/agents/orchestrator_agent/dashboard_prompts.py` instructs: *"EDIT ALL queries to INCLUDE THE FILTERS FROM THE DASHBOARD DATA"*. Consequence: widget queries must be valid **before** the dashboard filters are injected - do not pre-bake the slicing dimensions into the widget's query text.
### JSON keys Olly consumes
Source: `WHITELIST_DASHBOARD_KEYS` in `olly/apps/api/src/api/agent/handlers/utils/dashboard_enrichment.py`:
```
name, description, title, isVisible, columns, query, unit, scaleType,
variables, variablesV2, filters, relativeTimeFrame, annotations,
twoMinutes, slugName, actions, updatedAt, createdAt, updaterAuthorId,
updaterName, authorId, authorName, updatedOriginType, createdOriginType
```
Prefer these exact field names over synonyms. Keys outside this list are dropped from Olly's view of the dashboard.
### Allowed query languages in a widget
Source: `olly-knowledge-base/common/src/common/models/dashboard_context_models.py` (`QueryEmbedding.query_type`): only `dataprime`, `promql`, or `lucene`. Nothing else will be indexed or embedded downstream.
### Sibling Olly skill
The dashboard-analysis counterpart to this skill lives at `cx-olly/apps/api/src/api/agent/agents/skills_agent/skills/dashboards.md`. It covers the consume-side rules (accept dashboard links, honor `var-*` parameters, always add `filters` + `relativeTimeFrame` to derived queries). Our authoring rules above mirror its expectations.
### Coralogix docs
- Overview: <https://www.coralogix.com/docs/user-guides/custom-dashboards/introduction/>
- Widgets index: <https://www.coralogix.com/docs/user-guides/custom-dashboards/widgets/>
- Query builder: <https://www.coralogix.com/docs/user-guides/custom-dashboards/tutorials/query-builder/>
- Dashboard settings (slugs, folders): <https://www.coralogix.com/docs/user-guides/custom-dashboards/tutorials/manage-dashboard-settings/>
---
## 7. Dashboard-syntax checklist (apply before Phase 5)
- [ ] PromQL range vectors inside widgets use `[${__range}]`.
- [ ] `promqlQueryType` is `PROM_QL_QUERY_TYPE_INSTANT` for `gauge` / `pieChart` / `dataTable`; omitted for `lineChart`.
- [ ] DataPrime widget queries start with `source logs` or `source spans` (required for dashboard widgets; stripped during Phase 5 CLI verification).
- [ ] DataPrime `contains` is written as `.contains(...)`.
- [ ] DataPrime severity enums are unquoted (`ERROR`, `CRITICAL`, …).
- [ ] Success-rate denominators wrapped in `clamp_min(..., 1)`.
- [ ] Histogram queries use the correct suffix (`_sum`, `_count`, `_bucket`).
- [ ] No invented metric names - every PromQL metric appeared in `cx metrics search` during Phase 1.
- [ ] Widget queries remain valid **without** the dashboard-level `filters` (Coralogix/Olly inject them at render time).
- [ ] Widget query language is one of `dataprime`, `promql`, or `lucene` - nothing else.
- [ ] `relativeTimeFrame` in the final JSON matches the window used for Phase 5 verification.
references/spans-querying.md
# Span Querying Reference
Query and analyze distributed tracing data using the `cx spans` command with DataPrime syntax.
> **DataPrime syntax:** See `dataprime-reference.md` for the full query language reference.
## Understanding Spans in Coralogix
Spans are the fundamental unit of tracing data. **Traces are not stored as single entities** - they are logical groupings of spans that share the same `traceID`. To analyze a trace, you query its constituent spans.
This means:
- **Metadata (`$m.*`)** and **labels (`$l.*`)** are predictable - you can always filter on timestamp, duration, service name, and operation name without discovery.
- **User data (`$d.*`)** contains trace identifiers (`traceID`, `spanID`, `parentId`) and application-specific tags/attributes that vary by service. Exact attribute names depend on the instrumentation and can differ across tenants, so verify `$d` fields with a sample query before assuming they exist.
---
## CLI Command
```bash
cx spans '<dataprime_query>'
```
The `source spans` is automatically injected - do not include it in the query.
### Options
| Flag | Default | Description |
|------|---------|-------------|
| `--start` | `now-1h` | Start time (ISO 8601 or relative, e.g. `now-6h`) |
| `--end` | `now` | End time |
| `--limit` | `200` | Maximum number of results |
| `--tier` | `frequent` | Storage tier: `frequent` (hot/recent) or `archive` (cold/historical) |
| `-o, --output` | `text` | Output format: `text`, `json`, or `toon` |
---
## Span Data Model
### Standard Fields (Always Available)
| Field | Description |
|-------|-------------|
| `$m.timestamp` | Span start timestamp |
| `$m.duration` | Span duration in **microseconds** (see [Duration Units](#duration-units)) |
| `$l.applicationName` | Application name - highest-level label. Meaning varies by customer (environment, team, region) but it always exists. |
| `$l.subsystemName` | Subsystem name - second-level label. Typically maps to a component. |
| `$l.serviceName` | Service name - the logical service unit emitting the span. |
| `$l.operationName` | Operation name - the span title (e.g. "POST /checkout", "db.query"). |
| `$d.traceID` | Trace ID - groups spans into a single trace. |
| `$d.spanID` | Unique span identifier. |
| `$d.parentId` | Parent span ID. Root spans have `parentId == null` (not an empty string). |
| `$d.*` | Application-specific tags and attributes (see [Field Discovery](#field-discovery)). |
> **Note on label fields:** The meaning of `$l.applicationName` and `$l.subsystemName` varies by customer - they may represent environments, teams, regions, or something else entirely. Don't assume what they map to. Use `cx search-fields` or sample queries to verify actual values.
### Duration Units
`$m.duration` is in **microseconds**:
- 500ms = `500000`
- 1s = `1000000`
- 1min = `60000000`
When presenting duration values, always convert to human-readable units (milliseconds, seconds, or minutes) and include the unit. Never display raw microsecond values or the "µs" symbol.
```dataprime
# Computed field for milliseconds
create latency_ms from $m.duration / 1000
```
### Error Detection
Spans do not have a `$m.severity` field like logs. Errors are typically indicated by:
- `$d.tags.error == true` - the most common convention (OpenTelemetry/Jaeger)
- Status codes in custom fields (e.g. `$d.http.status_code`, `$d.grpc.status_code`)
- Other application-specific error tags
The exact field depends on the instrumentation library used. If `$d.tags.error` returns no results, inspect sample spans with `-o json` to discover how errors are tagged:
```bash
cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json
```
---
## Essential Query Examples
```bash
# Get all spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"
# Find spans for a service
cx spans "filter \$l.serviceName == 'checkout-service'"
# Find slow spans (> 1 second)
cx spans "filter \$m.duration > 1000000"
# Find error spans
cx spans "filter \$d.tags.error == true"
# Aggregate latency by operation
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc"
# Wider time range
cx spans "filter \$l.serviceName == 'api'" --start now-6h
```
### Wildfind Policy
`wildfind` runs the same **case-insensitive, token-based** match as `~` (whole tokens, not arbitrary substrings), but over the whole record instead of a single field — it matches tokens anywhere, in any field.
**Prefer a field-targeted `~`/`filter` when you know the field** — for **precision**: because `wildfind` matches tokens anywhere in the record, it can't be narrowed to a specific field.
**Performance:** `wildfind` with very short search terms (only a few characters) is much slower — prefer longer, more specific terms.
The **one exception**: when the user provides a specific string and you don't know which field contains it:
```bash
cx spans "wildfind 'connection refused'"
```
> **Tip:** `wildfind` can also serve as a last-resort field discovery method - when `cx search-fields` doesn't find what you need, run `wildfind` with a known value, then inspect the matching spans to see which fields contain it.
---
## Field Discovery
**Skip discovery when:**
- The query only uses standard fields (`$m.duration`, `$l.serviceName`, `$l.operationName`, `$d.traceID`)
- The user explicitly names the fields they want
- The fields have already been discovered earlier in the conversation
### 1. Infer from Source Code (Preferred)
If you have access to the application's source code, examine OpenTelemetry instrumentation, span attribute definitions, and tracing middleware to identify field names directly.
### 2. Semantic Search
```bash
cx search-fields "customer identifier" --dataset spans
cx search-fields "order ID" --dataset spans
cx search-fields "http response code" --dataset spans
```
Note: `cx search-fields` only has access to the most common fields. If it doesn't find what you need, fall back to sample query inspection.
### 3. Sample Query Inspection
```bash
cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json
```
Inspect the JSON output to see all available fields. Especially useful for discovering fields in unstructured or deeply nested data.
---
## Investigation Workflow
### 1. Understand the Request
Identify:
- Whether you have a trace ID, service name, or error description
- Time frame of interest
- Whether the question is about latency, errors, or request flow
### 2. Start with Known Information
**If you have a trace ID** - go straight to it:
```bash
cx spans "filter \$d.traceID == '<trace_id>'"
```
**If you have a service name** - query its spans:
```bash
cx spans "filter \$l.serviceName == '<service>'" --limit 50
```
**If you have neither** - start broad to find entry points:
```bash
# Find recent error spans
cx spans "filter \$d.tags.error == true" --limit 20
# Find the slowest spans in the last hour
cx spans "groupby \$l.serviceName, \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"
# Then extract trace IDs from interesting spans
cx spans "filter \$l.serviceName == '<service>' && \$m.duration > 1000000 | distinct \$d.traceID"
```
### 3. Troubleshooting
If a query returns no results, change **one thing at a time**. Keep the query window as **narrow** as possible and widen it deliberately — start from the window you already have rather than jumping to a huge range:
1. **Relax filters**: remove the most restrictive condition
2. **Check field availability**: the field you're filtering by may only exist in a subset of spans
3. **Verify field names**: run a sample query with `-o json` to inspect the actual schema
4. **Check service names**: service names are case-sensitive
5. **Extend the time range**: widen gradually from your current window (e.g. `now-1h` → `now-6h` → `now-24h`)
6. **Try archive tier**: for older data, add `--tier archive` and widen the window to cover the period you're after
---
## Common Query Patterns
### Trace Reconstruction
```bash
# All spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"
# Find root spans only (no parent)
cx spans "filter \$l.serviceName == 'api-gateway' | filter \$d.parentId == null"
# Find trace IDs for a service
cx spans "filter \$l.serviceName == 'payment-service' | distinct \$d.traceID"
```
### Latency Analysis
```bash
# Spans slower than 1 second
cx spans "filter \$m.duration > 1000000"
# Top 10 slowest operations by average duration
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"
# Average latency by service
cx spans "groupby \$l.serviceName aggregate avg(\$m.duration) as avg_latency"
# P95 latency by operation
cx spans "groupby \$l.operationName aggregate percentile(0.95, \$m.duration) as p95_latency"
```
### Latency Spike Detection
```bash
# Average latency per 15-minute window
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 15m) as interval aggregate avg(\$m.duration) as avg_latency | orderby interval"
# Find the time windows with highest latency
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 5m) as interval aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"
```
### Error Investigation
```bash
# All error spans
cx spans "filter \$d.tags.error == true"
# Error spans for a specific service
cx spans "filter \$l.serviceName == 'checkout' | filter \$d.tags.error == true"
# Error rate by service
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName aggregate count() as errors | orderby errors desc"
# Error rate over time
cx spans "filter \$d.tags.error == true | groupby roundTime(\$m.timestamp, 15m) as interval aggregate count() as errors"
```
### Sampling Error Types
```bash
# Group errors by operation with a sample
cx spans "filter \$d.tags.error == true | groupby \$l.operationName aggregate any_value(\$d) as sample, count() as total | orderby total desc | limit 5"
# Group by service and operation to see where errors concentrate
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName, \$l.operationName aggregate count() as errors | orderby errors desc | limit 10"
```
### Finding Unique Values
```bash
# List all services with spans
cx spans "distinct \$l.serviceName"
# List all operations for a service
cx spans "filter \$l.serviceName == 'api' | distinct \$l.operationName"
# Find unique trace IDs for error spans
cx spans "filter \$d.tags.error == true | distinct \$d.traceID"
```
### Correlating by Trace ID
```bash
# Find spans across services for the same trace
cx spans "filter \$d.traceID == 'abc123' | groupby \$l.serviceName aggregate count() as span_count, avg(\$m.duration) as avg_latency"
```
---
## Performance Tips
- Use `--limit` for exploratory queries
- Use `groupby` with aggregations instead of fetching raw spans when possible
- Filter by time first when dealing with large datasets
- Use specific filters (service name, operation) to reduce scan scope
- Don't rely solely on aggregations - retrieve sample spans to find information you didn't anticipate
- For large result sets, use `--output toon` which spills automatically:
```bash
cx spans "filter \$l.serviceName == 'api'" --start now-24h --limit 1000 -o toon
```
references/verification.md
# Phase 5: Live-verify every query through the cx CLI
Every PromQL and DataPrime query in the draft dashboard must successfully run through `cx` before Phase 8 ships it. This is where invented metric names, typoed field paths, and malformed DataPrime pipelines get caught.
---
## 1. Resolve the dashboard time range (PromQL only)
Parse `relativeTimeFrame` from the draft (default `"172800s"` = 48h) into a human token and call it `$RANGE`:
| `relativeTimeFrame` | `$RANGE` token |
|---|---|
| `3600s` | `1h` |
| `21600s` | `6h` |
| `86400s` | `24h` |
| `172800s` | `48h` |
| `604800s` | `7d` |
`$RANGE` is used **only** for PromQL verification (§2): range vectors are window-sensitive, so the CLI check has to match the window the dashboard will evaluate. DataPrime verification (§3) uses a fixed short window instead — see that section.
---
## 2. Verify each PromQL query
For every widget whose definition contains a `promqlQuery`, substitute `${__range}` in the expression with `[$RANGE]` (e.g. `[48h]`). Leave any other fixed window (`[5m]`, `[1h]`) untouched - those were placed intentionally for sliding-rate panels.
**Instant-style widgets** (`gauge` / `pieChart` / `dataTable` with `promqlQueryType: PROM_QL_QUERY_TYPE_INSTANT`):
```bash
cx metrics query '<expression-with-[$RANGE]-substituted>' -o toon
```
**Time-series widgets** (`lineChart`):
```bash
cx metrics query-range '<expression>' --start now-$RANGE --end now --step <auto> -o toon
```
Pick `<step>` proportional to `$RANGE`: `1m` for 1–6h, `5m` for 24h, `1h` for 7d+. Match any window used by a `*_over_time` / `rate` / `increase` inside the expression if it's narrower than `$RANGE`.
A query **passes** when the CLI returns a 200 response and either has data or an empty-but-well-formed result. **Fails** include unknown metric names, parse errors, non-200 responses, or `cx` error output.
On failure: consult the `cx-metrics-query` skill for PromQL help, re-search for the real metric name with `cx metrics search`, re-list labels with `cx metrics get-labels`, and fix the query in the draft JSON. Budget ≤5 retry attempts per query.
---
## 3. Verify each DataPrime query
For every widget whose definition contains a `dataprimeQuery`, pick the CLI command from the widget's source prefix and **strip the leading `source logs` / `source spans`** before handing the pipeline to `cx`:
| Widget prefix | CLI | What to pass |
|---|---|---|
| `source logs \| …` | `cx logs` | everything after `source logs \|` (trim the leading `\|` and whitespace) |
| `source spans \| …` | `cx spans` | everything after `source spans \|` |
The dashboard runtime requires the `source …` prefix inside the widget JSON (see `query-syntax.md` §3). `cx logs` and `cx spans` inject the source themselves; if you leave a leading `source …` in the pipeline they silently run against the command's own source, which masks pillar mismatches. Strip it for verification; restore nothing - the widget JSON keeps the prefix.
Verify against a **fixed short window** (`now-15m` → `now`), not the dashboard's `$RANGE`. The goal here is syntax / field / pipeline validation — proving the query parses and references real fields. The dashboard runs against `${__range}` itself at render time; we don't need to re-prove data presence on the dashboard's window during the build. A short window is faster, cheaper, and a clean fail signal (a query that fails on `now-15m` is broken regardless of range).
Choose the tier to verify:
- `--tier frequent` (default): hot storage, fast, recent data.
- `--tier archive`: cold/long-term storage, older data.
Use **Frequent Search** unless you have a reason to validate against Archive. Switch to **Archive** when:
- The dashboard is intended for long lookbacks (weekly/monthly trends, retrospectives).
- Frequent Search returns empty for known-good queries because the time range is beyond hot retention.
- The user explicitly says “this dashboard should work on archived data.”
**Log-backed widgets:**
```bash
cx logs '<pipeline-without-leading-source-logs>' --start now-15m --end now --limit 1 --tier <frequent|archive>
```
**Span-backed widgets:**
```bash
cx spans '<pipeline-without-leading-source-spans>' --start now-15m --end now --limit 1 --tier <frequent|archive>
```
Check both the exit code and the output — some errors surface only in the output. A query **passes** when `cx` exits 0 and the output is rows or `[]` with no error or warning lines (an empty result on a low-volume signal is fine). It **hard-fails** on a non-zero exit, an `error from profile '...': API request failed` line, or a `Compilation errors:` block — the query is broken. A `keypath does not exist` warning is a **soft fail**: the query parsed but no record in the window had the referenced field. Verify with `cx search-fields "<hint>" --dataset logs|spans`; if the field is real the query is fine (widen the window or accept the empty result), if it isn't, fix the field name. On failure: consult the `cx-dataprime` skill (`cx dataprime show <command>` for inline help), re-discover fields with `cx search-fields`, fix, retry. Budget ≤5 retry attempts per query.
---
## 4. Restore `${__range}` before Phase 6
Once every PromQL and DataPrime query passes, restore `${__range}` (and any other variables) in the emitted JSON. PromQL verification swapped `${__range}` for the concrete `[$RANGE]`; the final JSON keeps the injected variable intact. (DataPrime queries don't carry `${__range}`, so nothing to restore there beyond keeping the `source logs` / `source spans` prefix in the widget JSON.)
If any query can't be made to pass within the retry budget, surface it to the user with the CLI error verbatim - don't silently ship a broken widget.
references/widget-templates.md
# Coralogix Dashboard JSON - Widget Templates
Copy these templates when generating a dashboard. **Always replace every UUID** (`id.value`, query `id`) with a freshly generated UUID, and adapt queries to the target service.
For query-language rules:
- Dashboard-specific gotchas (`${__range}`, `promqlQueryType`, widget filters): [`query-syntax.md`](query-syntax.md).
- Full DataPrime syntax: `cx-dataprime` skill → `skills/cx-dataprime/references/dataprime-reference.md`.
- Full PromQL reference: `cx-metrics-query` skill → `skills/cx-metrics-query/references/promql-guidelines.md`.
> A "stat" widget in Coralogix docs is actually emitted as `gauge` in the JSON. There is no separate stat type.
> Every query below must pass live verification via `cx metrics query` / `cx logs` / `cx spans` before deploy - see SKILL.md Phase 5.
---
## Top-level skeleton
```json
{
"id": "<21-char-nanoid>",
"name": "<Service> - <Purpose>",
"layout": {
"sections": [
{ "id": {"value": "<uuid>"}, "rows": [ ... ], "options": { "custom": { "name": "Section name", "collapsed": false, "color": {"predefined": "SECTION_PREDEFINED_COLOR_UNSPECIFIED"} } } }
]
},
"variables": [],
"variablesV2": [],
"filters": [ /* see "Top-level filters" below */ ],
"relativeTimeFrame": "172800s",
"annotations": [],
"off": {},
"actions": []
}
```
A row always looks like:
```json
{
"id": {"value": "<uuid>"},
"appearance": {"height": 19},
"widgets": [ /* 1–2 widgets per row */ ]
}
```
---
## Widget: gauge (also used for "stat"/total)
Use for headline numbers: counts, percentages, success rates.
Gauge `min` and `max` must be numeric and `min < max`. This API constraint applies even when `thresholdType` is `THRESHOLD_TYPE_ABSOLUTE`; use a real range such as `0..100` for percentages, a known capacity/limit when available, or a max above the highest threshold.
```json
{
"id": {"value": "<uuid>"},
"title": "Success Rate",
"definition": {
"gauge": {
"query": {
"metrics": {
"promqlQuery": {
"value": "100 * sum(increase(foo_success_total[${__range}])) / clamp_min(sum(increase(foo_success_total[${__range}])) + sum(increase(foo_failure_total[${__range}])), 1)"
},
"aggregation": "AGGREGATION_UNSPECIFIED",
"filters": [],
"editorMode": "METRICS_QUERY_EDITOR_MODE_TEXT",
"promqlQueryType": "PROM_QL_QUERY_TYPE_INSTANT"
}
},
"min": 0,
"max": 100,
"showInnerArc": true,
"showOuterArc": true,
"unit": "UNIT_PERCENT",
"thresholds": [
{"from": 0, "color": "var(--c-visualization-red-05)"},
{"from": 80, "color": "var(--c-visualization-yellow-05)"},
{"from": 95, "color": "var(--c-visualization-green-05)"}
],
"dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED",
"thresholdBy": "THRESHOLD_BY_UNSPECIFIED",
"decimal": 2,
"thresholdType": "THRESHOLD_TYPE_ABSOLUTE",
"legend": {"isVisible": true, "columns": [], "groupByQuery": true, "placement": "LEGEND_PLACEMENT_AUTO"},
"legendBy": "LEGEND_BY_GROUPS",
"displaySeriesName": false,
"decimalPrecision": false
}
}
}
```
**For "count of bad things" (errors, DLQ):** use `unit: "UNIT_NUMBER"`, green at low values, red at high:
```json
"thresholds": [
{"from": 0, "color": "var(--c-visualization-green-05)"},
{"from": 1, "color": "var(--c-visualization-yellow-05)"},
{"from": 10, "color": "var(--c-visualization-red-05)"}
],
"thresholdType": "THRESHOLD_TYPE_ABSOLUTE"
```
**For DataPrime-driven count (e.g. error log count):** swap `metrics.promqlQuery` for `dataprime.dataprimeQuery`:
```json
"query": {
"dataprime": {
"dataprimeQuery": {"text": "source logs | filter $m.severity == ERROR || $m.severity == CRITICAL | agg count()"},
"filters": []
}
}
```
---
## Widget: pieChart
Use for small-cardinality breakdowns (≤8 slices).
```json
{
"id": {"value": "<uuid>"},
"title": "Messages Per Env",
"definition": {
"pieChart": {
"query": {
"metrics": {
"promqlQuery": {"value": "sum by (subsystem_name) (increase(foo_total[${__range}]))"},
"filters": [],
"groupNames": ["subsystem_name"],
"editorMode": "METRICS_QUERY_EDITOR_MODE_TEXT",
"promqlQueryType": "PROM_QL_QUERY_TYPE_INSTANT",
"aggregation": "AGGREGATION_UNSPECIFIED"
}
},
"maxSlicesPerChart": 8,
"minSlicePercentage": 1,
"stackDefinition": {"maxSlicesPerStack": 4},
"labelDefinition": {"labelSource": "LABEL_SOURCE_INNER", "isVisible": true, "showName": true, "showValue": true, "showPercentage": true},
"showLegend": true,
"unit": "UNIT_UNSPECIFIED",
"colorScheme": "classic",
"dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED",
"decimal": 2,
"legend": {"isVisible": true, "columns": [], "groupByQuery": true, "placement": "LEGEND_PLACEMENT_AUTO"},
"hashColors": false,
"decimalPrecision": false,
"showTotal": false
}
}
}
```
---
## Widget: lineChart
Use for anything over time (rates, latencies, counts per bucket).
```json
{
"id": {"value": "<uuid>"},
"title": "Latency P95 by Stage",
"definition": {
"lineChart": {
"legend": {"isVisible": true, "columns": [], "groupByQuery": true, "placement": "LEGEND_PLACEMENT_AUTO"},
"tooltip": {"showLabels": false, "type": "TOOLTIP_TYPE_ALL"},
"queryDefinitions": [
{
"id": "<uuid>",
"query": {
"metrics": {
"promqlQuery": {"value": "histogram_quantile(0.95, sum by (le, stage) (rate(foo_latency_bucket[${__range}])))"},
"filters": [],
"editorMode": "METRICS_QUERY_EDITOR_MODE_TEXT",
"seriesLimitType": "METRICS_SERIES_LIMIT_TYPE_BY_SERIES_COUNT"
}
},
"seriesCountLimit": "20",
"unit": "UNIT_SECONDS",
"scaleType": "SCALE_TYPE_LINEAR",
"isVisible": true,
"colorScheme": "classic",
"resolution": {"bucketsPresented": 96},
"dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED",
"decimal": 2,
"hashColors": false,
"decimalPrecision": false,
"intervalResolution": {"auto": {"minimumInterval": "15s", "maximumDataPoints": 96}}
}
],
"stackedLine": "STACKED_LINE_UNSPECIFIED",
"connectNulls": false
}
}
}
```
- For counts (not latency) use `"unit": "UNIT_UNSPECIFIED"`.
- Multiple lines in the same panel: add more objects to `queryDefinitions` (each with its own `id`).
---
## Widget: dataTable
Use for top-N tables and raw log listings.
**Metrics-backed table** (e.g. top accounts by count):
```json
{
"id": {"value": "<uuid>"},
"title": "Top Accounts by Message Count",
"definition": {
"dataTable": {
"query": {
"metrics": {
"promqlQuery": {"value": "sum by (account_id) (increase(foo_total[${__range}]))"},
"filters": [],
"editorMode": "METRICS_QUERY_EDITOR_MODE_TEXT",
"promqlQueryType": "PROM_QL_QUERY_TYPE_INSTANT"
}
},
"resultsPerPage": 10,
"rowStyle": "ROW_STYLE_UNSPECIFIED",
"columns": [
{"field": "account_id"},
{"field": "#value"}
],
"dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED"
}
}
}
```
**DataPrime-backed table** (e.g. last errors):
```json
{
"id": {"value": "<uuid>"},
"title": "Last errors",
"definition": {
"dataTable": {
"query": {
"dataprime": {
"dataprimeQuery": {"text": "source logs | filter $m.severity == ERROR || $m.severity == CRITICAL | orderby $m.timestamp desc"},
"filters": []
}
},
"resultsPerPage": 100,
"rowStyle": "ROW_STYLE_ONE_LINE",
"columns": [
{"field": "$m.severity", "width": 121},
{"field": "$m.timestamp", "width": 190},
{"field": "$d", "width": 600}
],
"dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED"
}
}
}
```
---
## Top-level filters
Add one entry per slicing dimension the user approved. Users fill in `values` at view time.
**Exclude non-prod environments** (replace the example values with whatever the target deployment actually uses, e.g. `["dev", "staging", "test"]`):
```json
{
"source": {
"metrics": {
"label": "subsystem_name",
"operator": {"notEquals": {"selection": {"list": {"values": ["<non-prod-env-1>", "<non-prod-env-2>"]}}}}
}
},
"enabled": true,
"collapsed": false,
"id": {"value": "<uuid>"}
}
```
**User-fillable slicing filter**:
```json
{
"source": {
"metrics": {
"label": "account_id",
"operator": {"equals": {"selection": {"list": {"values": []}}}}
}
},
"enabled": true,
"collapsed": false,
"id": {"value": "<uuid>"}
}
```
---
## Section template
```json
{
"id": {"value": "<uuid>"},
"rows": [ /* rows go here */ ],
"options": {
"custom": {
"name": "Errors",
"collapsed": true,
"color": {"predefined": "SECTION_PREDEFINED_COLOR_UNSPECIFIED"}
}
}
}
```
Set `collapsed: true` for logs/debug sections and any section that isn't the dashboard's primary purpose.
---
## Tier (`dataModeType`) quick-reference
`dataModeType` lives **per widget** under the widget's definition (e.g. `definition.gauge.dataModeType`, `definition.dataTable.dataModeType`, `definition.lineChart.queryDefinitions[].dataModeType`). It controls which storage tier the widget reads from at render time.
| Value | When |
|---|---|
| `DATA_MODE_TYPE_HIGH_UNSPECIFIED` | default — frequent (hot) search tier |
| `DATA_MODE_TYPE_ARCHIVE` | archive (cold) tier — for long lookbacks or when the user requests archive |
The templates above ship with `DATA_MODE_TYPE_HIGH_UNSPECIFIED`. When the user wants archive, replace `DATA_MODE_TYPE_HIGH_UNSPECIFIED` → `DATA_MODE_TYPE_ARCHIVE` on **every** widget before deploy (not just the dashboard root).
---
## Unit enum quick-reference
| Value | When |
|---|---|
| `UNIT_UNSPECIFIED` | raw counts, unitless ratios |
| `UNIT_NUMBER` | explicit integer count |
| `UNIT_SECONDS` | durations/latency |
| `UNIT_PERCENT` | percentages 0–100 |
| `UNIT_BYTES` | sizes |
---
## Threshold type
- `THRESHOLD_TYPE_ABSOLUTE` - thresholds compared against the raw value. Use for success rates and fixed-meaning counts.
- `THRESHOLD_TYPE_RELATIVE` - thresholds as % of min/max. Use when the scale is arbitrary.
Default to `ABSOLUTE` for rates and DLQ counts; `RELATIVE` only when the absolute scale is unknown.
SKILL.md
---
name: cx-dashboards
description: >
Build and deploy a Coralogix dashboard for a given service from its logs,
spans, metrics, and service specs. Discovers telemetry via cx CLI commands,
emits importable Coralogix JSON, verifies every PromQL and DataPrime query live
through the `cx` CLI, and creates or updates dashboards via `cx dashboards create`
and `cx dashboards replace`. Use whenever the user asks to create, build, generate,
deploy, update, replace, or modify a Coralogix dashboard, monitoring dashboard,
or observability dashboard for a service, app, or pipeline.
metadata:
version: "0.2.0"
---
# Create Coralogix Dashboard
Produces a Coralogix dashboard for a target service and deploys it via the `cx` CLI. Workflow: discover the service's telemetry, align on intent with the user, draft a plan, emit the JSON, live-verify every query through `cx`, then create the dashboard in a chosen folder.
Only use metric names, log fields, and span attributes you can cite from the service's code, README, configuration, or a live query that returned a result. Do not invent them.
---
## Reference files
Load these files for domain-specific guidance:
| Task | Reference |
|---|---|
| DataPrime query syntax | [`references/dataprime-reference.md`](references/dataprime-reference.md) |
| PromQL query syntax, counters vs gauges, histograms | [`references/promql-guidelines.md`](references/promql-guidelines.md) |
| Log field discovery, query patterns, wildfind policy | [`references/logs-querying.md`](references/logs-querying.md) |
| Span field discovery, latency analysis, trace queries | [`references/spans-querying.md`](references/spans-querying.md) |
| Dashboard-specific query gotchas (`${__range}`, `promqlQueryType`) | [`references/query-syntax.md`](references/query-syntax.md) |
| Widget JSON templates | [`references/widget-templates.md`](references/widget-templates.md) |
For choosing the right signal (metrics / logs / traces), use `cx-telemetry-querying`.
---
## Dashboard Management
Beyond creating dashboards, use these commands to manage existing ones:
| Command | Purpose |
|---|---|
| `cx dashboards catalog -o json` | List all dashboards in the catalog |
| `cx dashboards get <id> -o json` | Get a dashboard definition (useful as a template) |
| `cx dashboards folders list -o json` | List dashboard folders |
| `cx dashboards folders create --name "Name"` | Create a dashboard folder |
| `cx dashboards folders create --name "Sub" --parent-id <id>` | Create a nested folder |
| `cx dashboards replace --from-file dashboard.json` | Replace an existing dashboard with updated JSON |
| `cx dashboards check --from-file dashboard.json` | Validate a dashboard definition without persisting (server-side strict check; exits non-zero on errors) |
| `cx dashboards check <dashboard-id>` | Validate a stored dashboard by id |
To update an existing dashboard:
```bash
cx dashboards get <dashboard-id> -o json > dashboard.json
# Edit dashboard.json (change name, modify widgets, etc.)
cx dashboards replace --from-file dashboard.json
```
To duplicate a dashboard as a new copy:
```bash
cx dashboards get <dashboard-id> -o json > dashboard.json
# Remove the "id" field, then create as new:
cx dashboards create --from-file dashboard.json
```
---
## Workflow
Track progress through this checklist:
```
Dashboard Progress:
- [ ] Phase 1: Discover telemetry & business meaning
- [ ] Phase 2: Gather dashboard specifications from user
- [ ] Phase 3: Draft internal dashboard plan (sections/rows/widgets)
- [ ] Phase 4: Generate the Coralogix JSON
- [ ] Phase 5: Live-verify every query through the cx CLI
- [ ] Phase 6: Self-verify structure against the checklist
- [ ] Phase 7: Server-side validation via `cx dashboards check`
- [ ] Phase 8: Deploy via `cx dashboards create`
- [ ] Phase 9: Share the dashboard link with the user
```
Proceed in order. Don't jump to Phase 4 before the user approves the Phase 3 plan, and don't run Phase 8 before Phases 5–7 all pass. Phase 9 is mandatory — the workflow is not done until the user has a clickable link.
---
## Phase 1: Discover telemetry & business meaning
For the target service, gather:
1. **Business purpose** - read `README.md` and the top-level entrypoint (`main.*`, `index.*`, `cmd/main.go`, etc.). Summarize in 2–3 sentences what it does, its key stages, and what can go wrong.
2. **Metrics** - for each candidate keyword (service name, subsystem, verbs like `request`, `error`, `latency`, `dlq`) run `cx metrics search --name '*<keyword>*'`. When a metric looks promising, list its labels with `cx metrics get-labels <metric>`. Only use names `cx metrics search` returns - this is what prevents invented metrics from reaching Phase 5. Cross-check the service's instrumentation (`prometheus_client`, `promauto.NewCounter/Histogram/Gauge`, OTel meters, `prom-client`, Micrometer, `metrics.py`) for semantics and histogram buckets (`_sum`, `_count`, `_bucket`).
3. **Logs** - discover custom `$d.*` fields with `cx search-fields "<description>" --dataset logs` before assuming a field exists. Sample message templates and severity with `cx logs "filter \$l.applicationname == '<app>'" --limit 5 -o json`. Standard fields (`$m.severity`, `$m.timestamp`, `$l.applicationname`, `$l.subsystemname`) don't need discovery.
4. **Spans / traces** - discover span attributes with `cx search-fields "<description>" --dataset spans`. Sample with `cx spans "filter \$l.serviceName == '<svc>'" --limit 5 -o json`. Error conventions vary (`$d.tags.error`, `$d.http.status_code`); check samples before filtering.
5. **Message buses & DLQs** - grep for Kafka, RabbitMQ, SQS, Pub/Sub clients and any `dlq`/`DLQ` references. Note topic/queue names for DLQ panels.
6. **Service configuration** - check `meta.yaml`, Helm `values.yaml`, `Deployment`, `Dockerfile`, `chart.yaml`. Extract:
- The `applicationname` / `subsystemname` label values as they appear in Coralogix.
- Tenant/account/team identifiers used as metric or log labels.
- Deployment environments (`prod`, `staging`, `dev`, …).
If the signal for a question is ambiguous (e.g. "how much revenue last week"), delegate to `cx-telemetry-querying` first.
Produce a short internal summary before moving on. If critical telemetry is missing (e.g. no metrics), surface that to the user and ask whether they want a log-only or trace-only dashboard.
---
## Phase 2: Gather dashboard specifications
Ask the user a focused set (≤6). Prefer `AskQuestion`:
1. **Audience & use** - on-call triage, product/business tracking, capacity planning, customer success?
2. **Default time range** - typical viewing window (e.g. 24h, 7d). Queries still use `${__range}` so users can zoom.
3. **Slicing dimensions** - top-level filters (`tenant_id`, `account_id`, `subsystem_name`, `region`, `env`, …).
4. **Environment scope** - which environments to include/exclude (common default: exclude `dev`, `staging`, `test`).
5. **SLO-ish signals** - success-rate, latency, or throughput targets to highlight?
6. **Priorities** - what to see first (drives row ordering and which section is `collapsed: true`).
Don't block on answers you can reasonably infer - state the inference and continue.
---
## Phase 3: Draft the internal plan
Write a markdown plan the user can approve before JSON generation:
```
## Dashboard: <Service> - <Purpose>
### Section 1: <Overview> (collapsed: false)
- Row 1: [widget type] <title> - <what it shows> - source: metrics|logs|spans
- Row 2: ...
### Section 2: <Deep dive> (collapsed: false)
...
### Section N: <Logs & errors> (collapsed: true)
...
### Top-level filters
- <label> (<source>)
### Assumptions / gaps
- ...
```
**Section design**:
- First section: at-a-glance health (gauges + key rates), always expanded.
- Pair related time-series in the same row (rate + latency).
- Final section (raw logs, rare breakdowns): `collapsed: true`.
- Aim for 3–5 sections, 6–20 widgets total.
**Widget-type selection**:
| Signal | Widget type |
|---|---|
| Single headline number (count, % success, totals) | `gauge` (Coralogix calls this "stat") |
| Breakdown across ≤8 categories | `pieChart` |
| Change over time (rate, latency, count per bucket) | `lineChart` |
| Top-N tables, last errors, per-entity listings | `dataTable` |
Don't use other widget types unless the user asks.
Wait for the user to approve or adjust the plan before emitting JSON.
---
## Phase 4: Generate the Coralogix JSON
Produce a single JSON document following [`references/widget-templates.md`](references/widget-templates.md). Key rules:
1. **Top-level shape**:
```
{
"id": "<21-char-nanoid>",
"name": "<Dashboard Name>",
"layout": { "sections": [ ... ] },
"variables": [],
"variablesV2": [],
"filters": [ ... ],
"relativeTimeFrame": "<seconds>s",
"annotations": [],
"off": {},
"actions": []
}
```
2. **IDs** - fresh UUIDs for every `section`, `row`, `widget`, and query `id`.
3. **Row height** - `"appearance": { "height": 19 }` unless there's a reason to change.
4. **Section options** - include `options.custom.name`, `collapsed`, and `color.predefined: "SECTION_PREDEFINED_COLOR_UNSPECIFIED"`.
5. **Filters** - one entry per slicing dimension from Phase 2. Default operator `equals` with empty `values` so users can fill in. Use `notEquals` for environment exclusions (see [`references/widget-templates.md`](references/widget-templates.md)).
6. **relativeTimeFrame** - default `"172800s"` (48h) unless the user specified otherwise.
For query syntax follow [`references/query-syntax.md`](references/query-syntax.md); for the full query languages load [`references/dataprime-reference.md`](references/dataprime-reference.md) and [`references/promql-guidelines.md`](references/promql-guidelines.md).
---
## Phase 5: Live-verify every query through the cx CLI
Every PromQL and DataPrime query in the draft has to successfully run through `cx` before Phase 8. This catches invented metric names, typoed field paths, and malformed pipelines.
### Frequent vs Archive (what / when / where in JSON)
**What**:
- **Frequent** (`TIER_FREQUENT_SEARCH`): hot tier for fast search on recent logs/spans.
- **Archive** (`TIER_ARCHIVE`): cold tier for older logs/spans (long-term).
**When to choose**:
- Choose **Frequent** for on-call and recent investigations (hours/days).
- Choose **Archive** for long lookbacks (weeks/months) or when the time range is beyond hot retention.
The two languages are verified against different windows:
- **PromQL**: map `relativeTimeFrame` to a `$RANGE` token (e.g. `48h` for `172800s`), substitute `${__range}` with `[$RANGE]` for the CLI call, then restore `${__range}` in the JSON before Phase 6. Range vectors are window-sensitive, so the check has to match what the dashboard will evaluate.
- **DataPrime**: verify against a fixed short window (`now-15m` → `now`, `--limit 1`). The goal is syntax / field / pipeline validation, not data-presence on the dashboard's window — a short window is faster and a cleaner fail signal.
Full procedure (CLI invocations, `$RANGE` mapping table, retry budget, failure modes): [`references/verification.md`](references/verification.md).
If a query can't be made to pass within the retry budget, surface it to the user with the CLI error verbatim - don't ship a broken widget.
---
## Phase 6: Self-verify structure
Run this checklist against the final JSON. Fix and re-check if any item fails before Phase 7.
### Query syntax (dashboard-specific)
- [ ] Every PromQL range vector in a metrics widget uses `[${__range}]` - never `[$__range]`, never `[5m]` (unless the panel is intentionally a sliding window).
- [ ] `promqlQueryType` is `PROM_QL_QUERY_TYPE_INSTANT` for single-value widgets (gauge, pieChart, dataTable). Omitted for `lineChart`.
- [ ] DataPrime log queries use `$d.message` / `$l.applicationname` / unquoted severity enums (full rules: [`references/dataprime-reference.md`](references/dataprime-reference.md)).
- [ ] Every DataPrime widget query starts with `source logs` or `source spans` (dashboard widgets require the source prefix; Phase 5 verification strips it before handing the pipeline to `cx logs` / `cx spans`).
- [ ] Success-rate denominators wrapped in `clamp_min(..., 1)`.
- [ ] Histogram queries use the correct suffix (`_sum`, `_count`, `_bucket`).
- [ ] Widget queries are valid **without** the dashboard-level `filters` - Coralogix injects them at render time.
### Structure
- [ ] Each section has `id.value`, `rows`, and `options.custom`.
- [ ] Each row has `id.value`, `appearance.height`, and `widgets`.
- [ ] Each widget has a unique `id.value` and a `definition` with exactly one of `gauge` / `pieChart` / `lineChart` / `dataTable`.
- [ ] Every gauge has numeric `min` and `max`, and `min < max`.
- [ ] Success-rate gauges use `thresholdType: "THRESHOLD_TYPE_ABSOLUTE"` with green at high values; error/DLQ gauges use red at high values.
- [ ] "Total" / "stat" widgets are encoded as `gauge`, not as a stat type.
- [ ] Top-level `filters` includes each slicing dimension from Phase 2.
- [ ] All IDs are freshly generated UUIDs, unique within the document.
### Content
- [ ] Dashboard name is descriptive (`"<Service> - <Purpose>"`).
- [ ] Widget titles are short, human-readable, and match what the query computes.
- [ ] The logs/errors section is `collapsed: true` unless the user said otherwise.
---
## Phase 7: Server-side validation via `cx dashboards check`
Phase 6 caught structural issues by hand. This phase runs the whole dashboard through the Coralogix Dashboard Service's strict validator (`CheckDashboard`) — the same validation `create`/`replace` apply on write, plus a superset that compiles every PromQL/DataPrime query and enforces required ids for variables and filters.
1. Run `cx dashboards check --from-file /tmp/cx-dashboard-<slug>.json`.
2. If the command exits non-zero, the output lists issues with `severity`, `location` (an RFC 6901 JSON Pointer into the dashboard, e.g. `/sections/0/rows/1/widgets/2`), and `message`. Fix each issue in the JSON (loop back to Phase 4), re-run Phase 5 for any query that changed, then re-run this phase.
3. When `check` exits 0 (text output: `Dashboard is valid (no issues)`), proceed to Phase 8.
`check` is read-only — it never persists the dashboard. Warnings (`SEVERITY_WARNING`) print but do not fail the gate; only errors (`SEVERITY_ERROR`) cause a non-zero exit. In multi-profile fan-out, any profile returning errors fails the command.
---
## Phase 8: Deploy via `cx dashboards create`
Don't tell the user to paste JSON into the Coralogix UI - deploy it directly.
1. List folders: `cx dashboards folders list -o json`.
2. Suggest the best folder match (team, product area, or a folder named after the service). Default to root (omit `--folder`) if nothing fits.
3. Write the verified JSON to a temp file and run `cx dashboards create --from-file /tmp/cx-dashboard-<slug>.json --folder <id>`. The CLI generates the `requestId` envelope and prints the created dashboard ID.
Full procedure (folder-picking UX, command templates, idempotency note): [`references/deploy.md`](references/deploy.md).
On failure: show the CLI error verbatim and return to Phase 5. The most common cause is a query that parses locally but the live API rejects.
---
## Phase 9: Share the dashboard link
The workflow is **not done** until the user has a clickable link to the dashboard. Printing the ID alone forces the user to navigate the Coralogix UI by hand, which defeats the point of automating deployment.
After Phase 8 succeeds, capture the `View in Coralogix: <url>` line that `cx dashboards create` prints to stderr (see [`references/deploy.md`](references/deploy.md) § "Share the link" for when it's omitted) and emit the output template below. Render the dashboard **name** as the link text — that's what the user clicks.
Linking to a dashboard found via `cx dashboards catalog` (rather than one you just created) works differently: `catalog` prints only one link, to the catalog page, not a per-dashboard link. To link to one specific dashboard from that list, build `<base>/dashboards/<dashboard_id>`, where `<base>` is the console URL already seen in a `View in Coralogix: <base>/...` line printed by any `cx dashboards` command this session — never fabricate `<base>` yourself, and never invent it if no such line has been printed yet.
---
## Output format for the user
When `cx dashboards create` printed a `View in Coralogix:` link:
````
## Plan
<the approved Phase 3 plan>
## Verification
- PromQL queries verified: <N>/<N>
- DataPrime queries verified: <N>/<N>
## Deployed
- Dashboard: **[<Name>](<url from the View in Coralogix line>)**
- ID: `<id>`
- Folder: `<folder name or "root">`
- Profile: `<cx profile>`
Open it: [<Name>](<url from the View in Coralogix line>)
Adjust filter values (e.g. `account_id`) after opening it.
````
When `cx dashboards create` did **not** print a `View in Coralogix:` line (no console link could be resolved for the profile and no `console_url` override configured), omit the link entirely — do not invent a URL. Use this template instead:
````
## Plan
<the approved Phase 3 plan>
## Verification
- PromQL queries verified: <N>/<N>
- DataPrime queries verified: <N>/<N>
## Deployed
- Dashboard: **<Name>** (open via the Coralogix UI; ID `<id>`)
- ID: `<id>`
- Folder: `<folder name or "root">`
- Profile: `<cx profile>`
Adjust filter values (e.g. `account_id`) after opening it.
````
---
## References
- Dashboard query gotchas & cross-references: [`references/query-syntax.md`](references/query-syntax.md)
- Widget JSON templates: [`references/widget-templates.md`](references/widget-templates.md)
- Live-verification procedure: [`references/verification.md`](references/verification.md)
- Deploy procedure: [`references/deploy.md`](references/deploy.md)
- DataPrime language reference: [`references/dataprime-reference.md`](references/dataprime-reference.md)
- PromQL reference: [`references/promql-guidelines.md`](references/promql-guidelines.md)
- Log querying patterns: [`references/logs-querying.md`](references/logs-querying.md)
- Span querying patterns: [`references/spans-querying.md`](references/spans-querying.md)
- Inline DataPrime help: `cx dataprime list`, `cx dataprime show <command>`
- Coralogix Custom Dashboards docs: <https://www.coralogix.com/docs/user-guides/custom-dashboards/introduction/>
### Related Skills
- **`cx-observability-setup`** - full monitoring setup workflow (views, webhooks, notifications, integrations)
- **`cx-slos`** - SLO-connected reliability targets to surface on dashboards
- **`cx-cases`** - triage the cases raised against the services a dashboard monitors
- **`cx-telemetry-querying`** - discover the right telemetry signal before building dashboards