evals/evals.json
{
"skill_name": "signoz-creating-dashboards",
"evals": [
{
"id": 0,
"eval_name": "template-with-data-jvm",
"prompt": "Create a JVM dashboard so I can monitor my Java service.",
"expected_output": "Agent paginates signoz_list_dashboards and finds existing JVM dashboards (e.g. 'JVM Performance Metrics', 'JVM Metrics Dashboard'). Surfaces these duplicates with name and id and presents the three options: '(a) modify an existing one, (b) create a new one anyway, (c) stop'. When the user picks (b) create new, agent then calls signoz_list_dashboard_templates and picks the JVM catalog entry. Runs the Step 3b-i.1 no-data probe via signoz_list_metrics with searchText='jvm'; finds JVM metrics flowing, proceeds silently to signoz_import_dashboard with the chosen template path. On success, reports the created dashboard's title, panel count, sections, and variables, and offers a customization or alerting follow-up.",
"expectations": [
"Agent calls signoz_list_dashboards before any write operation and surfaces the existing JVM dashboards as duplicates",
"Agent presents modify / create new / stop options to the user before proceeding",
"Only after the user picks 'create new', agent calls signoz_list_dashboard_templates to look for a JVM template",
"Agent runs a no-data probe (signoz_list_metrics with a jvm-related searchText) before importing",
"Agent calls signoz_import_dashboard with the JVM template path",
"Agent reports the created dashboard title, panel count, or section breakdown"
],
"files": []
},
{
"id": 1,
"eval_name": "template-no-data-postgres",
"prompt": "Create a dashboard for my PostgreSQL database.",
"expected_output": "Agent paginates signoz_list_dashboards and finds existing Postgres dashboards (e.g. 'PostgreSQL', 'Postgres overview'). Surfaces these duplicates and presents '(a) modify, (b) create new, (c) stop'. When the user picks (b) create new, agent calls signoz_list_dashboard_templates and picks the postgresql/postgresql.json catalog entry. Runs the Step 3b-i.1 no-data probe via signoz_list_metrics searchText='postgresql'; the probe returns empty. CRITICAL: agent MUST warn the user that no postgres data was found and offer create-anyway-or-stop, then wait for the user's choice. Must NOT silently call signoz_import_dashboard.",
"expectations": [
"Agent calls signoz_list_dashboards and surfaces the existing Postgres dashboards as duplicates with modify/create/stop options",
"Only after the user picks 'create new', agent calls signoz_list_dashboard_templates and picks the postgresql template",
"Agent runs a no-data probe via signoz_list_metrics with a postgresql-related searchText",
"Agent emits a warning message noting data is not found before calling signoz_import_dashboard",
"Agent waits for user confirmation before calling signoz_import_dashboard (does not import silently)"
],
"files": []
},
{
"id": 2,
"eval_name": "broad-request-apm-category",
"prompt": "I need an APM dashboard - what do you have?",
"expected_output": "Agent recognises the request is broad and exploratory. Calls signoz_list_dashboards and may surface existing APM-tagged dashboards (e.g. 'Frontend Service - APM', 'Cart Service - APM') as related context. Calls signoz_list_dashboard_templates (no arguments needed — full catalog returns in one call), reads the catalog in-context, filters to APM entries (apm-metrics, db-calls-monitoring, http-api-monitoring), presents the narrowed list to the user, and asks them to pick before calling signoz_import_dashboard. Must NOT pick a template silently when the request is ambiguous.",
"expectations": [
"Agent calls signoz_list_dashboard_templates and reads the catalog",
"Agent presents multiple APM template options to the user and asks them to choose",
"Agent does NOT silently call signoz_import_dashboard without user selection"
],
"files": []
},
{
"id": 5,
"eval_name": "needs-input-missing-scope-custom-build",
"prompt": "Build me a dashboard for some k8s pods.",
"expected_output": "Request is too vague (no cluster, namespace, or pod set). Agent must NOT proceed to signoz_create_dashboard with a guessed scope. Either asks the user to clarify scope (using the host's clarification mechanism) with concrete candidates discovered via signoz_get_field_values, or proposes the k8s catalog template and asks the user to confirm scope before importing. Must explicitly ask the user to clarify cluster/namespace before any write tool.",
"expectations": [
"Agent does NOT call signoz_create_dashboard or signoz_import_dashboard without first clarifying scope with the user",
"Agent stops before any write and asks a clarifying question about the cluster or namespace",
"Agent calls signoz_get_field_values or signoz_list_dashboard_templates to discover candidates before prompting"
],
"files": []
},
{
"id": 6,
"eval_name": "custom-build-scylladb-no-near-template-shortcut",
"prompt": "I need a dashboard for our ScyllaDB cluster — show me read/write latency, pending compactions, and storage load per node.",
"expected_output": "ScyllaDB has NO catalog template (the catalog ships database templates for postgresql, mysql, mongodb, redis, elasticsearch, clickhouse, couchdb, memcached, snowflake, and a JMX/Cassandra template — but no Scylla, which is a distinct product with its own Prometheus exporter and metric names). Agent calls signoz_list_dashboards (no Scylla dashboard). Calls signoz_list_dashboard_templates, reads the catalog in-context, finds no scylla match. CRITICAL: agent must NOT shortcut by importing a near-neighbor template (jmx/cassandra.json, couchdb, mongodb) just because Scylla is Cassandra-compatible — must fall through to custom build. Calls signoz_list_metrics with searchText='scylla' as a no-data probe; if empty (likely on this instance), agent must surface the no-data warning from Step 3b-ii.2 and ask the user whether to proceed before building. If the user says proceed, agent uses signoz_get_field_keys with fieldContext=resource and signoz_get_field_values to surface per-node attributes (host.name, k8s.cluster.name where present). Reads signoz://dashboard/* MCP resources. Builds a payload with Grid sections (Overview, Latency, Compaction, Storage), groupBy host.name on per-node panels, OTel attribute names. Emits plain-language summary (no JSON dump) BEFORE signoz_create_dashboard.",
"expectations": [
"Agent calls signoz_list_dashboard_templates, reads the catalog, and finds no matching scylla template",
"Agent does NOT call signoz_import_dashboard with a near-neighbor template (jmx/cassandra, couchdb, mongodb) as a shortcut",
"Agent calls signoz_list_metrics with a scylla-related searchText as part of the no-data probe",
"When the metric search returns empty, agent surfaces a no-data warning and waits for user confirmation before signoz_create_dashboard",
"Agent reads at least one signoz://dashboard/ MCP resource (instructions, widgets-instructions, widgets-examples, or examples) before building JSON",
"Agent uses OTel resource attribute names (host.name, not 'host') in filters or groupBy",
"Agent emits a plain-language summary (no JSON dump) of the dashboard before calling signoz_create_dashboard",
"Agent runs signoz_execute_builder_query as a per-panel dry-run for every authored panel before signoz_create_dashboard",
"Every saved metric query has limit=100 plus order on the composed spaceAggregation(timeAggregation(metricName)) expression (or __result), and every dry-run metric builder_query has that same limit and order"
],
"files": []
},
{
"id": 7,
"eval_name": "template-kafka-consumer-pressure",
"prompt": "Quick — just build me a Kafka consumer-health dashboard with fetch rate and commit latency, grouped by client. Don't ask a bunch of questions, I'm in the middle of an incident.",
"expected_output": "Agent still paginates signoz_list_dashboards under incident pressure, then calls signoz_list_dashboard_templates and finds the Kafka Server Monitoring template at kafka/kafka-dashboard.json. It probes kafka metric ingestion, previews the template, and imports it with signoz_import_dashboard rather than custom-building a parallel dashboard. Because the user requested specific consumer panels/grouping, it reports the imported dashboard id and hands those customizations to signoz-modifying-dashboards; it does not update inline from the creation skill.",
"expectations": [
"Agent finds the Kafka Server Monitoring catalog entry with path kafka/kafka-dashboard.json",
"Agent probes ingestion with signoz_list_metrics using a kafka-related searchText before import",
"Agent calls signoz_import_dashboard with kafka/kafka-dashboard.json and does not call signoz_create_dashboard",
"Agent hands the requested fetch-rate, commit-latency, and client-grouping changes to signoz-modifying-dashboards using the imported dashboard id",
"Agent does not call signoz_update_dashboard or signoz_patch_dashboard inline from the creation skill",
"Agent does NOT bypass the duplicate check or skip signoz_list_dashboards under the incident-pressure framing"
],
"files": []
},
{
"id": 8,
"eval_name": "custom-build-checkout-funnel-span-attributes",
"prompt": "Create a checkout funnel dashboard for our OpenTelemetry demo store. We instrument the cart, checkout, and payment services with span attributes — I want orders-placed count, total order revenue, and a card-type breakdown for charges. Discover the actual attribute names on the spans rather than guessing.",
"expected_output": "Business-KPI dashboard with NO catalog template — must custom build on traces signal, not metrics. Agent paginates signoz_list_dashboards (none). signoz_list_dashboard_templates returns no match. CRITICAL: agent must recognise the signal is TRACES with span-level attributes, not metrics — calls signoz_get_field_keys with signal='traces' to DISCOVER the actual attribute names rather than guessing. On this instance the attributes are app.order.id, app.order.amount, app.order.items.count, app.payment.amount, app.payment.card_type, app.payment.charged. Panel builder queries use signal='traces' with count() for orders-placed (filtered to spans where app.order.id EXISTS, or name = 'oteldemo.CheckoutService/PlaceOrder'); sum(app.order.amount) for revenue; groupBy app.payment.card_type for the card-type breakdown. Reads signoz://traces/query-builder-guide and the signoz://dashboard/* resources. groupBy entries use the discovered span-attribute key with fieldContext='attribute' (span attributes are not resource attributes). Emits plain-language summary (no JSON dump) before signoz_create_dashboard.",
"expectations": [
"Agent calls signoz_list_dashboard_templates and finds no matching template before custom build",
"Agent calls signoz_get_field_keys with signal=traces to discover the actual span-attribute names (does not invent attributes like 'order.value' or 'checkout.step' that may not exist on the instance)",
"Panel builder queries use signal='traces' (not 'metrics') for the headline panels since the data lives on spans",
"Agent uses span-attribute keys discovered via field_keys (e.g. app.order.amount, app.payment.card_type) in filter.expression or groupBy with fieldContext='attribute', not 'resource'",
"Agent uses a sum aggregation over a numeric order-amount attribute for the revenue panel (not just a count)",
"Agent emits a plain-language summary (no JSON dump) before calling signoz_create_dashboard",
"Agent runs signoz_execute_builder_query as a per-panel dry-run for every authored panel before signoz_create_dashboard",
"Every saved builder query and formula entry has a positive limit plus non-empty order, and every dry-run builder_query/builder_formula sends those same values",
"Agent calls signoz_create_dashboard with the custom-built payload"
],
"files": []
},
{
"id": 9,
"eval_name": "custom-build-slo-error-budget-formula",
"prompt": "Build me an SLO dashboard for the checkout service — 99.9% availability target, 28-day rolling window, show error-budget burn rate and remaining budget.",
"expected_output": "SLO/error-budget dashboard has no catalog template. Agent paginates signoz_list_dashboards, checks signoz_list_dashboard_templates, discovers span metrics and the checkout service, then builds in Builder mode. With A=error requests and B=total requests, availability is (1-A/B)*100, remaining error budget is ((1-A/B)-0.999)/(1-0.999)*100, and burn rate is (A/B)/(1-0.999). Because a panel holds exactly one query, each computed panel wraps A, B, and the formula in a single signoz/CompositeQuery. The payload does not invent timeRange or refresh fields: panels follow the viewer-selected global range, so the summary tells the user to select 28d after creation. Filters use service.name, and every panel is dry-run before signoz_create_dashboard.",
"expectations": [
"Agent calls signoz_list_dashboard_templates and finds no matching SLO/error-budget template before custom build",
"Agent calls signoz_list_metrics to discover signoz_calls_total or signoz_latency_bucket span metrics (or equivalent) before authoring",
"Agent computes availability from error and total request counts and uses the 0.1% error-budget allowance consistently in remaining-budget and burn-rate formulas",
"Agent does not add timeRange, defaultTimeRange, or refresh fields to the dashboard payload and does not smuggle a PromQL range selector into a Builder query",
"Agent uses service.name (dotted, OTel resource attribute) in builder-mode filters and variables — not the underscored span-metric label 'service_name' (this skill builds in builder mode only, never PromQL) and not the bare shorthand 'service'",
"Agent emits a plain-language summary before creation that explains the formulas and tells the user to select the 28d global viewer range",
"The actual signoz_execute_builder_query call contains builder_query envelopes named A and B plus a sibling builder_formula envelope named F1 with its expression; the panel holds those same three members inside one signoz/CompositeQuery",
"Every saved base metric query referenced by the formula has limit=10000 plus order on its primary aggregation, while every formula-input builder_query in the dry-run has the same limit and order; saved and dry-run formulas each have limit=100 and use __result desc",
"Each dry-run base query carries the service restriction under filter.expression",
"Agent runs signoz_execute_builder_query as a per-panel dry-run for every authored panel before signoz_create_dashboard"
],
"files": []
},
{
"id": 11,
"eval_name": "duplicate-modify-hands-off",
"prompt": "I want to add a slow-query panel to my existing PostgreSQL dashboard.",
"expected_output": "Agent calls signoz_list_dashboards, finds the existing PostgreSQL dashboard, and presents the modify/create/stop options. When the user picks (a) modify, agent must hand off to signoz-modifying-dashboards with the dashboard's id and the user's intent — must NOT call signoz_get_dashboard, signoz_patch_dashboard, or signoz_update_dashboard from this skill. The skill's scope-boundary guardrail forbids inline modification.",
"expectations": [
"Agent calls signoz_list_dashboards and surfaces the existing PostgreSQL dashboard",
"Agent presents modify / create new / stop options to the user",
"When the user picks modify, agent hands off to signoz-modifying-dashboards (does NOT call signoz_update_dashboard, signoz_patch_dashboard, or signoz_get_dashboard from this skill)"
],
"files": []
},
{
"id": 12,
"eval_name": "shape-check-no-stringify",
"prompt": "Create a dashboard with one timeseries panel showing service request rate for the checkout service. Walk me through the create call.",
"expected_output": "Agent walks the standard custom-build path: list_dashboards, list_dashboard_templates (no exact 'request rate one panel' template), discovery via list_metrics + get_field_keys, reads dashboard MCP resources. Builds the JSON, dry-runs the panel via signoz_execute_builder_query, then emits a plain-language summary and calls signoz_create_dashboard. CRITICAL: the payload sent to signoz_create_dashboard must have spec, panels, layouts, tags, and variables as native JSON objects/arrays — NOT as JSON.stringify'd strings. Required top-level fields are schemaVersion and spec; the title lives at spec.display.name and the top-level name is omitted so the server derives it. Each panel must include kind and spec with display, plugin, and exactly one query. Agent must NOT wrap any of these in a JSON string.",
"expectations": [
"Payload sent to signoz_create_dashboard has spec.layouts as an array of objects (not a stringified JSON string)",
"Payload sent to signoz_create_dashboard has spec.panels as a map of objects keyed by panel id (not an array, not a stringified JSON string)",
"Each panel in the payload includes kind, spec.display, spec.plugin with kind='signoz/TimeSeriesPanel', and spec.queries — and no panelTypes field appears",
"Each grid item's content.$ref is '#/spec/panels/<the panel id>', never a bare panel id",
"The panel's single query envelope includes its kind and the complete active Builder spec; inactive PromQL and ClickHouse payloads need not be authored",
"Agent runs signoz_execute_builder_query as a dry-run for the panel before signoz_create_dashboard",
"The saved panel query has a positive limit plus non-empty order, and its dry-run builder_query sends those same values",
"The payload sets schemaVersion='v6', omits the top-level name so the server derives it, and uses {key, value} tag objects"
],
"files": []
},
{
"id": 13,
"eval_name": "import-failure-falls-back-to-custom",
"prompt": "Create a dashboard for our Redis cluster.",
"expected_output": "Agent paginates signoz_list_dashboards and finds an existing 'Redis - Overview' dashboard. Surfaces the duplicate and presents '(a) modify, (b) create new, (c) stop'. When the user picks (b) create new, agent calls signoz_list_dashboard_templates and picks the Redis catalog entry, runs the no-data probe. Calls signoz_import_dashboard. Simulated failure: signoz_import_dashboard returns an error (e.g. catalog path not found, server-side validation failure). CRITICAL: agent must NOT silently retry or claim success — must surface the import error to the user and either (a) ask whether to fall through to a custom build using the same template's signals, or (b) stop. Must NOT skip directly to signoz_create_dashboard with a fabricated payload without telling the user the import failed.",
"expectations": [
"Agent calls signoz_list_dashboards and surfaces the existing Redis dashboard as a duplicate before any import attempt",
"Only after the user picks 'create new', agent calls signoz_import_dashboard with the Redis template path",
"On import failure, agent surfaces the error to the user and does NOT silently claim success",
"Agent either offers a custom-build fallback OR stops — does not silently call signoz_create_dashboard with a guessed payload"
],
"files": []
},
{
"id": 10,
"eval_name": "custom-build-multi-service-user-journey",
"prompt": "Create a user-journey dashboard for our OpenTelemetry demo checkout flow — I want per-hop latency and error rate across frontend, cartservice, checkoutservice, paymentservice, and shippingservice, so I can see where transactions slow down or fail. Discover the exact service names from the instance rather than hard-coding them.",
"expected_output": "Multi-service journey dashboard, no catalog template (apm covers single-service). Agent paginates signoz_list_dashboards (none). signoz_list_dashboard_templates returns the catalog; agent reads it in-context and finds no user-journey match. Falls through to custom build. Calls signoz_get_field_keys with signal=traces fieldContext=resource to confirm service.name; signoz_get_field_values for service.name (or signoz_list_services) to verify the named services exist on the instance — the actual names are 'frontend', 'cartservice', 'checkoutservice', 'paymentservice', 'shippingservice' (plus others). Reads signoz://traces/query-builder-guide and the signoz://dashboard/* resources. Builds a payload with: filter.expression restricting service.name IN the discovered names; per-service latency panels (p50/p95/p99) with groupBy service.name (fieldContext resource); per-service error-rate panels using an A*100/B formula inside a signoz/CompositeQuery; a request-rate panel grouped by service.name. Variables are ListVariables with signoz/DynamicVariable plugins for service.name and the environment key that exists on the instance. Emits plain-language summary (no JSON dump) before signoz_create_dashboard.",
"expectations": [
"Agent calls signoz_list_dashboard_templates and finds no matching user-journey template before custom build",
"Agent calls signoz_list_services or signoz_get_field_values for service.name (signal=traces) to verify which services actually exist on the instance before authoring",
"Agent uses an IN-list filter.expression on service.name with the actual discovered service names (e.g. 'cartservice', 'checkoutservice', 'paymentservice', 'shippingservice') — not invented short names like 'cart' or 'payment'",
"Every groupBy entry is {name: 'service.name', fieldContext: 'resource', fieldDataType: 'string', signal: 'traces'} in both the saved panel and the dry-run, with no key/dataType/type aliases anywhere",
"Agent includes a formula combining error count and total count (A*100/B or equivalent) for per-service error rate, wrapped with its base queries in one signoz/CompositeQuery",
"Agent reads at least one signoz://dashboard/ or signoz://traces/ MCP resource before authoring JSON",
"Agent emits a plain-language summary (no JSON dump) before calling signoz_create_dashboard",
"Agent calls signoz_create_dashboard only after every query-bearing panel has an actual signoz_execute_builder_query dry-run",
"Formula-input base queries use limit=10000 with count() ordering and the formula result uses limit=100 with __result ordering, identically in the saved panel and the dry-run"
],
"files": []
},
{
"id": 14,
"eval_name": "custom-build-logs-signal-volume-and-errors",
"prompt": "Create a dashboard for our application log analytics — I want a timeseries of log volume per service, a value panel showing total ERROR/WARN count in the last hour, and a table of top 10 services by log volume broken down by severity. Use logs as the data source.",
"expected_output": "Logs-signal dashboard with NO catalog template — must custom build on the LOGS signal, not metrics or traces. Agent paginates signoz_list_dashboards (no log-analytics dashboard). signoz_list_dashboard_templates returns no match. Falls through to custom build. CRITICAL: agent must recognise the signal is LOGS (signal='logs' in each builder query spec). Calls signoz_get_field_keys with signal=logs to discover available log attributes (severity_text, service.name as resource attr, body, severity_number). Optionally calls signoz_aggregate_logs at discovery time to gauge log volume; the mandatory per-panel dry-run still goes through signoz_execute_builder_query. Reads signoz://dashboard/widgets-instructions, signoz://dashboard/widgets-examples, signoz://dashboard/examples, and signoz://logs/query-builder-guide. Builds three panels: (a) a signoz/TimeSeriesPanel with a time_series query, count(), groupBy service.name, legend {{service.name}}; (b) a signoz/NumberPanel with a scalar query, count(), filter.expression restricting severity_text IN ('ERROR','WARN','FATAL') — NO groupBy on a value panel; (c) a signoz/TablePanel with a scalar query, count() aliased 'Log Count', groupBy service.name AND severity_text. Emits plain-language summary (no JSON dump) before signoz_create_dashboard.",
"expectations": [
"Agent calls signoz_list_dashboard_templates and finds no matching log-analytics template before custom build",
"Agent uses signal='logs' in every builder query (not 'metrics' or 'traces') for the log panels",
"Agent calls signoz_get_field_keys with signal=logs to discover log attributes before authoring",
"Agent reads signoz://logs/query-builder-guide before authoring Builder log queries",
"Agent uses severity_text in a filter.expression for the ERROR/WARN value panel (not a guessed key like 'severity' or 'level')",
"Value panel is a signoz/NumberPanel with a scalar query and does NOT include groupBy (would produce multiple values instead of one)",
"Table panel is a signoz/TablePanel and includes groupBy with at least 'service.name' AND 'severity_text' to satisfy the per-service / per-severity breakdown",
"Timeseries panel includes legend with {{service.name}} matching the groupBy key",
"Aggregation columns are named with the aggregation's alias field, not an 'as ...' clause inside the expression",
"Agent emits a plain-language summary (no JSON dump) before calling signoz_create_dashboard",
"The actual signoz_execute_builder_query call places the severity restriction only in filter.expression and contains no filters field",
"The actual signoz_create_dashboard call keeps that same filter.expression, with no filters.items structure anywhere",
"Agent runs signoz_execute_builder_query (with signal=logs in each builder_query spec) as a per-panel dry-run before signoz_create_dashboard",
"Every saved log aggregate query has limit=100 plus order on its primary count aggregation, and every dry-run sends that same limit and order"
],
"files": []
},
{
"id": 15,
"eval_name": "custom-build-list-panel-select-fields",
"prompt": "Create a dashboard with one list panel that shows the most recent error traces — I want columns for service.name, span name, duration, http method, and response status code. Sort newest first.",
"expected_output": "Single-list-panel dashboard. No catalog template. Agent paginates signoz_list_dashboards (none). signoz_list_dashboard_templates returns no match. Falls through to custom build. Agent MUST: (a) read signoz://dashboard/widgets-examples for the list-panel shape; (b) author a signoz/ListPanel whose plugin spec carries selectFields for the displayed columns, with a single query of kind 'raw' whose signoz/BuilderQuery spec has signal='traces', filter.expression 'has_error = true', the same selectFields, order [{key:{name:'timestamp'},direction:'desc'}], and an explicit limit. Each selectFields entry uses name (never 'key'), fieldContext (resource for service.name; span for name/duration_nano/http_method/response_status_code), fieldDataType where applicable, and signal='traces'. Emits a plain-language summary (no JSON dump) before signoz_create_dashboard, after a per-panel signoz_execute_builder_query dry-run.",
"expectations": [
"Agent reads signoz://dashboard/widgets-examples (or widgets-instructions) before authoring the list-panel JSON",
"List panel uses plugin.kind='signoz/ListPanel', a query envelope of kind 'raw', and order on timestamp desc",
"selectFields is present on both the panel plugin spec and the query spec, and uses 'name' (not 'key') for each column entry",
"Each selectFields entry includes fieldContext (resource for service.name; span for span-level columns) and signal='traces'",
"Filter on the list panel restricts to error spans (e.g. has_error = true, or equivalent)",
"Agent does NOT use groupBy on the list panel (lists are unaggregated; groupBy would produce wrong behavior)",
"Agent emits a plain-language summary (no JSON dump) before signoz_create_dashboard",
"The actual signoz_execute_builder_query call uses requestType='raw' and sends the panel's query spec unchanged inside a builder_query envelope",
"The dry-run builder_query.spec contains none of pageSize, orderBy, or selectColumns, and no groupBy/selectFields entry contains dashboard-only key/dataType/type",
"The actual signoz_create_dashboard call retains that same limit, order, and selectFields — no pageSize, orderBy, or selectColumns anywhere",
"Agent runs signoz_execute_builder_query as a dry-run for the list panel before signoz_create_dashboard"
],
"files": []
},
{
"id": 16,
"eval_name": "custom-build-multi-panel-types-mixed",
"prompt": "Create an APM-style dashboard for the checkoutservice with: (1) a row section called 'Overview', (2) three value panels in a KPI row (total requests, error rate %, p99 latency), (3) a row section called 'Breakdowns', (4) a pie panel showing request distribution by operation, (5) a bar panel showing error count by status code, and (6) a table panel with operation, request count, and avg latency.",
"expected_output": "The user asks for 'row sections'; there is no row panel type, so those map onto separate Grid entries in spec.layouts. Multi-panel-type dashboard exercising sections and four panel plugin kinds. No catalog template — custom build on the traces signal. CRITICAL structure: sections are Grid layouts — there is no row panel type and no panelMap, so the payload has two entries in spec.layouts, each a {kind:'Grid', spec:{display:{title}, items:[...]}} with 'Overview' and 'Breakdowns' as titles. Panel rules: value panels are signoz/NumberPanel with a scalar query and NO groupBy; the pie panel is signoz/PieChartPanel with a scalar query, groupBy, and a matching legend; the bar panel is signoz/BarChartPanel with groupBy and legend; the table panel is signoz/TablePanel with groupBy and aggregation aliases for column names; the error-rate KPI wraps its base queries and the A*100/B formula in one signoz/CompositeQuery with the base queries disabled. Agent reads signoz://dashboard/widgets-examples and signoz://dashboard/examples, discovers the operation attribute (the 'name' span column, or rpc.method/http.route) via signoz_get_field_keys with signal=traces, dry-runs every panel, and emits a plain-language summary before signoz_create_dashboard.",
"expectations": [
"Dashboard payload includes exactly the two requested Grid sections titled Overview and Breakdowns, with no row panel and no panelMap",
"Value panels use plugin.kind='signoz/NumberPanel' and do NOT include groupBy",
"Pie panel uses plugin.kind='signoz/PieChartPanel' and includes groupBy AND legend with {{<key>}} matching the groupBy key",
"Bar panel uses plugin.kind='signoz/BarChartPanel' (not the timeseries kind) and includes groupBy on status_code or equivalent",
"Table panel uses groupBy and names its aggregation columns with the alias field",
"The actual table-panel dry-run emits separate V5 aggregation objects for count() and avg(duration_nano), preserving aliases, rather than one combined expression",
"Error-rate value panel uses an A*100/B formula with disabled:true on base queries, all inside one signoz/CompositeQuery envelope",
"Agent reads signoz://dashboard/widgets-examples before authoring (panel shapes vary per plugin kind)",
"The Overview Grid's items reference exactly the three KPI panels at a consistent y, and the Breakdowns Grid's items reference exactly the pie, bar, and table panels",
"Every grid item satisfies 0 <= x < 12, 1 <= width <= 12, and x+width <= 12, and no two items overlap",
"Every panel id in spec.panels is referenced by exactly one grid item content.$ref, and every $ref resolves to an existing panel",
"Agent emits a plain-language summary (no JSON dump) before signoz_create_dashboard",
"Agent runs signoz_execute_builder_query as a per-panel dry-run for every authored panel before signoz_create_dashboard",
"The error-rate formula's saved and dry-run base queries use limit=10000 with count() ordering, its formula result uses limit=100 with __result ordering, and standalone panels remain at limit=100"
],
"files": []
},
{
"id": 17,
"eval_name": "custom-build-variable-application-prompt",
"prompt": "Create a dashboard for our backend services with three panels: (1) request rate timeseries grouped by service, (2) p99 latency timeseries grouped by service, and (3) a global system health panel showing total error count across all services. I want a service.name dropdown variable to filter the dashboard.",
"expected_output": "Agent follows the custom-build discovery and resource-read flow, proposes a ListVariable backed by a signoz/DynamicVariable plugin on service.name, lists all three planned panels, and asks whether the variable should apply to all or a subset. It stops before wiring $service_name, dry-running, or creating the dashboard.",
"expectations": [
"Agent proposes a ListVariable whose plugin.kind is 'signoz/DynamicVariable' with plugin.spec.name='service.name' and a signal",
"Agent explicitly lists the panels and asks which the service.name variable should apply to before wiring any query",
"Agent does not wire $service_name into any panel before the user answers",
"Agent does not call signoz_execute_builder_query or signoz_create_dashboard before the user answers"
],
"files": []
},
{
"id": 18,
"eval_name": "duplicate-dashboard-on-second-page",
"prompt": "Create a PostgreSQL dashboard for prod.",
"expected_output": "In an offline simulated tool trace using the attached fixture, the first list page has no PostgreSQL match but reports total=51 against a 50-row page. Agent raises offset to 50, finds the existing dashboard on page two, and presents modify / create new / stop before any template lookup or write.",
"expectations": [
"The simulated trace invokes signoz_list_dashboards again with offset=50, derived from total and the returned page length, using an accepted integer or string value",
"Agent surfaces the second-page PostgreSQL dashboard as a duplicate",
"Agent presents modify / create new / stop before signoz_list_dashboard_templates, signoz_import_dashboard, or signoz_create_dashboard",
"Agent performs no dashboard write before the user chooses"
],
"files": ["evals/files/duplicate-on-second-page.json"]
},
{
"id": 19,
"eval_name": "custom-build-variable-application-selected-panels",
"prompt": "Apply it to Request Rate by Service and P99 Latency by Service, but keep Global System Health unfiltered.",
"expected_output": "Using the prior-turn fixture, agent wires $service_name only into the two selected panels' filter.expression, leaves Global System Health global, and produces an offline simulated dry-run trace for all three queries with the representative variable value. It prepares the plain-language summary but does not create a live dashboard in this offline eval.",
"expectations": [
"Variable reference $service_name appears in the Request Rate by Service and P99 Latency by Service filter expressions under draft.spec.panels",
"Variable reference $service_name does not appear in the Global System Health filter expression under draft.spec.panels",
"The prepared draft keeps spec.variables as an array and spec.panels as a map, with no widgets array or name-keyed variables object",
"Variable references use $service_name in filter expressions rather than bare service_name",
"The simulated trace runs signoz_execute_builder_query for every panel query with service_name=checkoutservice",
"Every prepared panel query has limit=100 plus order on its primary aggregation, and every dry-run builder_query sends that same limit and order",
"Agent prepares a plain-language summary that states which panels the variable filters",
"Agent does not call signoz_create_dashboard in the offline eval"
],
"files": ["evals/files/service-variable-scope-context.json"]
},
{
"id": 20,
"eval_name": "custom-build-having-array-validation-gap",
"prompt": "Create a logs table for ERROR logs grouped by service.name that shows only groups with count() greater than 10.",
"expected_output": "Agent discovers the log-side severity key, saves the panel with one filter.expression for the ERROR restriction and the group threshold as having.expression = 'count() > 10', then dry-runs that same spec. The saved having executes as stored, so a successful dry-run validates the panel — there is no clause-array form and no parity gap to warn about.",
"expectations": [
"The create payload keeps the ERROR restriction in filter.expression with the discovered severity key, and sends no filters.items structure",
"The actual signoz_execute_builder_query call sends that same filter.expression and no filters field",
"The saved panel expresses the threshold as having.expression = 'count() > 10', never a clause array such as [{columnName:'count()',op:'>',value:10}]",
"The dry-run sends the identical having.expression, and its success is treated as validating the saved panel rather than as a diagnostic probe",
"No having clause array is passed to signoz_execute_builder_query",
"The saved log query has limit=100 plus order on count(), and the dry-run sends that same limit and order",
"Agent calls signoz_create_dashboard only after the dry-run of the complete HAVING query succeeds"
],
"files": []
},
{
"id": 21,
"eval_name": "unsupported-builder-function-blocks-lossy-validation",
"prompt": "Create a metrics dashboard panel that uses the Builder timeShift function to compare the current series with one hour ago.",
"expected_output": "Agent reads the current signoz_execute_builder_query schema. If that schema cannot represent Builder functions, it surfaces the exact validation gap instead of deleting timeShift and claiming a successful dry-run; it waits for explicit acceptance before any unvalidated create.",
"expectations": [
"Agent checks the current signoz_execute_builder_query schema for functions support",
"When functions is unsupported, no actual dry-run silently omits timeShift and no such lossy call is described as successful",
"Agent identifies the panel as unvalidated and asks for explicit acceptance before signoz_create_dashboard",
"Agent preserves the requested functions field in the saved panel rather than weakening the query to make a dry-run pass"
],
"files": []
},
{
"id": 22,
"eval_name": "formula-order-limit-bounded-execution",
"prompt": "Create a table of the top 5 services by error percentage using A / B * 100 as formula F1, ordered by F1 descending.",
"expected_output": "Agent authors one signoz/CompositeQuery holding base queries A and B plus formula F1, gives the formula its deliberate top-5 bound and __result ordering, raises the base queries to 10000 so the top-5 cut happens after the formula rather than before it, dry-runs the complete composite, and creates only after that bounded execution succeeds. Because the saved query spec is already the execution spec, the same limits and order appear in both — there is no orderBy-to-order translation.",
"expectations": [
"The saved formula member keeps name='F1', expression='A / B * 100', limit=5, and order=[{key:{name:'__result'},direction:'desc'}]",
"Every saved base query uses limit=10000 plus order on its primary aggregation so the deliberate top-5 bound is applied only after the formula",
"The actual signoz_execute_builder_query call contains bounded builder_query siblings A and B plus builder_formula F1 in the same compositeQuery.queries array",
"The actual builder_formula spec is named F1, preserves expression='A / B * 100', has limit=5 and order=[{key:{name:'__result'},direction:'desc'}], and contains no queryName or orderBy",
"Every execution builder_query formula input uses limit=10000 plus non-empty order and contains no dashboard orderBy",
"Agent calls signoz_create_dashboard only after the complete bounded formula dry-run succeeds"
],
"files": []
},
{
"id": 23,
"eval_name": "trace-operator-sibling-envelope-contract",
"prompt": "Create a traces graph that counts flows where checkout span query A is a direct ancestor of payment span query B, using trace operator T1 with expression A => B.",
"expected_output": "Agent authors base trace queries A and B plus a trace operator T1 as three members of one signoz/CompositeQuery on a single panel, dry-runs the complete composite as sibling builder_query, builder_query, and builder_trace_operator envelopes, and only then creates the dashboard. The trace operator's saved spec and its execution spec are the same object — name, expression, and trace aggregations — so nothing is renamed on the way to the dry-run.",
"expectations": [
"The actual signoz_create_dashboard payload keeps base queries A and B plus T1 as three members of one signoz/CompositeQuery, with the operator named T1 and expression='A => B'",
"The actual signoz_execute_builder_query payload contains builder_query A, builder_query B, and builder_trace_operator T1 as siblings in the same compositeQuery.queries array",
"The builder_trace_operator spec has name='T1', expression='A => B', and trace aggregations such as [{expression:'count()'}]",
"The builder_trace_operator spec is passed through the current MCP raw-preservation contract and is never coerced into builder_query",
"Saved base trace queries A and B each have limit=100 plus order on count(), and their dry-run builder_query siblings send those same values",
"Neither the saved payload nor the execution spec contains queryName, dataSource, pageSize, orderBy, selectColumns, or queryTraceOperator",
"Agent does not call signoz_create_dashboard until the complete trace-operator execution call succeeds"
],
"files": []
},
{
"id": 24,
"eval_name": "saved-not-in-to-execution-not-in",
"prompt": "Create a logs table grouped by service that excludes checkout and frontend services.",
"expected_output": "Agent discovers the actual log-side service field, saves the panel with the canonical groupBy shape and one filter.expression using NOT IN, then dry-runs that same spec with a non-empty groupBy name over a short absolute Unix-ms window.",
"expectations": [
"Agent calls signoz_get_field_keys with signal=logs and uses the returned service field name rather than assuming service.name exists on logs",
"The signoz_create_dashboard payload expresses the exclusion as one filter.expression using NOT IN, with no filters.items structure and no NOT_IN underscore spelling",
"The actual signoz_execute_builder_query call uses filter.expression with NOT IN and sends no filters field",
"The actual dry-run outer query contains start and end as JSON integer Unix-millisecond values and spans a short representative window (roughly the last 30-60 minutes, not a multi-hour or multi-day display range)",
"Every execution groupBy entry has a non-empty name equal to the discovered attribute key and contains no dashboard-only key/dataType/type aliases",
"The saved filter.expression and the dry-run filter.expression are identical — the panel stores the execution spec"
],
"files": []
},
{
"id": 25,
"eval_name": "custom-build-raw-log-stable-order",
"prompt": "Create a dashboard with one list panel showing the 20 most recent ERROR logs, including timestamp, service.name, severity_text, and body columns.",
"expected_output": "Agent builds a signoz/ListPanel over the logs signal with an intentional 20-row bound, stable newest-first ordering across two keys, and selectFields for the requested columns. The single query envelope has kind='raw'. The dry-run reuses that spec with requestType='raw'.",
"expectations": [
"The saved list panel uses plugin.kind='signoz/ListPanel', signal='logs', limit=20, and order=[{key:{name:'timestamp'},direction:'desc'},{key:{name:'id'},direction:'desc'}]",
"The saved list panel includes selectFields for timestamp, service.name, severity_text, and body and filters to ERROR logs",
"The actual signoz_execute_builder_query call uses requestType='raw' and a logs builder_query with limit=20",
"The dry-run builder_query sends that same two-key order and contains no dashboard orderBy or pageSize",
"Agent calls signoz_create_dashboard only after the bounded raw-log dry-run succeeds"
],
"files": []
}
]
}
SKILL.md
---
name: signoz-creating-dashboards
description: >
Create a new SigNoz dashboard from a natural-language intent: import a
curated template (PostgreSQL, Redis, JVM, k8s, hostmetrics, APM, LLM,
etc.) when one fits, or build a custom dashboard from scratch with
metric / trace / log panels. Make sure to use this skill whenever the
user says "create a dashboard for…", "set up monitoring for…",
"build me a dashboard…", "I need observability for…", "import a
dashboard template", or asks to track / visualize a service, database,
cluster, or AI/LLM platform, even if they don't explicitly say
"dashboard". Also use it when someone wants to "monitor", "watch", or
"see metrics for" a technology and the natural answer is a dashboard.
argument-hint: <natural-language dashboard intent>
---
# Dashboard Create
## Prerequisites
This skill calls SigNoz MCP server tools (`signoz_create_dashboard`,
`signoz_list_dashboards`, `signoz_list_dashboard_templates`,
`signoz_import_dashboard`, `signoz_get_dashboard`,
`signoz_update_dashboard`, `signoz_list_metrics`,
`signoz_get_field_keys`, `signoz_get_field_values`,
`signoz_aggregate_logs`, `signoz_aggregate_traces`, etc.).
Before running the workflow, confirm the `signoz_*` tools are
available. If they are not, the SigNoz MCP server is not installed or
configured; run `signoz-mcp-setup` first to initialize or repair the MCP
connection. Do not fall back to raw HTTP calls or fabricate dashboard JSON
without the MCP tools.
## When to use
Use this skill when the user wants to:
- Create, set up, or build a new dashboard.
- "Monitor" or "set up observability" for a service, database,
infrastructure component, or AI/LLM platform.
- Import a curated dashboard template.
- Visualize a set of metrics / traces / logs together on one screen.
Do NOT use when the user wants to:
- Modify an existing dashboard → `signoz-modifying-dashboards`.
- Understand what an existing dashboard shows → `signoz-explaining-dashboards`.
- Run a one-off query without persisting it → `signoz-generating-queries`.
## Required inputs (strict)
Dashboard creation is a write operation. Guessing here clutters the
shared workspace with empty or wrongly-scoped dashboards someone else has
to clean up. The skill enforces a soft input contract; most fields have
sensible defaults, but a few cannot be guessed:
| Input | Required | Source if missing |
|---|---|---|
| Dashboard intent (NL goal) | yes | `$ARGUMENTS` or recent user turn |
| Technology / domain (e.g. PostgreSQL, Redis, "payment pipeline") | yes | parse from intent; otherwise ask |
| Confirmation to create (plus the modify-or-create choice when duplicates exist) | yes | ask the user (Step 2); still required with zero duplicates and under stated urgency |
| Resource scope for custom builds (service / namespace / cluster) | yes for custom builds | discover via `signoz_get_field_keys` + `signoz_get_field_values`; fall back to a dashboard variable |
| Specific metrics / signals for custom builds | inferred | derive from technology + MCP `signoz://dashboard/*` resources; surface in preview |
| Layout | inferred | apply defaults (see "Defaults" below) |
If a required input is missing and cannot be discovered, **stop before
calling any write tool** and ask the user. The host application decides
how the question is surfaced (a structured clarification tool, inline
`<assistant_question>` tags, an interactive prompt, etc.); follow the
host's UI rendering rules.
What to include in the question:
- **What is missing**: name the input concretely (e.g. "no service or
cluster specified for the custom build").
- **Candidate lists** populated from your discovery calls: concrete
values per attribute the user can pick from. Example shape:
`service.name` → `frontend`, `checkout`, `payments`, `inventory`;
`k8s.cluster.name` → `prod-us-east-1`, `staging`.
- **Allow free-form input** so the user can name a value you didn't
surface.
In autonomous mode (no human), escalate to the caller or fill the gap
from upstream context. Either way, do not proceed to
`signoz_create_dashboard` / `signoz_import_dashboard` with
a guessed value.
## Workflow
The create path starts **duplicate check → modify-or-create choice → template
lookup**. A matching template uses **no-data probe → preview → import**;
a custom build uses **no-data probe → build → per-panel dry-run → preview →
create**. Template lookup is internal; the user's only upfront choices are
modify or create.
### Step 1: Check for duplicates
Call `signoz_list_dashboards`. Most installs fit in the default
page (`limit=50`); narrow with the `filter` argument when the wording is
distinctive (see `signoz://dashboard/list-filter-guide`), and page by `offset`
until you have covered `total`; the schema accepts
integer or string `limit` / `offset` values.
**Match by relevance** Compare each existing
dashboard's lowercased `spec.display.name`, `.description`, and `tags` against the
user's technology/domain. Surface only matches a human would recognize
as the same thing: a "redis" dashboard does not match a "postgresql"
request just because both have a `database` tag. Collect each match's
`spec.display.name`, `id`, and `createdAt` for the next step.
### Step 2: Ask the user (modify or create)
Present exactly two options (no template-import as a separate top-level
choice; that's an internal decision in Step 3b):
- **Duplicates found:** "There are already these similar dashboards:
[list with name, id, created-at]. Want me to (a) modify one of
these, (b) create a new dashboard anyway, or (c) stop?"
- **No duplicates:** "I'll create a new dashboard for this. Proceed?"
(No "modify" option when there's nothing to modify.)
Wait for the user's choice. "modify" → Step 3a. "create new" / confirm
→ Step 3b. "stop" → stop.
### Step 3: Create or modify
#### Step 3a: Modify an existing dashboard
Hand off immediately to `signoz-modifying-dashboards` with the chosen
dashboard id and the user's intent. Do not call
`signoz_update_dashboard` or `signoz_patch_dashboard` from this skill;
modification is out of scope. (See "Scope boundary" in Guardrails.)
#### Step 3b: Create a new dashboard
Run the template lookup first. The user has already agreed to create
new; the lookup decides *how* we build it.
Call `signoz_list_dashboard_templates` once with no arguments.
The full catalog (~95 entries) returns in a single call; read it
in-context and pick the best match for the user's intent. When several
entries plausibly fit, present the top 3–5 and let the user choose.
Branch on the result:
- **Single clear template match**: proceed to Step 3b-i (template
import). Briefly tell the user "I found a pre-built [title] template
and will use it" so they know what's being created; do not block on
yes/no.
- **Multiple plausible matches**: present them and ask the user to
pick. Once picked, proceed to Step 3b-i.
- **Template matches the technology but not the requested signals**:
common for any specific ask ("Kafka, but I want consumer fetch rate by
client"). Not "no template": import it, then hand the extra panels to
`signoz-modifying-dashboards` with the new id. Building from scratch
discards the curated baseline for no gain.
- **No template**: proceed to Step 3b-ii (custom build). That means no
catalog entry for the technology, not an entry that looks imperfect or
aimed at a different metric family. Template bodies are not readable
before import, so a suspected mismatch is only a hypothesis, and the
Step 3b-i.1 probe sits *inside* the import path, so it cannot justify
leaving that path. Probe first, then decide.
#### Step 3b-i: Import the template
> **Tool guardrail** The only template tools are
> `signoz_list_dashboard_templates` and
> `signoz_import_dashboard`. Do not shell out, fetch raw GitHub
> URLs, or invent other tool names.
> `signoz_import_dashboard` takes the template `path` from the
> catalog entry and creates the dashboard in one call, so you do not need
> to fetch the JSON yourself or call `signoz_create_dashboard`
> afterwards.
##### Step 3b-i.1: Pre-flight no-data probe (fail fast)
Before calling `signoz_import_dashboard`, confirm the template's
signals are actually being ingested. The most common silent failure for
template imports is "the template imports cleanly but every panel reads
'No data' because the technology isn't being scraped": the user only
discovers it after clicking through to a useless dashboard.
Since we don't fetch the template body up front, base the probe on the
catalog entry's `category`, `title`, and `keywords` plus the user's
stated technology. Pick up to ~5 representative signals and check
them; keep the total small:
- **Metric-based templates** (most infra/runtime templates): call
`signoz_list_metrics` with `searchText` set to the technology
prefix (e.g. `searchText="postgresql"`). Empty result → metric family
is not being ingested. *Early out:* if this returns empty, declare
"None present" and skip the rest of the metric probes; they will all
return zero. Use `timeRange` for a relative window, or pass
`start`/`end` (unix-ms strings) when you need an exact window instead
of the server default.
- **Trace-based templates** (APM-style): call
`signoz_aggregate_traces` with `aggregation=count`,
`timeRange=1h`. No filter is needed for the "is anything flowing"
probe; adding `filter="service.name EXISTS"` is fragile and
unnecessary. Zero count → no traces flowing.
- **Log-based templates**: call `signoz_aggregate_logs` with
`aggregation=count`, `timeRange=1h`, no filter. Zero count → no logs.
- **Variable values** (when the template clearly relies on a resource
attribute, e.g. `service.name`, `k8s.cluster.name`): call
`signoz_get_field_values` to confirm there are values to pick
from. A dashboard whose top-level dropdown is empty is barely
better than one full of empty panels.
Branch on the probe result:
- **All signals present** → proceed silently to Step 3b-i.2.
- **Some present, some missing** → list which are missing and ask the
user to confirm before continuing. Many templates are useful even with
partial coverage; let them decide.
- **None present** → tell the user no data was found for this
technology in the probe window, explain the dashboard will show "No
data" until ingestion is set up, and offer to create it anyway or
stop. Wait for the user's choice.
This probe is cheap (a handful of queries, ~hundreds of ms total), and
catching the no-data case early avoids the worst UX failure mode of the
template path.
##### Step 3b-i.2: Preview, import, report
1. **Preview** Tell the user what's about to happen in one short
paragraph: which template (`title`, `path`), what category, what the
probe found. In autonomous mode the consumer proceeds; in interactive
mode the human can intervene.
2. **Import** Call `signoz_import_dashboard` with the `path`
from the chosen catalog entry (e.g. `postgresql/postgresql.json`).
The server fetches the JSON, validates it, and creates the dashboard
in one call.
3. **Report** Read the response and tell the user the dashboard's
title, panel count, and section breakdown. Surface the dashboard's
variables ("filter by `service.name`", "filter by
`k8s.cluster.name`") so the user knows what knobs they have. Offer
two follow-ups: "Want me to adjust panels, layout, or variables?"
and "Want me to wire alerts for any of these signals?
(`signoz-creating-alerts`)".
4. **Customization handling** If the user asks for any change to the
imported dashboard, hand off to `signoz-modifying-dashboards` with
the new dashboard's id and the requested changes. Do not call
`signoz_update_dashboard` from this skill.
#### Step 3b-ii: Custom build (no template, or import failed)
Run this path when the Step 3b template lookup found no match, the user
explicitly rejected the suggested template, or
`signoz_import_dashboard` failed.
##### Step 3b-ii.1: Gather requirements
Ask the user (skip questions whose answer is already clear from intent):
1. **Signals**: metrics, traces, logs, or a combination.
2. **Specific signals**: which metrics, which span attributes, which
log severities matter most.
3. **Resource scope**: which service(s), namespace(s), cluster(s), or
environment(s).
4. **Variables**: what should be a dropdown vs. a hard-coded filter
(typical: `service.name`, `deployment.environment.name`,
`k8s.cluster.name`).
5. **Sections**: group panels into Overview / Latency / Errors /
Saturation, or another structure that fits the domain.
If the user is non-specific ("just make me something useful for X"),
apply the defaults table below and surface them in the preview.
##### Step 3b-ii.2: Discover names and probe data
The MCP guideline applies: **always prefer resource-attribute filters**.
Before authoring panels, confirm the names you'll use exist and emit
data:
1. **Metrics**: call `signoz_list_metrics` with `searchText`
tied to the technology (e.g. `searchText="postgresql"`) to get the
*exact* OTel metric names. Catalog presence ≠ data flowing; for
any metric you intend to use, follow up with `signoz_query_metrics`
on a representative window to confirm it actually has datapoints.
2. **Resource attributes**: call `signoz_get_field_keys` with
`fieldContext=resource` for the relevant signal to enumerate
available attributes; call `signoz_get_field_values` on the
ones you'll use as variables to confirm concrete values exist. Note
that the live data may use older OTel semconv (e.g.
`deployment.environment` rather than `deployment.environment.name`);
always trust the discovered key over the one in the defaults
table.
If **none** of the discovered signals return data, tell the user the
dashboard's data isn't being ingested yet, explain the panels will
show "No data" until ingestion is set up, and offer to build anyway
or stop. Wait for the user's choice before building.
##### Step 3b-ii.3: Read the dashboard MCP resources
These are the source of truth for the JSON schema, panel types, query
builder shape, and layout rules; do not transcribe schema text into
this skill, it will rot out of sync with the server. Read the core
resources before authoring panel JSON.
> **Fallback when the MCP resource-reader is unavailable** Some MCP
> client harnesses do not expose a resource-reading tool. If you
> cannot read `signoz://...` URIs in this session, fall back to
> `signoz_list_dashboards` + `signoz_get_dashboard` on
> an existing dashboard of the same signal type (metrics / traces /
> logs) and read its `spec.panels` map for worked panel shapes.
- `signoz://dashboard/instructions`: title, tags, description,
layout, variables.
- `signoz://dashboard/widgets-instructions`: 7 panel types and layout
rules.
- `signoz://dashboard/widgets-examples`: complete panel configs with
all required fields (the most important resource; every panel must
include `kind`, `spec.display`, `spec.plugin`, and exactly one query).
- `signoz://dashboard/examples`: whole create payloads with panels,
layouts, and variables assembled.
- `signoz://dashboard/query-builder-example`: query builder reference.
Add signal-specific resources as needed:
- Metrics (PromQL): `signoz://promql/instructions`.
Saved PromQL may reference declared dashboard `$var` variables, but
`signoz_execute_builder_query` does not expand them: substitute representative
literals only for dry-runs, never in saved panels. Grafana-only
`$__rate_interval` / `$__interval` are invalid. Dotted OTel metric names use
Prometheus 3.x UTF-8 selectors such as `{"metric.name.with.dots"}`.
- Metrics (ClickHouse): `signoz://dashboard/clickhouse-schema-for-metrics`
+ `signoz://dashboard/clickhouse-metrics-example`.
- Metrics (Query Builder aggregation rules):
`signoz://metrics-aggregation-guide`: required for picking valid
`timeAggregation` / `spaceAggregation` per metric type.
- Traces (Query Builder): `signoz://traces/query-builder-guide`.
- Logs (Query Builder): `signoz://logs/query-builder-guide`.
- Traces (ClickHouse): `signoz://dashboard/clickhouse-schema-for-traces`
+ `signoz://dashboard/clickhouse-traces-example`.
- Logs (ClickHouse): `signoz://dashboard/clickhouse-schema-for-logs`
+ `signoz://dashboard/clickhouse-logs-example`.
##### Step 3b-ii.4: Build the dashboard JSON
Follow the schema documented in the resources above. Use OTel
semantic attribute names (not shorthand) in filters, groupBy, and
variables. Apply the defaults below unless the user specified otherwise.
Dashboard create/update payloads do not persist a default time range or
refresh interval. Panels follow the viewer-selected global range. If the user
asks for a specific window, mention that range in the final handoff instead of
inventing `timeRange`, `defaultTimeRange`, or `refresh` fields. Do not encode a
PromQL range selector inside a Builder query.
Use SigNoz kinds and JSON types exactly. `signoz/TimeSeriesPanel` means time series
(never Grafana `timeseries`); variables are `ListVariable` / `TextVariable` carrying a
`signoz/DynamicVariable`, `signoz/CustomVariable`, or `signoz/QueryVariable` plugin.
The envelope is `schemaVersion: "v6"` plus `spec`, with no top-level `name` on create;
the server derives that immutable machine label from `spec.display.name`. Tags are
`{key, value}` objects; defer full shapes to the resources.
Keep panel ids and grid items bijective; create/remove both entries together.
`spec.panels` is a map keyed by panel id, and `spec.layouts` positions those ids
through `content.$ref`. During import or rebuild, drop any grid item whose
`$ref` names a panel you did not carry over.
**Defaults the skill applies (and surfaces in the preview):**
| Field | Default | When to override |
|---|---|---|
| Section structure (APM/services) | Overview / Latency / Errors / Throughput | domain-specific (e.g. DB: Overview / Connections / Throughput / Slow Queries) |
| Section structure (infra/runtime) | Overview / Saturation / Errors / Latency | domain-specific |
| Headline panels (services) | request rate, error rate, p50/p95/p99 latency, throughput | omit those that don't apply |
| Headline panels (infra) | resource utilization (CPU, mem), saturation, error/restart counts, throughput | tailor to the technology |
| Counter render unit (rate vs. count) | per-second rate | per-interval **increase** count over a wider window (24h–7d) for any low-volume / bursty / human-paced counter (requests, **error counts**, restarts, OOM kills) where `/sec` renders as tiny decimals (e.g. `0.03/s`); gauges (CPU/mem/queue depth) are already absolute and unaffected; note `increase` rescales its y-axis with the selected range, so prefer it deliberately, not by reflex |
| Variables (services) | `service.name`, `deployment.environment` (or `deployment.environment.name`; verify which exists via `signoz_get_field_keys`) | add `k8s.cluster.name` / `k8s.namespace.name` when k8s-flavored |
| Variables (k8s/infra) | `k8s.cluster.name`, `k8s.namespace.name` (or `host.name` for hostmetrics) | drop `service.name`; it is rarely populated on infra signals |
| Layout | 2-column grid (`width: 6`), 12 columns wide; every item has `0 <= x < 12`, `1 <= width <= 12`, `x + width <= 12` | full-width (`x: 0, width: 12`) for tables and time-series with many series |
| GroupBy on per-service panels | `service.name` resource attribute | drop when filtering to a single service |
**Sections** A section is one Grid entry in `spec.layouts`, with its own
`spec.display.title` and its own `spec.items`. One Grid per section, in display
order; a panel belongs to a section by having its grid item in that Grid. Item
coordinates are per-Grid, so adding to an earlier section leaves later
sections untouched.
**Title and description** The dashboard title (`spec.display.name`) should name the
technology and the scope clearly: "PostgreSQL - prod-us-east-1", not
just "PostgreSQL". `spec.display.description` should answer "what is this for" in one
sentence. Tags are `{key, value}`: technology + signal types + environment when known.
##### Step 3b-ii.5: Shape check before save
`signoz://dashboard/widgets-examples` is the source of truth for panel
required fields, panel-type-specific shapes, plugin `kind` names, and
common write-shape errors. Re-skim it before serialising any custom panel
JSON.
Every builder query and formula entry must carry a positive `limit` and non-empty
`order`. Raw list and trace-request panels default to 100 with timestamp-desc ordering
(raw logs add id); a deliberately smaller list page may lower `limit`. Aggregate panels use 100 with
the primary aggregation desc. Formula outputs use 100 with `__result desc`; every referenced base query uses 10000
because base limits apply before formula evaluation. Find those inputs from every formula expression, including
formulas with `disabled: true`, following references until every base `builder_query` leaf is reached. This dependency
walk chooses bounds only; it does not establish deterministic formula-to-formula evaluation order, so dry-run the
complete composite payload. A metrics `order` key is the composed `spaceAggregation(timeAggregation(metricName))`
expression; the bare metric name is rejected, while `__result` and groupBy keys are accepted.
Time-series top-N ranks groups over the whole window and can omit a short-lived local spike. Narrow filters/grouping
if formula-input cardinality can exceed 10000.
Two rules `widgets-examples` does not call out, but
`signoz_create_dashboard` enforces: **no `JSON.stringify` on
arrays/objects** (`spec`, `panels`, `layouts`, `tags`, and `variables`
are native JSON) and **one query per panel**, so a panel plotting two
series carries a single `signoz/CompositeQuery` envelope holding both.
##### Step 3b-ii.6: Dry-run before save (mandatory)
For every query-bearing panel, read the compact
[`dashboard-to-query-builder-v5` reference](./references/dashboard-to-query-builder-v5.md).
The panel already stores the execution spec, so lift it into the outer envelope
and call `signoz_execute_builder_query` with that payload, never panel JSON.
Dry-run over a short absolute Unix-ms window (usually the last 30-60 minutes),
never the panel's display range by reflex; apply the reference's dry-run hygiene
rules before widening or retrying after a timeout. Use representative variable
values in the dry-run copy and keep `$var` in `signoz_create_dashboard`.
If the reference's safety gate finds an unsupported execution field, report the
panel as unvalidated and continue only after explicit user acceptance. Server or
validation errors block. Unexpected empty results block unless the user already
accepted absent telemetry.
##### Step 3b-ii.7: Preview, save, report
1. **Preview** Emit a one-paragraph plain-language summary of what
will be created; no JSON dump. A 20–30 panel payload is hundreds
of lines the user cannot meaningfully review in chat. Call out any
validation gap the user explicitly accepted.
> **Summary**: This dashboard tracks [signals] for [scope], with
> sections [list]. Variables: [list].
> Dry-run: [N] panels passed. Unvalidated: [none / accepted gaps].
> Data: [confirmed / pending ingestion by explicit user choice].
2. **Save** Call `signoz_create_dashboard` with the payload.
3. **Report** Tell the user:
- The created dashboard's id and title.
- Panel count and section breakdown.
- Which variables are wired.
- Two follow-up offers: "Want me to adjust panels, layout, or
variables?" and "Want me to wire alerts for any of these signals?
(`signoz-creating-alerts`)".
## Guardrails
- **Strict inputs over guessing** Resource scope is required for custom
builds. If missing, stop and ask the user (see *Required inputs*
above). A guessed scope on a shared dashboard is harder to clean up
than asking.
- **Always paginate `signoz_list_dashboards`** Stopping at page
1 misses duplicates and produces clutter.
- **Duplicate check first** The user's only two upfront options are
"modify an existing one" or "create a new one"; never offer
template-import as a separate top-level choice.
- **Template-first on the create path** Once the user has chosen to
create, always run `signoz_list_dashboard_templates` before any
`signoz_create_dashboard` call. If a matching template exists,
import it via `signoz_import_dashboard` (just inform the user);
only build from scratch when no template matches.
- **No-data probe is mandatory before save** Run the pre-flight probe
(Step 3b-i.1 / Step 3b-ii.2) before `signoz_import_dashboard`
/ `signoz_create_dashboard`. A "No data" dashboard is a worse
outcome than one extra confirmation prompt. Skip only if the user has
explicitly opted out for this request.
- **Validate custom builds before save** Follow Step 3b-ii.6; never treat a
dry-run that omits active query semantics as validated.
- **Preview before save on custom builds** Emit the plain-language
summary before `signoz_create_dashboard` so the human can
intervene on intent.
- **Prefer OTel attribute names** `service.name` not `service`,
`host.name` not `host`. Wrong names produce empty panels. Verify the
exact key (`deployment.environment` vs `deployment.environment.name`,
for instance) against `signoz_get_field_keys` rather than guessing;
installs running classic OTel semconv emit the no-`.name` form.
- **No metric guessing** For custom builds, verify metric names with
`signoz_list_metrics` before authoring. Wrong names produce
empty panels and the user only finds out later.
- **Valid JSON shapes only** Follow the schema documented in
`signoz://dashboard/*` MCP resources. Required panel and query
fields are listed in `signoz://dashboard/widgets-instructions` and
`signoz://dashboard/widgets-examples`. Never wrap arrays/objects in
`JSON.stringify`; enforce the panel/grid-item bijection, field types, and
plugin kinds from Step 3b-ii.4.
- **Scope boundary** This skill creates dashboards. The moment the
user asks to modify, edit, rearrange, or extend an existing dashboard
(including immediately after import), hand off to
`signoz-modifying-dashboards`. Do not call
`signoz_update_dashboard` or `signoz_patch_dashboard` from this skill.
## Examples
Four canonical flows (template happy path, template choice, duplicate
found, custom build) live in [`references/examples.md`](references/examples.md).