signoz-reducing-telemetry-cost

어떤 스킬인가요?

Investigate and reduce SigNoz telemetry ingestion cost and metric cardinality across metrics, logs, and traces. Find what drives SigNoz spend (via the Cost Meter), which metrics have runaway or unbounded label cardinality, and safe, dashboard-, alert-, and Infra-page-aware ways to cut volume. Make sure to use this skill whenever the user asks "why is my SigNoz bill so high", "what's driving my ingestion cost", "reduce telemetry volume", "which metrics cost the most", "cardinality health check", or "what can I safely drop", or otherwise asks about telemetry spend, ingestion volume, or metric cardinality, even if they don't say "cost" or "optimize" explicitly.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

{
  "skill_name": "signoz-reducing-telemetry-cost",
  "evals": [
    {
      "id": 0,
      "eval_name": "full-investigation-covers-every-signal",
      "prompt": "Run a full SigNoz ingestion-cost investigation. The Cost Meter has data for all three signals. Metrics are the largest current cost contributor, logs are second, and traces are third. I want one prioritized action list, not three separate reports.",
      "expected_output": "Agent starts with Cost Meter totals, keeps metrics as the primary driver, and investigates metrics, logs, and traces in current-cost order. The final answer consolidates findings into one prioritized list and does not repeat the full per-signal decision rules.",
      "expectations": [
        "Agent starts with signoz_execute_builder_query Cost Meter queries before signal-specific analysis",
        "Agent discovers Cost Meter groupBy keys with signoz_get_field_keys signal=metrics source=meter",
        "Every raw builder groupBy entry in the actual signoz_execute_builder_query payload losslessly copies name, fieldDataType, fieldContext, and signal from the selected signoz_get_field_keys result",
        "Every Cost Meter builder_query has limit=100 plus Query Builder v5 order by __result desc and no dashboard orderBy",
        "Agent analyzes metrics, logs, and traces because every signal has Cost Meter data",
        "Agent orders the investigation by current weighted cost: metrics, then logs, then traces",
        "Agent returns one consolidated prioritized action list without repeating the per-signal playbooks"
      ],
      "files": []
    },
    {
      "id": 1,
      "eval_name": "retained-metric-cardinality-and-interval",
      "prompt": "Metric samples are my main cost. `debug_request_id_total` is unused and I agree to drop it. `checkout_request_duration` backs a dashboard and alert, so I must keep it; it is exported every 10 seconds and I could tolerate one-minute resolution. Check cardinality and suggest the next safe lever. Do not change the metric's delta/cumulative behavior.",
      "expected_output": "Agent checks usage before cardinality, skips cardinality analysis for the agreed drop, and inspects cardinality only for the retained checkout metric. It may recommend increasing the appropriate collection, scrape, or SDK export interval after identifying the source, while preserving dashboard/alert resolution and explicitly avoiding any temporality change.",
      "expectations": [
        "Agent runs or relies on signoz_check_metric_usage before classifying either metric",
        "Agent does not call signoz_check_metric_cardinality for debug_request_id_total after the user agrees to drop it",
        "Agent calls signoz_check_metric_cardinality for checkout_request_duration because the user must retain it",
        "Agent identifies the metric source before choosing collection_interval, scrape_interval, or OTEL_METRIC_EXPORT_INTERVAL",
        "Agent considers the dashboard and alert's required resolution before recommending a longer interval",
        "Agent does not recommend switching between delta and cumulative temporality"
      ],
      "files": []
    },
    {
      "id": 2,
      "eval_name": "cost-meter-uses-explicit-builder-sum",
      "prompt": "Show me my total SigNoz span, log, and metric ingestion for the last seven days. The Cost Meter response includes an edge datapoint with partial=true. Walk me through the query tool and unit conversion you use for the totals.",
      "expected_output": "Agent first discovers the live meter metrics, units, and temporalities with signoz_list_metrics source=meter. It uses signoz_execute_builder_query with the full outer query wrapper, Unix-ms start/end, requestType=time_series, stepInterval=3600, formatOptions, variables, the discovered temporality, and explicit raw sum aggregation for each discovered Cost Meter metric. It excludes partial datapoints and sums complete hourly buckets, converts byte units with the decimal 1e9 GB divisor, does not use signoz_query_metrics for totals, and does not fabricate totals when response values are absent.",
      "expectations": [
        "Agent calls signoz_list_metrics with source=meter and uses the returned metric names and units rather than assuming a fixed meter list",
        "Agent calls signoz_execute_builder_query with an outer query object containing Unix-millisecond start and end, requestType=time_series, formatOptions, and variables",
        "Agent uses source=meter, the discovered metricName and temporality, stepInterval=3600, and timeAggregation=sum for Cost Meter totals",
        "Every Cost Meter builder_query has limit=100 plus Query Builder v5 order by __result desc and no dashboard orderBy",
        "Agent does not use signoz_query_metrics for Cost Meter totals",
        "Agent excludes datapoints marked partial=true and sums the remaining complete hourly buckets",
        "Agent converts log and span bytes to GB with the decimal 1e9 divisor",
        "Agent does not fabricate totals when the prompt provides no complete response values",
        "Agent does not cite a fixed 6 percent or other universal undercount figure"
      ],
      "files": []
    },
    {
      "id": 3,
      "eval_name": "trace-sampling-is-never-recommended",
      "prompt": "Trace volume is high. Configure 10% tail sampling. We understand that SigNoz APM values may become approximate and accept that tradeoff.",
      "expected_output": "Agent does not recommend or configure trace sampling even though the user accepts approximate APM values. It explains that built-in APM/Services metrics cover only retained traces, so absolute request counts and rates undercount real traffic even though latency trends and error spikes may remain useful. It does not treat processor placement as an exception to the no-sampling policy. It identifies confirmed noise operations first, then may suggest a targeted SDK exclusion or Collector filter while stating the APM impact.",
      "expectations": [
        "Agent does not recommend or provide a configuration for head, probabilistic, tail, or any other trace sampling",
        "Agent states that sampled built-in APM/Services metrics cover only retained traces and absolute request counts or rates undercount real traffic",
        "Agent acknowledges that latency trends and error spikes may remain useful without treating that as permission to recommend sampling",
        "Agent does not present Collector processor placement as an exception to the no-sampling policy",
        "Agent investigates confirmed noise operations instead of treating sampling as a cost-reduction lever",
        "Agent states that targeted SDK exclusions or Collector filters remove those operations from APM before configuring them"
      ],
      "files": []
    },
    {
      "id": 4,
      "eval_name": "infra-identity-overrides-empty-usage-and-cardinality",
      "prompt": "Metric samples are my main cost. `k8s.pod.cpu.usage` is a top-volume metric. signoz_check_metric_usage returns dashboards=[], alerts=[], error=\"\". Its highest-cardinality attributes are `k8s.pod.uid`, `k8s.pod.name`, and `k8s.pod.start_time`. A custom metric, `debug_queue_depth`, also returns empty dashboards and alerts, but its usage result has error=\"timeout\". Tell me what I can safely drop or aggregate.",
      "expected_output": "Agent treats k8s.pod.cpu.usage as an Infra-page dependency despite its empty dashboard/alert usage. It does not recommend dropping the metric or aggregating away pod UID/name/start-time because they power entity resolution and Pod Age. It marks debug_queue_depth as Needs one check first because the usage timeout makes the empty lists unreliable.",
      "expectations": [
        "Agent labels k8s.pod.cpu.usage as an Infra-page dependency rather than a safe drop",
        "Agent does not recommend aggregating away k8s.pod.uid or k8s.pod.name from the Infra metric",
        "Agent preserves k8s.pod.start_time on the Infra metric because the Pods page uses it for Pod Age",
        "Agent treats the timeout for debug_queue_depth as unknown usage and marks it Needs one check first",
        "Agent does not classify debug_queue_depth as unused from empty lists returned with an error"
      ],
      "files": []
    },
    {
      "id": 5,
      "eval_name": "warning-logs-block-blanket-drop",
      "prompt": "Logs are my main cost. The checkout service emits 80% INFO, 5% DEBUG, 10% WARNING, and 5% ERROR logs. Drop the whole service or give me the safest filter.",
      "expected_output": "Agent counts WARNING as high-signal, calculates a 15% high-signal share, and refuses a blanket service drop or any filter that includes WARN/ERROR. It treats LOG_LEVEL=WARN or an INFO/DEBUG-only Collector filter as candidates until it quantifies the service's Cost Meter share and checks every page of log alerts.",
      "expectations": [
        "Agent includes WARNING when calculating the high-signal percentage",
        "Agent calculates or recognizes that high-signal logs are 15 percent of the service total",
        "Agent does not recommend dropping the whole checkout service",
        "Agent does not recommend a severity filter that removes WARNING or ERROR logs",
        "Agent quantifies checkout's GB share with the Cost Meter grouped by service.name before finalizing a reduction",
        "Agent paginates through every log alert and checks whether it depends on checkout INFO or DEBUG records",
        "Agent keeps LOG_LEVEL=WARN and any INFO/DEBUG-only filter as candidates until those checks complete"
      ],
      "files": []
    }
  ]
}

# Cost Meter query templates

## Contents
- Why `signoz_execute_builder_query` (not `signoz_query_metrics`) for totals
- Discover meter metrics
- Per-signal total (template)
- Breakdown by environment / service
- Converting and reconciling the numbers

## Why `signoz_execute_builder_query` (not `signoz_query_metrics`) for totals

Cost Meter data lives in the metrics store under `source: "meter"`. Query it with
`signoz_execute_builder_query` and an explicit `timeAggregation: "sum"`.

Do **not** use `signoz_query_metrics` for Cost Meter totals. Use
`signoz_execute_builder_query`, which honors the explicit raw builder `sum` required for these
meter metrics.

## Discover meter metrics

Always call `signoz_list_metrics` with `source: "meter"` before querying. Treat its returned
metric names, types, temporalities, and units as the live source of truth; the meter set evolves.
Select the discovered size, datapoint, or record-count metric that matches the requested signal.
Names such as `signoz.meter.log.size` and `signoz.meter.span.count` are examples, not an
exhaustive table. Do not query an example name that discovery did not return. Copy the selected
metric's returned `temporality` into the raw builder aggregation.

## Per-signal total (template)

Call `signoz_execute_builder_query` once per discovered meter metric. Replace
`<discovered_meter_metric_name>` with the live name returned by `signoz_list_metrics`; replace
the example `start` and `end` integers with the requested range in Unix milliseconds.

```json
{
  "query": {
    "schemaVersion": "v1",
    "start": 1751932800000,
    "end": 1752537600000,
    "requestType": "time_series",
    "compositeQuery": {
      "queries": [
        {
          "type": "builder_query",
          "spec": {
            "name": "A",
            "signal": "metrics",
            "source": "meter",
            "stepInterval": 3600,
            "limit": 100,
            "order": [
              {"key": {"name": "__result"}, "direction": "desc"}
            ],
            "aggregations": [
              {
                "metricName": "<discovered_meter_metric_name>",
                "temporality": "<discovered_temporality>",
                "timeAggregation": "sum",
                "spaceAggregation": "sum"
              }
            ],
            "disabled": false
          }
        }
      ]
    },
    "formatOptions": {
      "formatTableResultForUI": false,
      "fillGaps": false
    },
    "variables": {}
  }
}
```

The 100-group bound ranks groups across the whole requested window. A group
with a short-lived local spike can fall outside the returned top N; narrow the
window or choose a deliberate positive override when that matters. Query Range
uses `order`; `orderBy` is only for dashboard editor payloads.

Meter buckets are hourly, so keep `stepInterval: 3600`. Sum all complete hourly values across
every returned series, excluding datapoints with `partial: true`; they are incomplete edge
buckets. Use the unit returned by discovery when labeling or converting the total.

## Breakdown by environment / service

Call `signoz_get_field_keys` with `signal: "metrics"` and `source: "meter"` before grouping.
Use only a returned key and losslessly copy its `name`, `fieldDataType`, `fieldContext`, and
`signal`; raw builder queries do not reliably infer an omitted or ambiguous field descriptor.
Add the complete returned field to the same spec:

```json
"groupBy": [
  {
    "name": "<returned_field_name>",
    "fieldDataType": "<returned_field_data_type>",
    "fieldContext": "<returned_field_context>",
    "signal": "<returned_signal>"
  }
]
```

## Converting and reconciling the numbers

- GB divisor = 1,000,000,000 (`1e9`). "M samples" divisor = 1,000,000 (`1e6`).
- A grouped sum can differ from the ungrouped total. Use the ungrouped total for absolute cost
  figures; use grouped values only for percentages and ranking.

# Infra-page "do-not-drop" metrics: the safety guard

## Contents
- The rule (Infra page uses built-in queries, not dashboards)
- Hosts view: families
- Kubernetes view: enumerated names + workload/volume/namespace/cluster families
- APM / Services page: span-derived `signoz_*` RED metrics
- Required attributes and page metadata (resolution / filtering)
- Why the do-not-drop list matters
- Caveat / discrepancy to keep in mind

The SigNoz **Infrastructure** page (Hosts and Kubernetes views) queries these metrics
via **built-in queries, not dashboards**, and fails silently when the data is missing.
So `signoz_check_metric_usage` will report them as having **no dashboards/alerts**, yet
dropping them **breaks the Infra page.** This list is the authoritative exception to the
"unused ⇒ drop candidate" rule.

**Rule:** if a drop candidate matches anything below, do NOT present it as "safe to
drop." Present it as **"Infra-page dependency: dropping breaks the Hosts/Kubernetes
view; confirm you don't use that view before dropping."** This overrides the empty
usage-check result.

Bias intentionally conservative (family-level); over-protecting a metric costs a little
saving; under-protecting silently breaks a product page.

## Hosts view: families (treat whole prefix as load-bearing)
- `system.cpu.*` (incl. `system.cpu.load_average.1m/5m/15m`)
- `system.memory.*`
- `system.disk.*`
- `system.network.*`
- `system.filesystem.*`
- `system.paging.*`
- `system.processes.*`
- `host.cpu.usage` (used for host entity validation)

Exact `system.*` names are NOT enumerated in SigNoz docs; definitive list lives in the
frontend Hosts query builders. Family-level protection is the safe default; refine from
frontend source only if a user needs to drop a specific `system.*` metric.

## Kubernetes view: enumerated names (from k8s-metrics.mdx, high confidence)
Entity resolution (critical; missing = entities don't resolve / click errors):
- `k8s.pod.cpu.usage`, `k8s.node.cpu.usage`
Pods: `k8s.pod.cpu.usage`, `k8s.pod.cpu_request_utilization`, `k8s.pod.cpu_limit_utilization`,
  `k8s.pod.memory_request_utilization`, `k8s.pod.memory_limit_utilization`, `k8s.pod.memory.usage`,
  `k8s.pod.uptime`, `k8s.pod.status_reason`
Containers: `container.cpu.usage`, `container.uptime`,
  `k8s.container.cpu_request_utilization`, `k8s.container.cpu_limit_utilization`,
  `k8s.container.memory_request_utilization`, `k8s.container.memory_limit_utilization`
Nodes: `k8s.node.cpu.usage`, `k8s.node.condition`, `k8s.node.uptime`
The Workload (Deployments/StatefulSets/DaemonSets/Jobs/ReplicaSets), Volumes, Namespaces, and
Clusters tabs each render from their own metric families: `k8s.replicaset.*`, `k8s.deployment.*`,
`k8s.volume.*`, and the rest listed below are all load-bearing for those tabs.

**Families safe to treat whole** (family-level protection is the safe default; exact names in
`k8s-metrics.mdx`):
`k8s.pod.*`, `k8s.container.*`, `k8s.node.*`, `container.*`, `k8s.deployment.*`,
`k8s.replicaset.*`, `k8s.statefulset.*`, `k8s.daemonset.*`, `k8s.job.*`, `k8s.cronjob.*`,
`k8s.hpa.*`, `k8s.volume.*`, `k8s.namespace.*`, `k8s.cluster.*`

## APM / Services page: powered exclusively by span-derived `signoz_*` RED metrics
The SigNoz APM / Services pages are built from span-derived RED metrics that the
`signozspanmetrics` collector processor generates from the traces pipeline, not from any user
dashboard.

The Services list and service-detail charts (rate, error %, p50/p90/p99 latency, apdex, DB
calls, external calls) query ONLY these span-derived metrics:
- `signoz_calls_total`
- `signoz_latency_bucket`, `signoz_latency_count`, `signoz_latency_sum`
- `signoz_db_latency_count`, `signoz_db_latency_sum`
- `signoz_external_call_latency_count`, `signoz_external_call_latency_sum`
- dotted variants (`signoz_latency.bucket`, …) when `dotMetricsEnabled` is on (same data).

**Handling in the skill:**
- These already fall under the internal-`signoz_`/`signoz.` exclusion → never in the drop
  candidate list. Keep it that way; if a user asks about them, explain they power the APM
  page (do not drop).
- **OTel-native `http.server.*` / `rpc.*` metrics do NOT back the APM page in SigNoz.** Treat
  them as ordinary metrics: normal usage-check applies; if unused and not in a dashboard/alert
  they are genuinely droppable (histograms: trim buckets rather than hard-drop). No special APM
  protection.
- **Trace-layer dependency:** because these RED metrics are generated from spans, trace sampling
  limits the built-in APM/Services metrics to retained traces. Absolute request counts and rates
  undercount real traffic; latency trends and error spikes may remain useful. Never recommend
  trace sampling as a cost-reduction lever.
  Targeted SDK exclusions and Collector filters also remove those operations from APM, so state
  that impact before recommending them. Dropping the `signoz_*` metrics themselves breaks APM
  entirely. Treat that as a separate lever.

## Required attributes and page metadata (not metrics)
- Hosts: `host.name` (required; missing = host not clickable), `host.id` (fallback)
- Kubernetes entity resolution: each entity's `.uid` + `.name`:
  `k8s.pod.uid`/`k8s.pod.name`,
  `k8s.node.uid`/`k8s.node.name`, `k8s.deployment.name`, `k8s.namespace.name`,
  `k8s.statefulset.name`, `k8s.daemonset.name`, `k8s.job.name`, `k8s.cronjob.name`,
  and `k8s.container.name`. Container rows use `k8s.pod.uid` + `k8s.container.name`, not
  `container.id`.
  Missing UID → clicking the entity raises an internal error.
- Kubernetes page metadata: keep `k8s.pod.start_time` (Pod Age) and `k8s.pod.ip` on Pod metrics;
  keep `k8s.volume.type` on volume metrics.

## Why the do-not-drop list matters
`signoz_check_metric_usage` does a real dashboard/alert lookup, so it correctly reports these
metrics as referenced by no dashboard and no alert. That is exactly the trap: the Infra page
queries them directly rather than through a dashboard, so "no usage" does **not** mean "safe to
drop." Top-volume `k8s.*` metrics like `k8s.node.condition` and the `k8s.container.*_utilization`
family are common offenders, frequently the single biggest metric-volume contributors, and
exactly what a naive drop pass would remove first.

## Caveat / discrepancy to keep in mind
SigNoz docs write some resolution metrics as `pod.cpu.usage` / `host.cpu.usage` (no `k8s.` /
`system.` prefix) while the Kubernetes metrics reference uses the `k8s.*` form. When matching by
exact name, allow both prefix forms.

_Source: SigNoz Infrastructure Monitoring docs (Kubernetes metric names are authoritative). The
exact Hosts `system.*` names live in the frontend Hosts query builders; family-level protection
is the safe default._

# OTel Attribute Cardinality Reference

## Contents
- Profiles (UNBOUNDED, ACCUMULATING, BOUNDED, DEPLOYMENT_DEPENDENT, IDENTIFIER)
- HTTP / Network
- Database
- Messaging / Queues
- RPC / gRPC
- Kubernetes / Infrastructure
- Process / Runtime
- Errors / Exceptions
- Tracing / Correlation IDs
- Cloud / Provider
- Service / SDK
- How to classify an unknown attribute
- Fix reference

Used by the agent to classify any attribute encountered during cardinality analysis.
For each attribute: profile, why it matters, and what to do about it.

## Profiles

- **UNBOUNDED**: grows without ceiling. Every new URL, query, ID, or stack trace creates a new series permanently. These are the most dangerous.
- **ACCUMULATING**: grows with infrastructure churn (pod restarts, deploys). Does not reflect concurrent active series; a metric with 20,000 `k8s.pod.uid` values may only have 50 active pods right now.
- **BOUNDED**: fixed or near-fixed set of values. Safe to keep as a label dimension.
- **DEPLOYMENT_DEPENDENT**: cardinality depends on the deployment scale. Fine at 10 services, dangerous at 500.
- **IDENTIFIER**: always a unique ID per request/event. Should never appear as a metric label. Immediate red flag.

---

## HTTP / Network

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `url.full` | UNBOUNDED | Full URL including query string and path params. Each unique URL = new series. `/user/123` and `/user/456` are different series. | Aggregate with metricstransform or use `http.route` instead |
| `http.url` | UNBOUNDED | Older convention, same problem as `url.full` | Same |
| `http.target` | UNBOUNDED | URL path + query string. Path params make this unbounded | Use `http.route` (templated path) instead |
| `url.path` | UNBOUNDED | Raw path without query, but still unbounded if path contains IDs | Use `http.route` |
| `url.query` | UNBOUNDED | Query string. Completely unbounded | Drop entirely |
| `http.route` | BOUNDED | Templated route e.g. `/user/{id}`. Fixed set per application | Safe to keep |
| `http.method` | BOUNDED | GET, POST, PUT, DELETE etc. Fixed set | Safe |
| `http.request_method` | BOUNDED | Same as `http.method`, newer convention | Safe |
| `http.status_code` | BOUNDED | 200, 404, 500 etc. Fixed set | Safe |
| `http.response_status_code` | BOUNDED | Same, newer convention | Safe |
| `http.scheme` | BOUNDED | http or https | Safe |
| `http.flavor` | BOUNDED | 1.0, 1.1, 2.0, 3.0 | Safe |
| `network.protocol.version` | BOUNDED | Protocol version, small fixed set | Safe |
| `network.protocol.name` | BOUNDED | http, grpc, etc. | Safe |
| `net.peer.name` | DEPLOYMENT_DEPENDENT | Hostname of remote peer. Fine if calling fixed upstream services, unbounded if calling arbitrary user-supplied hosts | Review actual cardinality |
| `net.peer.port` | UNBOUNDED | Ephemeral client ports are unique per connection (range 32768–60999). Server ports are bounded but client ports are not. | Check if client or server port; drop client ports |
| `net.host.port` | BOUNDED | Server-side listening port. Small fixed set | Safe |
| `client.port` | UNBOUNDED | Client-side ephemeral port. Always unbounded | Drop (no diagnostic value) |
| `client.address` | DEPLOYMENT_DEPENDENT | Client IP. Bounded in internal service mesh, unbounded for public-facing APIs | Check cardinality |
| `server.address` | DEPLOYMENT_DEPENDENT | Upstream hostname. Usually bounded | Review |
| `server.port` | BOUNDED | Server-side port. Fixed set | Safe |
| `network.peer.address` | DEPLOYMENT_DEPENDENT | IP of remote peer. Bounded in internal mesh, unbounded externally | Check cardinality |
| `network.local.address` | BOUNDED | Local interface IP. Small fixed set per host | Safe |

---

## Database

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `db.query.text` | UNBOUNDED | Raw SQL or query string. Every unique query = new series. Parameterized queries with different values = different series | Drop entirely or replace with `db.operation.name` |
| `db.statement` | UNBOUNDED | Older convention for raw query text. Same problem | Drop |
| `db.operation` | BOUNDED | SELECT, INSERT, UPDATE, DELETE etc. | Safe |
| `db.operation.name` | BOUNDED | Same, newer convention | Safe |
| `db.name` | DEPLOYMENT_DEPENDENT | Database name. Usually small fixed set | Safe at low counts |
| `db.sql.table` | DEPLOYMENT_DEPENDENT | Table name. Bounded if schema is fixed | Usually safe |
| `db.collection.name` | DEPLOYMENT_DEPENDENT | Collection/table name | Usually safe |
| `db.system` | BOUNDED | postgresql, mysql, redis, mongodb etc. | Safe |
| `db.redis.database_index` | BOUNDED | Redis DB index 0-15 | Safe |
| `db.cassandra.keyspace` | DEPLOYMENT_DEPENDENT | Keyspace name | Usually safe |
| `db.hbase.namespace` | DEPLOYMENT_DEPENDENT | HBase namespace | Usually safe |
| `db.mongodb.collection` | DEPLOYMENT_DEPENDENT | Collection name | Usually safe |

---

## Messaging / Queues

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `messaging.message_id` | IDENTIFIER | Unique per message. Should never be a metric label | Drop immediately |
| `messaging.destination.name` | DEPLOYMENT_DEPENDENT | Topic/queue name. Bounded if topics are fixed, unbounded if dynamically created per user/tenant | Check cardinality |
| `messaging.kafka.message_key` | UNBOUNDED | Message key (can be any string) | Drop |
| `messaging.kafka.partition` | BOUNDED | Partition number. Fixed per topic | Safe |
| `messaging.kafka.consumer_group` | DEPLOYMENT_DEPENDENT | Consumer group name. Usually bounded | Usually safe |
| `messaging.rabbitmq.routing_key` | DEPLOYMENT_DEPENDENT | Routing key. Bounded if fixed routes, unbounded if dynamic | Check cardinality |
| `messaging.operation` | BOUNDED | publish, receive, process etc. | Safe |
| `messaging.system` | BOUNDED | kafka, rabbitmq, sqs etc. | Safe |

---

## RPC / gRPC

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `rpc.method` | BOUNDED | gRPC method name. Fixed set per service definition | Safe |
| `rpc.service` | BOUNDED | gRPC service name. Fixed set | Safe |
| `rpc.system` | BOUNDED | grpc, thrift, etc. | Safe |
| `rpc.grpc.status_code` | BOUNDED | 0-16 fixed gRPC status codes | Safe |

---

## Kubernetes / Infrastructure

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `k8s.pod.uid` | ACCUMULATING | New UID per pod restart. Accumulates over time; a 7-day window captures every pod that ever ran, not just current pods | Keep on Infra-page metrics; on unrelated custom metrics, aggregate only after identity and usage review |
| `k8s.pod.name` | ACCUMULATING | Pod names with random suffixes change on every restart/deploy. Same accumulation problem as pod.uid | Keep on Infra-page metrics; apply the generic fix only to unrelated custom metrics |
| `k8s.pod.start_time` | ACCUMULATING | Timestamp of pod start. Unique per pod lifecycle; the Pods page uses it for Pod Age | Keep on Pod Infra metrics; remove from unrelated custom metrics only after usage review |
| `container.id` | ACCUMULATING | Container runtime ID. New per restart; current Infra container identity uses `k8s.pod.uid` + `k8s.container.name` | Aggregate away after usage review |
| `k8s.namespace.name` | BOUNDED | Namespace. Fixed small set in most deployments | Safe |
| `k8s.deployment.name` | BOUNDED | Deployment name. Fixed set | Safe |
| `k8s.statefulset.name` | BOUNDED | StatefulSet name. Fixed set | Safe |
| `k8s.daemonset.name` | BOUNDED | DaemonSet name. Fixed set | Safe |
| `k8s.job.name` | DEPLOYMENT_DEPENDENT | Job names may include timestamps or IDs if dynamically generated | Check cardinality |
| `k8s.cronjob.name` | BOUNDED | CronJob name. Fixed set | Safe |
| `k8s.node.name` | DEPLOYMENT_DEPENDENT | Node name. Bounded by cluster size. Fine at 10 nodes, notable at 1000 | Check actual count |
| `k8s.cluster.name` | BOUNDED | Cluster name. Usually 1-5 values | Safe |
| `k8s.replicaset.name` | ACCUMULATING | Includes hash suffix, changes per deploy | Aggregate away |
| `k8s.container.name` | BOUNDED | Container name within pod spec. Fixed | Safe |

---

## Process / Runtime

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `process.pid` | UNBOUNDED | Process ID. New per process restart. Unbounded over time | Drop from metrics |
| `process.command_line` | UNBOUNDED | Full command with args. Can contain paths, flags, secrets | Drop entirely |
| `process.command_args` | UNBOUNDED | Array of command arguments | Drop entirely |
| `process.executable.path` | BOUNDED | Executable path. Usually fixed per service | Usually safe |
| `process.executable.name` | BOUNDED | Executable name. Fixed | Safe |
| `process.runtime.name` | BOUNDED | go, python, jvm, dotnet etc. | Safe |
| `process.runtime.version` | DEPLOYMENT_DEPENDENT | Runtime version. Bounded if versions are controlled | Usually safe |
| `process.owner` | BOUNDED | Process owner user. Fixed set | Safe |
| `thread.id` | UNBOUNDED | Thread ID. Unique per thread, changes on restart | Drop from metrics |
| `thread.name` | DEPLOYMENT_DEPENDENT | Thread name. Bounded if thread pool has fixed names, unbounded for request-scoped threads | Check cardinality |

---

## Errors / Exceptions

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `exception.stacktrace` | UNBOUNDED | Full stack trace as string. Every unique trace = new series. Should never be a metric label | Drop immediately |
| `exception.message` | UNBOUNDED | Error message. Can contain IDs, values, dynamic content | Drop or cap with metricstransform |
| `exception.type` | BOUNDED | Exception class name. Fixed set per application | Safe |
| `error.type` | BOUNDED | Error type/class. Fixed set | Safe |
| `error.stack` | UNBOUNDED | Stack trace. Same as exception.stacktrace | Drop immediately |

---

## Tracing / Correlation IDs (should never appear on metrics)

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `span.id` | IDENTIFIER | Unique per span. Catastrophic as metric label | Drop immediately |
| `trace.id` | IDENTIFIER | Unique per trace. Catastrophic as metric label | Drop immediately |
| `trace_id` | IDENTIFIER | Same | Drop immediately |
| `span_id` | IDENTIFIER | Same | Drop immediately |
| `request.id` | IDENTIFIER | Unique per request | Drop from metrics |
| `session.id` | IDENTIFIER | Unique per session | Drop from metrics |
| `user.id` | IDENTIFIER | Unique per user. Unbounded as metric label | Drop from metrics |
| `enduser.id` | IDENTIFIER | Same | Drop from metrics |

---

## Cloud / Provider

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `cloud.provider` | BOUNDED | aws, gcp, azure etc. | Safe |
| `cloud.region` | BOUNDED | us-east-1, eu-west-1 etc. Fixed set | Safe |
| `cloud.availability_zone` | BOUNDED | AZ name. Small fixed set per region | Safe |
| `cloud.account.id` | DEPLOYMENT_DEPENDENT | Account/project ID. Bounded by org structure | Usually safe |
| `cloud.resource_id` | UNBOUNDED | Full ARN or resource path. Includes instance IDs | Drop or restrict |
| `aws.lambda.invoked_arn` | UNBOUNDED | Full ARN with region and account | Drop |
| `faas.instance` | UNBOUNDED | Function instance ID. Unique per cold start | Drop |
| `faas.id` | UNBOUNDED | Function instance identifier | Drop |
| `faas.name` | BOUNDED | Function name. Fixed set | Safe |
| `faas.version` | DEPLOYMENT_DEPENDENT | Function version. Bounded if controlled | Usually safe |

---

## Service / SDK

| Attribute | Profile | Notes | Fix |
|-----------|---------|-------|-----|
| `service.name` | BOUNDED | Service name. Fixed set; this is a required attribute | Safe, keep always |
| `service.version` | DEPLOYMENT_DEPENDENT | Service version. Bounded if versioning is controlled. Can grow if CI deploys unique versions per commit | Check cardinality |
| `service.instance.id` | DEPLOYMENT_DEPENDENT | Instance identifier. Bounded = one per running pod. But if it includes a UUID or timestamp, it becomes ACCUMULATING | Check format: if it contains hyphens/long random strings, treat as ACCUMULATING |
| `service.namespace` | BOUNDED | Logical grouping of services. Fixed | Safe |
| `deployment.environment` | BOUNDED | prod, staging, dev, uat etc. | Safe |
| `telemetry.sdk.name` | BOUNDED | opentelemetry etc. | Safe |
| `telemetry.sdk.version` | BOUNDED | SDK version. Fixed per deployment | Safe |
| `telemetry.sdk.language` | BOUNDED | go, python, java etc. | Safe |

---

## How to classify an unknown attribute

If you encounter an attribute not listed above, reason through these questions:

1. **Can the value be user-supplied, request-scoped, or contain IDs?** → Likely UNBOUNDED. Drop.
2. **Does the value change every time a pod restarts or container is replaced?** → ACCUMULATING. Aggregate away.
3. **Is it a timestamp, UUID, hash, or has segments separated by hyphens/underscores of varying length?** → IDENTIFIER or UNBOUNDED. Drop.
4. **Is it chosen from a fixed vocabulary defined at deploy time?** → Probably BOUNDED. Check actual cardinality to confirm.
5. **Does the cardinality roughly equal the number of running instances/nodes/pods?** → DEPLOYMENT_DEPENDENT. Flag if count is high but not an immediate problem.

When uncertain, report the attribute name, its observed cardinality, and the value pattern (e.g. "looks like UUIDs", "looks like IP:port pairs") and let the user verify.

---

## Fix reference

For Infra-page metric families, the required identity attributes and page metadata in
`infra-do-not-drop.md` override the generic fixes below.

| Problem | Fix | Where |
|---------|-----|-------|
| UNBOUNDED label on existing metric | `metricstransform` processor: aggregate the label away (merges series) | OTel Collector |
| ACCUMULATING label | Same: `metricstransform` aggregate | OTel Collector |
| IDENTIFIER label (span.id, trace.id, user.id) | `metricstransform` `aggregate_labels` to merge series, or stop emitting it as a metric label | OTel Collector / SDK |
| Raw SQL / stack traces as labels | Instrument correctly: use `db.operation` not `db.query.text` | SDK / instrumentation config |
| Client port (net.peer.port, client.port) | `metricstransform` `aggregate_labels` to merge series, or stop emitting it at the SDK | OTel Collector / SDK |
| Too many service instances | `service.instance.id` is expected to be high; only a problem if it contains timestamps/UUIDs that don't reflect real instance count | Check format |

**Important: cardinality reduction means fewer samples, and samples are the billable cost.**
Use the `metricstransform` processor's `aggregate_labels` action to *merge* the
series that share the remaining labels; that is what actually cuts the series and sample count.
Do **not** use the `transform` processor's `delete_key`: removing a label key without merging
leaves the same number of samples and produces colliding series (SigNoz sums them), so cost does
not drop. If a label is essential to the metric's identity, drop the whole metric or stop
emitting the label at the SDK instead.

Docs: https://signoz.io/docs/metrics-management/dropping-metric-labels/
Docs: https://signoz.io/docs/logs-management/guides/remove-resource-attributes/

---
name: signoz-reducing-telemetry-cost
description: >
  Investigate and reduce SigNoz telemetry ingestion cost and metric
  cardinality across metrics, logs, and traces. Find what drives SigNoz
  spend (via the Cost Meter), which metrics have runaway or unbounded
  label cardinality, and safe, dashboard-, alert-, and Infra-page-aware
  ways to cut volume. Make sure to use this skill whenever the user asks
  "why is my SigNoz bill so high", "what's driving my ingestion cost",
  "reduce telemetry volume", "which metrics cost the most", "cardinality
  health check", or "what can I safely drop", or otherwise asks about
  telemetry spend, ingestion volume, or metric cardinality, even if they
  don't say "cost" or "optimize" explicitly.
argument-hint: <investigation focus, such as metrics cost or cardinality health>
---

## Prerequisites

This skill calls SigNoz MCP server tools heavily (`signoz_list_metrics`,
`signoz_get_field_keys`, `signoz_execute_builder_query`,
`signoz_get_top_metrics`, `signoz_check_metric_usage`, `signoz_check_metric_cardinality`,
`signoz_aggregate_logs`, `signoz_aggregate_traces`, `signoz_search_logs`,
`signoz_list_alert_rules`, `signoz_get_alert`, `signoz_get_service_top_operations`). Before
running the workflow, confirm the `signoz_*` tools are available. If they are not, the SigNoz
MCP server is not installed or configured; run `signoz-mcp-setup` first. The whole
investigation is grounded in these queries; without the server there is nothing to analyze.

Read both reference files before drawing conclusions:
- `references/otel-attribute-cardinality.md`: classify any metric label you encounter.
- `references/infra-do-not-drop.md`: the metrics that power the built-in Infrastructure page
  **and the APM/Services page** (span-derived `signoz_*` RED metrics); never present these as
  "safe to drop" even when usage shows them unused.

## Workflow

Always start with the Cost Meter snapshot (Step 1). For a full cost investigation, run the
metrics, logs, and traces steps for every signal with data, ordered by current cost contribution
(primary, secondary, tertiary). Finish with the report (Step 5).

### Step 1: Cost Meter snapshot

Establish the cross-signal cost picture first. Always call `signoz_list_metrics` with
`source: "meter"`; treat its returned metric names, types, temporalities, and units as the live
source of truth because the meter set evolves. Then query each relevant discovered metric with
`signoz_execute_builder_query` (`source: "meter"`, `requestType: "time_series"`,
`stepInterval: 3600`, the discovered `temporality`, and `timeAggregation: "sum"`); see
`references/cost-meter-queries.md` for the full tool-argument template. Sum complete hourly
buckets and exclude every datapoint marked `partial: true`. Do **not** use
`signoz_query_metrics` for Cost Meter totals or grouped total attribution.
Report only values returned by successful queries. If a query fails or returns no usable values
after the MCP tools are available, show the intended query and say that the total could not be
computed; never invent a total. If the tools are unavailable, follow the prerequisite instead.

For a rolling 7-day window (`end` = now, `start` = end − 7 days), get the per-signal totals
(span size, log size, metric datapoints), then compute and report:

- **Primary cost driver.** The signal with the highest *dollar* weight: traces/logs at
  $0.30/GB, metrics at $0.10/M samples (orientation only; never quote dollar savings). This
  picks by cost, not by raw volume.
- **Bytes per record.** span.size ÷ span.count and log.size ÷ log.count, which tells you whether a
  signal is a payload-size problem or a volume problem.

Then break the primary signal down by environment and service. First call
`signoz_get_field_keys` with `signal: "metrics"` and `source: "meter"`; use only keys it returns
and copy each key's `name`, `fieldDataType`, `fieldContext`, and `signal` into the raw
builder `groupBy` without translating or dropping fields. Run the same meter query with that
complete `groupBy` and report the top
~10 per group with their share. If a non-prod environment (`staging`, `dev`, `test`, `qa`,
`sandbox`, `preview`, `uat`, …) is > 40% of volume, recommend Ingestion Limits on that key
before any signal-level change: https://signoz.io/docs/ingestion/signoz-cloud/keys/

### Step 2: Metrics

Run when the Cost Meter shows metric data, ordered by its cost contribution, or when the user
explicitly asks about metric cost or cardinality.

**2a. Rank by volume with `signoz_get_top_metrics`.** Returns the top 100 metrics by ingested
samples with percentages pre-computed and `totalValue` sample counts (pass `start`/`end`). This
is the volume-ranked worklist. Histogram metrics (`.bucket` suffix) are usually the top
contributors, since each bucket boundary is a separate sample per scrape.

**2b. Check usage with `signoz_check_metric_usage`.** Pass the top metric names (batch of ≤ 50 per
call). Returns `{ dashboards, alerts, error }` per metric. A metric is a drop candidate only when
its `error` is empty **and** both `dashboards` and `alerts` are empty. If `error` is non-empty the
lookup is incomplete (a timeout, or an older SigNoz that lacks the endpoint) and the returned
lists are unreliable, so never treat that metric as unused; mark it **Needs one check first** (verify
its usage manually). A clean lookup with both lists empty is a drop candidate, except for the guard
below.

> **Do-not-drop guard (mandatory).** Before calling any empty-usage metric a "safe drop", check
> it against `references/infra-do-not-drop.md`. The Infrastructure page (Hosts / Kubernetes)
> queries `system.*` and many `k8s.*` / `container.*` metrics through built-in queries, *not*
> dashboards, so usage-check reports them empty even though dropping them breaks that page. If
> a candidate matches the do-not-drop set, present it as **"Infra-page dependency: breaks the
> Hosts/Kubernetes view; confirm you don't use that view before dropping,"** never as "safe to
> drop." This overrides the empty usage result. Also exclude internal `signoz_` / `signoz.`
> metrics (auto-generated RED metrics that power the APM page, not customer-controlled).

**2c. Inspect cardinality with `signoz_check_metric_cardinality`.** Run this for metrics that are
not drop candidates and for any drop candidate the user chooses to retain. Cardinality analysis
adds no value for a metric the user has agreed to drop. The tool returns attribute keys sorted
highest-cardinality first, each with `valueCount` and sample `values`. Classify each with
`references/otel-attribute-cardinality.md`:

- **UNBOUNDED** (`url.full`, `http.target`, `db.query.text`, `client.port`, `trace.id`,
  `exception.stacktrace`, …): grow without ceiling; flag regardless of current count.
- **ACCUMULATING** (`container.id`, `k8s.pod.uid`, `k8s.pod.name`, `k8s.pod.start_time`):
  `valueCount` reflects historical pod churn, not active series; explain the distinction.
- **HIGH but bounded** (`valueCount` ≳ 100): check whether dashboards/alerts actually filter on
  that label before recommending aggregation.

> **Infra identity override (mandatory).** For a metric protected by
> `references/infra-do-not-drop.md`, preserve the identity attributes and page metadata used by
> that metric's Infra entity/view. Do not aggregate or remove entity UID/name attributes when they
> resolve that entity. Keep `k8s.pod.start_time` on Pod metrics because the Pods page uses it for
> Pod Age. This overrides the generic ACCUMULATING fixes in the cardinality reference.

To reduce cardinality use the `metricstransform` processor's `aggregate_labels` action to *merge*
series (samples are the billable cost, so merging is what actually cuts it), not the `transform`
processor's `delete_key`, which leaves the same sample count and creates colliding series. If a
label is essential to the metric's identity, drop the whole metric or fix it at the SDK instead.
For histograms, reducing bucket boundaries cuts samples with little P99 impact. Docs:
https://signoz.io/docs/userguide/drop-metrics/ ·
https://signoz.io/docs/metrics-management/dropping-metric-labels/

**2d. Review the collection interval.** For a high-volume metric that must be kept, identify how
it is produced and its current interval before recommending a change. A longer interval reduces
ingested datapoints but also lowers time resolution, so preserve the resolution required by its
dashboards and alerts. Use the source's own control: a receiver `collection_interval` for
Collector-generated metrics, the scrape interval for Prometheus-scraped metrics, or
`OTEL_METRIC_EXPORT_INTERVAL` for SDK push metrics when that SDK supports it. Never recommend
switching a metric between delta and cumulative temporality; changing temporality for the same
metric can break SigNoz queries.

### Step 3: Logs

Run when the Cost Meter shows log data, ordered by its cost contribution, or when the user
explicitly asks about log cost.

**3a. Total + attribution decides the path.**
- Total log GB (the absolute cost figure): use `signoz_execute_builder_query` with the discovered
  meter metric whose live unit and meaning represent log bytes, summed as in Step 1.
- Attribution: run the **same meter query grouped by `service.name`**. This returns one group per
  service plus an unset/empty-`service.name` group for logs with no attribution. Compute the ratio
  entirely from THIS grouped result so numerator and denominator share one basis; a grouped sum can
  differ from the ungrouped total, so never divide the grouped attributed sum by the ungrouped
  total:
  - attributed GB = sum of the groups with a non-empty `service.name`.
  - grouped total = sum of *all* groups (including the empty one).
  - **Attribution % = attributed ÷ grouped total.**
- This is a hard branch:
  - **≥ 10% → Path A (service mode).**
  - **< 10% → Path B (namespace mode).** Logs come from an infra forwarder (Fluent Bit /
    Fluentd / Vector), not OTel SDKs, so `service.name` isn't set. Path B is a less common
    setup, so sanity-check its numbers.

**3b. Analyze the selected attribution path and identify candidate fixes.** The fixes in this
step are candidates only. Complete the alert check in Step 3c before recommending any log-volume
reduction, including a source log-level change.

**Path A, service mode (≥ 10%).** Get the severity mix with `signoz_aggregate_logs`,
`aggregation: count`, `groupBy: "service.name,severity_text"`. Classify severities: REDUCIBLE =
INFO, INFORMATION, DEBUG, TRACE, VERBOSE; HIGH-SIGNAL = ERROR, FATAL, CRITICAL, WARN, WARNING.
For each top service by GB:
- Reducible-dominant + own service code → candidate: set `LOG_LEVEL=WARN` (stops generation at
  source).
- Reducible-dominant + third-party library → candidate: Collector filter on
  `instrumentation_scope.name`
  (read the scope from a `signoz_search_logs` sample's `scope_name`).
- No source access → candidate: Collector filter on `severity_text` matching **only the reducible
  set** (INFO/DEBUG/TRACE), never a range that also catches WARN+.
- High-signal-dominant (WARN/ERROR > 50%) → the logs are worth keeping; a high error rate may be
  a real problem, so flag it separately, do not recommend filtering it away.

> **Service severity guard.** The candidate fixes above (`LOG_LEVEL=WARN`, or a Collector filter
> scoped to INFO/DEBUG only) preserve WARN+ by construction, so the severity mix does not block
> them; Step 3c still determines whether an alert depends on the records. The guard applies only
> to actions that would *also* drop high-signal logs: a **blanket service drop**, or a
> `severity_text` filter whose range includes WARN/ERROR/FATAL/CRITICAL. Before recommending one
> of those, compute high-signal %
> = (all HIGH-SIGNAL severities, including ERROR, FATAL, CRITICAL, WARN, **and WARNING**, matched
> case-insensitively) ÷ service total. **If it is > 1% (or a non-trivial absolute count), do not
> take the blanket action**: keep the reduction scoped to INFO/DEBUG and flag the errors as a real
> signal.

**Path B, namespace mode (< 10%).**
- Top namespaces: `signoz_aggregate_logs`, `count`,
  `groupBy: "k8s.namespace.name,deployment.environment"`, order by count desc, limit ~10. Use a
  **24h** window (7-day namespace scans time out on large tenants; count is the proxy, since the Cost
  Meter has no namespace dimension).
- Severity per namespace: `signoz_aggregate_logs`, `count`,
  `groupBy: "k8s.namespace.name,severity_text"`.
- Samples: `signoz_search_logs`, `filter: "k8s.namespace.name = '<ns>'"`, small limit; read
  `body`, `severity_text`, `scope_name`.

> **Namespace severity guard.** high-signal % = (all HIGH-SIGNAL severities, including ERROR, FATAL,
> CRITICAL, WARN, **and WARNING**, matched case-insensitively) ÷ namespace
> total. **If it is > 1% (or a non-trivial absolute count), do not consider dropping or
> filtering the whole namespace.** Scope the filter to the specific noisy pattern (a log
> category, a `severity_text` match, or a component). Never drop a namespace that carries active
> errors. Only namespaces that are essentially all INFO/DEBUG are wholesale-filter candidates,
> subject to Step 3c.

- Empty `severity_text` on samples → the forwarder ships raw lines unparsed; the fix is a
  json/regex parser in the OTel Collector log pipeline (without it, severity filtering is
  impossible).
- `k8s.event.*` logs are often high-volume / low-value → droppable if not alerted on.

**3c. Log alerts: check before any log-reduction recommendation.** `signoz_list_alert_rules`,
**paginating through every page** (follow `pagination.nextOffset` until `pagination.hasMore` is
false; do not stop at the first page, or an alert on a later page is missed and a filter looks
safe when it isn't). Keep `alertType == "LOGS_BASED_ALERT"`. For each, `signoz_get_alert(id)` and read
`condition.compositeQuery.queries[].spec.filter.expression` + `groupBy` to see which service /
severity / namespace it guards. If a filter would blind an alert, mark it **Will break alert
coverage** and name the alert. Docs: https://signoz.io/docs/logs-management/guides/drop-logs/

### Step 4: Traces

Run when the Cost Meter shows span data, ordered by its cost contribution, or when the user
explicitly asks about span cost.

**4a. Global operation-name view, read this first.** `signoz_aggregate_traces`,
`aggregation: count`, `groupBy: "name"`, `orderBy: "count() desc"`, limit 20, **no service
filter**. This surfaces auto-instrumentation noise (health checks, SQL, cache, sidecars) across
all services at once.

**4b. Cost per service + ops per service.** Span GB by service: use
`signoz_execute_builder_query` with the discovered meter metric whose live unit and meaning
represent span bytes, summed as in Step 1 and grouped by the `service.name` field returned by
`signoz_get_field_keys` (copying its `name`, `fieldDataType`, `fieldContext`, and `signal`).
Compute each service's % against the **grouped total (sum of all service groups from this same
query)**, not a top-N sum and not the separately-fetched ungrouped total; keep numerator and
denominator on one grouped basis. For each top-3
service by GB, get its dominant operations with `signoz_get_service_top_operations` (or
`signoz_aggregate_traces` with `service: "<svc>"` + `groupBy: "name"`). The op breakdown is
top-3 for brevity, but the error-rate check (4c) and the APM guard below apply to **every**
service you consider reducing.

**4c. Error rate per service.** `signoz_aggregate_traces`, `count`, `groupBy: "service.name"`
(total), then again with `error: true`. Error rate = errors ÷ total, per service.

**4d. Trace alerts.** `signoz_list_alert_rules`, **paginating through every page** (follow
`pagination.nextOffset` until `pagination.hasMore` is false). Keep
`alertType == "TRACES_BASED_ALERT"`; `signoz_get_alert(id)` for what each guards.

**Classify the dominant operations.**
- Common noise candidates: health/liveness (`/health`, `/ping`, `/ready`,
  `grpc.health.v1.Health/Check`); proxy/sidecar (`envoy.*`, `istio.*`, `linkerd.*`). Treat these as
  candidates, not automatically safe removals; the APM guard below still applies.
- Research before concluding, and do not assume: SQL fragments (the language decides the library:
  Java → JDBC, Python → SQLAlchemy, Node → pg/mysql2/sequelize, .NET → EF/Dapper); cache commands
  (HMGET/GET/SET… → ioredis, redis-py, Jedis, go-redis…); unfamiliar gRPC methods. Search first;
  if still unidentified after searching, say so.

**Fix layer.** For a confirmed noise operation, prefer the deployed SDK or instrumentation
library's documented disable/exclusion control so generation stops at source. Identify the
language and library before naming a setting; Java, Python, Node.js, and .NET use different
controls. If no SDK control exists, use a Collector filter on the operation name.

> **Span → APM guard (mandatory).** Never recommend or configure head, probabilistic, tail, or
> any other trace sampling as a cost-reduction lever. When a user asks for sampling, state both
> effects: the built-in APM/Services metrics cover only retained traces, so absolute request counts
> and rates undercount real traffic; latency trends and error spikes may remain useful, but the APM
> page no longer represents all requests. This limited usefulness does not make sampling an allowed
> lever, and processor placement is not an exception.
>
> Use SDK exclusions and Collector filters for confirmed noise. State that the removed operation
> will disappear from APM before giving a configuration.
Docs: https://signoz.io/docs/traces-management/guides/drop-spans/

### Step 5: Report what you found

Lead with a **TL;DR**: concise, no headers, two parts:
1. **Cost orientation**: which signal is the primary driver, how much, and what generates it
   (name the service, environment, or metric). If several things together dominate, name them
   with individual shares and combined impact.
2. **Prioritized action list**: every actionable finding in priority order, each with its
   status inline: **Safe to implement** / **Needs one check first** (state what) / **Will break
   dashboards** (name them) / **Will break alert coverage** (name the alert) / **Infra-page
   dependency** (breaks the Hosts/K8s view). For metric drops, include the volume % so the
   reader knows the cost impact.

After the TL;DR, add only evidence or context that was not already stated. Rank the actions using
the decision order in Steps 2–4, but state each finding and guardrail once instead of repeating
the action list or the per-signal playbooks.

Keep it conversational: a prioritized triage list, not a formatted report with headers and
tables. Cover every signal with Cost Meter data in that single list; do not ask whether to inspect
a signal the workflow already analyzed.

## Guardrails

- **Never declare a drop or filter "safe" on a partial check.** Paginate
  `signoz_list_alert_rules` fully (through `pagination.hasMore`) before ruling out alert impact,
  and treat a non-empty `error` from `signoz_check_metric_usage` as *unknown* usage (needs a
  manual check), not as "unused." An incomplete lookup is not a green light.
- **Volume comes from discovered Cost Meter metrics and their live units:** bytes for span/log
  volume or samples for metric volume. Never cite span/log record counts as volume.
- **Cost totals and grouped total attribution use `signoz_execute_builder_query`** with raw
  `timeAggregation: sum`, hourly `stepInterval: 3600`, and complete datapoints only, never
  `signoz_query_metrics`.
- Every Cost Meter `builder_query` sent through that raw escape hatch includes
  `limit: 100` and Query Builder v5 `order: [{key:{name:"__result"},
  direction:"desc"}]`. This is wire `order`, not dashboard `orderBy`; preserve
  it in grouped queries. The limit ranks groups over the whole window, so call
  out the possibility that a short-lived group falls outside the top N.
- **Never present an Infra-page or APM metric as safe to drop** (see
  `references/infra-do-not-drop.md`), even when usage-check shows no dashboards or alerts.
- **Logs:** never recommend dropping/filtering a whole service or namespace that carries active
  ERROR/WARN (> 1% high-signal, or a non-trivial absolute count); scope the filter to the noisy
  pattern. Fix order: at source (`LOG_LEVEL=WARN`) → Collector scope filter
  (`instrumentation_scope.name`) → `severity_text` filter.
- **Logs attribution is a hard threshold:** ≥ 10% → Path A; < 10% → Path B. Compute it.
- **Traces:** never recommend or configure head, probabilistic, tail, or any other sampling.
  Sampling limits the built-in APM/Services metrics to retained traces, so absolute request counts
  and rates no longer represent all traffic.
- **UNBOUNDED and IDENTIFIER labels are always worth flagging**: the problem is trajectory, not
  just current count.
- **Never suggest changing retention. Never estimate dollar savings.** Never call a dashboard or
  alert unused/noisy/redundant; report counts only.
- **Anchor claims to query results.** If a signal has no Cost Meter data, say so; do not
  substitute count-based proxies. Don't re-list metrics already shown in the breakdown; metrics
  outside the top ~20 by volume are individually negligible.

## Additional resources

- `references/cost-meter-queries.md`: Cost Meter discovery, the full
  `signoz_execute_builder_query` template, unit conversion, and the grouped-vs-ungrouped caveat.
- `references/infra-do-not-drop.md`: the metrics behind the built-in Infrastructure and
  APM/Services pages that must never be recommended for dropping.
- `references/otel-attribute-cardinality.md`: reference for classifying metric labels
  (UNBOUNDED / ACCUMULATING / BOUNDED / IDENTIFIER).
- `signoz-generating-queries` skill: for the ad-hoc follow-up queries this investigation points to.