coralogix/cx-cli · GitHub

cx-alerts

2.2천회 설치

어떤 스킬인가요?

This skill should be used when the user asks to "manage alerts", "create alert", "list alerts", "delete alert", "check alert status", "enable alert", "disable alert", "investigate firing alerts", "check which alerts are active", "find alerting rules", "set up an alert", "configure alerting", "mute an alert", "silence an alert", "see alert definitions", "check alert priority", or wants to manage Coralogix alert definitions using the cx CLI.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

npx skills add coralogix/cx-cli --skill cx-alerts

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

`references/alert-schemas.md`

# Alert Definition Schemas Reference

Complete JSON schema reference for all Coralogix alert types, using the **actual REST API wire format**. Use this when constructing `alertDefProperties` payloads for `cx alerts create`.

> **Tip:** The easiest way to create a new alert is to fetch an existing one with `cx alerts get <id> -o json`, modify the JSON, and pipe it into `cx alerts create --from-file -`.

## Common Structure

Every alert definition has this top-level shape. The alert type config (e.g. `logsThreshold`) is a **sibling** of `type`, `name`, `priority`, etc. - NOT nested inside `type`.

```json
{
  "alertDefProperties": {
    "name": "My Alert (required)",
    "description": "What this alert monitors",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_THRESHOLD",
    "enabled": true,
    "groupByKeys": [],
    "entityLabels": {},
    "phantomMode": false,
    "activeOn": { ... },
    "incidentsSettings": { ... },
    "notificationGroup": { ... },
    "logsThreshold": { ... }
  }
}
```

### Priority values

| Wire value | Meaning |
|---|---|
| `ALERT_DEF_PRIORITY_P1` | Critical |
| `ALERT_DEF_PRIORITY_P2` | High |
| `ALERT_DEF_PRIORITY_P3` | Medium |
| `ALERT_DEF_PRIORITY_P4` | Low |
| `ALERT_DEF_PRIORITY_P5_OR_UNSPECIFIED` | Info |

### Type values

| Wire value | Alert type config key |
|---|---|
| `ALERT_DEF_TYPE_LOGS_IMMEDIATE_OR_UNSPECIFIED` | `logsImmediate` |
| `ALERT_DEF_TYPE_LOGS_THRESHOLD` | `logsThreshold` |
| `ALERT_DEF_TYPE_LOGS_ANOMALY` | `logsAnomaly` |
| `ALERT_DEF_TYPE_LOGS_RATIO_THRESHOLD` | `logsRatioThreshold` |
| `ALERT_DEF_TYPE_LOGS_NEW_VALUE` | `logsNewValue` |
| `ALERT_DEF_TYPE_LOGS_UNIQUE_COUNT` | `logsUniqueCount` |
| `ALERT_DEF_TYPE_LOGS_TIME_RELATIVE_THRESHOLD` | `logsTimeRelativeThreshold` |
| `ALERT_DEF_TYPE_METRIC_THRESHOLD` | `metricThreshold` |
| `ALERT_DEF_TYPE_METRIC_ANOMALY` | `metricAnomaly` |
| `ALERT_DEF_TYPE_TRACING_IMMEDIATE` | `tracingImmediate` |
| `ALERT_DEF_TYPE_TRACING_THRESHOLD` | `tracingThreshold` |
| `ALERT_DEF_TYPE_FLOW` | `flow` |

---

## Common Sub-Objects

### Activity Schedule (`activeOn`)

```json
{
  "dayOfWeek": ["DAY_OF_WEEK_MONDAY_OR_UNSPECIFIED", "DAY_OF_WEEK_TUESDAY"],
  "startTime": { "hours": 8, "minutes": 0 },
  "endTime": { "hours": 18, "minutes": 0 }
}
```

Day values: `DAY_OF_WEEK_MONDAY_OR_UNSPECIFIED`, `DAY_OF_WEEK_TUESDAY`, `DAY_OF_WEEK_WEDNESDAY`, `DAY_OF_WEEK_THURSDAY`, `DAY_OF_WEEK_FRIDAY`, `DAY_OF_WEEK_SATURDAY`, `DAY_OF_WEEK_SUNDAY`

### Incident Settings (`incidentsSettings`)

```json
{
  "minutes": 60,
  "notifyOn": "NOTIFY_ON_TRIGGERED_ONLY_UNSPECIFIED"
}
```

`notifyOn` values: `NOTIFY_ON_TRIGGERED_ONLY_UNSPECIFIED`, `NOTIFY_ON_TRIGGERED_AND_RESOLVED`

### Notification Group (`notificationGroup`)

```json
{
  "groupByKeys": [],
  "destinations": [
    {
      "connectorId": "uuid",
      "presetId": "uuid",
      "notifyOn": "NOTIFY_ON_TRIGGERED_AND_RESOLVED"
    }
  ],
  "webhooks": [
    {
      "integration": { "integrationId": 123 },
      "minutes": 15,
      "notifyOn": "NOTIFY_ON_TRIGGERED_ONLY_UNSPECIFIED"
    }
  ],
  "router": {
    "id": "uuid",
    "notifyOn": "NOTIFY_ON_TRIGGERED_ONLY_UNSPECIFIED"
  }
}
```

### Logs Filter (`logsFilter`)

Used by all log-based alert types:

```json
{
  "simpleFilter": {
    "luceneQuery": "severity:ERROR AND service:api",
    "labelFilters": {
      "applicationName": [
        { "operation": "LOG_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "value": "my-app" }
      ],
      "subsystemName": [
        { "operation": "LOG_FILTER_OPERATION_TYPE_STARTS_WITH", "value": "backend" }
      ],
      "severities": ["LOG_SEVERITY_WARNING", "LOG_SEVERITY_ERROR", "LOG_SEVERITY_CRITICAL"]
    }
  }
}
```

**Label filter operations:** `LOG_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED`, `LOG_FILTER_OPERATION_TYPE_INCLUDES`, `LOG_FILTER_OPERATION_TYPE_STARTS_WITH`, `LOG_FILTER_OPERATION_TYPE_ENDS_WITH`

**Severities:** `LOG_SEVERITY_VERBOSE_UNSPECIFIED`, `LOG_SEVERITY_DEBUG`, `LOG_SEVERITY_INFO`, `LOG_SEVERITY_WARNING`, `LOG_SEVERITY_ERROR`, `LOG_SEVERITY_CRITICAL`

### Tracing Filter (`tracingFilter`)

Used by tracing-based alert types:

```json
{
  "simpleFilter": {
    "latencyThresholdMs": "1000",
    "tracingLabelFilters": {
      "applicationName": [
        { "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "values": ["my-app"] }
      ],
      "serviceName": [
        { "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "values": ["api-gateway"] }
      ],
      "operationName": [],
      "subsystemName": [],
      "spanFields": [
        {
          "key": "http.status_code",
          "filterType": {
            "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED",
            "values": ["500"]
          }
        }
      ]
    }
  }
}
```

**Tracing filter operations:** `TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED`, `TRACING_FILTER_OPERATION_TYPE_INCLUDES`, `TRACING_FILTER_OPERATION_TYPE_STARTS_WITH`, `TRACING_FILTER_OPERATION_TYPE_ENDS_WITH`, `TRACING_FILTER_OPERATION_TYPE_IS_NOT`

### Undetected Values Management

```json
{
  "triggerUndetectedValues": true,
  "autoRetireTimeframe": "AUTO_RETIRE_TIMEFRAME_HOUR_1"
}
```

Values: `AUTO_RETIRE_TIMEFRAME_NEVER_OR_UNSPECIFIED`, `AUTO_RETIRE_TIMEFRAME_MINUTES_5`, `AUTO_RETIRE_TIMEFRAME_MINUTES_10`, `AUTO_RETIRE_TIMEFRAME_HOUR_1`, `AUTO_RETIRE_TIMEFRAME_HOURS_2`, `AUTO_RETIRE_TIMEFRAME_HOURS_6`, `AUTO_RETIRE_TIMEFRAME_HOURS_12`, `AUTO_RETIRE_TIMEFRAME_HOURS_24`

---

## Alert Type Schemas

Each section shows only the alert-type-specific config block. This block is placed as a sibling of `type`, `name`, `priority`, etc. inside `alertDefProperties`.

### 1. Logs Threshold (`logsThreshold`)

Trigger when log count crosses a threshold in a time window.

```json
{
  "alertDefProperties": {
    "name": "High Error Rate",
    "description": "Alert when error logs exceed threshold",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_LOGS_THRESHOLD",
    "enabled": true,
    "logsThreshold": {
      "logsFilter": {
        "simpleFilter": {
          "luceneQuery": "severity:ERROR",
          "labelFilters": {
            "applicationName": [
              { "operation": "LOG_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "value": "my-app" }
            ]
          }
        }
      },
      "rules": [{
        "condition": {
          "conditionType": "LOGS_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 100,
          "timeWindow": {
            "logsTimeWindowSpecificValue": "LOGS_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED"
          }
        },
        "override": { "priority": "ALERT_DEF_PRIORITY_P1" }
      }],
      "notificationPayloadFilter": [],
      "undetectedValuesManagement": null,
      "evaluationDelayMs": 0
    }
  }
}
```

**conditionType:** `LOGS_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED`, `LOGS_THRESHOLD_CONDITION_TYPE_LESS_THAN`

**logsTimeWindowSpecificValue:** `LOGS_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED`, `LOGS_TIME_WINDOW_VALUE_MINUTES_10`, `LOGS_TIME_WINDOW_VALUE_MINUTES_15`, `LOGS_TIME_WINDOW_VALUE_MINUTES_20`, `LOGS_TIME_WINDOW_VALUE_MINUTES_30`, `LOGS_TIME_WINDOW_VALUE_HOUR_1`, `LOGS_TIME_WINDOW_VALUE_HOURS_2`, `LOGS_TIME_WINDOW_VALUE_HOURS_4`, `LOGS_TIME_WINDOW_VALUE_HOURS_6`, `LOGS_TIME_WINDOW_VALUE_HOURS_12`, `LOGS_TIME_WINDOW_VALUE_HOURS_24`, `LOGS_TIME_WINDOW_VALUE_HOURS_36`

### 2. Logs Immediate (`logsImmediate`)

Trigger instantly on every matching log entry. No rules or time windows.

```json
{
  "alertDefProperties": {
    "name": "OOM Killer Detected",
    "description": "Alert immediately when OOM killer runs",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_LOGS_IMMEDIATE_OR_UNSPECIFIED",
    "enabled": true,
    "logsImmediate": {
      "logsFilter": {
        "simpleFilter": {
          "luceneQuery": "\"Out of memory\" OR \"OOM\"",
          "labelFilters": {}
        }
      },
      "notificationPayloadFilter": []
    }
  }
}
```

### 3. Logs Anomaly (`logsAnomaly`)

ML-based anomaly detection on log volume.

```json
{
  "alertDefProperties": {
    "name": "Unusual Log Volume",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_ANOMALY",
    "enabled": true,
    "logsAnomaly": {
      "logsFilter": { "simpleFilter": { "luceneQuery": "*", "labelFilters": {} } },
      "rules": [{
        "condition": {
          "conditionType": "LOGS_ANOMALY_CONDITION_TYPE_MORE_THAN_USUAL_OR_UNSPECIFIED",
          "minimumThreshold": 10,
          "timeWindow": {
            "logsTimeWindowSpecificValue": "LOGS_TIME_WINDOW_VALUE_HOUR_1"
          }
        }
      }],
      "anomalyAlertSettings": { "percentageOfDeviation": 50 },
      "notificationPayloadFilter": [],
      "evaluationDelayMs": 0
    }
  }
}
```

**conditionType:** `LOGS_ANOMALY_CONDITION_TYPE_MORE_THAN_USUAL_OR_UNSPECIFIED`

### 4. Logs Ratio Threshold (`logsRatioThreshold`)

Alert based on ratio between two log queries.

```json
{
  "alertDefProperties": {
    "name": "Error Rate Ratio",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_LOGS_RATIO_THRESHOLD",
    "enabled": true,
    "logsRatioThreshold": {
      "numerator": { "simpleFilter": { "luceneQuery": "severity:ERROR", "labelFilters": {} } },
      "numeratorAlias": "Errors",
      "denominator": { "simpleFilter": { "luceneQuery": "*", "labelFilters": {} } },
      "denominatorAlias": "All Logs",
      "rules": [{
        "condition": {
          "conditionType": "LOGS_RATIO_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 0.1,
          "timeWindow": {
            "logsRatioTimeWindowSpecificValue": "LOGS_RATIO_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED"
          }
        }
      }],
      "groupByFor": "LOGS_RATIO_GROUP_BY_FOR_BOTH_OR_UNSPECIFIED",
      "ignoreInfinity": true,
      "notificationPayloadFilter": []
    }
  }
}
```

**conditionType:** `LOGS_RATIO_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED`, `LOGS_RATIO_CONDITION_TYPE_LESS_THAN`

**groupByFor:** `LOGS_RATIO_GROUP_BY_FOR_BOTH_OR_UNSPECIFIED`, `LOGS_RATIO_GROUP_BY_FOR_NUMERATOR_ONLY`, `LOGS_RATIO_GROUP_BY_FOR_DENUMERATOR_ONLY`

**logsRatioTimeWindowSpecificValue:** `LOGS_RATIO_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED`, `..._MINUTES_10`, `..._MINUTES_15`, `..._MINUTES_30`, `..._HOUR_1`, `..._HOURS_2`, `..._HOURS_4`, `..._HOURS_6`, `..._HOURS_12`, `..._HOURS_24`, `..._HOURS_36`

### 5. Logs Time Relative Threshold (`logsTimeRelativeThreshold`)

Compare current log volume to a past time period.

```json
{
  "alertDefProperties": {
    "name": "Spike vs Yesterday",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_TIME_RELATIVE_THRESHOLD",
    "enabled": true,
    "logsTimeRelativeThreshold": {
      "logsFilter": { "simpleFilter": { "luceneQuery": "severity:ERROR", "labelFilters": {} } },
      "rules": [{
        "condition": {
          "conditionType": "LOGS_TIME_RELATIVE_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 1.5,
          "comparedTo": "LOGS_TIME_RELATIVE_COMPARED_TO_SAME_HOUR_YESTERDAY"
        }
      }],
      "ignoreInfinity": true,
      "notificationPayloadFilter": []
    }
  }
}
```

**conditionType:** `LOGS_TIME_RELATIVE_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED`, `LOGS_TIME_RELATIVE_CONDITION_TYPE_LESS_THAN`

**comparedTo:** `LOGS_TIME_RELATIVE_COMPARED_TO_PREVIOUS_HOUR_OR_UNSPECIFIED`, `..._SAME_HOUR_YESTERDAY`, `..._SAME_HOUR_LAST_WEEK`, `..._YESTERDAY`, `..._SAME_DAY_LAST_WEEK`, `..._SAME_DAY_LAST_MONTH`

### 6. Logs Unique Count (`logsUniqueCount`)

Alert when unique value count in a field crosses a threshold.

```json
{
  "alertDefProperties": {
    "name": "Too Many Unique IPs",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_UNIQUE_COUNT",
    "enabled": true,
    "logsUniqueCount": {
      "logsFilter": { "simpleFilter": { "luceneQuery": "*", "labelFilters": {} } },
      "uniqueCountKeypath": "remote_addr",
      "maxUniqueCountPerGroupByKey": "1000",
      "rules": [{
        "condition": {
          "maxUniqueCount": "500",
          "timeWindow": {
            "logsUniqueValueTimeWindowSpecificValue": "LOGS_UNIQUE_VALUE_TIME_WINDOW_VALUE_MINUTES_5"
          }
        }
      }],
      "notificationPayloadFilter": []
    }
  }
}
```

**logsUniqueValueTimeWindowSpecificValue:** `LOGS_UNIQUE_VALUE_TIME_WINDOW_VALUE_MINUTE_1_OR_UNSPECIFIED`, `..._MINUTES_5`, `..._MINUTES_10`, `..._MINUTES_15`, `..._MINUTES_20`, `..._MINUTES_30`, `..._HOURS_1`, `..._HOURS_2`, `..._HOURS_4`, `..._HOURS_6`, `..._HOURS_12`, `..._HOURS_24`, `..._HOURS_36`

### 7. Logs New Value (`logsNewValue`)

Alert when a value not previously seen appears in a field.

```json
{
  "alertDefProperties": {
    "name": "New IP Address",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_NEW_VALUE",
    "enabled": true,
    "logsNewValue": {
      "logsFilter": { "simpleFilter": { "luceneQuery": "*", "labelFilters": {} } },
      "rules": [{
        "condition": {
          "keypathToTrack": "ip_address",
          "timeWindow": {
            "logsNewValueTimeWindowSpecificValue": "LOGS_NEW_VALUE_TIME_WINDOW_VALUE_HOURS_24"
          }
        }
      }],
      "notificationPayloadFilter": []
    }
  }
}
```

**logsNewValueTimeWindowSpecificValue:** `LOGS_NEW_VALUE_TIME_WINDOW_VALUE_HOURS_12_OR_UNSPECIFIED`, `..._HOURS_24`, `..._HOURS_48`, `..._HOURS_72`, `..._WEEK_1`, `..._MONTH_1`, `..._MONTHS_2`, `..._MONTHS_3`

### 8. Metric Threshold (`metricThreshold`)

Trigger when a PromQL expression crosses a threshold.

```json
{
  "alertDefProperties": {
    "name": "CPU Usage Critical",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_METRIC_THRESHOLD",
    "enabled": true,
    "metricThreshold": {
      "metricFilter": {
        "promql": "avg(cpu_usage_percent{service=\"api\"})"
      },
      "rules": [{
        "condition": {
          "conditionType": "METRIC_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 90,
          "ofTheLast": {
            "dynamicDuration": "5m"
          },
          "forOverPct": 100
        }
      }],
      "missingValues": {
        "replaceWithZero": true,
        "minNonNullValuesPct": 0
      },
      "undetectedValuesManagement": null,
      "evaluationDelayMs": 0
    }
  }
}
```

**conditionType:** `METRIC_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED`, `..._MORE_THAN_OR_EQUALS`, `..._LESS_THAN`, `..._LESS_THAN_OR_EQUALS`

**dynamicDuration:** any PromQL duration string (e.g. `5m`, `1h`, `24h`) within 1-2160 minutes. This is a free-form string, not an enum.

**forOverPct:** percentage of data points that must breach (0-100).

### 9. Metric Anomaly (`metricAnomaly`)

ML-based anomaly detection on metrics.

```json
{
  "alertDefProperties": {
    "name": "Unusual Request Rate",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_METRIC_ANOMALY",
    "enabled": true,
    "metricAnomaly": {
      "metricFilter": {
        "promql": "rate(http_requests_total[5m])"
      },
      "rules": [{
        "condition": {
          "conditionType": "METRIC_ANOMALY_CONDITION_TYPE_MORE_THAN_USUAL_OR_UNSPECIFIED",
          "threshold": 10,
          "ofTheLast": {
            "specificValue": "METRIC_TIME_WINDOW_VALUE_HOUR_1"
          },
          "forOverPct": 100,
          "minNonNullValuesPct": 50
        }
      }],
      "anomalyAlertSettings": { "percentageOfDeviation": 50 },
      "evaluationDelayMs": 0
    }
  }
}
```

**conditionType:** `METRIC_ANOMALY_CONDITION_TYPE_MORE_THAN_USUAL_OR_UNSPECIFIED`, `METRIC_ANOMALY_CONDITION_TYPE_LESS_THAN_USUAL`

**specificValue (timeWindow):** `METRIC_TIME_WINDOW_VALUE_MINUTES_1_OR_UNSPECIFIED`, `..._MINUTES_5`, `..._MINUTES_10`, `..._MINUTES_15`, `..._MINUTES_20`, `..._MINUTES_30`, `..._HOUR_1`, `..._HOURS_2`, `..._HOURS_4`, `..._HOURS_6`, `..._HOURS_12`, `..._HOURS_24`, `..._HOURS_36`

### 10. Tracing Immediate (`tracingImmediate`)

Trigger instantly on matching trace spans.

```json
{
  "alertDefProperties": {
    "name": "Slow API Call",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_TRACING_IMMEDIATE",
    "enabled": true,
    "tracingImmediate": {
      "tracingFilter": {
        "simpleFilter": {
          "latencyThresholdMs": "1000",
          "tracingLabelFilters": {
            "serviceName": [
              {
                "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED",
                "values": ["api-gateway"]
              }
            ],
            "applicationName": [],
            "operationName": [],
            "subsystemName": [],
            "spanFields": []
          }
        }
      },
      "notificationPayloadFilter": []
    }
  }
}
```

### 11. Tracing Threshold (`tracingThreshold`)

Trigger when span count crosses a threshold.

```json
{
  "alertDefProperties": {
    "name": "High Span Volume",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_TRACING_THRESHOLD",
    "enabled": true,
    "tracingThreshold": {
      "tracingFilter": {
        "simpleFilter": {
          "latencyThresholdMs": "0",
          "tracingLabelFilters": {
            "serviceName": [
              {
                "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED",
                "values": ["api-gateway"]
              }
            ],
            "applicationName": [],
            "operationName": [],
            "subsystemName": [],
            "spanFields": []
          }
        }
      },
      "rules": [{
        "condition": {
          "conditionType": "TRACING_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "spanAmount": 100,
          "timeWindow": {
            "tracingTimeWindowSpecificValue": "TRACING_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED"
          }
        }
      }],
      "notificationPayloadFilter": []
    }
  }
}
```

**conditionType:** `TRACING_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED`

**tracingTimeWindowSpecificValue:** `TRACING_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED`, `..._MINUTES_10`, `..._MINUTES_15`, `..._MINUTES_20`, `..._MINUTES_30`, `..._HOUR_1`, `..._HOURS_2`, `..._HOURS_4`, `..._HOURS_6`, `..._HOURS_12`, `..._HOURS_24`, `..._HOURS_36`

### 12. SLO Threshold (`sloThreshold`)

Monitor error budget consumption or burn rate. Exactly one of `errorBudget` or `burnRate` must be set.

#### Error Budget variant:

```json
{
  "alertDefProperties": {
    "name": "SLO Budget Low",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_SLO_THRESHOLD",
    "enabled": true,
    "sloThreshold": {
      "sloDefinition": { "sloId": "uuid" },
      "errorBudget": {
        "rules": [{
          "condition": { "threshold": 50 },
          "override": { "priority": "ALERT_DEF_PRIORITY_P1" }
        }]
      }
    }
  }
}
```

#### Burn Rate variant:

```json
{
  "alertDefProperties": {
    "name": "SLO Burn Rate High",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_SLO_THRESHOLD",
    "enabled": true,
    "sloThreshold": {
      "sloDefinition": { "sloId": "uuid" },
      "burnRate": {
        "rules": [{
          "condition": { "threshold": 2.0 },
          "override": { "priority": "ALERT_DEF_PRIORITY_P1" }
        }],
        "single": {
          "timeDuration": { "duration": "1", "unit": "DURATION_UNIT_HOURS" }
        }
      }
    }
  }
}
```

`unit` values: `DURATION_UNIT_UNSPECIFIED`, `DURATION_UNIT_HOURS`

---

## Important Notes

- **groupByKeys for metric alerts**: Leave empty to let the API infer from the PromQL `by` clause. If provided, keys must be in **alphabetical order** (the API infers alphabetically, not in query order).
- **Priority**: Always ask the user -- never pick a default.
- **override in rules**: Optional per-rule priority override using the same `ALERT_DEF_PRIORITY_*` enum values. Omit to use the alert-level priority.
- **notificationPayloadFilter**: List of log/span field paths to include in notifications (e.g. `["obj.field"]`).
- **Best practice**: Fetch an existing alert with `cx alerts get <id> -o json` to see the exact response shape, then use it as a template for creating new alerts.

`references/dataprime-reference.md`

# DataPrime Query Language Reference

## Query Structure

A DataPrime query is a pipeline of commands separated by `|`. Each command transforms the output of the previous one:

```dataprime
filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors
```

### Source Handling

Every query targets a **source** (`logs`, `spans`, etc.). The source is set by whichever `cx` command you use. A full query with an explicit source looks like:

```dataprime
source <logs|spans> | filter ... | groupby ...
```

When running via a source-specific command (e.g. `cx logs`, `cx spans`), the source is injected automatically - omit it from the query. When running via `cx dataprime query`, use the `--source` flag or include `source` in the query itself.

The examples below focus on the DataPrime query language and omit the source and CLI command prefix.

### Comments

Comments are supported with `#` or `//`:

```dataprime
filter $m.severity == ERROR  # only errors
| limit 10                   // cap results
```

## Data Prefixes

All fields are accessed through three namespaces:

| Prefix | Description | Examples |
|--------|-------------|----------|
| `$m` | Metadata (system-managed) | `$m.timestamp`, `$m.severity`, `$m.duration` |
| `$l` | Labels (indexed key-value pairs) | `$l.applicationname`, `$l.subsystemname`, `$l.serviceName` |
| `$d` | User data (application payload) | `$d.message`, `$d.user_id`, `$d.traceID` |

`$d` is the default prefix and can sometimes be omitted, but being explicit avoids ambiguity.

## Data Types

| Type | Description | Example |
|------|-------------|---------|
| `string` | Text, enclosed in **single quotes** | `'some_text'` |
| `number` | Numeric value | `123`, `3.14` |
| `boolean` | True or false | `true`, `false` |
| `timestamp` | Date and time (nanoseconds since epoch) | `1714636800000000000` |
| `interval` | Time duration | `1h`, `1d`, `1w` |
| `array` | List of values | `[1, 2, 3]` |
| `object` | Key-value pairs | `{"name": "John"}` |
| `null` | Missing value or key | `null` |

## Commands

### Filtering and Selection

| Command | Description | Example |
|---------|-------------|---------|
| `filter` | Keep rows matching a condition | `filter $m.severity == ERROR` |
| `choose` | Select specific fields | `choose $m.timestamp, $d.message` |
| `limit` | Cap the number of results | `limit 10` |
| `wildfind` | Token match across the whole record (see note below) | `wildfind 'connection refused'` |
| `lucene` | Filter using Lucene syntax (`field:value`, field names relative to `$d`, combine with `AND`/`OR`/parens) | `lucene 'field:"value"'` |

> **Note on `wildfind`:** It is a standalone command, not a condition within `filter`. You cannot combine it with other filter expressions - use it as its own pipeline stage. `wildfind` with very short search terms (only a few characters) is much slower — prefer longer, more specific terms.

### Aggregation

| Command | Description | Example |
|---------|-------------|---------|
| `groupby` | Group rows and apply aggregations | `groupby $l.subsystemname aggregate count() as n` |
| `multigroupby` | Group by multiple field sets | `multigroupby a, b aggregate count()` |
| `count` | Count all rows | `count` |
| `countby` | Count rows grouped by a field | `countby $l.applicationname` |
| `distinct` | Return unique values of a field | `distinct $l.subsystemname` |

### Transformation

| Command | Description | Example |
|---------|-------------|---------|
| `create` | Add a computed field | `create latency_ms from $m.duration / 1000` |
| `orderby` | Sort results | `orderby $d.timestamp desc` |
| `extract` | Parse fields with regex or JSON | See [Text Extraction](#text-extraction) |
| `dedupeby` | Remove duplicates by a field (cost grows with number of distinct keys) | `dedupeby $m.templateid` |

## Operators

| Operator | Description | Example |
|----------|-------------|---------|
| `==` | Equals | `filter $m.severity == ERROR` |
| `!=` | Not equals | `filter $l.subsystemname != 'test'` |
| `>`, `<`, `>=`, `<=` | Comparison | `filter $d.response_time > 1000` |
| `~` | Case-insensitive token match (matches whole tokens, not arbitrary substrings) | `filter $d.message ~ 'timeout'` |
| `&&` | AND | `filter $m.severity == ERROR && $l.applicationname == 'api'` |
| `\|\|` | OR | `filter $m.severity == ERROR \|\| $m.severity == CRITICAL` |
| `!= null` | Field exists | `filter $d.some_field != null` |

> **`~` vs `contains()`:** `~` is a **case-insensitive, token-based** match — it matches whole tokens (words), so `~ 'timeout'` also matches `TIMEOUT` and `Timeout`. `contains()` is a **case-sensitive raw substring** match — `contains('time')` matches inside `timeout`, but only in that exact case. Use `~` for word/term search; use `contains()` when you need an exact-case partial-string match.

## Type Conversions

Cast fields inline with `:type`:

```dataprime
filter $d.http_error_code:number == 500
```

Supported types: `bool`, `number`, `string`, `timestamp`, `interval`, `array`, `object`

## Field Access

```dataprime
# Chained field names (dot notation)
filter $d.tags.user_context.email == 'test@example.com'

# Special characters require brackets
filter $d.http['status/code'] == 500
```

## Aggregation Functions

| Function | Description |
|----------|-------------|
| `count()` | Count rows |
| `sum($field)` | Sum values |
| `avg($field)` | Average |
| `min($field)` | Minimum |
| `max($field)` | Maximum |
| `percentile(0.95, $field)` | Percentile |
| `median($field)` | Median value |
| `stddev($field)` | Standard deviation |
| `variance($field)` | Variance |
| `distinct_count($field)` | Count unique values (exact) |
| `approx_count_distinct($field)` | Approximate count of unique values |
| `any_value($field)` | Random sample value |
| `collect($field)` | Collect values into an array |

> **`distinct_count` vs `approx_count_distinct`:** Exact `distinct_count` can be slow or run out of memory on high-cardinality fields. When an exact count isn't required, use `approx_count_distinct` instead.

Example - full CLI invocation:
```bash
cx dataprime query --source logs 'groupby $l.subsystemname aggregate count() as error_count, avg($d.response_time) as avg_response | orderby error_count desc'
```

## Utility Functions

### firstNonNull - Field Coalescing

Return the first non-null value from a list of fields. Useful when the same data may appear in different fields across log sources:

```dataprime
# Merge fields
create message from firstNonNull($d.error_message, $d.msg, $d.body)

# Use inside groupby
groupby firstNonNull($d.error_message, $d.msg) as message aggregate count() as n
```

### Template Sampling

Find top error patterns with a sample message for each:

```dataprime
filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5
```

## Time-Based Grouping

Use `roundTime()` to bucket timestamps:

```dataprime
# Group by hour
groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count

# Error rate over 15-minute intervals
filter $m.severity == ERROR | groupby roundTime($m.timestamp, 15m) as interval aggregate count() as errors
```

## Multi-Value Matching

Use `arrayContains` to match against a set of values:

```dataprime
# Match multiple subsystems
filter ['api', 'web', 'worker'].arrayContains($l.subsystemname)

# Match multiple severity levels
filter [ERROR, CRITICAL].arrayContains($m.severity)
```

## Text Extraction

### Regex Extraction

```dataprime
# Extract with unnamed capture group
extract $d.email into domain using regexp(e=/@(.*)/) | distinct $d.domain._0

# Named capture groups
extract $d.email into extracted using regexp(e=/(?<username>[a-zA-Z0-9._%+-]+)@(?<domain>.*)/) | choose $d.extracted.username, $d.extracted.domain
```

### JSON String Parsing

```dataprime
# Parse a JSON string field into an object for further querying
extract $d.json_payload into parsed using jsonobject() | filter $d.parsed.status == 'failed'
```

## Deduplication

```dataprime
# Remove duplicates by log template
dedupeby $m.templateid

# Keep one row per key (cost grows with the number of distinct keys)
dedupeby $d.session_id
```

> **Performance:** `dedupeby` cost scales with the number of **distinct keys** it tracks, so it gets slow and memory-heavy on high-cardinality or near-unique keys (e.g. a request id). Cardinality depends on your data and time window — a key like session or trace id can still be large. Keep `dedupeby` when you genuinely need one representative row per key; to cut cost, narrow the input first (tighter `filter`, smaller time window) rather than swapping in `filter`/`limit` (which does **not** keep one row per key) or an aggregation (which changes the row shape) unless that different output is acceptable. The `dedupeby <key> orderby ...` form (keep the latest row per key) is heavier still.

## Built-In Documentation

For the full list of commands and functions with detailed syntax:

```bash
cx dataprime list                              # List all commands and functions
cx dataprime list --filter commands             # Commands only
cx dataprime list --filter functions --name time # Search functions by name
cx dataprime show filter                        # Detailed help for a specific command
cx dataprime show groupby
```

## Validating a DataPrime query

A query that looks right can still fail on a typoed field path, an invented function, or a malformed pipeline stage. Validate before trusting the output — a short-window run through the CLI is cheap and catches almost all of these:

```bash
cx logs  '<pipeline>' --start now-15m --end now --limit 1
cx spans '<pipeline>' --start now-15m --end now --limit 1
```

`now-15m` is a good default; widen it only if 15 minutes is unlikely to exercise the pipeline. Per "Source Handling" above, omit any leading `source logs` / `source spans` — `cx logs` and `cx spans` inject the source themselves.

Check both the exit code and the output — some errors surface only in the output.

**Pass** = exit 0 and the output is rows or `[]` with no error or warning lines.

**Hard fail** — query is broken, fix it:
- non-zero exit
- `error from profile '...': API request failed` — HTTP error from the API
- `Compilation errors:` — parse error, unknown function, malformed expression

**Soft fail** (needs investigation):
- `keypath does not exist` — the query parsed, but no record in the window had the referenced field. This is ambiguous: the field name might be a typo, or it might be real but absent from records in this 15-minute slice. Confirm with `cx search-fields "<field hint>" --dataset logs` (or `--dataset spans`). If the field is real, the query is fine — try a wider window or accept the empty result. If it isn't, fix the field name.

On fail: re-discover fields with `cx search-fields`, look up command syntax with `cx dataprime show <command>`, fix, re-run.

`references/logs-querying.md`

# Log Querying Reference

Query and analyze Coralogix logs using the `cx logs` command with DataPrime syntax.

> **DataPrime syntax:** See `dataprime-reference.md` for the full query language reference.

## Understanding Logs in Coralogix

Logs in Coralogix are **largely unstructured**. Every log entry has a small structured envelope - metadata and labels - but the actual application payload (`userData`) is free-form and varies entirely by application. There is no universal schema for `$d.*` fields.

This means:
- **Metadata (`$m.*`)** and **labels (`$l.*`)** are predictable - you can always filter on severity, timestamp, application name, and subsystem name without discovery.
- **User data (`$d.*`)** is not predictable - field names, nesting, and types depend on whatever the application chose to log. Always verify `$d` fields before assuming they exist.

---

## CLI Command

```bash
cx logs '<dataprime_query>'
```

The `source logs` prefix is automatically injected if the query doesn't already include a `source` command.

### Options

| Flag | Default | Description |
|------|---------|-------------|
| `--start` | `now-1h` | Start time (ISO 8601 or relative, e.g. `now-6h`) |
| `--end` | `now` | End time |
| `--limit` | `100` | Maximum number of results |
| `--tier` | `frequent` | Storage tier: `frequent` (hot/recent) or `archive` (cold/historical) |
| `-o, --output` | `text` | Output format: `text`, `json`, or `toon` |

---

## Log Data Model

### Standard Fields (Always Available)

| Field | Description |
|-------|-------------|
| `$m.timestamp` | Log timestamp |
| `$m.severity` | Severity level (see below) |
| `$m.templateid` | Log template identifier (groups structurally similar logs) |
| `$l.applicationname` | Application name - the highest-level label. All data in Coralogix is tagged with it. Meaning varies by customer (environment, team, region) but it always exists. |
| `$l.subsystemname` | Subsystem name - second highest-level label. All data is tagged with it. Typically maps to a service or component. |
| `$d.*` | User data - free-form, application-specific (see [Field Discovery](#field-discovery)) |

### Severity Values

Severity keywords are used **without quotes** in DataPrime:

`VERBOSE` | `DEBUG` | `INFO` | `WARNING` | `ERROR` | `CRITICAL`

```bash
cx logs 'filter $m.severity == ERROR'
cx logs 'filter [ERROR, CRITICAL].arrayContains($m.severity)'
```

---

## Essential Query Examples

```bash
# Filter by severity
cx logs 'filter $m.severity == ERROR'

# Text search in a known field
cx logs "filter \$d.message ~ 'timeout'"

# Filter by application and subsystem
cx logs "filter \$l.applicationname == 'api' && \$l.subsystemname == 'auth'"

# Aggregate errors by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'

# Wider time range and archive tier
cx logs "filter \$l.subsystemname == 'payments'" --tier archive --start now-7d
```

> **`~` (text match):** `~` is a **case-insensitive, token-based** match — `~ 'timeout'` also matches `TIMEOUT`/`Timeout`, and it matches whole tokens (words), not arbitrary substrings. For a **case-sensitive raw substring** match, use `contains()` instead. See the operator table in `dataprime-reference.md`.

### Wildfind Policy

`wildfind` runs the same **case-insensitive, token-based** match as `~` (whole tokens, not arbitrary substrings), but over the whole record instead of a single field — it matches tokens anywhere, in any field.

**Prefer a field-targeted `~`/`filter` when you know the field** — for **precision**: because `wildfind` matches tokens anywhere in the record, it can't be narrowed to a specific field and will match the term in fields you didn't intend.

**Performance:** `wildfind` with very short search terms (only a few characters) is much slower — prefer longer, more specific terms.

The **one exception**: when the user provides a specific, quoted error message or log string and you don't know which field contains it:

```bash
# User says: "Find logs with 'connection refused'"
cx logs "wildfind 'connection refused'"
```

In all other cases, use `filter` with known fields (`$m.severity`, `$l.subsystemname`, `$d.<field>`) or discover field names first with `cx search-fields`.

---

## Field Discovery

**Skip discovery when:**
- The query only uses standard fields (`$m.severity`, `$m.timestamp`, `$l.applicationname`, `$l.subsystemname`)
- The user explicitly names the fields they want (e.g., "filter by `$d.customer_id`")
- You're searching for a specific error message - use `wildfind` directly
- The fields have already been discovered earlier in the conversation

For customer-specific `$d.*` fields that need discovery, use one of these approaches:

### 1. Infer from Source Code (Preferred)

If you have access to the application's source code, examine logger calls, structured logging configs, and log format templates to identify field names directly.

### 2. Semantic Search

```bash
cx search-fields "customer identifier" --dataset logs
cx search-fields "http response code" --dataset logs
```

Returns DataPrime paths with similarity scores:

```
+------------------------+-----------------------------------+-----------+
| DataPrime path         | Description                       | Similarity|
+------------------------+-----------------------------------+-----------+
| $d.customer_id         | Unique customer identifier        | 0.89      |
| $d.user.account_id     | Customer account reference        | 0.85      |
+------------------------+-----------------------------------+-----------+
```

### 3. Sample Query Inspection

```bash
cx logs "filter \$l.subsystemname == 'api'" --limit 5 -o json
```

Inspect the JSON output to see all available fields in the actual data.

---

## Investigation Workflow

### 1. Understand the Request

Identify:
- What type of logs are needed (errors, info, specific events)
- Time frame of interest
- Key entities (services, users, transactions)

### 2. Start with Standard Fields

For basic queries, use standard fields directly:

```bash
# Recent errors - no discovery needed
cx logs 'filter $m.severity == ERROR | limit 20'

# Errors in a specific subsystem
cx logs "filter \$m.severity == ERROR && \$l.subsystemname == 'payment-service'"
```

### 3. Build and Execute Query

Start simple, add complexity:

```bash
# Step 1: Check if data exists
cx logs "filter \$l.subsystemname == 'checkout'" --limit 10

# Step 2: Add filters
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR"

# Step 3: Add aggregation
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR | groupby \$d.error_type aggregate count() as occurrences"
```

### 4. Troubleshooting

If a query returns no results, change **one thing at a time**. Keep the query window as **narrow** as possible and widen it deliberately — start from the window you already have rather than jumping to a huge range:

1. **Relax filters**: remove the most restrictive condition
2. **Verify field names**: run a sample query with `-o json` to inspect the actual schema
3. **Extend the time range**: widen gradually from your current window (e.g. `now-1h` → `now-6h` → `now-24h`)
4. **Try archive tier**: for older data, add `--tier archive` and widen the window to cover the period you're after

---

## Common Query Patterns

### Error Investigation

```bash
# All errors in last hour
cx logs 'filter $m.severity == ERROR'

# Critical errors only
cx logs 'filter $m.severity == CRITICAL'

# Errors with text search
cx logs "filter \$m.severity == ERROR && \$d.message ~ 'database connection'"
```

### Aggregation by Service

```bash
# Error count by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'

# Error count by application and subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.applicationname, $l.subsystemname aggregate count() as errors'
```

### Time-Based Analysis

```bash
# Errors per hour
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count'

# Find error spikes in 5-minute windows
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 5m) as interval aggregate count() as count | orderby count desc | limit 10'
```

### Finding Unique Values

```bash
# List all subsystems with errors
cx logs 'filter $m.severity == ERROR | distinct $l.subsystemname'

# List unique error types
cx logs 'filter $m.severity == ERROR | distinct $d.error_type'
```

### Fetching Sample Logs by Template

Find top error patterns with sample messages:

```bash
cx logs 'filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5'
```

---

## Performance Tips

- Use `--limit` for exploratory queries
- Use `groupby` with aggregations instead of fetching all raw logs
- Filter by time first when dealing with large datasets
- Use specific filters (application, subsystem) to reduce scan scope
- For large result sets, use `--output toon` which spills to a temp file automatically:

```bash
cx logs 'filter $m.severity == ERROR' --start now-24h --limit 1000 -o toon
```

`references/promql-guidelines.md`

# PromQL Guidelines

## Core Principles

1. **Pick the right query type**
   - **Instant queries** (`cx metrics query`) evaluate an expression at a single timestamp (now, or a given `--time`). Use when the question requires **one number or one vector** *as of* a moment - essentially any query that does not require results over different timeframes.
   - **Range queries** (`cx metrics query-range`) evaluate the expression **repeatedly** across `[--start, --end]` at a given `--step`. Use for **time series** over a period (e.g., daily active users per day).
     - Note: Range queries **evaluate the expression repeatedly** at each step. If `--step=1d`, `--start=now-1d`, `--end=now`, and the query is `max_over_time(metric[1d])`, the query evaluates at `now-1d` and `now` - two evaluations covering two days of data.
   - Prefer instant queries over range queries for most questions, except when comparing different timeframes.

2. **Understand PromQL value types**
   - **Instant vector** - set of series with 1 sample each at eval time
   - **Range vector** - series with many samples over a window `[t-range, t]`
   - **Scalar** - single number
   - **String** - rare
   - Functions like `*_over_time()` **require a range vector**. Aggregations like `sum/max/min/avg ... by(...)` **consume instant vectors**.
   - **Important**: When using `*_over_time()` functions with range queries, be aware that the query also evaluates at the `--start` time and includes the window specified in the function.
   - **Example**: If `max_over_time(metric[1d])` is used with `--start=now-1d`, `--end=now`, `--step=1d`, the query evaluates at `now-1d` and `now` - the result is the max over `[now-2d, now]`. This is a common mistake. If a user asks "What is the max of x between 2025-01-01 and 2025-01-07?" and `max_over_time(x[7d])` is used with `--start=2025-01-01`, `--end=2025-01-07`, `--step=1d`, the evaluation at `2025-01-01` includes `[2024-12-25, 2025-01-01]` - which is wrong. Use an instant query with `--time` to avoid this.

3. **Separation of concerns**
   - Use `*_over_time()` for **temporal reductions** across a window (e.g., `max_over_time`, `avg_over_time`, `quantile_over_time`).
   - Use `sum/max/min/avg by (...)` for **label-set aggregation** across series at the eval point.
   - Chain them as needed (temporal reduction first, then label aggregation, or vice versa).

4. **Counters vs. gauges**
   - **Counters** (monotonic, suffixed `_total`) → use `rate()`/`irate()` or `increase()` over a window.
   - **Gauges** (current value) → use `avg_over_time`, `max_over_time`, etc., or plain `avg(...)` depending on intent.

5. **Suffix conventions**
   - Canonical: `_total` (counter), `_bucket/_sum/_count` (histogram), `_sum/_count` (summary), `_created`.
   - Non-standard: `_avg`, `_mean`, etc. Prefer computing averages via PromQL unless the exporter dictates otherwise.

---

## CLI Usage

### Instant Query

```bash
cx metrics query '<expr>'
cx metrics query '<expr>' --time 2024-01-01T12:00:00Z
cx metrics query '<expr>' --output json
```

**Example: absolute max over last 24h (single result)**

```bash
cx metrics query 'max by () (max_over_time(http_requests_in_flight[24h]))'
```

### Range Query

```bash
cx metrics query-range '<expr>' --start now-7d --end now --step 1d
```

**Example: absolute max per day over the last 7 days**

```bash
cx metrics query-range 'max by () (max_over_time(metric[1d]))' \
  --start now-7d --end now --step 1d
```

**IMPORTANT**: Align `--step` with any window used in temporal reduction functions. If using `max_over_time(metric[1d])`, set `--step 1d`.

---

## PromQL Fundamentals

### Label Matching & Aggregation

- Matchers: `{label="v"}`, `{label!="v"}`, `{label=~"re.*"}`, `{label!~"re"}`
- Aggregate **by** labels to keep them; use **without** to drop them.

```promql
sum by (job) (rate(http_requests_total[5m]))
sum without (instance) (up)
```

### Temporal Reductions (range → instant)

```promql
max_over_time(cpu_usage[1h])
avg_over_time(node_memory_Active_bytes[30m])
quantile_over_time(0.99, queue_length[1h])
```

### Counters: Rates, Increases, Windows

Per-instance RPS:
```promql
rate(http_requests_total[5m])
```

Total RPS across fleet:
```promql
sum by () (rate(http_requests_total[5m]))
```

Events in last day (per user, then count actives):
```promql
count( sum by (user_id) (increase(api_call_count[24h])) > 0 )
```

### Histograms & Summaries

p95 from a histogram:
```promql
histogram_quantile(
  0.95,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)
```

Average from summary parts:
```promql
sum(rate(req_duration_seconds_sum[5m]))
/
sum(rate(req_duration_seconds_count[5m]))
```

### Max over a Period

Correct - temporal reduction, then aggregation:
```promql
max by () (max_over_time(metric[4d]))
```

Per-label max:
```promql
max by (label) (max_over_time(metric[4d]))
```

Incorrect - `max()` cannot take a range vector:
```promql
max(metric[4d])   ← error
```

### Top-k / Ranking

```promql
topk(5, sum by (instance) (rate(http_requests_total[5m])))
```

---

## Common Tasks (ready to adapt)

1. **Absolute peak per instance over 7d, then pick the winner**

```promql
topk(1, max by (instance) (max_over_time(my_metric[7d])))
```

Run as instant query (no `--time` needed - defaults to now).

2. **Global CPU usage % (avg across cores & hosts)**

```promql
avg by () (
  rate(process_cpu_seconds_total[5m])
) * 100
```

3. **Error rate (%) per route**

```promql
100 * sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
    / sum by (route) (rate(http_requests_total[5m]))
```

4. **Daily active users over a week (time series)**

Expression:
```promql
count(count by (user_id) (increase(api_call_count[1d]) > 0))
```

Run as range query with `--step 1d --start now-6d --end now`. (Starting from 6 days ago because `increase` looks back one full day from each evaluation point.)

---

## Performance & Safety Guidelines

- Prefer **short windows** for `rate()` (e.g., 1–5m) unless data is bursty or sparse.
- Avoid unbounded fan-out (e.g., joining massive label sets).
- Keep **cardinality** under control; aggregate early (`sum by (...)`) when only totals are needed.
- Use `clamp_max`/`clamp_min` to tame outliers when needed.
- For histograms, **always** aggregate buckets (`sum by (le, ...)`) before `histogram_quantile`.
- Be mindful of counter resets; `rate()`/`increase()` handle resets automatically.

---

## Frequent Gotchas (and fixes)

- **"Why am I getting a time series when I only want one number?"**
  Use `cx metrics query` (instant) instead of `cx metrics query-range`.

- **"`max(metric[...])` errors."**
  `max()` can't take a range vector. Use `max_over_time(metric[...])`, then aggregate with `max by () (...)`.

- **"`_over_time(metric[...]) by (label)` errors."**
  `_over_time` aggregations cannot include a `by` clause. Use `max_over_time(metric[...])`, then `max by (label) (...)`.

- **"Avg looks wrong for counters."**
  Counters need `rate()`/`increase()`, not `avg_over_time`.

- **"p95 from a summary?"**
  Summaries expose quantiles directly via the `quantile` label. For histograms, use `histogram_quantile` on bucket rates.

- **"Results show empty label values."**
  Add `{label!=""}` to the selector to filter out empty label values. Example: `max by (deployment) (rate(cpu_usage{deployment!=""}[5m]))`.

---

## Mini Cheat-Sheet

| Goal | PromQL |
|---|---|
| Rate of a counter | `rate(x_total[5m])` |
| Increase last 24h | `increase(x_total[24h])` |
| Avg of a gauge over 1h | `avg_over_time(x[1h])` |
| Max over 4d (absolute) | `max by () (max_over_time(x[4d]))` |
| Top 5 by RPS | `topk(5, sum by (instance) (rate(x_total[5m])))` |
| p95 latency (histogram) | `histogram_quantile(0.95, sum by (le) (rate(x_bucket[5m])))` |
| Filter labels | `{env="prod", job=~"api\|web"}` |
| Drop a label in agg | `sum without (instance) (x)` |
| Time travel | `expr @ <unix_ts>` or `expr offset 1h` |

`references/spans-querying.md`

# Span Querying Reference

Query and analyze distributed tracing data using the `cx spans` command with DataPrime syntax.

> **DataPrime syntax:** See `dataprime-reference.md` for the full query language reference.

## Understanding Spans in Coralogix

Spans are the fundamental unit of tracing data. **Traces are not stored as single entities** - they are logical groupings of spans that share the same `traceID`. To analyze a trace, you query its constituent spans.

This means:
- **Metadata (`$m.*`)** and **labels (`$l.*`)** are predictable - you can always filter on timestamp, duration, service name, and operation name without discovery.
- **User data (`$d.*`)** contains trace identifiers (`traceID`, `spanID`, `parentId`) and application-specific tags/attributes that vary by service. Exact attribute names depend on the instrumentation and can differ across tenants, so verify `$d` fields with a sample query before assuming they exist.

---

## CLI Command

```bash
cx spans '<dataprime_query>'
```

The `source spans` is automatically injected - do not include it in the query.

### Options

| Flag | Default | Description |
|------|---------|-------------|
| `--start` | `now-1h` | Start time (ISO 8601 or relative, e.g. `now-6h`) |
| `--end` | `now` | End time |
| `--limit` | `200` | Maximum number of results |
| `--tier` | `frequent` | Storage tier: `frequent` (hot/recent) or `archive` (cold/historical) |
| `-o, --output` | `text` | Output format: `text`, `json`, or `toon` |

---

## Span Data Model

### Standard Fields (Always Available)

| Field | Description |
|-------|-------------|
| `$m.timestamp` | Span start timestamp |
| `$m.duration` | Span duration in **microseconds** (see [Duration Units](#duration-units)) |
| `$l.applicationName` | Application name - highest-level label. Meaning varies by customer (environment, team, region) but it always exists. |
| `$l.subsystemName` | Subsystem name - second-level label. Typically maps to a component. |
| `$l.serviceName` | Service name - the logical service unit emitting the span. |
| `$l.operationName` | Operation name - the span title (e.g. "POST /checkout", "db.query"). |
| `$d.traceID` | Trace ID - groups spans into a single trace. |
| `$d.spanID` | Unique span identifier. |
| `$d.parentId` | Parent span ID. Root spans have `parentId == null` (not an empty string). |
| `$d.*` | Application-specific tags and attributes (see [Field Discovery](#field-discovery)). |

> **Note on label fields:** The meaning of `$l.applicationName` and `$l.subsystemName` varies by customer - they may represent environments, teams, regions, or something else entirely. Don't assume what they map to. Use `cx search-fields` or sample queries to verify actual values.

### Duration Units

`$m.duration` is in **microseconds**:
- 500ms = `500000`
- 1s = `1000000`
- 1min = `60000000`

When presenting duration values, always convert to human-readable units (milliseconds, seconds, or minutes) and include the unit. Never display raw microsecond values or the "µs" symbol.

```dataprime
# Computed field for milliseconds
create latency_ms from $m.duration / 1000
```

### Error Detection

Spans do not have a `$m.severity` field like logs. Errors are typically indicated by:
- `$d.tags.error == true` - the most common convention (OpenTelemetry/Jaeger)
- Status codes in custom fields (e.g. `$d.http.status_code`, `$d.grpc.status_code`)
- Other application-specific error tags

The exact field depends on the instrumentation library used. If `$d.tags.error` returns no results, inspect sample spans with `-o json` to discover how errors are tagged:

```bash
cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json
```

---

## Essential Query Examples

```bash
# Get all spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"

# Find spans for a service
cx spans "filter \$l.serviceName == 'checkout-service'"

# Find slow spans (> 1 second)
cx spans "filter \$m.duration > 1000000"

# Find error spans
cx spans "filter \$d.tags.error == true"

# Aggregate latency by operation
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc"

# Wider time range
cx spans "filter \$l.serviceName == 'api'" --start now-6h
```

### Wildfind Policy

`wildfind` runs the same **case-insensitive, token-based** match as `~` (whole tokens, not arbitrary substrings), but over the whole record instead of a single field — it matches tokens anywhere, in any field.

**Prefer a field-targeted `~`/`filter` when you know the field** — for **precision**: because `wildfind` matches tokens anywhere in the record, it can't be narrowed to a specific field.

**Performance:** `wildfind` with very short search terms (only a few characters) is much slower — prefer longer, more specific terms.

The **one exception**: when the user provides a specific string and you don't know which field contains it:

```bash
cx spans "wildfind 'connection refused'"
```

> **Tip:** `wildfind` can also serve as a last-resort field discovery method - when `cx search-fields` doesn't find what you need, run `wildfind` with a known value, then inspect the matching spans to see which fields contain it.

---

## Field Discovery

**Skip discovery when:**
- The query only uses standard fields (`$m.duration`, `$l.serviceName`, `$l.operationName`, `$d.traceID`)
- The user explicitly names the fields they want
- The fields have already been discovered earlier in the conversation

### 1. Infer from Source Code (Preferred)

If you have access to the application's source code, examine OpenTelemetry instrumentation, span attribute definitions, and tracing middleware to identify field names directly.

### 2. Semantic Search

```bash
cx search-fields "customer identifier" --dataset spans
cx search-fields "order ID" --dataset spans
cx search-fields "http response code" --dataset spans
```

Note: `cx search-fields` only has access to the most common fields. If it doesn't find what you need, fall back to sample query inspection.

### 3. Sample Query Inspection

```bash
cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json
```

Inspect the JSON output to see all available fields. Especially useful for discovering fields in unstructured or deeply nested data.

---

## Investigation Workflow

### 1. Understand the Request

Identify:
- Whether you have a trace ID, service name, or error description
- Time frame of interest
- Whether the question is about latency, errors, or request flow

### 2. Start with Known Information

**If you have a trace ID** - go straight to it:
```bash
cx spans "filter \$d.traceID == '<trace_id>'"
```

**If you have a service name** - query its spans:
```bash
cx spans "filter \$l.serviceName == '<service>'" --limit 50
```

**If you have neither** - start broad to find entry points:
```bash
# Find recent error spans
cx spans "filter \$d.tags.error == true" --limit 20

# Find the slowest spans in the last hour
cx spans "groupby \$l.serviceName, \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

# Then extract trace IDs from interesting spans
cx spans "filter \$l.serviceName == '<service>' && \$m.duration > 1000000 | distinct \$d.traceID"
```

### 3. Troubleshooting

If a query returns no results, change **one thing at a time**. Keep the query window as **narrow** as possible and widen it deliberately — start from the window you already have rather than jumping to a huge range:

1. **Relax filters**: remove the most restrictive condition
2. **Check field availability**: the field you're filtering by may only exist in a subset of spans
3. **Verify field names**: run a sample query with `-o json` to inspect the actual schema
4. **Check service names**: service names are case-sensitive
5. **Extend the time range**: widen gradually from your current window (e.g. `now-1h` → `now-6h` → `now-24h`)
6. **Try archive tier**: for older data, add `--tier archive` and widen the window to cover the period you're after

---

## Common Query Patterns

### Trace Reconstruction

```bash
# All spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"

# Find root spans only (no parent)
cx spans "filter \$l.serviceName == 'api-gateway' | filter \$d.parentId == null"

# Find trace IDs for a service
cx spans "filter \$l.serviceName == 'payment-service' | distinct \$d.traceID"
```

### Latency Analysis

```bash
# Spans slower than 1 second
cx spans "filter \$m.duration > 1000000"

# Top 10 slowest operations by average duration
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

# Average latency by service
cx spans "groupby \$l.serviceName aggregate avg(\$m.duration) as avg_latency"

# P95 latency by operation
cx spans "groupby \$l.operationName aggregate percentile(0.95, \$m.duration) as p95_latency"
```

### Latency Spike Detection

```bash
# Average latency per 15-minute window
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 15m) as interval aggregate avg(\$m.duration) as avg_latency | orderby interval"

# Find the time windows with highest latency
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 5m) as interval aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"
```

### Error Investigation

```bash
# All error spans
cx spans "filter \$d.tags.error == true"

# Error spans for a specific service
cx spans "filter \$l.serviceName == 'checkout' | filter \$d.tags.error == true"

# Error rate by service
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName aggregate count() as errors | orderby errors desc"

# Error rate over time
cx spans "filter \$d.tags.error == true | groupby roundTime(\$m.timestamp, 15m) as interval aggregate count() as errors"
```

### Sampling Error Types

```bash
# Group errors by operation with a sample
cx spans "filter \$d.tags.error == true | groupby \$l.operationName aggregate any_value(\$d) as sample, count() as total | orderby total desc | limit 5"

# Group by service and operation to see where errors concentrate
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName, \$l.operationName aggregate count() as errors | orderby errors desc | limit 10"
```

### Finding Unique Values

```bash
# List all services with spans
cx spans "distinct \$l.serviceName"

# List all operations for a service
cx spans "filter \$l.serviceName == 'api' | distinct \$l.operationName"

# Find unique trace IDs for error spans
cx spans "filter \$d.tags.error == true | distinct \$d.traceID"
```

### Correlating by Trace ID

```bash
# Find spans across services for the same trace
cx spans "filter \$d.traceID == 'abc123' | groupby \$l.serviceName aggregate count() as span_count, avg(\$m.duration) as avg_latency"
```

---

## Performance Tips

- Use `--limit` for exploratory queries
- Use `groupby` with aggregations instead of fetching raw spans when possible
- Filter by time first when dealing with large datasets
- Use specific filters (service name, operation) to reduce scan scope
- Don't rely solely on aggregations - retrieve sample spans to find information you didn't anticipate
- For large result sets, use `--output toon` which spills automatically:

```bash
cx spans "filter \$l.serviceName == 'api'" --start now-24h --limit 1000 -o toon
```

`SKILL.md`

---
name: cx-alerts
description: This skill should be used when the user asks to "manage alerts", "create alert", "list alerts", "delete alert", "check alert status", "enable alert", "disable alert", "investigate firing alerts", "check which alerts are active", "find alerting rules", "set up an alert", "configure alerting", "mute an alert", "silence an alert", "see alert definitions", "check alert priority", or wants to manage Coralogix alert definitions using the cx CLI.
metadata:
  version: "0.1.0"
---

# Alert Management Skill

Use this skill to list, inspect, create, delete, enable, and disable Coralogix alert definitions using the `cx alerts` CLI commands.

## CLI Commands

| Command | Purpose | Key flags |
|---|---|---|
| `cx alerts list` | List all alert definitions | `--name <filter>` |
| `cx alerts get <id>` | Get a single alert definition by ID | - |
| `cx alerts create` | Create an alert from a JSON definition | `--from-file <path>` (default: stdin) |
| `cx alerts delete <id>` | Delete an alert | - |
| `cx alerts enable <id>` | Enable an alert | - |
| `cx alerts disable <id>` | Disable an alert | - |
| `cx alerts events` | List events; use alert-version scoped endpoint when filtering | `--alert-version-id`, `--start`, `--end` |
| `cx alerts event-stats` | Get alert event statistics | - |
| `cx alerts suppression-rules list` | List suppression rules | - |
| `cx alerts suppression-rules get <id>` | Get a suppression rule | - |
| `cx alerts suppression-rules create` | Create a suppression rule | `--from-file <path>` |
| `cx alerts suppression-rules update` | Update a suppression rule | `--from-file <path>` |
| `cx alerts suppression-rules delete <id>` | Delete a suppression rule | - |

**Output format:** append `-o json` or `-o toon` to `list`, `get`, and `create` commands for machine-readable output.

**Multi-profile:** use `-p <profile>` (repeatable) to target multiple profiles simultaneously.

## Alert Types Reference

Coralogix supports 12 alert types:

| Type enum | Human name | Description |
|---|---|---|
| `ALERT_DEF_TYPE_LOGS_IMMEDIATE` | Logs Immediate | Trigger on every matching log entry |
| `ALERT_DEF_TYPE_LOGS_THRESHOLD` | Logs Threshold | Trigger when log count exceeds a threshold in a time window |
| `ALERT_DEF_TYPE_LOGS_ANOMALY` | Logs Anomaly | ML-based anomaly detection on log volume |
| `ALERT_DEF_TYPE_LOGS_RATIO_THRESHOLD` | Logs Ratio Threshold | Trigger on ratio between two log queries |
| `ALERT_DEF_TYPE_LOGS_NEW_VALUE` | Logs New Value | Trigger when a new value appears in a field |
| `ALERT_DEF_TYPE_LOGS_UNIQUE_COUNT` | Logs Unique Count | Trigger on unique value count threshold |
| `ALERT_DEF_TYPE_LOGS_TIME_RELATIVE_THRESHOLD` | Logs Time Relative | Compare current vs past time window |
| `ALERT_DEF_TYPE_METRIC_THRESHOLD` | Metric Threshold | Trigger when a PromQL expression crosses a threshold |
| `ALERT_DEF_TYPE_METRIC_ANOMALY` | Metric Anomaly | ML-based anomaly detection on metrics |
| `ALERT_DEF_TYPE_TRACING_IMMEDIATE` | Tracing Immediate | Trigger on every matching span |
| `ALERT_DEF_TYPE_TRACING_THRESHOLD` | Tracing Threshold | Trigger when span count exceeds a threshold |
| `ALERT_DEF_TYPE_FLOW` | Flow | Sequence-based alert combining multiple conditions |

## Priority Levels

Always ask the user what priority to use when creating alerts:

| Priority | Use case |
|---|---|
| P1 | Critical - pages on-call immediately |
| P2 | High - needs attention within the hour |
| P3 | Medium - investigate during business hours |
| P4 | Low - informational, check when convenient |
| P5 | Info - logging/tracking only |

## Create Workflow

1. Ask the user what they want to alert on (logs, metrics, traces)
2. Ask for priority (P1–P5)
3. Build the JSON payload with `alertDefProperties` - use the **API wire format** (see `references/alert-schemas.md` for all enum values)
4. **Tip:** use `cx alerts get <existing-id> -o json` to get a working template, modify it, and pipe into create
5. Create using: `echo '<json>' | cx alerts create` or `cx alerts create --from-file alert.json`
6. Verify with `cx alerts list --name "<alert name>"`

**Important structural note:** The `type` field is a **string enum** (e.g. `"ALERT_DEF_TYPE_LOGS_THRESHOLD"`), and the alert type config (e.g. `"logsThreshold": {...}`) is a **sibling** field at the same level - NOT nested inside `type`.

### Example: Logs Threshold Alert

```json
{
  "alertDefProperties": {
    "name": "High Error Rate",
    "description": "Alert when error logs exceed threshold",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_LOGS_THRESHOLD",
    "enabled": true,
    "logsThreshold": {
      "logsFilter": {
        "simpleFilter": {
          "luceneQuery": "severity:ERROR",
          "labelFilters": {
            "applicationName": [
              { "operation": "LOG_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "value": "my-app" }
            ]
          }
        }
      },
      "rules": [{
        "condition": {
          "conditionType": "LOGS_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 100,
          "timeWindow": {
            "logsTimeWindowSpecificValue": "LOGS_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED"
          }
        }
      }]
    }
  }
}
```

### Example: Metric Threshold Alert

```json
{
  "alertDefProperties": {
    "name": "CPU Usage Critical",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_METRIC_THRESHOLD",
    "enabled": true,
    "metricThreshold": {
      "metricFilter": { "promql": "avg(cpu_usage_percent)" },
      "rules": [{
        "condition": {
          "conditionType": "METRIC_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 90,
          "ofTheLast": { "dynamicDuration": "5m" },
          "forOverPct": 100
        }
      }]
    }
  }
}
```

### Example: Logs Immediate Alert

```json
{
  "alertDefProperties": {
    "name": "OOM Killer Detected",
    "description": "Alert immediately when OOM killer runs",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_LOGS_IMMEDIATE_OR_UNSPECIFIED",
    "enabled": true,
    "logsImmediate": {
      "logsFilter": {
        "simpleFilter": {
          "luceneQuery": "\"Out of memory\" OR \"OOM\"",
          "labelFilters": {}
        }
      }
    }
  }
}
```

## Investigation Workflow

### Find firing alerts

```bash
# List all alerts and look for ALERTING status
cx alerts list -o json | jq '.[] | select(.status == "ALERTING")'

# Filter by name
cx alerts list --name "error"
```

### Inspect a specific alert

```bash
cx alerts get <alert-id>
cx alerts get <alert-id> -o json
```

### Disable a noisy alert (temporary mute)

```bash
cx alerts disable <alert-id>
# Later, re-enable:
cx alerts enable <alert-id>
```

## Suppression Rules

Manage alert suppression rules that mute alerts during maintenance windows or known noisy periods.

| Command | Purpose |
|---|---|
| `cx alerts suppression-rules list` | List all suppression rules |
| `cx alerts suppression-rules get <id>` | Get a suppression rule by ID |
| `cx alerts suppression-rules create --from-file` | Create a suppression rule |
| `cx alerts suppression-rules update --from-file` | Update a suppression rule |
| `cx alerts suppression-rules delete <id>` | Delete a suppression rule |

```bash
# List suppression rules
cx alerts suppression-rules list -o json

# Create from template
cx alerts suppression-rules get <existing-id> -o json > suppression-rule.json
# Edit suppression-rule.json
cx alerts suppression-rules create --from-file suppression-rule.json
```

## Key Principles

- **Always ask for priority** (P1–P5) when creating alerts - never assume
- **Use `--name` filter** for large accounts with many alerts
- **Use `-o json` with `jq`** for filtering and transformation
- **Use `--from-file -`** to pipe JSON from stdin when constructing alerts programmatically
- **Verify after create** - always list or get the alert after creation to confirm
- **Disable, don't delete** - prefer disabling alerts over deletion for auditability
- **Link to a specific alert** - `cx alerts list` prints only one "View in
  Coralogix" link, to the alerts overview page, not a per-alert link. To
  link a user to one specific alert, build `<base>/alerts/<alert_id>`,
  where `<base>` is the console URL already seen in a `View in Coralogix:
  <base>/...` line printed by any `cx alerts` command this session - never
  fabricate `<base>` yourself.

---

## Additional Resources

### Reference Files

- **[`references/alert-schemas.md`](references/alert-schemas.md)** - Complete JSON schema reference for all 12 alert types: field names, enum values (condition types, time windows, filter operations), common sub-objects (logs filter, tracing filter, notification groups, activity schedules), and important gotchas
- **[`references/dataprime-reference.md`](references/dataprime-reference.md)** - DataPrime query language reference for log-based and span-based alert conditions (filter syntax, operators, severity values)
- **[`references/logs-querying.md`](references/logs-querying.md)** - Log data model, field discovery, and query patterns for building log alert conditions
- **[`references/promql-guidelines.md`](references/promql-guidelines.md)** - PromQL reference for metric-based alert conditions (counters, gauges, histograms, threshold patterns)
- **[`references/spans-querying.md`](references/spans-querying.md)** - Span data model, duration units, and query patterns for building tracing alert conditions

### Related Skills

- **`cx-cases`** - triage the cases that group alert events into investigations
- **`cx-slos`** - the SLO definitions whose error-budget burn raises alerts
- **`cx-observability-setup`** - setting up notification routing and webhook integrations for alerts
- **`cx-telemetry-querying`** - investigate the telemetry behind a firing alert