references/changelog-patterns.md
# changelog-patterns
This reference captures practical `changelog.yml` patterns used in this repository and the package spec constraints enforced by `elastic-package lint`.
## Schema baseline
Source: `docs/extend/changelog-spec.md` (from the `elastic/integrations` upstream repo)
```yaml
spec:
type: array
items:
type: object
additionalProperties: false
properties:
version:
$ref: "./manifest.spec.yml#/definitions/version"
changes:
type: array
items:
type: object
additionalProperties: false
properties:
description:
type: string
type:
type: string
enum:
- "breaking-change"
- "bugfix"
- "enhancement"
link:
type: string
required:
- description
- type
- link
required:
- version
- changes
```
Operational rules:
- `changelog.yml` is required in each package
- top-level is a list of version sections
- each version section contains one or more `changes`
- only `enhancement`, `bugfix`, and `breaking-change` are valid `type` values
## Entry ordering and grouping
Repository convention:
```yaml
# newer versions go on top
```
Patterns:
- one section per package version
- all changes for the same package version are grouped under that section
- prepend new versions at the top to preserve descending order
## Representative entry patterns
### Enhancement
From `packages/nginx/changelog.yml` (elastic/integrations):
```yaml
- version: "2.3.0"
changes:
- description: Use links panel in Dashboards.
type: enhancement
link: https://github.com/elastic/integrations/pull/14380
```
### Bugfix
From `packages/nginx/changelog.yml` (elastic/integrations):
```yaml
- version: "2.3.2"
changes:
- description: Remove unused agent files.
type: bugfix
link: https://github.com/elastic/integrations/pull/14995
```
### Breaking change
From `packages/apache/changelog.yml` (elastic/integrations):
```yaml
- version: "3.0.0"
changes:
- description: Remove third-party pipeline for previously removed 'third-party REST API' input.
type: breaking-change
link: https://github.com/elastic/integrations/pull/16133
```
### Multi-change release section
Use this pattern when several changes ship in the same version:
```yaml
- version: "1.18.0"
changes:
- description: Prepare package for serverless.
type: enhancement
link: https://github.com/elastic/integrations/pull/9818
- description: Remove duplicated and ambiguous field definitions.
type: bugfix
link: https://github.com/elastic/integrations/pull/9818
```
### Multi-line description
From `packages/wiz/changelog.yml` (elastic/integrations):
```yaml
- version: "4.0.0"
changes:
- description: |
As `sourceRule` is deprecated by the Wiz Get Issue API, this version removes the deprecated `source_rule` field from the issue data stream.
Previous versions added the new `source_rules` field to the issue data stream.
Users should update their custom-user artifacts if they are using the deprecated `source_rule` field to use the new `source_rules` field.
type: breaking-change
link: https://github.com/elastic/integrations/pull/16892
```
## Link conventions
Preferred links:
- PR URL: `https://github.com/elastic/integrations/pull/<number>`
- issue URL: `https://github.com/elastic/integrations/issues/<number>`
Guidelines:
- always provide a resolvable URL
- prefer the PR that introduces the change when available
- use issue links only when a PR link is not the right source of detail
## Versioning guidance
Apply semver aligned with user impact:
- patch (`x.y.Z`): backward-compatible bug fixes
- minor (`x.Y.z`): backward-compatible enhancements
- major (`X.y.z`): breaking changes
When a version bump is driven by a `format_version` or ECS reference upgrade rather than by the feature itself (e.g., adding a data stream to an existing package required bumping the spec version), the changelog description must say so explicitly:
```yaml
- version: "2.4.0"
changes:
- description: Add audit data stream. Bumps format_version to 3.4.2 to support <reason>.
type: enhancement
link: https://github.com/elastic/integrations/pull/12345
```
Reviewers need to assess whether the bump is justified without reading the full diff — a generic description that buries the version change is insufficient.
When selecting `breaking-change`, check for:
- field type changes (mapping conflicts for existing data)
- field removals or renames
- ECS field mapping collisions or incompatible remapping
- required config/auth changes that break existing policies
- data stream split/merge/restructure changes
- default behavior changes that alter collected/normalized output
## CLI workflows
`elastic-package changelog add` can create entries in the expected format.
Add entry for next patch/minor/major:
```bash
# run inside packages/<package_name>/
elastic-package changelog add \
--type bugfix \
--description "Fix parser for empty status field" \
--link "https://github.com/elastic/integrations/pull/12345" \
--next patch
```
Add entry for explicit version:
```bash
elastic-package changelog add \
--type enhancement \
--description "Add support for additional API region" \
--link "https://github.com/elastic/integrations/pull/12345" \
--version 1.14.0
```
Validate after changelog updates:
```bash
elastic-package lint
# or full sequence:
elastic-package check
```
## CI and automation pattern
Automation in `.github/workflows/docs-edit-automation.yml` (elastic/integrations) uses:
```bash
elastic-package changelog add \
--type "$CHANGE_TYPE" \
--description "$CHANGELOG_DESC" \
--link "https://github.com/${GITHUB_REPOSITORY}/pull/$PR_NUMBER" \
--next "$VERSION_BUMP"
```
This pattern is useful for scripted, repeatable changelog updates where the change type and version bump can be inferred from workflow context.
references/format-version-features.md
# Package spec format_version features
Curated table of features by `format_version`, sourced from the [elastic/package-spec](https://github.com/elastic/package-spec) repository's `spec/changelog.yml`. Use this to determine the minimum `format_version` a package needs based on the features it uses.
## Feature-to-version table
### Core features (pre-3.4.2)
| Feature | Minimum format_version | Notes |
|---------|----------------------|-------|
| Basic package structure | 1.0.0 | Manifest, data streams, ingest pipelines, fields |
| Input-level variables | 2.0.0 | `policy_templates[].inputs[].vars` |
| `elasticsearch.privileges` | 2.3.0 | Required privileges for Fleet |
| `routing_rules.yml` support | 2.9.0 | Dynamic dataset/namespace routing |
| `lifecycle` field | 3.0.0 | Data stream lifecycle configuration |
| Secret variables (`secret: true`) | 3.0.0 | Variables masked in Fleet UI |
| `elasticsearch.source_mode` | 3.0.3 | Synthetic source, TSDB mode |
### Current standard for new packages: 3.6.5
This is the standard `format_version` for new integrations. All features up to 3.6.5 are available, and the 3.6.0 pipeline validators apply. It stays within the spec range supported by the new-package Kibana floor `^9.4.0` (see `../SKILL.md`).
### Recent additions (3.5.0+)
Existing packages below these versions must bump to use these features. Only bump an existing package if it specifically needs the feature.
| Feature | Minimum format_version | Package-spec reference |
|---------|----------------------|----------------------|
| Pipeline tag validations (enforced) | 3.6.0 | [#1010](https://github.com/elastic/package-spec/pull/1010) |
| Pipeline global on_failure validations (enforced) | 3.6.0 | [#1038](https://github.com/elastic/package-spec/pull/1038) |
| Deprecation support (packages, inputs, data streams, variables) | 3.6.0 | [#1053](https://github.com/elastic/package-spec/pull/1053) |
| ES\|QL query assets | 3.6.0 | [#1028](https://github.com/elastic/package-spec/pull/1028) |
| Time series index mode for input packages | 3.6.0 | [#1066](https://github.com/elastic/package-spec/pull/1066) |
| Multiple template paths | 3.6.0 | [#1089](https://github.com/elastic/package-spec/pull/1089) |
| Package dependencies (`requires` field) | 3.6.0 | [#1071](https://github.com/elastic/package-spec/pull/1071) |
| OTel input type | 3.6.0 | [#1091](https://github.com/elastic/package-spec/pull/1091) |
| `dynamic_signal_types` (otelcol only) | 3.6.0 | [#1067](https://github.com/elastic/package-spec/pull/1067), [#1091](https://github.com/elastic/package-spec/pull/1091) |
| Input type migration | 3.6.0 | [#1021](https://github.com/elastic/package-spec/pull/1021) |
| `var_groups` (policy template and input levels) | 3.6.1 | [#1120](https://github.com/elastic/package-spec/pull/1120) |
| Named inputs in policy templates | 3.6.1 | [#1135](https://github.com/elastic/package-spec/pull/1135) |
| Fleet-reserved variable validation | 3.6.1 | [#1134](https://github.com/elastic/package-spec/pull/1134) |
| `geo_shape` field type | 3.6.1 | [#1132](https://github.com/elastic/package-spec/pull/1132) |
| `sections` for Fleet UI layout | 3.6.1 | [#1133](https://github.com/elastic/package-spec/pull/1133) |
| `show_divider` on inputs | 3.6.1 | [#1133](https://github.com/elastic/package-spec/pull/1133) |
| Transform `num_failure_retries` | 3.6.1 | [#1124](https://github.com/elastic/package-spec/issues/1124) |
| ML modules in content packages | 3.6.2 | [#1149](https://github.com/elastic/package-spec/pull/1149) |
| `provider_permissions` declarations (IAM role generation for Federated Identity) | 3.6.4 | [#1180](https://github.com/elastic/package-spec/pull/1180) |
| `semantic_text` field type | 3.7.0 (unreleased) | [#807](https://github.com/elastic/package-spec/pull/807) |
## `dynamic_signal_types` is otelcol-only
`dynamic_signal_types: true` lets one policy template accept every signal type at
once. It is restricted to the `otelcol` input in **every** package type, and no
released spec version lifts that — the restriction is unchanged through
`3.7.0-next`. Below `format_version: 3.6.0` the field is not recognised at all
(`Additional property dynamic_signal_types is not allowed`).
Where it sits differs, and the sibling `type:` means different things:
| Package type | Field | Sibling `type:` |
|---|---|---|
| input | `policy_templates[].dynamic_signal_types` | the signal type — must be **absent** |
| integration | `policy_templates[].inputs[].dynamic_signal_types` | the input name — must be **`otelcol`** |
| integration | `streams[].dynamic_signal_types` (data stream manifest) | the input name, via `streams[].input` |
So the input-package rule "`type` must not be set" does not carry over: in an
integration package the neighbouring `type` is the input name and has to be
there. The validator words them differently too — `type field must not be set
when dynamic_signal_types is true` for input packages, `input type "<type>":
dynamic_signal_types is only allowed when input is 'otelcol'` for integration
packages.
All five packages using the field today (`otlp_input_otel`, `kafka_input_otel`,
`elasticapm_input_otel`, `mysql_input_otel`, `sql_server_input_otel`) are input
packages; no integration package uses it yet. For a non-otelcol input that needs
several signal types, see `create-integration/references/package-layout.md`.
## Breaking changes at 3.6.0
Spec version 3.6.0 introduced **stricter validations** that are breaking changes for packages that were previously valid:
1. **Pipeline tag validations**: every processor in an ingest pipeline must have a `tag` field. Packages at 3.6.0+ that lack tags will fail `elastic-package check`.
2. **Pipeline on_failure validations**: the pipeline-level `on_failure` block must follow the expected structure. Missing or malformed `on_failure` blocks fail validation.
New packages (default `3.6.5`) are subject to these validations from the start; pipelines built per the `ingest-pipelines` skill already comply. Existing packages below 3.6.0 are NOT subject to them, and bumping to 3.6.0+ adds these requirements. Only bump an existing package if it needs a 3.6.0+ feature AND the pipeline already complies with tag and on_failure rules.
**Federated Identity note:** enabling Cloud Connectors requires `provider_permissions` and therefore `format_version >= 3.6.4`, so the 3.6.0 validators apply. Expect a possible hygiene PR before the federation change — see **Floors and hygiene** in `input-configurations` -> `references/federated-identity-aws.md`.
## Upstream source
The authoritative source is `spec/changelog.yml` in the [elastic/package-spec](https://github.com/elastic/package-spec) repository. When new spec versions are released, update this table by reviewing that changelog for features relevant to integration developers.
The spec versioning model:
- A package at `format_version: x.y.z` must be valid against specs in `[x.y.z, X.0.0)` where `X = x + 1`
- Patch versions (x.y.Z) may add stricter validations
- Minor versions (x.Y.z) add new feature support
- Major versions (X.y.z) are reserved for significant format changes
references/manifest-rules.md
# Manifest Rules Reference
Comprehensive rules for `manifest.yml` validation in Elastic integration packages. Covers format_version selection, variable scoping, Handlebars template declarations, routing rules, and YAML structure.
---
## Variable shadowing
### The rule
Fleet variables can be declared at four levels:
1. **Package level** -- `manifest.yml` top-level `vars:`
2. **Policy template level** -- `manifest.yml` under `policy_templates[].vars:`
3. **Input level** -- `manifest.yml` under `policy_templates[].inputs[].vars:`
4. **Data stream level** -- `data_stream/*/manifest.yml` under `streams[].vars:`
A variable declared in an inner scope MUST NOT reuse the name of a variable in an outer scope. This is **variable shadowing** and is rejected by `elastic-package` validation.
### Why it is rejected
When a variable name appears at multiple levels, Fleet cannot determine which value the user intended. The inner declaration shadows the outer one, leading to:
- confusing UI where the same variable appears twice
- ambiguous Handlebars template resolution
- validation failures in `elastic-package check`
### Correct vs incorrect
```yaml
# WRONG -- "api_key" declared at package level AND input level
# manifest.yml
vars:
- name: api_key # Package-level
type: password
title: API Key
required: true
policy_templates:
- name: events
inputs:
- type: cel
vars:
- name: api_key # Shadows package-level "api_key"
type: password
title: API Key
```
```yaml
# CORRECT -- unique names at each scope
# manifest.yml
vars:
- name: api_key
type: password
title: API Key
required: true
policy_templates:
- name: events
inputs:
- type: cel
vars:
- name: initial_interval # Different name, no shadow
type: text
title: Initial Interval
```
### Common shadowing patterns
| Outer scope | Inner scope | Typical variable |
|-------------|------------|-----------------|
| Package | Input | `api_key`, `url` |
| Package | Data stream | `proxy_url`, `ssl` |
| Policy template | Input | `interval`, `tags` |
### How to detect and fix
1. List all variable names at each scope level across the root `manifest.yml` and all `data_stream/*/manifest.yml` files.
2. Check that no name appears in both an outer and inner scope.
3. If shadowing is found, rename the inner variable to be more specific (e.g., `events_api_key` instead of `api_key`).
---
## Handlebars variable declarations
### The rule
Every Handlebars variable referenced in agent input templates (`*.yml.hbs` files) MUST be declared as a variable in the corresponding manifest. Undeclared variables silently resolve to empty strings, causing broken configs at runtime.
Variables can be declared at any manifest scope:
- Package-level: `manifest.yml` -> `vars:`
- Policy template: `manifest.yml` -> `policy_templates[].vars:`
- Input-level: `manifest.yml` -> `policy_templates[].inputs[].vars:`
- Data stream: `data_stream/*/manifest.yml` -> `streams[].vars:`
### How to check
Work per stream, in both directions. A manual scan that only looks for `{{variable_name}}` misses the references that cause real bugs, so collect every form:
1. List every variable the `.yml.hbs` file references. Include all of these forms, not only the plain mustache:
- plain expressions: `{{x}}`, `{{ x }}` (spaces are common), `{{{x}}}`
- block helper arguments: `{{#if x}}`, `{{#unless x}}`, `{{else if x}}`, `{{#each x}}`, `{{#contains "value" x}}`
- helper arguments inside expressions: `{{escape_string x}}`, `{{url_encode x}}`
Exclude helper names, block parameters declared with `as |p|` and paths starting from them, string literals, comments, and the built-in variables listed below.
2. List every variable the stream's manifest declares, at all four scopes above. For scopes in the root manifest, use the `policy_templates[].inputs[].vars` block whose `type` matches the stream's `input`.
3. Compare both ways. A template variable missing from every scope is undeclared. A stream-level manifest variable that no template references is unused.
4. Pair the results. An unused manifest variable next to an undeclared template variable with a similar name in the same stream is almost always a rename or typo, and the user-facing setting does nothing. Report that pairing as HIGH. A single-direction result on its own is often harmless (a `{{#if x}}` guard in a template shared by streams that declare `x` elsewhere, or a leftover branch for a removed setting) and needs judgment.
### Correct vs incorrect
```yaml
# CORRECT -- all template variables are declared
# data_stream/events/agent/stream/cel.yml.hbs
interval: {{interval}}
resource.url: {{url}}/api/v1/events
{{#if proxy_url}}
resource.proxy_url: {{proxy_url}}
{{/if}}
{{#if ssl}}
resource.ssl: {{ssl}}
{{/if}}
# data_stream/events/manifest.yml -- variables declared
streams:
- input: cel
vars:
- name: interval
type: text
title: Polling Interval
default: 5m
- name: url
type: text
title: API URL
required: true
- name: proxy_url
type: text
title: Proxy URL
- name: ssl
type: yaml
title: SSL Configuration
```
```yaml
# WRONG -- "api_version" used in template but not declared
# data_stream/events/agent/stream/cel.yml.hbs
resource.url: {{url}}/api/{{api_version}}/events
# ^^^^^^^^^^^^^^^^ NOT in any manifest!
```
```yaml
# WRONG -- manifest declares "interval", template reads "bucket_list_interval"
# data_stream/audit/manifest.yml
streams:
- input: aws-s3
vars:
- name: interval
type: text
title: Bucket List Interval
default: 2m
# data_stream/audit/agent/stream/aws-s3.yml.hbs
{{#if bucket_list_interval }}
bucket_list_interval: {{ bucket_list_interval }}
{{/if}}
```
The template never reads `interval`, and `bucket_list_interval` is declared nowhere, so the `{{#if}}` is always false and the user's setting is ignored with no error. The reference sits inside a block helper argument with spaces around the name, which a scan for `{{variable_name}}` does not find. Policy tests do not catch it either: `elastic-package test policy --generate` records the broken rendering as the expected output. The fix is to make the names agree, usually by renaming the manifest variable to what the template reads.
### Built-in variables
Some variables are provided by Fleet automatically and do NOT need declaration in manifests:
- `{{data_stream.type}}` -- logs, metrics, etc.
- `{{data_stream.dataset}}` -- the data stream dataset name
- `{{data_stream.namespace}}` -- the namespace
- `{{output}}` -- output configuration
### Handlebars helpers (not variables)
These are control-flow helpers, not variable references. Do not flag the helper names themselves, but do check their arguments: `var`, `items`, and `array` below are variable references and must be declared.
- `{{#if var}}...{{/if}}`
- `{{#unless var}}...{{/unless}}`
- `{{#each items as |item|}}...{{/each}}` (`item` is a block parameter and needs no declaration)
- `{{#contains "value" array}}...{{/contains}}`
### Common mistakes
1. **Copy-paste from another integration** -- template references a variable from the source integration that was never added to this package's manifest.
2. **Renamed variable** -- manifest variable was renamed but the template still uses the old name. Shows up as an unused manifest variable paired with an undeclared template variable in the same stream.
3. **Template copied from a sibling stream** -- the new stream's manifest was written by hand while the template was copied, so the two disagree on a name. Diff the new template against the one it was copied from and check any variable name that differs.
4. **Wrong scope** -- variable declared at package level but template expects it at data stream level. This works at runtime, but may confuse users when reviewing the code.
---
## Routing rules configuration
### The rule
When a data stream uses `routing_rules.yml` to route documents to different backing indices based on field values, the data stream manifest **must** declare both:
```yaml
elasticsearch:
dynamic_dataset: true
dynamic_namespace: true
```
Without these flags, Elastic Agent does not have write permissions to the dynamically-named data streams that routing rules create. Documents are rejected with permission errors at index time.
### Why both flags are required
- `dynamic_dataset: true` -- allows the data stream name's dataset component to vary based on routing rules (e.g., `logs-mypackage.alerts` vs `logs-mypackage.events`).
- `dynamic_namespace: true` -- allows the namespace component to vary (e.g., routing to a different namespace per tenant).
Even if only one dimension varies, both flags are typically needed because Fleet's permission model grants write access based on the full `{type}-{dataset}-{namespace}` triple.
### Correct vs incorrect
```yaml
# WRONG -- routing_rules.yml exists but manifest lacks dynamic flags
# data_stream/events/manifest.yml
title: Events
type: logs
streams:
- input: cel
title: Events
# data_stream/events/routing_rules.yml exists
```
```yaml
# CORRECT -- dynamic flags declared alongside routing_rules
# data_stream/events/manifest.yml
title: Events
type: logs
elasticsearch:
dynamic_dataset: true
dynamic_namespace: true
streams:
- input: cel
title: Events
# data_stream/events/routing_rules.yml
- source_dataset: events
rules:
- target_dataset: alerts
if: ctx.severity == "critical"
default_dataset: events
```
### Routing rules file structure
```yaml
# data_stream/*/routing_rules.yml
- source_dataset: <original_dataset>
rules:
- target_dataset: <new_dataset>
if: <painless condition>
namespace: <optional namespace override>
default_dataset: <fallback_dataset>
```
### Checklist
1. Does the data stream have a `routing_rules.yml` file?
2. If yes, does the data stream manifest declare `dynamic_dataset: true`?
3. Does it also declare `dynamic_namespace: true`?
4. Are both flags under a properly nested `elasticsearch:` key (not dot-notation)?
5. Are the routing rule conditions valid Painless expressions?
6. Is there a `default_dataset` fallback for unmatched documents?
---
## YAML structure
### The rule
YAML keys must use proper dictionary nesting, not dot-separated flat keys. While some YAML parsers treat `elasticsearch.dynamic_dataset` as a literal key name, Fleet and `elastic-package` expect nested structure.
Dot-notation keys create a single key whose name contains a literal dot, rather than a nested object. This can cause validation failures or silent misinterpretation.
### Correct vs incorrect
```yaml
# WRONG -- dot in key name, creates a single key "elasticsearch.dynamic_dataset"
elasticsearch.dynamic_dataset: true
elasticsearch.dynamic_namespace: true
```
```yaml
# CORRECT -- proper YAML nesting
elasticsearch:
dynamic_dataset: true
dynamic_namespace: true
```
```yaml
# WRONG
elasticsearch.source_mode: synthetic
elasticsearch.index_mode: time_series
```
```yaml
# CORRECT
elasticsearch:
source_mode: synthetic
index_mode: time_series
```
### Where this applies
This rule applies to all manifest files in the package:
- `manifest.yml` (top-level package manifest)
- `data_stream/*/manifest.yml` (data stream manifests)
- any YAML file validated by `elastic-package`
The most common occurrence is the `elasticsearch` section in data stream manifests, where `dynamic_dataset`, `dynamic_namespace`, `source_mode`, and `index_mode` must be nested under an `elasticsearch:` parent key.
---
## Input names are not validated
### The rule
The *name* of an input is a free-form string in the spec. Nothing checks it
against what the agent can actually run:
| Package type | Field holding the input name |
|---|---|
| integration | `policy_templates[].inputs[].type`, and `streams[].input` in the data stream manifest |
| input | `policy_templates[].input` |
All three are declared `type: string` with no `enum` and no `pattern`
(`spec/integration/manifest.spec.yml`, `spec/integration/data_stream/manifest.spec.yml`,
`spec/input/manifest.spec.yml`). A nonexistent name passes `elastic-package
lint` **and** `elastic-package check` -- the latter still builds the zip.
### Do not confuse it with the enum-backed `type`
One nearby field *is* enum-backed: the data stream's own `type`, and its
`policy_templates[].type` equivalent in an input package. A bad value there is
rejected outright:
```
field type: type must be one of the following: "metrics", "logs", "synthetics", "traces", "profiles"
```
(`profiles` is marked technical preview in the spec.)
This matters most in an integration package, where the field holding the *input
name* is also called `type` -- just one level deeper, under `inputs`. It is not
the enum:
```yaml
# lints and checks green, builds a zip -- the input does not exist
policy_templates:
- name: nginx
inputs:
- type: bogus_input_xyz
```
### How to check
Confirm an input against the agent's own capabilities -- the "Detected available
inputs and outputs" line in the agent log -- not against a passing build. The
scaffolder's `--inputs` allowlist is no substitute either; it is narrower than
what the agent supports (see
`create-integration/references/scaffold-commands.md`).
*(Verified with elastic-package v0.125.1 on a `format_version: "3.0.2"`
integration package.)*
---
## format_version selection
### The rule
The `format_version` field in `manifest.yml` declares which package spec version the package conforms to. Always use the **minimum version that supports the features the package actually uses**, not the latest available spec version.
New packages use `"3.6.5"` (see `../SKILL.md`). For existing packages, higher versions are correct when the package uses features that require them.
### Why minimum matters
Using the latest spec version without needing its features:
- forces users to run a newer Kibana than necessary
- breaks backward compatibility for no reason
- makes it harder to determine which features the package actually depends on
### Feature-to-version mapping
Only bump when the package uses a feature introduced in a newer spec version:
| Feature | Minimum format_version |
|---------|----------------------|
| Basic package structure | 1.0.0 |
| Input-level variables | 2.0.0 |
| `elasticsearch.privileges` | 2.3.0 |
| `routing_rules.yml` support | 2.9.0 |
| `lifecycle` field | 3.0.0 |
| Secret variables (`secret: true`) | 3.0.0 |
| `elasticsearch.source_mode` | 3.0.3 |
| `var_groups` | 3.6.1 |
| `provider_permissions` | 3.6.4 |
See `format-version-features.md` for the full 3.5.0+ table, and `var-groups-and-provider-permissions.md` for schema details.
### When bumping is justified
A `format_version` bump is justified only when the PR also introduces a feature that requires the higher version. Review questions:
1. What `format_version` is declared?
2. Does the package use any feature that requires this version?
3. Could a lower version work?
4. If the PR bumps `format_version`, does it also introduce a feature that requires the bump?
**Worked example — Federated Identity:** introducing `provider_permissions` (and usually `var_groups`) requires `format_version: "3.6.4"`. That jump activates 3.6.0 pipeline tag / `on_failure` validators on files the federation change may not touch — apply only the bump first, run `elastic-package lint`, and land hygiene as a separate PR if needed before the federation change.
---
## Review checklist
- [ ] format_version matches package needs -- **HIGH** if wrong
- [ ] No variable shadowing across scopes -- **HIGH**
- [ ] Every `{{var}}` in templates declared in manifest -- **HIGH**
- [ ] Routing rules have dynamic_dataset + dynamic_namespace -- **HIGH** if routing_rules.yml present but flags missing
- [ ] YAML uses nested structure, not dotted keys -- **MEDIUM**
- [ ] Input names verified against agent capabilities, not a green lint -- **MEDIUM**
- [ ] All variables have title, description, type, required -- **MEDIUM**
- [ ] Defaults present for optional variables -- **LOW**
- [ ] Deprecated streams/inputs set `enabled: false` so the replacement is the default collection path (precedent: `o365` `audit`, `island_browser` `admin_actions`; formal `deprecated:` field available from format_version 3.6.0) -- **MEDIUM**
- [ ] If `var_groups` includes `identity_federation`: `vars` is `[role_arn, supports_identity_federation]` (no `external_id`) and `iac_template_url` points at `cloudformation-federated-identity-aws-<Kibana-floor>.yml` with no `param_ElasticResourceId` — not the GuardDuty-only CFT -- **HIGH** when in scope (see `input-configurations` -> `references/federated-identity-aws.md`)
- [ ] If `provider_permissions` is declared: `format_version >= 3.6.4` and IAM action names match the collector's real API calls -- **HIGH** when in scope
references/var-groups-and-provider-permissions.md
# `var_groups` and `provider_permissions`
Schema and floors for credential selectors and IAM declarations in `manifest.yml`. This file is the package-spec authority for these constructs.
For the Federated Identity (Cloud Connectors) procedure on an AWS integration, load `input-configurations` -> `references/federated-identity-aws.md`.
## Floors
| Construct | Minimum `format_version` | Notes |
|-----------|--------------------------|-------|
| `var_groups` | 3.6.1 | Credential / setup method selector in Fleet UI |
| `provider_permissions` | 3.6.4 | Declarations that feed IaC Provider IAM role rendering |
New packages default to `3.6.5`, which already supports these features. Bump an existing package only when introducing them.
## `var_groups`
Groups related vars into named options so Fleet shows a credential-method selector (e.g. Identity Federation vs access keys).
Typical shape (package level, before `policy_templates:`):
```yaml
var_groups:
- name: credential_type
required: true
title: Setup Access
selector_title: Preferred method
options:
- name: identity_federation
title: Identity Federation
vars: [role_arn, supports_identity_federation]
hide_in_deployment_modes: [default]
provider: aws
iac_template_url: https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://elastic-cspm-cft.s3.eu-central-1.amazonaws.com/cloudformation-federated-identity-aws-<KIBANA_FLOOR_MINOR>.yml
- name: direct_access_key
title: Direct Access Keys
vars: [access_key_id, secret_access_key]
```
On `identity_federation`, `iac_template_url` is required for the CloudFormation fallback. Replace `<KIBANA_FLOOR_MINOR>` with the package Kibana floor (currently `9.6.0`). There is no `param_ElasticResourceId=RESOURCE_ID` query parameter any more — the external ID was removed from the trust model. The GuardDuty-only `cloudformation-cloud-connectors-guardduty-*.yml` URL is legacy; every shipped package (`aws`, `aws_logs`, `aws_mq`, `aws_securityhub`, `aws_bedrock`) uses the federated-identity template.
Validator rules:
- Every name in `options[].vars` must exist as a var at some level (package, policy template, input, or data stream).
- Vars referenced by a var_group must not have `required: true`.
- Use `hide_in_deployment_modes` so agent-only methods stay hidden in agentless mode (and federation stays hidden in default mode when appropriate).
Related input gating (not part of `var_groups` itself):
- `hide_in_var_group_options` on an input hides that input when a given option is selected. Introduced in the first `aws` federation rollout and removed again once those inputs became federation-eligible; no shipped package uses it today.
- Input-level `deployment_modes: ["default"]` pins an input to agent-based mode only — this is how `aws-s3` is handled in `aws_logs` and `aws_bedrock_agentcore`.
## `provider_permissions`
Declares the IAM actions (and optional managed roles) the integration needs. The IaC Provider renders the onboard-time IAM role template from these entries. Declare at the narrowest level that needs them (package, policy_template, input, or data_stream); entries accumulate and deduplicate across levels.
```yaml
provider_permissions:
- provider: aws
description: Security Hub read access for findings collection.
permissions:
- name: securityhub:GetFindings
```
Optional `roles` attach AWS managed policies by ARN alongside inline `permissions`.
Do not guess IAM action names — derive them from the collector's real API calls and verify against the AWS API Reference (authorization names can differ from API operation names).
## Conditions with these features
When a package uses Federated Identity / `use_cloud_connectors`:
- `format_version` >= `"3.6.4"` (new packages: `"3.6.5"`)
- `conditions.kibana.version: "^9.6.0"` (or higher) — external-ID-free connector flow
- `conditions.agent.version: "^9.4.0"` (or higher) — `auth.aws` + libbeat cloud connectors shipped in agent 9.4.0, so the agent floor is deliberately lower than the Kibana floor ([elastic/integrations#21007](https://github.com/elastic/integrations/pull/21007))
If raising the Kibana floor would abandon a still-supported stack line (e.g. dropping 8.x), escalate — do not silent-bump. See **Floors and hygiene** in `input-configurations` -> `references/federated-identity-aws.md`.
## Handoff
| Need | Where |
|------|-------|
| Schema / floors / validators for these fields | This file + `format-version-features.md` |
| Enable Federated Identity on an AWS package | `input-configurations` -> `references/federated-identity-aws.md` |
| `auth.aws` / `use_cloud_connectors` in stream templates | Same federation reference (**Stream template**); CEL *program* logic stays in `cel-programs` |
SKILL.md
---
name: package-spec
description: >-
Package specification compliance for Elastic integration packages. Covers
manifest structure (format_version, conditions, variables, var_groups,
provider_permissions, routing rules), changelog schema and semantic version
bumps, and alignment with the upstream elastic/package-spec. Use when building
or reviewing manifest.yml, changelog.yml, or debugging elastic-package
lint/check errors on package metadata.
license: Apache-2.0
metadata:
author: elastic
version: "1.0"
synced_from: sit-llm/knowledge/skills/manifest/
---
# package-spec
## Skill authority
The rules and patterns defined in this skill and its reference files are the **authoritative source of truth**. When examining existing integrations in the `elastic/integrations` repository for reference, you may encounter patterns that conflict with what is specified here -- many integrations contain legacy patterns that predate current standards. **Always follow this skill over patterns observed in other integrations.** If a reference integration uses a deprecated or prohibited pattern, do not copy it.
## When to use
Use this skill when tasks include:
- building or reviewing `manifest.yml` at root or data stream level
- adding or validating `changelog.yml` entries
- selecting the correct change type and semantic version bump
- configuring policy templates, inputs, and variable declarations
- declaring `var_groups` or `provider_permissions` (schema and floors)
- debugging `elastic-package lint` or `elastic-package check` errors on manifests or changelogs
- reviewing variable scoping across package, policy template, input, and data stream levels
- validating Handlebars template variables against manifest declarations
- configuring routing rules and their required manifest flags
- determining which `format_version` is needed for a package's features
## When NOT to use
- Package scaffolding and directory layout (`create-integration`)
- End-to-end Federated Identity / Cloud Connectors enablement on an AWS package (`input-configurations` -> `references/federated-identity-aws.md`)
- Ingest pipeline design (`ingest-pipelines`)
- Field mapping and ECS compliance (`ecs-field-mappings`)
- CEL programs (`cel-programs`)
- Transform configuration (see `review-integration` skill's `references/transform-guide.md`)
## Handoff
For package directory layout and required files, see `create-integration` -> `references/package-layout.md`. For `elastic-package` CLI commands and troubleshooting, see `elastic-package-cli`. For enabling Federated Identity (Cloud Connectors) on an AWS integration, see `input-configurations` -> `references/federated-identity-aws.md`.
---
## format_version
The `format_version` field in `manifest.yml` declares which [elastic/package-spec](https://github.com/elastic/package-spec) version the package conforms to.
**New packages:** use `"3.6.5"`. It is supported by the new-package Kibana floor (`^9.4.0`, see below): Kibana 9.4 serverless Fleet accepts specs up to `3.6`. Do not go above the spec range the Kibana floor supports, and never use a `-next` (unreleased) version. 3.6.0+ enforces processor `tag` and global `on_failure` validation on ingest pipelines; pipelines built per the `ingest-pipelines` skill already comply.
**Existing packages:** use the minimum version that supports the features the package actually uses, not the latest available spec version. Bumping without needing new features:
- forces users to run a newer Kibana than necessary
- breaks backward compatibility for no reason
- makes it harder to determine which features the package depends on
Only bump an existing package when it uses a feature introduced in a newer spec version:
| Feature | Minimum format_version |
|---------|----------------------|
| Basic package structure | 1.0.0 |
| Input-level variables | 2.0.0 |
| `elasticsearch.privileges` | 2.3.0 |
| `routing_rules.yml` support | 2.9.0 |
| `lifecycle` field | 3.0.0 |
| Secret variables (`secret: true`) | 3.0.0 |
| `elasticsearch.source_mode` | 3.0.3 |
| `var_groups` | 3.6.1 |
| `provider_permissions` (Federated Identity IAM declarations) | 3.6.4 |
**Federated Identity:** packages that declare `provider_permissions` (and typically `var_groups`) need `format_version` >= `"3.6.4"`. New packages already meet this with `3.6.5`. For an existing package, the bump activates 3.6.0+ pipeline validators; land pipeline hygiene first if lint surfaces pre-existing violations. See `references/var-groups-and-provider-permissions.md` and `input-configurations` -> `references/federated-identity-aws.md`.
See `references/format-version-features.md` for the full feature-to-version table including recent spec additions (3.6.0+), and `references/manifest-rules.md` for the review procedure.
---
## conditions.kibana.version
**New packages:** use `"^9.4.0"`. Newly added integrations must require Kibana 9.4 or later. Reviewers flag a new package whose constraint admits anything below 9.4.
**Existing packages:** keep the current constraint. Only raise it when the package uses a feature that needs a newer stack. Do not bump an existing package to `^9.4.0` just to match the new-package floor.
This is set in the **root** `manifest.yml` only -- data stream manifests must NOT set their own `conditions`.
When an integration uses features that require a newer agent (e.g., CEL functions introduced in v9.3.0), the constraint must be adjusted accordingly. For systematic version verification of CEL features, see the `review-integration` skill's version check references.
**Justified exception — Federated Identity / `use_cloud_connectors`:** set `conditions.kibana.version` to `"^9.6.0"` and `conditions.agent.version` to `"^9.4.0"` (or higher). If the current Kibana constraint still covers a line that `^9.6.0` would drop, do **not** bump silently — escalate per **Floors and hygiene** in `input-configurations` -> `references/federated-identity-aws.md`.
---
## Variable scoping
Fleet variables exist at four levels:
1. **Package level** -- `manifest.yml` top-level `vars:`
2. **Policy template level** -- `manifest.yml` under `policy_templates[].vars:`
3. **Input level** -- `manifest.yml` under `policy_templates[].inputs[].vars:`
4. **Data stream level** -- `data_stream/*/manifest.yml` under `streams[].vars:`
A variable declared in an inner scope **must not** reuse the name of a variable in an outer scope. This is variable shadowing and is rejected by `elastic-package` validation.
See `references/manifest-rules.md` -> **Variable shadowing** for full rules, examples, and common patterns.
---
## Manifest rules (brief)
- **Every Handlebars `{{var}}` must be declared in a manifest** -- undeclared variables silently resolve to empty strings. Handlebars helpers (`{{#if}}`, `{{#each}}`, `{{#unless}}`, `{{#contains}}`) and built-in variables (`{{data_stream.type}}`, `{{data_stream.dataset}}`, `{{data_stream.namespace}}`, `{{output}}`) are exempt.
- **Routing rules require dynamic flags** -- when a data stream uses `routing_rules.yml`, the data stream manifest must declare `elasticsearch.dynamic_dataset: true` and `elasticsearch.dynamic_namespace: true`.
- **Use proper YAML nesting, not dotted keys** -- `elasticsearch.dynamic_dataset` as a literal key name creates a single flat key, not a nested object. Use nested `elasticsearch:` -> `dynamic_dataset:` structure.
- **A green lint does not prove an input exists** -- the input *name* is an
unvalidated string everywhere it appears: `policy_templates[].inputs[].type`
and the data stream's `streams[].input` in an integration package,
`policy_templates[].input` in an input package. Do not confuse it with the
enum-backed data stream `type`, which in an integration package is a
different field with the same name. See `references/manifest-rules.md` ->
**Input names are not validated**.
See `references/manifest-rules.md` for complete rules, correct/incorrect examples, and the review checklist.
---
## Changelog schema
`changelog.yml` is a version-grouped array; newer versions go on top:
```yaml
- version: "1.2.0"
changes:
- description: Added example parsing for edge-case payloads.
type: enhancement
link: https://github.com/elastic/integrations/pull/12345
```
Each entry requires `description`, `type`, and `link`. Valid types: `enhancement`, `bugfix`, `breaking-change`.
## Version bump rules
- **patch** (`x.y.Z`): bug fixes and low-risk fixes
- **minor** (`x.Y.z`): new content -- new data streams, new fields, new features
- **major** (`X.y.z`): breaking changes -- field type changes or removals on existing integrations, ECS mapping conflicts, required config/auth changes that break existing policies, data stream restructuring, default behavior changes that alter collected or normalized data
### When a bump is NOT required
Internal metadata-only changes that do not alter what ships to users need **no version bump and no changelog entry**: `owner.github` reassignment, CODEOWNERS updates, CI/dev-tooling files outside the built package. Reviewers must not flag the absence of a bump for such diffs.
Related bump conventions from elastic/integrations practice (see `review-integration/references/repo-conventions.md` for the dated details):
- `owner.type` changes ARE published package metadata -- **minor** bump.
- Adding the top-level `group:` manifest field -- **patch** bump + changelog entry.
- Bot-authored `requires:` dependency bumps arrive with their own patch bump and changelog entry; never ask a human to redo or justify them.
- `kibana.version` constraint updates are NOT breaking changes.
### Changelog `type` calibration (for reviewers)
Flag an entry's `type` only when the diff or the entry's own description shows an observable compatibility or behavior change: dropped stack support, dataset/field rename or removal, field type change, default-behavior change, policy-breaking config moves. Re-bucketing between `bugfix` and `enhancement`, or splitting/merging entries, is editorial judgment for the author -- not a review finding.
## Adding changelog entries
Edit `changelog.yml` directly, or use `elastic-package changelog add` (see `elastic-package-cli` skill for command flags and `--next patch|minor|major` usage).
## Common changelog pitfalls
- Adding the entry under the wrong version or not at the top
- Missing or wrong `link` field -- the link must be the pull request URL of the PR that introduces the change (`https://github.com/elastic/integrations/pull/<n>`), not an issue URL. `elastic-package lint` only validates that the number is a positive integer and **rejects** `pull/0`; use a real PR number or `pull/99999` as a development placeholder and replace before merge. Review tooling recognizes placeholder shapes (a repeated digit such as `1` or `9999`, zeros, `99999`, `12345`) and CI fails any number other than the PR's own, so both keep flagging a placeholder until it is replaced -- that is expected pre-merge behavior, not noise
- Bumping manifest/package version inconsistently with changelog intent
- **Forgetting to swap the placeholder link back in** -- see below
See `references/changelog-patterns.md` for detailed patterns, breaking-change checklist, and CI examples.
## Updating the changelog link after PR creation
The changelog `link` field must point to the PR that introduces the change, but the
real PR number is only known once a PR has actually been opened -- everything up to
that point necessarily uses the `pull/99999` placeholder from the pitfall above. This
step is easy to skip because it happens *after* the rest of the package work is done,
outside the usual build/test loop.
Once the PR exists on GitHub:
1. Get the real PR number (from `gh pr create` output or the PR URL).
2. In `changelog.yml`, replace every placeholder link
(`https://github.com/elastic/integrations/pull/99999`) with
`https://github.com/elastic/integrations/pull/<real-number>`.
3. Run `elastic-package lint` to confirm the link now validates.
4. Commit and push the update to the same PR branch -- the placeholder must not be
present in the version that gets merged.
---
## Upstream: elastic/package-spec
The [elastic/package-spec](https://github.com/elastic/package-spec) repository is the upstream authority for package structure, manifest schema, and validation rules. The `spec/changelog.yml` in that repo documents which features were added in each spec version.
Key points from the package-spec versioning model:
- Packages must specify `format_version` in root `manifest.yml`
- A package at `format_version: x.y.z` must be valid against specs in the range `[x.y.z, X.0.0)` where `X = x + 1`
- Patch versions may add stricter validations (e.g., 3.6.0 added pipeline tag and on_failure validation)
- Minor versions add new feature support
- Major versions are reserved for significant format changes
See `references/format-version-features.md` for the curated feature-to-version table.
## Reference files
| File | Contains |
|------|----------|
| `references/manifest-rules.md` | Full rules for format_version selection, variable shadowing, Handlebars variable declarations, routing rules, YAML structure, and severity-tagged review checklist |
| `references/changelog-patterns.md` | Changelog entry patterns, semver rules, breaking-change checklist, CI examples |
| `references/format-version-features.md` | Feature-to-version table sourced from elastic/package-spec, including recent spec additions |
| `references/var-groups-and-provider-permissions.md` | Schema for `var_groups` and `provider_permissions`, floors, validators; handoff to Federated Identity procedure |