references/data-model-notes.md
# Log Data Model Notes
Source: [Dynatrace Log Semantic Dictionary](https://docs.dynatrace.com/docs/semantic-dictionary/model/log)
Extensions: [Dynatrace SD Fields](https://docs.dynatrace.com/docs/semantic-dictionary/fields)
Baseline: `samples/audit-logs.json`, `samples/http-logs.json`
## TOC
- [Core Fields](#core-fields)
- [Audit Namespace](#audit-namespace)
- [Actor and Client Fields](#actor-and-client-fields)
- [Device and Browser Fields](#device-and-browser-fields)
- [Result Fields](#result-fields)
- [HTTP Namespace](#http-namespace)
- [URL Namespace](#url-namespace)
- [Server Namespace](#server-namespace)
- [Geo Namespace](#geo-namespace)
- [Cloud Context Fields](#cloud-context-fields)
- [loglevel Enum](#loglevel-enum)
- [status Enum](#status-enum)
- [SD Extension Namespaces](#sd-extension-namespaces)
- [Log Class Field Requirements](#log-class-field-requirements)
- [Known Discrepancies and Bugs](#known-discrepancies-and-bugs)
- [Sparse Mapping Policy](#sparse-mapping-policy)
---
## Core Fields
| Field | Type | SD Status | Required | Notes |
|---|---|---|---|---|
| `timestamp` | timestamp | stable | yes | Auto-generated by Dynatrace on ingest — represents ingest time, NOT the vendor event time. Do not map from vendor payload. Vendor event time → `audit.time`. |
| `content` | string | stable | yes | Raw vendor log as a JSON string. Auto-enriched by Dynatrace on ingest — do not map manually. MUST be string on final ingested events. |
| `log.source` | string | stable | yes | Human-readable integration name as a constant (e.g. `"JFrog"`, `"Okta"`, `"CyberArk"`). NOT the raw vendor identifier from the payload. |
| `loglevel` | string | stable | yes | See enum — MUST be normalized |
| `status` | string | experimental | recommended | See enum — mirrors `loglevel` in practice |
| `event.type` | string | stable | recommended | Always `"LOG"` for audit logs; often absent for HTTP logs |
---
## Audit Namespace
Required for authentication, authorization, and user_action log classes. Not used for HTTP logs.
| Field | Type | SD Status | Notes |
|---|---|---|---|
| `audit.action` | string | stable | What happened. Use vendor-native value. Examples: `"Multifactor challenge"`, `"Create okta group"`, `"export"` |
| `audit.identity` | string | stable | Who did it — email or username. Required when identifiable. |
| `audit.result` | string | stable | Outcome. Convention from samples: **`Succeeded`** / **`Failed`** (title case). SD does not define an enum. |
| `audit.status` | string | stable | Lifecycle state of the audited action. SD enum: `Started`, `In Progress`, `Succeeded`, `Failed`, `Active`, `Resolved`. MUST NOT be set to a loglevel value (e.g. `"INFO"`) — that is a mapping error. |
| `audit.time` | timestamp | experimental | When the action occurred if different from `timestamp`. MUST be ISO8601 with timezone. |
---
## Actor and Client Fields
| Field | Type | SD Status | Notes |
|---|---|---|---|
| `actor.ips` | ipAddress[] | stable | Client IP(s) as array. Primary IP field for audit logs — populate when vendor provides any IP. |
| `actor.geo.city.name` | string | experimental | Actor city — for audit logs where the actor is the subject |
| `actor.geo.country.name` | string | experimental | Actor country name or 2-letter code |
| `actor.geo.location.lat` | float | experimental | WGS84 latitude. Prefer float/number — string is accepted but degrades geo queries |
| `actor.geo.location.lon` | float | experimental | WGS84 longitude |
| `client.ip` | ipAddress | experimental | Single client IP. Consistently present alongside `actor.ips` in audit logs — acceptable duplication. |
| `client.app.name` | string | experimental | Client application name (e.g. `"Confluence"`, `"Identity"`) |
---
## Device and Browser Fields
| Field | Type | SD Status | Notes |
|---|---|---|---|
| `browser.name` | string | resource stable | Browser type (e.g. `"Chrome"`) |
| `browser.version` | string | resource stable | Browser version |
| `browser.user_agent` | string | — | Full raw user-agent string. Not in log SD but consistently observed in samples — acceptable extension. |
| `device.id` | string | experimental | Device GUID |
| `device.name` | string | experimental | Device display name |
| `device.os.name` | string | experimental | OS name (e.g. `"Mac"`, `"Windows 10"`) |
---
## Result Fields
| Field | Type | SD Status | Notes |
|---|---|---|---|
| `result.code` | **long** | experimental | SD type is `long`. String values observed in samples are **bugs** — coerce to numeric on ingest (e.g. `"50076"` → `50076`). Non-numeric vendor codes (e.g. `"IDP2013"`, `"SUCCESS"`) should use `result.message` instead. |
| `result.message` | string | experimental | Brief outcome message |
| `result.detail` | string | experimental | Detailed result information. SD field name is `result.detail` (singular). `result.details` (plural) observed in samples is a **bug** — use `result.detail`. |
---
## HTTP Namespace
Used for HTTP/WAF log classes. Not used for audit logs.
| Field | Type | Notes |
|---|---|---|
| `http.request.method` | string | HTTP verb (`"GET"`, `"POST"`, `"PUT"`, `"DELETE"`, etc.) |
| `http.response.status_code` | integer | HTTP status code — MUST be integer. String values (e.g. `"204"`) are bugs; coerce to int. |
| `http.response.body.size` | long | Response body size in bytes |
| `http.request.header.<name>` | string | Individual request headers — header name MUST be lowercase (e.g. `http.request.header.user-agent`, `http.request.header.content-type`) |
| `http.response.header.<name>` | string | Individual response headers — header name MUST be lowercase (e.g. `http.response.header.content-type`) |
---
## URL Namespace
Used for HTTP log classes.
| Field | Type | Notes |
|---|---|---|
| `url.domain` | string | Hostname / domain (e.g. `"api.sampledomain.com"`) |
| `url.path` | string | Request path (e.g. `"/api/v1/groups"`) |
| `url.port` | string / integer | Port number. Prefer integer. |
| `url.scheme` | string | Protocol (e.g. `"https"`, `"HTTP/1.1"`) |
---
## Server Namespace
Used for HTTP log classes where the request target is a named server.
| Field | Type | Notes |
|---|---|---|
| `server.address` | string | Target server hostname or FQDN |
---
## Geo Namespace
`geo.*` fields can appear at root level or nested under another SD namespace to clarify whose location is being described. Choose the placement based on the subject:
| Namespace | Use for | Fields |
|---|---|---|
| `geo.*` | Geo-location of the network request or event origin (IP-derived). Root-level. Use for HTTP/WAF logs and any log where the IP source location is the primary geo signal. | `geo.city.name`, `geo.continent.name`, `geo.country.name`, `geo.region.code` |
| `actor.geo.*` | Geo-location of the actor (person/service performing the action). Nested under `actor.*`. Use in audit logs when the vendor separately reports actor location. | `actor.geo.city.name`, `actor.geo.country.name`, `actor.geo.location.lat`, `actor.geo.location.lon` |
Do not mix both namespaces in the same event unless the vendor separately reports network origin geo and actor geo.
---
## Cloud Context Fields
| Field | Type | Notes |
|---|---|---|
| `cloud.provider` | string | `"aws"`, `"azure"`, `"gcp"` |
| `azure.tenant.id` | string | Azure tenant GUID |
| `azure.tenant.name` | string | Azure tenant display name |
| `aws.account.id` | string | AWS account number |
---
## loglevel Enum
MUST be one of: `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`, `NONE`
Derivation guidance:
| Condition | loglevel |
|---|---|
| HTTP 5xx | `ERROR` |
| HTTP 4xx | `WARN` |
| HTTP 2xx/3xx | `INFO` |
| `audit.result = "Failed"` (auth/access failure) | `ERROR` |
| `audit.result = "Failed"` (non-critical action) | `WARN` |
| `audit.result = "Succeeded"` | `INFO` |
| No severity signal available | `NONE` |
---
## status Enum
MUST be one of: `INFO`, `WARN`, `ERROR`, `NONE`
In all samples, `status` mirrors `loglevel`. Use the same derivation logic.
---
## SD Extension Namespaces
Log records can carry fields from any SD-documented namespace as extensions. The most common for audit and HTTP logs:
| Namespace | Use case | Representative fields |
|---|---|---|
| `product.*` | Product/vendor identity | `product.name`, `product.vendor` |
| `object.*` | Target resource in an audit action | `object.id`, `object.name`, `object.type` |
| `browser.*` | Browser context | `browser.name`, `browser.version`, `browser.user_agent` |
| `http.*` | HTTP request/response | `http.request.method`, `http.response.status_code`, `http.request.header.*` |
| `url.*` | URL decomposition | `url.domain`, `url.path`, `url.port`, `url.scheme` |
| `server.*` | Request target server | `server.address` |
| `geo.*` | Network geo-location (HTTP logs) | `geo.city.name`, `geo.country.name`, `geo.continent.name` |
| `host.*` | Log-shipping host | `host.name`, `host.hostname`, `host.fqdn`, `host.id` |
| `db.*` | Database operations | `db.system`, `db.query.text` |
| `code.*` | Source code context | `code.function`, `code.filepath` |
| `network.*` | Network layer | `network.protocol`, `network.transport` |
| `device.*` | Client device | `device.id`, `device.name`, `device.os.name` |
| `user.*` | User identity (where `audit.identity` is insufficient) | `user.id`, `user.name`, `user.email` |
Less commonly used: `app.*`, `aws.*`, `azure.*`, `gcp.*`, `cicd.*`, `container.*`, `k8s.*`
All SD-documented namespace fields are acceptable — they are not vendor extensions and do not need to pass the duplication check. Vendor-specific namespaces (`akamai.*`, `cyberark.*`, `okta.*`) follow the separate vendor namespace duplication rule.
Full field list: https://docs.dynatrace.com/docs/semantic-dictionary/fields
---
## Log Class Field Requirements
| Class | Required semantic fields | Typical namespaces |
|---|---|---|
| `authentication` | `audit.action`, `audit.identity`, `audit.result`, `actor.ips` | `audit.*`, `actor.*`, `browser.*`, `device.*`, `cloud.*` |
| `authorization` | `audit.action`, `audit.identity`, `audit.result` | `audit.*`, `actor.*`, `object.*` |
| `user_action` | `audit.action`, `audit.identity`, `audit.result` | `audit.*`, `actor.*`, `object.*`, `product.*` |
| `http` | `http.request.method`, `http.response.status_code`, `url.path` | `http.*`, `url.*`, `server.*`, `geo.*`, `client.*` |
Minimum always-required (all classes): `timestamp`, `log.source`, `content`, `loglevel`
---
## Known Discrepancies and Bugs
Fields that diverge from the SD. Bugs are mapping errors to fix; accepted deviations are expected patterns.
| Field | SD Definition | Observed | Verdict |
|---|---|---|---|
| `result.code` string values | `long` | `"IDP2013"`, `"SUCCESS"`, `"50076"` in audit-logs.json | **Bug** — non-numeric codes belong in `result.message`; numeric codes must be coerced to `long` |
| `result.details` (plural) | `result.detail` (singular) | `result.details` in audit-logs.json | **Bug** — use `result.detail` |
| `audit.status = "INFO"` | `Started/In Progress/Succeeded/Failed/Active/Resolved` | CyberArk sample | **Bug** — `"INFO"` is a loglevel value, not an audit status |
| `http.response.status_code` as string | integer | `"204"` in http-logs.json | **Bug** — coerce to integer |
| `http.response.body.size` as string | long | `"0"` in http-logs.json | **Bug** — coerce to long |
| `client.ip` alongside `actor.ips` | — | All audit-log samples | Accepted — minor redundancy, do not flag as error |
| `actor.geo.location.lat/lon` as string | float | audit-logs.json mixed types | Minor — prefer float; accept string |
| `audit.time` missing timezone | ISO8601 with timezone | GitHub sample | Minor — flag, append `Z` |
| `event.type = "LOG"` | SD optional | audit-log samples | Accepted — consistent convention |
| `product.name`, `product.vendor` | Absent from log SD | CyberArk, Okta | Accepted SD extension |
| `object.id/name/type` | Absent from log SD | Okta, Azure, CyberArk | Accepted SD extension |
| `browser.user_agent` | Absent from log SD | CyberArk, Okta | Accepted SD extension |
| `host.name/id/hostname` | Absent from log SD | log-shipping metadata | Accepted — standard ECS/OTel host fields |
| `authentication.*` | Absent from log SD | CyberArk, Azure | Accepted vendor extension |
| `loglevel: "INFO"` / `status: "INFO"` for HTTP 204 | HTTP 2xx → `INFO` | Akamai SIEM sample | Aligned — derived from HTTP status code per enum table |
| `url.scheme: "https"` | Protocol scheme (e.g. `"https"`) | Akamai SIEM sample | Aligned — normalized to scheme component only |
| `url.port` as integer | integer preferred | Akamai SIEM sample | Aligned — normalized numeric port |
---
## Sparse Mapping Policy
Not every field is required for every integration.
**Minimum required (all classes):** `timestamp`, `log.source`, `content`, `loglevel`, `audit.action`, `audit.identity`
*(HTTP class: replace `audit.action`/`audit.identity` with `http.request.method`, `http.response.status_code`, `url.path`)*
A sparse mapping covering the minimum is `medium` confidence. Add recommended fields to reach `high`.
references/mapping-workflow.md
# Mapping Workflow
## TOC
- [Before You Start — Intake](#before-you-start--intake)
- [Workflow A — Suggest a New Mapping](#workflow-a--suggest-a-new-mapping)
- [Workflow B1 — Static Validation](#workflow-b1--static-validation)
- [Shared: Content Field Analysis](#shared-content-field-analysis)
- [Shared: Field Priority Order](#shared-field-priority-order)
- [Mapping Table Template](#mapping-table-template)
---
## Before You Start — Intake
Confirm before any mapping or validation:
1. Log target: `fetch logs` (not `security.events`).
2. `log.source` value(s) the vendor uses.
3. At least 1 raw vendor log sample; 3+ recommended per log class.
4. `content` field format: flat string or JSON object?
5. Audit log class: authentication, authorization, user_action, or access/request.
6. For B1: is the pasted event a final ingested Dynatrace record or a pre-ingest draft?
7. For B2: confirm `log.source` to query and that live tenant access is available.
If inputs are incomplete, continue with a partial draft but list missing evidence explicitly.
---
## Workflow A — Suggest a New Mapping
Triggered when the user provides a raw vendor log payload and no existing mapping.
### A.1 Parse the Input
1. If the input is a raw vendor log (entire JSON becomes the Dynatrace `content` string on ingest): reference all fields by their original vendor path — top-level fields by name (e.g. `host.name`, `service.name`), nested fields in dot notation (e.g. `content.user_name`, `content.ip_address`). Do NOT use any pipeline variable prefix (e.g. `c.`) in mapping tables — that is an OpenPipeline implementation detail for Phase 2 only.
2. If `content` is already a JSON string on a pre-ingest draft: deserialize and reference sub-fields as `content.<field>`.
3. If `content` is a flat string: note the format; extraction requires a log parsing rule.
4. Identify audit log class from field names and values.
### A.2 Run Content Field Analysis
See [Shared: Content Field Analysis](#shared-content-field-analysis).
### A.3 Build Mapping Candidate
Map fields to semantic targets using [Shared: Field Priority Order](#shared-field-priority-order).
Rules:
- `log.source` MUST be the human-readable integration name as a constant (e.g. `"JFrog"`, `"Okta"`) — not the raw vendor identifier from the payload.
- `timestamp` is auto-generated by Dynatrace on ingest — do NOT map from the vendor payload. Map the vendor's event time to `audit.time` instead (audit log classes only).
- `content` is auto-enriched by Dynatrace on ingest — do NOT include it in the mapping table.
- Map `content` sub-fields to top-level semantic attributes where a match exists.
- Extract a vendor field to the top level only if it meets at least one of:
1. Maps to an SD-canonical field (always extract).
2. Provides valuable additional context not captured by SD fields (extract to `<vendor>.<field>`).
- Limit vendor namespace (`<vendor>.*`) extractions to ~5 fields per integration — only the highest-value data points not covered by SD fields.
- Do NOT extract fields that are redundant, null, or carry no operational value (e.g. `-1` sentinel values, duplicates already covered by SD fields).
- Discard string sentinel values — fields whose value is a meaningless placeholder (e.g. `"UNKNOWN"`, `"N/A"`, `"-"`, `"null"`) should be omitted from the mapping, not promoted.
- Verify truly unknown fields against the live SD before classifying as vendor-extension:
```dql-snippet
fetch dt.semantic_dictionary.fields
| filter name == "<field_name>"
| fields name, type, description, stability
```
### A.4 Phase 1 Output — Present and Wait for Approval
1. Mapping table (see [Mapping Table Template](#mapping-table-template)).
2. Content field promotion plan: source → target → transform → OpenPipeline rule needed?
3. Gap summary: required fields that cannot be mapped.
4. Discrepancies list: critical → major → minor.
5. Confidence tier.
**Stop here. Do not produce Phase 2 until the user approves.**
### A.5 Phase 2 Output (after approval)
1. One sample mapped JSON log record.
2. Inline annotations for every transform applied.
3. OpenPipeline processor sketch for promoted content fields.
OpenPipeline extraction pattern for a serialized-`content` field (JSON string). Do **not** use `parseJson` — it is not an OpenPipeline function. Use `parse content, "json:content"` (the `json` DPL token is case-insensitive), then access fields via subscript notation `content[key]` in `fieldsAdd`. `content` always retains the raw payload, so re-parse it in each processor that needs buried values — do not assume an earlier processor's parse persists. Also check for **root-level fields** already parsed at ingest (e.g. `` `service.name` ``) and reference those directly. See `openpipeline-constraints.md`.
```dql-snippet
| parse content, "json:content" // turn JSON string into a structured record
| fieldsAdd audit.identity = content[user_name] // subscript access: content[key] for JSON keys
| fieldsAdd actor.ips = array(toIp(content[ip_address])) // → ipAddress[]
| fieldsAdd audit.action = content[action]
| fieldsAdd audit.result = if(content[success] == "True", "Succeeded", else: "Failed")
| fieldsAdd loglevel = if(content[success] == "True", "INFO", else: "ERROR")
| fieldsAdd status = loglevel
```
---
## Workflow B1 — Static Validation
Triggered when the user pastes an ingested log event (or a pre-ingest mapping draft).
### B1.0 Determine Input Mode
- **Final ingested event**: top-level semantic fields + `content` as JSON string are both present. Use `content` (deserialized) as the source payload for comparison.
- **Pre-ingest draft**: `content` may still be a JSON object. Do not fail for this; flag as a reminder to serialize.
### B1.1 Parse Content
Deserialize `content` (if it is a JSON string). Build a side-by-side view: `content.*` sub-fields vs top-level semantic fields on the log record.
### B1.2 Run Content Field Analysis
See [Shared: Content Field Analysis](#shared-content-field-analysis).
### B1.3 Apply Validation Rules
Run all rules from `validation-rules.md`. For each missing required semantic field, check `content` for a candidate source path and add a `➕ add` entry with the proposed OpenPipeline extraction.
### B1.4 Produce Diff-Highlighted Mapping Table
| Source Field | Current Target | Suggested Target | Transform | Status | Reason |
|---|---|---|---|---|---|
| `content.user_name` | — | `audit.identity` | direct | ➕ add | required audit field not promoted |
| `content.ip_address` | `client.ip` | `actor.ips` | string→array | ⚠ change | use array field; keep `client.ip` as acceptable duplicate |
| `loglevel` | `loglevel` | `loglevel` | normalize enum | ✅ ok | |
| `content.internalRef` | `internal.ref` | — | — | ❌ remove | not in SD or samples |
Marker legend: ✅ ok · ⚠ change · ➕ add · ❌ remove
---
## Shared: Content Field Analysis
Apply in both A and B1.
**Inventory every `content` sub-field and classify:**
| Classification | Criteria | Action |
|---|---|---|
| `✅ promoted` | Top-level semantic field has same/equivalent value | Note — acceptable |
| `➕ promotable` | Maps to a known SD field; not yet at top level | Add to mapping + OpenPipeline plan |
| `🏷 vendor-specific` | No SD equivalent; unique vendor context | Keep in content; optionally add as `<vendor>.<field>` |
| `ℹ️ duplicate` | Identical value already at top level | Leave in content (raw payload); do NOT add a second top-level copy |
| `❓ unknown` | Not in SD, not in samples, purpose unclear | Flag for clarification |
**Report:** `X of Y content fields promoted | Z promotable gaps remaining`
---
## Shared: Field Priority Order
Map in this order:
1. `log.source` (constant — integration name), `loglevel`, `status`, `event.type`
2. `audit.action`, `audit.identity`, `audit.result`, `audit.status`, `audit.time`
3. `actor.ips`, `client.ip`, `client.app.name`
4. `actor.geo.*`
5. `browser.name`, `browser.version`, `device.*`
6. `object.id`, `object.name`, `object.type`
7. `result.code`, `result.message`, `result.detail`
8. `cloud.provider`, `azure.tenant.*`, `aws.account.*`, `gcp.organization.*`
9. Vendor namespace (`<vendor>.*`) — must add information not in SD-canonical fields above
---
## Common Mapping Patterns
Reusable patterns observed across vendor integrations. Apply where applicable.
### host.* as connector metadata
When `host.*` fields originate from the log shipper or SaaS connector (not the target host being audited), do NOT map to `host.*` SD fields. Move to vendor namespace (e.g. `<vendor>.instance.name`, `<vendor>.instance.id`). Recognition signal: `host.name` is a cloud service name (e.g. `"jfrog-cloud"`), constant across all log records from the integration.
### service.name as audit object
When the vendor provides a `service.name` or equivalent that identifies the system being audited, and no explicit object ID/name exists in the payload, map to `object.name` + `object.id` with `object.type = "service"`.
### audit.action from code combination
When the vendor uses separate event-type and subtype codes (e.g. `event = "TKN"`, `event_type = "C"`), combine as `event:subtype` → `audit.action` (e.g. `"TKN:C"`).
### result.code and result.message
- `result.code` (long): use when the vendor provides a numeric status or error code (e.g. HTTP status, error number). Coerce string→long if needed.
- `result.message` (string): use the vendor's outcome string when no numeric code exists (e.g. `"ACCEPTED"`, `"DENIED"`). For change-tracking audit logs, the change-type key name (e.g. `"added"`, `"removed"`, `"modified"`) is a good result.message candidate.
### audit.time vs timestamp
`timestamp` is always Dynatrace ingest time — never map from vendor. The vendor's event time (when the action occurred) maps to `audit.time` for audit log classes. For http class logs, event time stays in `content`.
---
## Mapping Table Template
**Workflow A — Suggestion:**
| Source Field | Target Field | Transform | Required | Sample Value | Notes |
|---|---|---|---|---|---|
| `content.username` | `audit.identity` | direct | yes | `jsmith@acme.com` | |
| `content.source` | `actor.ips` | string→array | yes | `["44.198.0.1"]` | wrap in array |
| `content.action` | `audit.action` | direct | yes | `"login"` | |
| `content.success` | `audit.result` | `"True"`→`"Succeeded"`, `"False"`→`"Failed"` | recommended | `"Succeeded"` | |
| derived | `loglevel` | from `audit.result` | yes | `"INFO"` | ERROR if Failed |
| — | `log.source` | constant | yes | `"CyberArk"` | hardcoded per integration |
**Workflow B1 — Diff:**
| Source Field | Current Target | Suggested Target | Transform | Status | Reason |
|---|---|---|---|---|---|
| `log.source` | `log.source` | `log.source` | direct | ✅ ok | |
| `content.username` | — | `audit.identity` | direct | ➕ add | required field not promoted |
references/openpipeline-constraints.md
# OpenPipeline Transform Constraints
Mappings suggested by this skill are implemented as **OpenPipeline processors** at ingest. OpenPipeline runs a **restricted subset of DQL** — fewer commands and fewer functions than the DQL you use to *query* `fetch logs` in Grail. A transform that is valid in a Grail query is **not** guaranteed to be valid in an OpenPipeline processor.
Apply these constraints whenever you propose a transform for a buried `content` field or any other extraction rule.
**Authority:** the live Dynatrace documentation is the source of truth for the exact OpenPipeline command and function set. Do **not** assume a full-DQL function is available in OpenPipeline — verify against the references below before relying on it.
- Commands: https://docs.dynatrace.com/docs/platform/openpipeline/reference/dql/openpipeline-dql-commands
- Functions: https://docs.dynatrace.com/docs/platform/openpipeline/reference/dql/openpipeline-dql-functions
- Operators: https://docs.dynatrace.com/docs/platform/openpipeline/reference/dql/openpipeline-dql-operators
- Matcher: https://docs.dynatrace.com/docs/platform/openpipeline/reference/dql/dql-matcher-in-openpipeline
## TOC
- [Supported processing commands](#supported-processing-commands)
- [Supported function classes](#supported-function-classes)
- [Two field sources](#two-field-sources-root-level-fields-and-the-content-blob)
- [`content` field extraction](#content-field-extraction--do-not-use-parsejson)
- [Iterative operators](#iterative-operators--per-element-array-processing)
- [Matching criteria (processor gating)](#matching-criteria-processor-gating)
- [How to apply](#how-to-apply)
---
## Supported processing commands
| Command | Use |
|---|---|
| `parse` | Parse a field with a DPL pattern into one or more fields |
| `fields` / `fieldsKeep` | Keep only the specified fields |
| `fieldsAdd` | Evaluate an expression and append or replace a field |
| `fieldsRename` | Rename a field |
| `fieldsRemove` | Remove fields |
| `fieldsFlatten` | Extract/flatten fields from a nested record |
Commands outside this set (`fetch`, `filter`, `summarize`, `sort`, `dedup`, `join`, `lookup`, `makeTimeseries`) are query/aggregation constructs — do not propose them as ingest transforms.
---
## Supported function classes
| Class | Notes |
|---|---|
| String | `concat`, `contains`, `replacePattern`, `splitString`, `trim`, `lower`, `upper`, `substring`, `matchesPhrase`, `matchesValue`, and others |
| Conversion / cast | `toString`, `toLong`, `toDouble`, `toBoolean`, `toTimestamp`, `toDuration`, `toIp`, `asString`, `asLong`, `asIp`, `asRecord`, `asArray`, and others |
| Conditional | `coalesce`, `if` |
| Boolean | `isNull`, `isNotNull`, `isTrueOrNull`, `isFalseOrNull` |
| Array | `arraySort`, `arraySum`, `arrayAvg`, `arrayDistinct`, `arrayRemoveNulls`, `arraySize`, and others |
| Network | `ip()`, `ipIn`, `ipIsPrivate`, `isIpV4`, `isIpV6`, and others |
| Time | timestamp operations, duration creation, date extraction |
| Math | `abs`, `sqrt`, `round`, `log`, `power`, and others |
| Hash / bitwise | `hashMd5`, `hashSha256`, bitwise operators |
| General | `in`, `exists`, `record` |
**Not available:** `parseJson`, `jsonPath`, and any aggregation/query function that requires `summarize` (e.g. `count`, `sum`, `avg`) — processors are per-record; there is no grouping or aggregation stage.
**Confirmed idioms (from production processors):** `timestampFromUnixMillis(toLong(x))` (ms epoch → timestamp), `toLong(x)` (status codes / epochs), `array(toIp(x))` (→ `ipAddress[]`), `concat(a, " ", b)`, `coalesce(a, b, …)` (fallback across source paths), nested `if(cond, x, else: if(cond2, y, else: z))`, and operators `==`/`!=`/`>=`/`>`/`<=`/`<`. Assign the built-in `timestamp` to `audit.time` when no vendor timestamp is available. `matchesValue(field, "*pat*")` is usable **inside `fieldsAdd`** (to compute a boolean flag), not only as a matcher. Compute reusable values into a `_`-prefixed helper field and `fieldsRemove` it when done. Backtick-quote field names with dots/hyphens (`` `audit.action` ``, `` `http.request.header.user-agent` ``).
---
## Two field sources: root-level fields and the `content` blob
An ingested log event can expose vendor data in **two places at once** — they are not mutually exclusive:
- **Root-level fields** — some fields are already parsed to the top level of the event at ingest (e.g. `service.name`, `company.name`). Reference them **directly** in `fieldsAdd`, backtick-quoting names with dots or hyphens: `` fieldsAdd `object.id` = `service.name` ``. No parsing needed.
- **The `content` field** — always carries the **full raw (unparsed)** vendor payload. Parse it to reach anything not already at root level.
Prefer a root-level field when the value is already there; parse `content` for buried values.
## `content` field extraction — do NOT use `parseJson`
The natural instinct is `parseJson content, prefix:"c."` — but **`parseJson` is not an OpenPipeline function**.
**Canonical OpenPipeline-valid pattern:**
```dql-snippet
| parse content, "json:content" // json DPL token is case-insensitive; parses into a record
| fieldsAdd audit.identity = content[user_name] // subscript: content[key] or content[key][subkey] for nested
| fieldsAdd loglevel = if(content[response] == "DENIED", "ERROR", else: "INFO")
// ... other fieldsAdd statements ...
```
Key points:
- **DPL literal** `"json:<varname>"` — the `json`/`JSON` token is case-insensitive. Two conventions: parse into `content` itself (`"json:content"`, overwrites the working field; the persisted event keeps the raw payload so any processor can re-parse) or into a new record (`"JSON:c"`, then `fieldsRemove c` in cleanup).
- Use **subscript notation** `content[key]` or `content[key][subkey]` (any depth) for nested access — no `fieldsFlatten` needed for targeted extraction.
- **Do not assume a parsed record persists across processors.** Because `content` always retains the raw payload, re-parse `content` in each processor that needs buried values (production integrations commonly do exactly this).
- **`fieldsFlatten content, prefix:"c."`** is an alternative when every JSON key should be promoted to `c.<key>` — less common when targeting specific fields.
---
## Iterative operators — per-element array processing
Three operators are available for per-element array transforms:
| Operator | Use |
|---|---|
| `iAny` | Check a per-element boolean expression; true if satisfied at least once |
| `iCollectArray` | Collect per-element expression results into a new array |
| `iIndex` | Access the current element's index |
Use `iCollectArray` when casting an array of strings to a typed array (e.g. `ipAddress[]` fields like `actor.ips`).
---
## Matching criteria (processor gating)
Processors fire only for records matching a gating condition. Available:
| Feature | Notes |
|---|---|
| `matchesValue(field, "pattern")` | Supports `*` wildcards |
| `matchesPhrase(field, "phrase")` | Case-insensitive; wildcards at phrase start/end only |
| `isNull(field)` / `isNotNull(field)` | Null testing |
| `AND`, `OR`, `NOT` | Logical combination |
| Numeric comparators | `<`, `>`, `==`, `<=`, `>=` |
| `iAny(condition)` | Per-element condition check |
**Gotcha:** `==` is case-sensitive with no wildcards — use `matchesValue()` when casing could vary.
---
## How to apply
- **Workflow A (suggest):** every Transform column entry for a buried `content` field must be expressible with `parse` → `fieldsFlatten`/subscript (not `parseJson`/`jsonPath`). Use `fieldsAdd` with supported function classes only.
- **Workflow B (validate):** flag any provided transform that uses `parseJson`, `jsonPath`, or an unavailable function/command as a major discrepancy and propose the valid alternative.
references/report-format.md
# Report Format
## TOC
- [Workflow A — Phase 1](#workflow-a--phase-1)
- [Workflow A — Phase 2 (after approval)](#workflow-a--phase-2-after-approval)
- [Workflow B1](#workflow-b1)
- [Workflow B2](#workflow-b2)
---
## Workflow A — Phase 1
### Mapping Summary
- Vendor / `log.source`:
- Log class:
- Sample count:
- Confidence: `high | medium | low`
### Mapping Table
| Source Field | Target Field | Transform | Required | Sample Value | Notes |
|---|---|---|---|---|---|
| `content.username` | `audit.identity` | direct | yes | `jsmith@acme.com` | |
| `content.source` | `actor.ips` | string→array | yes | `["44.198.0.1"]` | |
| `content.success` | `audit.result` | bool→title-case | recommended | `"Succeeded"` | |
| derived | `loglevel` | from `audit.result` | yes | `"INFO"` | |
| — | `log.source` | constant | yes | `"CyberArk"` | |
### Content Field Promotion Plan
| content sub-field | Target SD field | Transform | OpenPipeline rule needed |
|---|---|---|---|
| `content.username` | `audit.identity` | direct | yes |
| `content.ip_address` | `actor.ips` | string→array | yes |
### OpenPipeline Processor Sketch
```dql-snippet
// parseJson is NOT an OpenPipeline function — use parse content, "json:content" + subscript access instead.
| parse content, "json:content" // turn JSON string into a structured record
| fieldsAdd audit.identity = content[username] // subscript access: content[key] for JSON keys
| fieldsAdd actor.ips = array(toIp(content[ip_address])) // → ipAddress[]
| fieldsAdd audit.result = if(content[success] == "True", "Succeeded", else: "Failed")
| fieldsAdd loglevel = if(content[success] == "True", "INFO", else: "ERROR")
| fieldsAdd status = loglevel
```
### Gap Summary
| Required Field | Status | Reason |
|---|---|---|
| `audit.result` | ⚠ derived | not explicit — derived from `content.success` |
### Discrepancies
| Severity | Issue | Suggested Fix |
|---|---|---|
| major | `audit.identity` not promoted from `content.username` | add OpenPipeline extraction |
| minor | `loglevel` absent | derive from outcome or set constant `"INFO"` |
---
## Workflow A — Phase 2 (after approval)
One mapped sample JSON per log class. Annotate every transform inline.
```json
// CyberArk Identity — authentication (MFA challenge)
// Transforms:
// content.username → audit.identity [direct]
// content.source → actor.ips [string→array]
// content.success "False" → audit.result "Failed" [bool map]
// audit.result "Failed" → loglevel "ERROR" [outcome map]
{
"timestamp": "2026-05-08T16:23:07.539000000Z",
"log.source": "CyberArk",
"content": "{\"username\":\"i.rodriguez@example.com\",\"source\":\"207.162.45.12\",\"action\":\"Multifactor challenge\",\"success\":\"False\"}",
"loglevel": "ERROR",
"status": "ERROR",
"event.type": "LOG",
"audit.action": "Multifactor challenge",
"audit.identity": "i.rodriguez@example.com",
"audit.result": "Failed",
"actor.ips": ["207.162.45.12"],
"client.ip": "207.162.45.12",
"actor.geo.city.name": "Montreal",
"actor.geo.country.name": "Canada",
"browser.name": "Chrome",
"browser.version": "147.0.0.0",
"device.os.name": "Mac",
"cloud.provider": "aws",
"cyberark.service": "Identity"
}
```
---
## Workflow B1
### Diff-Highlighted Mapping Table
Marker legend: ✅ ok · ⚠ change · ➕ add · ❌ remove
| Source Field | Current Target | Suggested Target | Transform | Status | Reason |
|---|---|---|---|---|---|
| `log.source` | `log.source` | `log.source` | direct | ✅ ok | |
| `content.username` | — | `audit.identity` | direct | ➕ add | required field not promoted |
| `content.ip_address` | `client.ip` | `actor.ips` | string→array | ⚠ change | primary IP field is array |
| `loglevel` | — | `loglevel` | derive from outcome | ➕ add | required field absent |
| `content.internalRef` | `internal.ref` | — | — | ❌ remove | not in SD or samples |
### Content Burial Report
| content sub-field | Top-level semantic field? | Target | Status |
|---|---|---|---|
| `content.username` | no | `audit.identity` | ➕ promote |
| `content.ip_address` | yes — `client.ip` | `actor.ips` | ⚠ promote to array field |
| `content.action` | no | `audit.action` | ➕ promote |
| `content.trace_id` | yes — `trace_id` | — | ✅ promoted |
**Burial score:** 2 of 4 content fields promotable, 2 gaps remaining.
### Required-Field Matrix
| Field | Status | Notes |
|---|---|---|
| `timestamp` | ✅ pass | |
| `log.source` | ✅ pass | |
| `content` | ⚠ warn | JSON object — must be serialized to string |
| `loglevel` | ❌ fail | absent — derive from outcome |
| `audit.action` | ❌ fail | buried in `content.action` |
| `audit.identity` | ❌ fail | buried in `content.username` |
### Discrepancies
| Severity | Issue | Suggested Fix |
|---|---|---|
| critical | `content` is JSON object, not string | serialize before ingest |
| major | `audit.identity` not promoted | extract from `content.username` via OpenPipeline |
| major | `loglevel` absent | derive from `audit.result` |
### OpenPipeline Improvement Plan
1. Extract `audit.identity` from `content.username`.
2. Derive `loglevel` / `status` from outcome.
3. Serialize `content` to JSON string.
---
## Workflow B2
B2 output uses the B1 format above, applied to fetched live records. Prefix the report with:
**Runtime Context:**
- `log.source` queried:
- Time window:
- Records fetched: N
- Execution method: live DQL execution
Then run B1 validation on the fetched records. Produce the same Diff-Highlighted Mapping Table, Content Burial Report, Required-Field Matrix, and Discrepancies sections.
Add a **Validation Summary** table:
| Check | Result | Evidence |
|---|---|---|
| Logs found for `log.source` | `🟢 pass` / `🔴 fail` | record count |
| `content` is string | `🟢 pass` / `🔴 fail` | type on sample |
| `loglevel` valid enum | `🟢 pass` / `🔴 fail` | distinct values seen |
| `audit.action` populated | `🟢 pass` / `🟡 warn` | null count |
| `audit.identity` populated | `🟢 pass` / `🟡 warn` | null count |
| `audit.result` populated | `🟢 pass` / `🟡 warn` | null count |
| Content burial score | `🟢 0` / `🟡 1–2` / `🔴 3+` | promotable field list |
Status legend: `🟢 pass` · `🟡 warn` · `🔴 fail`
references/runtime-validation.md
# Runtime Validation (Workflow B2)
B2 = fetch live log records → run B1 static validation on them.
No DQL query pack is embedded here. All validation logic lives in `mapping-workflow.md § Workflow B1` and `validation-rules.md`.
## Procedure
### Step 0 — Confirm Inputs
1. `log.source` value to query (e.g. `"CyberArk"`, `"Okta"`, `"SignInLogs"`).
2. Time window (default: `now()-24h`).
3. Execution method: live DQL execution against the connected tenant.
### Step 1 — Fetch Sample Records
Execute a DQL query against the live tenant to fetch 3–5 recent log records for the given `log.source`.
Query pattern:
```dql-snippet
fetch logs, from:now()-24h
| filter log.source == "<LOG_SOURCE>"
| sort timestamp desc
| limit 5
```
If 0 records returned: report `🔴 fail — no logs found`. Ask the user to verify `log.source` and time window before continuing.
### Step 2 — Run B1 on Each Fetched Record
Apply the full Workflow B1 procedure from `mapping-workflow.md` to each fetched record.
Treat the fetched records as **final ingested events** — `content` is expected to be a string, top-level semantic fields are expected to be promoted.
### Step 3 — Aggregate and Report
Consolidate B1 findings across all fetched records:
- If the same issue appears on all records: report as a systemic gap.
- If an issue appears on only some records: note variability and which conditions trigger it.
Produce the Validation Summary table and the full B1 report sections from `report-format.md § Workflow B2`.
## Notes
- The B2 query is always a simple `fetch logs | filter log.source == "..."` — do not add complex DQL unless the user requests it.
- If the user asks to narrow the query (e.g. by time, by audit class, by specific `audit.action`), adjust the filter accordingly and note it in the Runtime Context section.
- Do not interpret fetched records beyond what B1 covers. B2 scope = fetch + B1.
references/validation-rules.md
# Validation Rules
## TOC
- [Required Fields](#required-fields)
- [content Field Rules](#content-field-rules)
- [loglevel and status Rules](#loglevel-and-status-rules)
- [Audit Namespace Rules](#audit-namespace-rules)
- [Content Burial Rules](#content-burial-rules)
- [Type and Value Rules](#type-and-value-rules)
- [Discrepancy Severity Reference](#discrepancy-severity-reference)
- [Acceptance Criteria](#acceptance-criteria)
---
## Required Fields
| Field | Severity if absent | Notes |
|---|---|---|
| `timestamp` | critical | Auto-set on ingest — not required in pre-ingest draft validation |
| `log.source` | critical | Must be non-null |
| `content` | critical | Must be a string (not an object) on final ingested events |
| `loglevel` | critical | Must be a valid enum value |
| `audit.action` | major | Required when action is identifiable from the payload |
| `audit.identity` | major | Required when a user or service account is identifiable |
---
## content Field Rules
| Rule | Severity |
|---|---|
| `content` is not a string on a final ingested event | critical |
| `content` is a JSON object on a pre-ingest draft | major — remind to serialize before ingest |
| `content` is absent entirely | critical |
| `content` cannot be parsed as JSON when it appears to be JSON | minor — note in report, use raw string analysis |
---
## loglevel and status Rules
| Rule | Severity |
|---|---|
| `loglevel` absent | critical |
| `loglevel` value outside `ERROR/WARN/INFO/DEBUG/TRACE/NONE` | critical |
| `loglevel` set to un-normalized vendor value (e.g. `critical`, `fatal`) | major |
| `status` absent | minor |
| `status` value outside `INFO/WARN/ERROR/NONE` | major |
| `loglevel` and `status` semantically inconsistent | minor — flag and document intent |
---
## Audit Namespace Rules
| Rule | Severity |
|---|---|
| `audit.action` absent when action is derivable from payload (audit log classes only) | major |
| `audit.identity` absent when user/service is in payload (audit log classes only) | major |
| `audit.result` absent when outcome (success/failure) is in payload | major |
| `audit.result` value not using title-case convention (`Succeeded`/`Failed`) | minor |
| `audit.status` set to a `loglevel` value (e.g. `"INFO"`, `"ERROR"`) | major — mapping error; valid values are `Started`, `In Progress`, `Succeeded`, `Failed`, `Active`, `Resolved` |
| `audit.status` value outside SD enum (`Started/In Progress/Succeeded/Failed/Active/Resolved`) | major |
| `audit.time` missing timezone offset (e.g. `"2026-05-08T15:37:05"` without Z) | minor |
---
## Content Burial Rules
A field is "buried" when it exists in parsed `content` JSON but has no top-level semantic counterpart.
| Buried fields | Severity |
|---|---|
| 0 promotable buried fields | pass |
| 1–2 promotable buried fields | minor |
| 3+ promotable buried fields | major |
| Buried required field (`audit.identity`, `audit.action`) | major per field |
For each buried field: state the source path, the target SD field, the transform, and the OpenPipeline extraction rule.
---
## Type and Value Rules
| Rule | Severity |
|---|---|
| `actor.ips` is a plain string instead of `ipAddress[]` | major |
| `result.code` is a string (SD type is `long`) | major — coerce to long; non-numeric codes (e.g. `"SUCCESS"`, `"IDP2013"`) belong in `result.message` instead |
| `result.details` (plural) used instead of `result.detail` (singular) | minor — rename to `result.detail` |
| `http.response.status_code` is a string instead of integer | major — coerce to integer |
| `http.response.body.size` is a string instead of long | minor — coerce to long |
| `actor.geo.location.lat/lon` as string instead of float | minor — prefer float for correct geo-query behavior |
| `http.request.header.<Name>` or `http.response.header.<Name>` uses mixed/Pascal case header name | minor — rename to lowercase (e.g. `user-agent` not `User-Agent`) |
| Field value is a string sentinel (`"UNKNOWN"`, `"N/A"`, `"-"`, `"null"`) | minor — omit field; do not promote sentinel strings to top-level semantic attributes |
| `timestamp` mapped from vendor payload | major — `timestamp` is Dynatrace ingest time; vendor event time belongs in `audit.time` |
| `log.source` set to raw vendor identifier instead of integration name | minor — use human-readable integration name constant (e.g. `"JFrog"` not `"jfrog_artifactory"`) |
| `host.*` fields populated from connector/shipper metadata (not target host) | minor — move to vendor namespace (e.g. `<vendor>.instance.*`) |
| Vendor-namespace field populated while its SD-canonical counterpart is null | major — backfill the SD field |
| Vendor-namespace field is an exact duplicate of an SD-canonical field value | minor — remove vendor field |
| More than ~5 vendor-namespace fields extracted | minor — review and retain only highest-value fields not covered by SD |
---
## Discrepancy Severity Reference
| Severity | Criteria |
|---|---|
| `critical` | Missing required field; `content` not a string; invalid `loglevel` enum |
| `major` | Missing `audit.action`/`audit.identity`/`audit.result` when derivable; 3+ buried promotable fields; `audit.status` contains a `loglevel` value; type mismatch on required fields |
| `minor` | 1–2 buried promotable fields; missing `audit.time` timezone; `audit.result` not title-case; `loglevel`/`status` inconsistency; vendor-namespace duplication |
| `info` | Deviation already documented in `data-model-notes.md § Known Discrepancies` |
---
## Acceptance Criteria
A mapping passes when:
1. All required fields are present or auto-derivable.
2. No critical discrepancies unresolved.
3. All major discrepancies fixed or documented.
4. `loglevel` and `status` use valid SD enum values.
5. `audit.action` and `audit.identity` populated.
6. `content` is a string (or will be on ingest).
7. Content burial score is 0, or remaining buried fields are documented as intentional.
samples/audit-logs.json
[
{
"timestamp": "2026-05-08T16:23:07.539000000Z",
"content": "{\"uuid\": \"34f1eea3-3acf-44e4-bff7-096048c81757\", \"tenantId\": \"000000-5d4e-4ecb-a970-e1d7f9e46047\", \"timestamp\": 1778257364455, \"username\": \"i.rodriguez@example.com\", \"applicationCode\": \"IDP\", \"auditCode\": \"IDP2013\", \"auditType\": \"Info\", \"action\": \"Multifactor challenge\", \"userId\": \"d3d24da8-8395-46c2-e188-95f4e3108089\", \"source\": \"207.162.45.12\", \"actionType\": \"Retrieve\", \"component\": \"Identity\", \"serviceName\": \"Identity\", \"accessMethod\": null, \"accountId\": null, \"target\": null, \"command\": null, \"sessionId\": null, \"message\": \"cloud.core.login.multifactorchallenge\", \"customData\": {\"source_ip_address\": \"{template}\", \"uuid\": \"{template}\", \"user_guid\": \"{template}\", \"device_os\": \"Mac\", \"browser_name\": \"Chrome\", \"authentication_method\": \"MultiAuth\", \"when_occurred\": \"5/8/2026 4:22:44 PM\", \"tenant_id\": \"ABC1234\", \"thread_type\": \"RestCall\", \"mobile_device\": \"False\", \"level\": \"Info\", \"hostname\": \"10.10.123.45\", \"username\": \"i.rodriguez@example.com\", \"internal_session_id\": \"00000bWt7w1aE4Ojc_P_Zm31x57RLHtLs_3z7_Ztd981\", \"directory_service_uuid\": \"00000000-6CE8-465F-AB03-65766D33B05E\", \"user_agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36\", \"mechanism\": \"Email\", \"challenge_redirected\": \"False\", \"target\": \"i.rodriguez@example.com\", \"success\": \"True\", \"geoip_latitude\": \"45.5019\", \"geoip_longitude\": \"-73.5674\", \"geoip_city_name\": \"Montreal\", \"geoip_country_name\": \"Canada\", \"geoip_country_code\": \"CA\", \"client_ip_address\": \"207.162.45.12\", \"cookie_session\": \"null\"}, \"cloudProvider\": \"aws\", \"cloudWorkspacesAndRoles\": [], \"cloudIdentities\": null, \"cloudAssets\": null, \"safe\": null, \"accountName\": null, \"targetPlatform\": null, \"targetAccount\": null, \"identityType\": \"HUMAN\", \"correlationId\": null, \"isDr\": false, \"originRegion\": \"eu-central-1\"}",
"event.type": "LOG",
"status": "INFO",
"loglevel": "INFO",
"actor.geo.city.name": "Montreal",
"actor.geo.country.name": "CA",
"actor.geo.location.lat": "45.5019",
"actor.ips": [
"207.162.45.12"
],
"audit.action": "Multifactor challenge",
"audit.identity": "i.rodriguez@example.com",
"audit.result": "Succeeded",
"audit.status": "Succeeded",
"audit.time": "2026-05-08T16:22:44.455000+00:00",
"browser.name": "Chrome",
"browser.user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
"browser.version": "147.0.0.0",
"client.app.name": "Identity",
"client.ip": "207.162.45.12",
"cloud.provider": "aws",
"cyberark.service": "Identity",
"device.os.name": "Mac",
"log.source": "CyberArk",
"object.name": "Identity: Identity",
"product.name": "CyberArk",
"product.vendor": "CyberArk",
"result.detail": "Multifactor challenge",
"result.message": "IDP2013",
"actor.geo.location.lon": "-73.5674",
"authentication.is_multifactor": true
},
{
"timestamp": "2026-05-08T16:22:36.363000000Z",
"content": "{\"actor\": {\"id\": \"00u11bsm8zpSQvS0M698\", \"type\": \"User\", \"alternateId\": \"user2@example.com\", \"displayName\": \"Example User\", \"detailEntry\": null}, \"client\": {\"userAgent\": {\"rawUserAgent\": \"python-requests/2.25.1\", \"os\": \"Unknown\", \"browser\": \"UNKNOWN\"}, \"zone\": \"null\", \"device\": \"Unknown\", \"id\": null, \"ipAddress\": \"44.203.244.130\", \"geographicalContext\": {\"city\": \"Ashburn\", \"state\": \"Virginia\", \"country\": \"United States\", \"postalCode\": \"20149\", \"geolocation\": {\"lat\": 39.0469, \"lon\": -77.4903}}}, \"device\": null, \"authenticationContext\": {\"authenticationProvider\": null, \"credentialProvider\": null, \"credentialType\": null, \"issuer\": null, \"interface\": null, \"authenticationStep\": 0, \"rootSessionId\": \"trsdARsJ0AjSkW37kFoXl-R9g\", \"externalSessionId\": \"trsdARsJ0AjSkW37kFoXl-R9g\"}, \"displayMessage\": \"Create okta group\", \"eventType\": \"group.lifecycle.create\", \"outcome\": {\"result\": \"SUCCESS\", \"reason\": null}, \"published\": \"2026-05-08T16:20:43.994Z\", \"securityContext\": {\"asNumber\": 14618, \"asOrg\": \"amazon data services northern virginia\", \"isp\": \"amazon.com inc.\", \"domain\": \"amazonaws.com\", \"isProxy\": false, \"ipDetails\": {\"asNumber\": 14618, \"asOrg\": \"amazon data services northern virginia\", \"isp\": \"amazon.com inc.\", \"domain\": \"amazonaws.com\"}}, \"severity\": \"INFO\", \"debugContext\": {\"debugData\": {\"requestId\": \"c61d47b0d030a842a740515483f60cc9\", \"dtHash\": \"0ddb6b1fdd21f4bd573c4d0b45b75af573b91ca8d6ee2ee199da66342a8c8391\", \"requestUri\": \"/api/v1/groups\", \"url\": \"/api/v1/groups?\"}}, \"legacyEventType\": \"group.lifecycle.create\", \"transaction\": {\"type\": \"WEB\", \"id\": \"c61d47b0d030a842a740515483f60cc9\", \"detail\": {\"rootApiTokenId\": \"00T3u94nuvXTk7aHJ697\", \"requestApiTokenId\": \"00T3u94nuvXTk7aHJ697\"}}, \"uuid\": \"dcfaec59-4af9-11f1-88d5-c9bac4c5670d\", \"version\": \"0\", \"request\": {\"ipChain\": [{\"ip\": \"44.203.244.130\", \"geographicalContext\": {\"city\": \"Ashburn\", \"state\": \"Virginia\", \"country\": \"United States\", \"postalCode\": \"20149\", \"geolocation\": {\"lat\": 39.0469, \"lon\": -77.4903}}, \"version\": \"V4\", \"source\": null, \"ipDetails\": {\"asNumber\": 14618, \"asOrg\": \"amazon data services northern virginia\", \"isp\": \"amazon.com inc.\", \"domain\": \"amazonaws.com\"}}]}, \"target\": [{\"id\": \"00g12syy766mp9qw5698\", \"type\": \"UserGroup\", \"alternateId\": \"unknown\", \"displayName\": \"could\", \"detailEntry\": null}]}",
"event.type": "LOG",
"loglevel": "INFO",
"status": "INFO",
"actor.geo.city.name": "Ashburn",
"actor.geo.country.name": "United States",
"actor.geo.location.lat": "39.0469",
"actor.geo.location.lon": "-77.4903",
"actor.ips": [
"44.203.244.130"
],
"audit.action": "Create okta group",
"audit.identity": "user2@example.com",
"audit.result": "Succeeded",
"audit.time": "2026-05-08T16:20:43.994Z",
"browser.user_agent": "python-requests/2.25.1",
"client.ip": "44.203.244.130",
"instance": "integrator-8493742-admin.okta.com",
"log.source": "Okta",
"log.uuid": "dcfaec59-4af9-11f1-88d5-c9bac4c5670d",
"object.id": "c61d47b0d030a842a740515483f60cc9",
"object.name": "/api/v1/groups",
"object.type": "Request URI",
"product.name": "Okta",
"product.vendor": "Okta",
"result.message": "SUCCESS"
},
{
"timestamp": "2026-05-08T16:24:07.877000000Z",
"content": "{\"time\":\"2026-05-08T17:09:48.815Z\",\"resourceId\":\"/tenants/34243efe-c30f-49fc-8035-eceb48ff9a4f/providers/Microsoft.aadiam\",\"operationName\":\"Sign-in activity\",\"category\":\"SignInLogs\",\"tenantId\":\"34243efe-c30f-49fc-8035-eceb48ff9a4f\",\"properties\":{\"createdDateTime\":\"2026-05-08T17:09:48.815Z\",\"userDisplayName\":\"Bruce Wayne\",\"userPrincipalName\":\"bruce.wayne@example.com\",\"userId\":\"255a621a-4457-4523-9501-c798f020edca\",\"appId\":\"74655a85-ea8d-465e-84e5-6b7fe15d2fb6\",\"appDisplayName\":\"Confluence\",\"ipAddress\":\"104.26.0.1\",\"clientAppUsed\":\"Browser\",\"correlationId\":\"63cea183-bc93-4a91-bc35-f6b94ac3f25f\",\"authenticationRequirement\":\"multiFactorAuthentication\",\"resourceDisplayName\":\"Power Apps API\",\"resourceId\":\"331cc017-5973-4173-b270-f0042fddfd75\",\"status\":{\"errorCode\":50076,\"failureReason\":\"MFA required but not satisfied.\",\"additionalDetails\":\"Check credentials and try again.\"},\"deviceDetail\":{\"deviceId\":\"device-b4f2c9d8\",\"displayName\":\"User Device\",\"operatingSystem\":\"Windows 10\",\"browser\":\"Chrome 115.0.0\"},\"location\":{\"city\":\"Linz\",\"countryOrRegion\":\"AT\",\"geoCoordinates\":{\"altitude\":null,\"latitude\":48.303,\"longitude\":14.291}},\"appliedConditionalAccessPolicies\":[]}}",
"loglevel": "ERROR",
"status": "ERROR",
"cloud.provider": "azure",
"audit.time": "2026-05-08T17:09:48.815000000Z",
"audit.action": "Sign-in activity",
"audit.identity": "bruce.wayne@example.com",
"audit.result": "Failed",
"log.source": "SignInLogs",
"object.id": "331cc017-5973-4173-b270-f0042fddfd75",
"object.name": "Power Apps API",
"object.type": "Cloud Application",
"result.code": 50076,
"result.message": "MFA required but not satisfied.",
"result.detail": "Check credentials and try again.",
"client.app.name": "Confluence",
"client.ip": "104.26.0.1",
"actor.ips": [
"104.26.0.1"
],
"actor.geo.city.name": "Linz",
"actor.geo.country.name": "AT",
"actor.geo.location.lat": 48.303,
"actor.geo.location.lon": 14.291,
"browser.name": "Chrome",
"browser.version": "115.0.0",
"device.id": "device-b4f2c9d8",
"device.name": "User Device",
"device.os.name": "Windows 10",
"azure.tenant.id": "34243efe-c30f-49fc-8035-eceb48ff9a4f",
"authentication.is_multifactor": true
},
{
"timestamp": "2026-05-08T16:12:56.110000000Z",
"content": "{\"timestamp\": \"2026-05-08T15:12:27.535Z\", \"remoteIpAddress\": \"82.177.196.146\", \"forwarded\": \"82.177.196.146\", \"userAgent\": \"python-requests/2.32.5\", \"username\": \"user3@example.com\", \"domain\": \"audit-log\", \"type\": \"export\"}",
"loglevel": "INFO",
"status": "INFO",
"event.type": "LOG",
"audit.action": "export",
"audit.identity": "user3@example.com",
"audit.time": "2026-05-08T15:12:27.535Z",
"extension.config.name": "demo live",
"log.source": "Sonatype Lifecycle"
},
{
"timestamp": "2026-05-08T13:39:32.846000000Z",
"content": "{\"@timestamp\": 1778247425821, \"_document_id\": \"qOalRRj-b7bDr4NBZGO_iQ\", \"action\": \"workflows.completed_workflow_run\", \"actor\": \"github-actions[bot]\", \"actor_id\": 41898282, \"actor_is_agent\": false, \"actor_is_bot\": true, \"business\": \"dynatrace-partner-demo\", \"business_id\": 314815, \"completed_at\": \"2026-05-08T13:37:05.000Z\", \"conclusion\": \"success\", \"created_at\": 1778247425821, \"event\": \"dynamic\", \"head_branch\": \"main\", \"head_sha\": \"c7a6f51df9ada1183b42be9a1311d3a719ede0d3\", \"name\": \"Push on main\", \"operation_type\": \"modify\", \"org\": \"DynatraceAppSecDemo\", \"org_id\": 225466100, \"public_repo\": true, \"repo\": \"DynatraceAppSecDemo/unguard\", \"repo_id\": 1034686578, \"run_attempt\": 1, \"run_number\": 691, \"started_at\": \"2026-05-08T13:34:57.000Z\", \"topic\": \"cp1-iad.ingest.github.actions.v0.WorkflowUpdate\", \"trigger_id\": null, \"workflow_id\": 183081562, \"workflow_run_id\": 25558551186}",
"audit.action": "workflows.completed_workflow_run",
"audit.identity": "github-actions[bot]",
"audit.result": "Succeeded",
"audit.time": "2026-05-08T13:37:05.821000Z",
"log.source": "GitHub Advanced Security",
"loglevel": "INFO",
"status": "INFO",
"event.type": "LOG"
}
]samples/http-logs.json
[{
"timestamp": "2026-01-30T12:32:54.432000000Z",
"content": "{\"type\":\"akamai_siem\",\"format\":\"json\",\"version\":\"1.0\",\"attackData\":{\"configId\":\"86508\",\"policyId\":\"APRE_186629\",\"clientIP\":\"148.64.8.134\",\"appliedAction\":\"allow\",\"rules\":\"Qk9ULTYwODA2MTgx%3bNjAxOTY5MDI%3d%3b\",\"ruleVersions\":\"%3b%3b\",\"ruleMessages\":\"U2FtcGxlIEludGVybmFsIFRyYWZmaWM%3bVmlzaWJpbGl0eQ%3d%3d%3b\",\"ruleTags\":\"QUtBTUFJL0JPVC9DVVNUX0RFRklORURfQk9UUw%3d%3d%3bVGVzdA%3d%3d%3b\",\"ruleData\":\"SW50ZXJuYWwgU2FtcGxlIFRyYWZmaWM%3b%3b\",\"ruleSelectors\":\"%3b%3b\",\"ruleActions\":\"YWxsb3c%3d%3bYWxlcnQ%3d%3b\",\"apiId\":\"API_826456\",\"apiKey\":\"\"},\"httpMessage\":{\"requestId\":\"4940e898\",\"start\":\"1769776343\",\"protocol\":\"HTTP/1.1\",\"tls\":\"tls1.3\",\"method\":\"POST\",\"host\":\"atlas-pre-01-channel-api.sampledomain.com\",\"port\":\"443\",\"path\":\"/cws-main-api-service/v1/feedback-logs\",\"requestHeaders\":\"Host%3a%20atlas-pre-01-channel-api.sampledomain.com%0d%0aUser-Agent%3a%20python-requests%2f2.32.4%0d%0aAccept-Encoding%3a%20gzip,%20deflate,%20br%0d%0aAccept%3a%20application%2fjson%0d%0aConnection%3a%20keep-alive%0d%0aContent-Type%3a%20application%2fjson%0d%0ax-lbg-brand%3a%20SAMPLE%0d%0ax-lbg-channel%3a%20DIGITAL%0d%0ax-lbg-org%3a%20SAMPLE%0d%0ax-lbg-client-id%3a%20example_client_id%0d%0ax-lbg-client-secret%3a%20xxx%0d%0ax-lbg-session-id%3a%20example_session_id%0d%0aCookie%3a%20akaalb_Failover1%3d%7eop%3d%7erv%3d77%7em%3d%7eos%3d%7eid%3dexample_cookie_id%0d%0aContent-Length%3a%20132%0d%0aVia%3a%20threatpulse%0d%0aX-Forwarded-For%3a%205.64.103.80%0d%0aremove-dup-edge-ctrl-headers-rollout-enabled%3a%201%0d%0a\",\"status\":\"204\",\"bytes\":\"0\",\"responseHeaders\":\"Content-Type%3a%20application%2fjson%0d%0ax-lbg-txn-correlation-id%3a%20example_txn_correlation_id%0d%0ax-lbg-api-version%3a%201.0%0d%0aAccess-Control-Allow-Origin%3a%20*%0d%0ax-content-type-options%3a%20nosniff%0d%0ax-request-id%3a%20example_request_id%0d%0aDate%3a%20Fri,%2030%20Jan%202026%2012%3a32%3a23%20GMT%0d%0aConnection%3a%20keep-alive%0d%0aStrict-Transport-Security%3a%20max-age%3d31536000%20%3b%20includeSubDomains%0d%0a\"},\"geo\":{\"continent\":\"EU\",\"country\":\"GB\",\"city\":\"LONDON\",\"regionCode\":\"EN\",\"asn\":\"5607\"},\"identity\":{\"tlsFingerprintV2\":\"example_tls_fp_v2\",\"tlsFingerprintV3\":\"example_tls_fp_v3\"}}",
"akamai.attackdata.apiid": "API_826456",
"akamai.attackdata.apikey": "",
"akamai.attackdata.appliedaction": "allow",
"akamai.attackdata.clientip": "148.64.8.134",
"akamai.attackdata.configid": "86508",
"akamai.attackdata.policyid": "APRE_186629",
"akamai.attackdata.ruleselectors": "",
"akamai.attackdata.ruleversions": "",
"akamai.config.ids": "86508",
"client.ip": "148.64.8.134",
"extension.config.name": "test",
"geo.city.name": "London",
"geo.continent.name": "Europe",
"geo.country.name": "United Kingdom",
"geo.region.code": "EN",
"host.fqdn": [
"atlas-pre-01-channel-api.sampledomain.com"
],
"http.request.header.accept": "application/json",
"http.request.header.accept-encoding": "gzip, deflate, br",
"http.request.header.connection": "keep-alive",
"http.request.header.content-length": "132",
"http.request.header.content-type": "application/json",
"http.request.header.cookie": "akaalb_Failover1=~op=~rv=77~m=~os=~id=example_cookie_id",
"http.request.header.host": "atlas-pre-01-channel-api.sampledomain.com",
"http.request.header.user-agent": "python-requests/2.32.4",
"http.request.header.via": "threatpulse",
"http.request.header.x-forwarded-for": "5.64.103.80",
"http.request.header.remove-dup-edge-ctrl-headers-rollout-enabled": "1",
"http.request.header.x-lbg-brand": "SAMPLE",
"http.request.header.x-lbg-channel": "DIGITAL",
"http.request.header.x-lbg-client-id": "example_client_id",
"http.request.header.x-lbg-client-secret": "xxx",
"http.request.header.x-lbg-org": "SAMPLE",
"http.request.header.x-lbg-session-id": "example_session_id",
"http.request.method": "POST",
"http.response.body.size": 0,
"http.response.header.access-control-allow-origin": "*",
"http.response.header.connection": "keep-alive",
"http.response.header.content-type": "application/json",
"http.response.header.date": "Fri, 30 Jan 2026 12:32:23 GMT",
"http.response.header.strict-transport-security": "max-age=31536000 ; includeSubDomains",
"http.response.header.x-content-type-options": "nosniff",
"http.response.header.x-lbg-api-version": "1.0",
"http.response.header.x-lbg-txn-correlation-id": "example_txn_correlation_id",
"http.response.header.x-request-id": "example_request_id",
"http.response.status_code": 204,
"log.source": "Akamai SIEM",
"loglevel": "INFO",
"server.address": "atlas-pre-01-channel-api.sampledomain.com",
"status": "INFO",
"url.domain": "atlas-pre-01-channel-api.sampledomain.com",
"url.path": "/cws-main-api-service/v1/feedback-logs",
"url.port": 443,
"url.scheme": "https",
"akamai.attackdata.ruleactions": "allow",
"akamai.attackdata.ruledata": "Internal Sample Bot Traffic",
"akamai.attackdata.rulemessages": "Sample Internal Bot Traffic",
"akamai.attackdata.rules": "BOT-60806181",
"akamai.attackdata.ruletags": "AKAMAI/BOT/CUST_DEFINED_BOTS"
}
]
SKILL.md
---
name: dt-obs-log-semantic-mapping
description: "Suggest and validate semantic dictionary (SD) mappings for audit log integrations using raw vendor log payloads or live ingested events. Use when: mapping a vendor audit log feed, authentication logs, user activity logs to the Dynatrace SD; checking required semantic fields; proposing OpenPipeline processor extraction rules based on DQL; running runtime validation (fetches live logs by log.source, then applies static validation)."
license: Apache-2.0
---
# dt-obs-log-semantic-mapping
Build and validate semantic-dictionary-aligned mappings for audit log integrations.
## Purpose
Use this skill when a user wants to:
- **Suggest** a mapping from a raw vendor audit log payload to Dynatrace `fetch logs` fields (Workflow A).
- **Validate** a mapping against a pasted ingested log event (Workflow B1 — static).
- **Validate** against live tenant data via live tenant access (Workflow B2 — runtime: fetches logs by `log.source`, then runs B1 on the result).
## Log Classes
| Class | Description | Key namespaces | Example sources |
|---|---|---|---|
| `authentication` | Login, logout, MFA, token | `audit.*`, `actor.*`, `browser.*`, `device.*` | CyberArk, Okta, Azure SignInLogs |
| `authorization` | Access decisions, permission changes | `audit.*`, `actor.*`, `object.*` | CyberArk, Okta |
| `user_action` | CRUD on platform resources | `audit.*`, `actor.*`, `object.*`, `product.*` | Okta, GitHub, Sonatype |
| `http` | HTTP request/response (WAF, network devices) | `http.*`, `url.*`, `server.*`, `geo.*`, `client.*` | Akamai SIEM, Cloudflare |
## Workflows
| Mode | Input | Source |
|---|---|---|
| **Workflow A** — Suggest mapping | Raw vendor log payload | `references/mapping-workflow.md § Workflow A` |
| **Workflow B1** — Static validation | Pasted ingested log event | `references/mapping-workflow.md § Workflow B1` |
| **Workflow B2** — Runtime validation | `log.source` value + live tenant access | `references/runtime-validation.md` — fetches logs, then runs B1 |
## Key Concepts
**Content field burial:** The primary validation concern. Fields in `content` (the raw vendor payload) that could be promoted to top-level semantic attributes but are not. The skill always inventories buried vs promoted fields and proposes OpenPipeline extraction rules to fix gaps.
> **Prerequisite:** When proposing OpenPipeline processor extraction rules, load the `dt-dql-essentials` skill first. OpenPipeline processors use DQL functions (`parse`, `fieldsAdd`, `splitString`, etc.) — using non-DQL syntax produces invalid rules.
**Sparse mappings are valid:** Integrations like GitHub or Sonatype may only populate core fields. Minimum required: `timestamp`, `log.source`, `content`, `loglevel`, `audit.action`, `audit.identity`.
## References
- `references/data-model-notes.md` — Log SD field taxonomy, audit namespace, enums, sample-derived patterns and known discrepancies
- `references/mapping-workflow.md` — Intake checklist, Workflow A and B1 procedures, content field analysis, field priority order
- `references/validation-rules.md` — Required fields, content/enum/type rules, discrepancy severity
- `references/openpipeline-constraints.md` — OpenPipeline processor command/function/operator/matcher restrictions; `parseJson` unavailability + `parse`→`fieldsFlatten` alternative; iterative operators for array casting
- `references/report-format.md` — Mapping table, diff table, OpenPipeline sketch, Validation Summary templates
- `references/runtime-validation.md` — Workflow B2: fetch live records, then run B1
- `samples/audit-logs.json` — Mapped samples: CyberArk, Okta, Azure SignInLogs, Sonatype, GitHub
- `samples/http-logs.json` — Mapped samples: Akamai SIEM (WAF/HTTP class)
- [Dynatrace Log Semantic Dictionary](https://docs.dynatrace.com/docs/semantic-dictionary/model/log)