references/completeness-and-root-cause.md
# Completeness and root cause
Use this to decide whether a finding is complete and whether a root cause holds: confirming
whether a failure belongs to one entity or the whole fleet, reporting counts, totals or "no
evidence of X", and performing and reporting root cause analysis. It applies to any source
(logs, API listings, data stores); the pagination rules below name the CloudWatch Logs calls.
Commands use AWS CLI syntax. The AWS MCP server is recommended for sandboxed execution and
audit logging; the same calls work through the CLI without it.
## Completeness and scope
### Search completeness
Before concluding that an event, request, or record doesn't exist, make sure you searched
everywhere it could be:
1. **AWS resources are regional.** If resource names include a region or marketplace marker,
or the data could be served by more than one regional deployment, list and query the
matching resources in every plausible region, not just the default one.
2. **Never report "no evidence of X" as a finding** until you have checked every scope. If
searches across all scopes still find nothing, say exactly which scopes you checked.
3. **For identity or authorization failures**, find and compare the actual values on both
sides, such as the identity the invitation or grant was issued to and the identity that
authenticated. Base the root cause and fix on that comparison, not on guesses about
infrastructure.
### Pagination requirements
When you read logs or list resources through AWS APIs, paginate until the data is exhausted
before you count, aggregate, or state a scope:
1. **For CloudWatch GetLogEvents**, keep following `nextForwardToken` until it returns the
same token twice.
2. **For FilterLogEvents and List/Describe calls**, keep following `nextToken` until none is
returned.
3. **Treat any pagination or truncation warning** as proof that the data is incomplete.
4. **If one call returns too much output**, process the pages and return only summaries. Do
not shrink the read to a single page.
5. **Before you report counts, totals, or "every X" claims**, check that the last event's
timestamp matches the stream's `lastEventTimestamp`. Also look for entities whose results
look partial, since that often means a page was cut off.
6. **If you can't confirm the data is complete**, say the figures are partial.
### Coverage of the requested range
When you compute a metric over a range (dates, hours, partitions, regions):
1. Check which values in the range have data, separately for each entity being compared.
2. In the answer, state for each entity whether its data covers the whole range, or name the
missing values.
3. Never present an aggregate as covering the full range when part of it has no data.
4. If the entities cover different subsets, say so, because it affects whether the comparison is
fair, and consider also giving the metric over the overlap they share.
### Scope verification for failures
When asked to confirm from logs whether a failure belongs to a particular entity (store,
account, tenant, resource), do not filter only on that entity. Get the wider context too:
1. **Run an aggregated query**, such as `stats count by entity, sub-ID, and outcome`, across
the same operation and time window. This shows whether the same error also hits other
entities, so you can say whether it is specific to this entity or happening everywhere.
2. **For the target entity, list every unit of work in the relevant run**, both successes and
failures, so you can see the full scope of the batch. Pay particular attention to the
operation that matters for the root cause.
3. **If a query returns too much output**, narrow it with aggregation, fields/parse, or
smaller time windows. Do not add filters that would hide the comparison you need.
4. **In the final answer**, state whether the failure is unique to the entity or shared, and
roughly how widely it is shared. Say which related operations succeeded. Explicitly correct
any earlier claims in the thread that the logs contradict.
5. **When asked why one record is failed or in an exception state**, widen the analysis to the
whole population the root cause touches: every category or rule pairing involved, every
entity, account or ID the user listed, exact totals per category and per side, how many of the
listed entities are affected and which are not, and the largest entity in each category.
## Root-cause analysis
### Consistency checks
Before reporting a root cause, check that it fits every fact you have already found,
especially your own conclusions about how the system is designed (for example, whether work
is pushed or pulled):
1. **If something expected seems to be missing** (such as a poller, loop, handler or log
entry), do not treat the gap as proof of a defect. First ask whether the design you already
found explains why it is absent.
2. **Label any mechanism you could not confirm** from data, configuration or logs as an
unverified hypothesis, never as the root cause. In particular, never claim how a processing
engine behaves ("rule X is being applied") unless you have read the rule definition.
3. **If the evidence points elsewhere** (such as a resource whose state changed or a reference
that now points to a newer record), build the diagnosis on that.
### Missing features as root causes
Treat a missing feature as a root cause only after you confirm the component was meant to
have it:
1. **A missing feature can be** a missing handler, polling loop, subscriber, or log line.
Before calling its absence a defect, work out from the evidence which component owns that
responsibility.
2. **In distributed or cross-region systems**, check the data's target or destination fields
(such as target region, account, or consumer) to see whether a different service or region
does that work.
3. **If you can't confirm who owns it**, report the absence as an open question and name the
next system to check. Don't present it as the root cause.
### Bug reports and code defects
When investigating a bug report against code and logs, make sure your final verdict follows
from your evidence:
1. **If your trace shows that a computation gives a wrong or self-inconsistent result** (for
example, the same quantity is counted twice, or a valid request is rejected even though the
resource is available), classify it as a code defect.
2. **Do not call it "working as designed"**, and do not offer only workarounds such as process
changes, extra user steps, or better error messages.
3. **Name the specific faulty expression or line** and propose a concrete code correction that
would have made the reported case succeed. Workarounds can be listed as extra suggestions
after that.
4. **Take the intended behavior from the code and from the user's stated expectations**, not
from your own view of what the system should do. If a code path exists to handle a case
(such as a fallback, retry, or alternate branch), treat that case as supported unless you
have clear evidence otherwise.
### Recommendation quality
In root-cause reports:
1. **Do not say the case should be rejected**, or call the failure "working as designed," when
your own diagnosis shows a defect.
2. **Recommend only the minimal fix** that follows from the evidence, and do not add extra
policy changes on top of it.
3. **Remove or correct anything that contradicts** your main finding, the user's stated
expectation, or the purpose of the code.
4. **If the user points to a runbook or playbook**, go through every remediation it lists
(retry, rollback, scaling, backfill, escalation) and say for each whether it addresses the
confirmed cause and why, including the ones you reject. Name the options that only treat the
symptom, such as a retry for a deterministic limit or a code defect. Recommend the change that
fixes the cause.
### Answering the user's question
When a user asks a specific question or names a suspected cause (for example, "why isn't X
being applied?" or "is it enforced per A or per B?"):
1. **Answer that question**, and back the answer with evidence from the
system.
2. **Check the actual configuration or enforcement records**, such as limit or quota
settings, throttle or decision logs, and the keys or dimensions they record.
3. **Say plainly whether the configuration was in effect** during the incident window, what
scope it applied to, and what value was enforced.
4. **Rule out each competing explanation** the user or other stakeholders raised, using
evidence rather than assumption.
5. **Don't hedge or offer speculative alternatives** your evidence doesn't support (such as
limits reverting to defaults or being split across hosts or partitions).
6. **If you can't confirm something**, label it as unverified instead of implying it.
## Closing checklist
Run the closing checklist in the skill's `SKILL.md` before you report.
references/data-investigation.md
# Data investigation
Use this when the answer is in the account's data, logs or configuration: counts, breakdowns,
extracts, joins, or why records are failed or unmatched. Whether a finding is complete (pagination,
range coverage, the whole affected population) and whether a root cause holds is in
[completeness-and-root-cause.md](completeness-and-root-cause.md).
## Missing or ambiguous parameters
If a parameter is ambiguous or missing (end of a date range, number of iterations, grouping
convention), discover it from the live environment first: query the available partitions or the
latest dates in the data, list existing resources, and choose the most natural default (for
example consecutive fixed-size windows from the stated start date up to the latest available
data). State the choice in the answer.
## Finding the data source
If a request names data, fields or values without saying where they are stored, find the source
before doing anything else:
1. List likely stores in the account and region: DynamoDB tables, S3 buckets and prefixes,
Glue and Athena databases and tables, RDS and Redshift, OpenSearch, Bedrock knowledge bases.
2. Look for names that match the request's terms.
3. Sample items, describe schemas or scan with filters to confirm which attributes hold the values
the user mentioned.
## Configuration plus data: failed, unmatched or exception records
When asked why records are in an exception, unmatched or failed state, investigate the
configuration as well as the data:
1. **Find the rules first.** Search every reachable store: S3 configuration or rule objects outside
the data prefixes, DynamoDB tables, SSM parameters, Secrets Manager and AppConfig, Lambda, Step
Functions and Glue job definitions and environment variables, and rule tables in the catalog.
2. For each category and subcategory involved, check whether a matching or processing rule exists
and is enabled. Report any category without a rule as a candidate root cause.
3. Break the largest contributors down by category and subcategory, outcome or status, and
secondary dimensions such as currency, with exact counts for each breakdown, not only totals.
## Counting
- **Exact aggregates.** Get numbers with aggregate queries (COUNT or SUM with GROUP BY) run in the
query engine. Do not pull raw rows and estimate, and never describe counts as approximate
patterns.
- **Alarms and requests.** Scoping counts to an alarm's dimensions and counting unique failed
requests are in [multi-service-logs.md](multi-service-logs.md); widening from one failed record
to the whole affected population is in
[completeness-and-root-cause.md](completeness-and-root-cause.md).
## Extracting and filtering records
1. **Split the conditions.** Break the request into separate filters (reason or category, current
state, subset of a reference list).
2. **Map each to the source data.** List the distinct category values, read the registry's own
category definitions, and check how the current state is recorded (latest event per item,
reinstatement records).
3. **Use exactly the subsets the user named,** and leave out neighbouring categories they did not
ask for.
4. **Re-derive, don't copy.** Do not copy filters or row sets from earlier output files without
deriving them again from the source.
5. **Verify before delivering.** Confirm that a sample of included rows meets every condition, look
for rows wrongly left out (case or spacing differences, items with several records), and report
the count after each filtering step.
## Joining tables for complete rows
When the user names one table as the authority for which records count:
1. Use that table to decide which rows are included.
2. If it lacks a requested field, inspect the schemas of related tables, find the shared keys
(IDs, source identifiers, batch or group IDs, array columns that need unnesting) and join to get
it.
3. Never output "N/A" or "could not be derived" for a requested field until you have tried those
joins.
4. Do not collapse rows that share a key: if several entities map to one identifier, list each.
5. Copy identifier values from query results instead of retyping them, and check that every row has
every requested field before writing the answer.
## Output
- **Literal values.** When the user asks for records, logs or query results rather than an
analysis, give every value (timestamps, identifiers, URLs, codes, headers) exactly as the source
produced it, never as categories, prefixes, inferred buckets or paraphrases. If the output is too
large, write the complete raw rows to a CSV or JSON file and point to it. Collapse only rows that
are identical on every selected field, showing the full values and a count. Summaries come in
addition to the raw values, never instead of them.
- **Exact error evidence.** Quote the status code and the error code and message from the failing
response payload word for word, not a summary such as "HTTP 500".
- **Every requested artifact.** Produce every file or plan asked for (Excel workbooks, CSV files,
EXPLAIN plans), run every step, and assume any role or credentials you were told to use.
references/deployed-artifacts.md
# Read the deployed artifact, don't infer it
A question about what a system *currently does* is answered only by the artifact that governs
runtime. Anything else (the newest stored version, a draft, a tag, a description field, a parameter
spec, or your own paraphrase of the code's query) is a **proxy**. Proxies are allowed as leads,
never as evidence.
## Core procedure
1. **Name the governing artifact before fetching anything.** Ask what the runtime actually
consults. Typical answers:
- the version a *deployment record* says is live, not the latest stored version;
- the configuration *served to clients*, not the hosted store;
- the code that *executes* for the question (the implementing function or executor), not the
card, schema, wiring or docstring that references it;
- the packaged code attached to the live version or alias, not the repository HEAD.
2. **Retrieve that artifact and record where it came from.** Keep a provenance note per claim:
`claim <- artifact (identifier, version, timestamp)`.
3. **Check the gap between proxy and artifact explicitly.** If you also saw a newer version, a
description or a spec, state whether it is live, pending or only descriptive.
4. **Reproduce queries verbatim, not from intuition.** When code filters, sorts or limits data and
you want to check live data, copy the exact predicate set, ordering and limit from the code.
Reconcile any difference before concluding.
5. **When a route is blocked, try the alternatives before concluding.** Only after every route fails
may you report the claim, and then as *unverified*.
6. **Answer with staged and live separated.** Never let an undeployed version, a draft or a
descriptive field appear as live behaviour.
## Proxy traps and the artifact that replaces them
| Proxy you reached first | Why it is not evidence | Artifact to read instead |
|---|---|---|
| Latest hosted or stored configuration version | Creating a version does not deploy it; deployment is a separate step per environment | The environment's deployment history: the `ConfigurationVersion` of the most recent `COMPLETE` deployment, or the configuration actually served to clients |
| Repository HEAD, `$LATEST` or branch code | Clients may invoke a published version or alias that points elsewhere | The package attached to the version or alias actually invoked |
| Source template in a repository | The running stack may differ | The template the stack is running (`GetTemplate` on the stack) |
| Description, tag, label, schema field, card or wiring definition | Descriptive unless something enforces it | The implementing code that reads and acts on the value |
| Your own rewritten query ("status = X") | Narrower or wider than the code's predicate; a different order and limit change the "first result" and the counts | The code's exact filter set, sort direction and limit |
| One failed data route plus an empty log search | Absence of evidence from one route is not evidence of absence | Other routes (below), or label the claim as unverified |
## Reproducing a code path's query
- Extract from the code every status or state value in the `IN` set, every additional predicate
(tenant, date range, soft-delete flags), the `ORDER BY` column and direction, and the `LIMIT` or
page size.
- Run exactly that. Do not substitute the "obvious" status, drop the limit or change ascending to
descending.
- If your count or first row differs from what the code would observe, explain the difference
before building a root cause on it. A customer-specific root cause built on the wrong first row is
a wrong root cause.
- Say "N rows matched the code's query" only after running the code's query; otherwise say "my
approximation returned N".
## When the first route is blocked
Try these, roughly in order, and record what each returned:
1. Deployment or release records (which version is live, since when, in what state).
2. Configuration as served to clients (the data-plane read path, not the control-plane store).
3. The running stack's template or the packaged artifact of the live version.
4. Alternative data sources (replicas, exports, metrics, cached responses).
5. Broader log windows, other log groups, and structured fields instead of free text.
If all fail, mark each dependent conclusion **UNVERIFIED** with the routes tried. Do not turn "I
could not see it" into "it is silently failing".
## Answer template
```text
Live behaviour (read from <artifact + identifier>): ...
Staged / pending / undeployed (read from <artifact>): ... <- NOT live
Parameter semantics (read from implementing code at <path/function>): ...
Query reproduced verbatim from <file:line>: <predicates, order, limit> -> N rows; first row = ...
Unverified: <claim> - routes tried: <list>
```
## AWS routes to the governing artifact
### AWS AppConfig: what is actually being served
Creating a hosted configuration version only uploads it; a separate deployment to an environment
makes it live. The newest version number in a profile says nothing about what clients receive.
Control plane, the deployment history of the environment:
- `ListDeployments(ApplicationId, EnvironmentId)` returns deployments in descending
`DeploymentNumber` order, each with `ConfigurationProfileId`, `ConfigurationVersion`, `State`,
`PercentageComplete`, `StartedAt` and `CompletedAt`.
- `GetDeployment(ApplicationId, EnvironmentId, DeploymentNumber)` adds `EventLog` and
`VersionLabel`.
- `State` is one of `BAKING`, `VALIDATING`, `DEPLOYING`, `COMPLETE`, `ROLLING_BACK`,
`ROLLED_BACK`, `REVERTED`.
Interpretation:
- The live version of a profile in an environment is the `ConfigurationVersion` of the most recent
deployment of that profile whose `State` is `COMPLETE`.
- `DEPLOYING`, `BAKING` or `VALIDATING` means a partial rollout: report the prior complete version
and the in-flight one, with `PercentageComplete`.
- `ROLLED_BACK` or `REVERTED` means the targeted version is not live; fall back to the previous
`COMPLETE` deployment.
- A profile with hosted versions but no `COMPLETE` deployment in the environment has nothing live
there; say so instead of reporting the newest version's content.
- Repeat per environment; one environment's history says nothing about another.
Data plane, the configuration as clients receive it: `StartConfigurationSession` then
`GetLatestConfiguration` returns the deployed configuration for the application, environment and
profile. It can return empty data when the caller already holds the latest version, so use a fresh
session.
### AWS Lambda: which code actually runs
- Clients usually invoke an **alias** that points to a published **version**, optionally weighted
across two versions through `RoutingConfig`. `$LATEST` and the repository HEAD may differ from
what the alias serves.
- `GetFunction(FunctionName, Qualifier=<version or alias>)` returns that version's configuration
and `Code.Location`, a presigned URL (valid 10 minutes) for the deployment package, or
`Code.ImageUri` and `ResolvedImageUri` for container images. Read the package behind the invoked
qualifier, not the unqualified function.
- `Configuration.CodeSha256` tells you whether two versions, or a version and a build, are the same
code.
### AWS CloudFormation: what the stack is really running
`GetTemplate(StackName, TemplateStage=Original|Processed)` returns the template the running stack
was created or updated with; `Processed`, the default, is the post-transform form. It works for
running stacks and, by stack ID, for deleted stacks up to 90 days. Use it instead of a repository
template when you need the deployed definition (inline function code, parameter wiring, resource
properties).
### Descriptive fields that are not enforced
`Description`, `VersionLabel`, tags and similar fields on AppConfig deployments and hosted versions,
Lambda versions and aliases, and CloudFormation resources are user-authored metadata. They show
intent, not behaviour. Cite them only as "labelled as ...", and read the implementing code or the
deployment record for the actual meaning.
references/multi-service-logs.md
# Multi-service logs investigation
Use this when a request, alarm or incident spans several services or a pipeline of components.
## 1. Read every service's own logs
Find the log group of every service the request passes through: the caller, the service you were
asked about, and each downstream service. Query each one around the timestamp or by the request or
correlation ID, and take each service's outcome from its own logs, never from another service's.
## 2. Report fields exactly, per layer
When similar fields appear at different layers (request-context parameters and token claims,
request scopes and OAuth scopes, a service's own authorization result and its caller's), report
each one separately with its source, verbatim, including `null`, empty or missing values.
## 3. Count unique failed requests within the alarm's scope
1. Read the alarm's dimensions (operation, caller, endpoint, stage) and its breach window from the
alarm definition and history.
2. Keep only log records that match all dimensions and fall inside the window.
3. Group them by request or correlation ID and follow each ID to its final outcome; a request that
succeeded on a retry is not a failure.
4. Report the number of unique failed requests separately from the number of raw error records,
say that repeated records of one ID are retry attempts, and list the qualifying IDs.
## 4. Place the failure boundary per event
For each event, decide separately where it was lost: it never reached the service (upstream loss),
or the service received it and handed it off but the customer saw no result (downstream loss). If
the events differ, report both boundaries and both escalation paths. Never rule out the downstream
team when your logs show a successful handoff without a customer result.
SKILL.md
---
name: aws-investigation
description: >-
Investigates what is happening in an AWS account from evidence already recorded there. Not for
capturing new data from running code (snapshots of function arguments on a live service,
Dynamic Instrumentation), questions answered from the CloudWatch Omni context graph or
Application Signals service map, or building or changing infrastructure. Covers reading
deployment records instead of the newest stored version (completed AppConfig deployments, the
version an alias points to, a stack's current template), following a request through every
service's own logs (exact per-layer fields, unique failed requests in an alarm's breach window,
failure boundary per event), and answering questions from the account's data (exact counts and
breakdowns, coverage of the requested range, exact filters and joins, the rule behind failed or
unmatched records, verbatim values). Use to triage an alarm, error spike or failed request, trace
a request across services, or count, break down, extract or join records.
metadata:
version: "1"
---
# AWS Investigation
An investigation answers a question about what an AWS account is doing, and every claim in the
answer must come from something you read: a deployed artifact, a service's own log record, or a
query result. Inference from names, descriptions, the newest stored version or one service's view
of another is a lead, never evidence.
## Pick the reference
Read the reference that matches the question before you start; most investigations need more than
one.
| The question is about | Read |
|---|---|
| What a deployed system currently does, serves or executes: live configuration, feature flags, parameters, which code runs, reproducing a code path's query, or a first data route that is blocked | [references/deployed-artifacts.md](references/deployed-artifacts.md) |
| A request, alarm or incident that spans several services: tracing one request, counting failed requests, deciding where an event was lost | [references/multi-service-logs.md](references/multi-service-logs.md) |
| Facts in the account's data: finding where data lives, exact counts and breakdowns, filtering, joining or exporting records, the rules behind failed or unmatched records | [references/data-investigation.md](references/data-investigation.md) |
| Whether a finding is complete and a root cause holds: searching every region and scope, paginating before counting, coverage of a date range, one entity vs the whole affected population, consistency of the root cause, whether a runbook fixes the cause, answering the user's own question | [references/completeness-and-root-cause.md](references/completeness-and-root-cause.md) |
## Rules for every investigation
1. **Explore before you give up.** Assume the credentials are the intended environment. Treat
unfamiliar or internal terms (service nicknames, airport-code regions such as PDX = us-west-2,
IAD = us-east-1) as search hints: list resources, log groups and data stores in the likely
regions and search for the identifiers in the request before saying something cannot be done.
2. **Separate evidence from inference.** Keep a short provenance note per claim
(`claim <- artifact, record or query`). Label anything you did not read as a hypothesis or
**UNVERIFIED**, with the routes you tried.
3. **Count exactly.** Use aggregate queries run in the engine, scoped to exactly what was asked, and
report unique entities separately from raw records.
4. **Quote, don't paraphrase.** Give identifiers, field values, error codes and messages exactly as
the source produced them, including `null` and empty values.
5. **Finish the task.** Produce every requested artifact (files, tables, plans), and state the
assumptions you made in the final answer.
## Closing checklist
Before submitting your analysis:
1. Have you searched all relevant regions and scopes?
2. Have you paginated through all results, and said which part of the requested range has data?
3. Have you verified the scope of failures (entity-specific vs fleet-wide, the whole affected
population)?
4. Does your root cause fit all the facts you found?
5. Have you labelled every mechanism you could not confirm as unverified, and reported a
missing component as a defect only if it was meant to exist?
6. Did your claims about live configuration come from the deployed artifact, not the latest
stored version ([deployed-artifacts.md](references/deployed-artifacts.md))?
7. For a request that crosses services, did you read each service's own logs and place the
failure boundary per event ([multi-service-logs.md](references/multi-service-logs.md))?
8. Have you answered the user's specific question directly?
9. Before you finish, have you reread your conclusion against each piece of evidence you
collected and removed any statement that contradicts it?
10. Have you removed any contradictions between your findings and recommendations, and checked
each runbook option against the confirmed cause?