references/architectural-patterns.md
# Trace Design Patterns for Non-Request/Response Architectures
Most tracing models fit request/response architectures well: a client sends a request,
the service does work, and the trace captures the full waterfall. The further your
architecture drifts from request/response, the more intentional your trace design
needs to be.
This reference covers patterns for streaming, async jobs, long-running ETL, and
serverless — drawn from Chapter 7 of *Observability Engineering* (2nd edition).
## Streaming (Kafka / Queues / Message Buses)
Stream processing is the second most common architecture after request/response.
Tracing it is tricky because:
1. The smallest instrumentable "unit of work" isn't always the most useful unit to optimize
2. The questions you want to answer often involve trace *shape* (stage dwell times,
success/failure ratios) rather than individual spans
### Context Propagation
Serialize trace context into the message envelope when publishing. Extract it when
consuming. For Kafka and similar systems, message envelopes make this straightforward.
```
Producer:
orders-api creates span "messaging.publish" (kind: PRODUCER)
→ attributes: messaging.system, messaging.destination, message.id, order.id, customer.id
→ context serialized into message envelope of OrderPlaced event
Consumer:
payment-svc extracts context from envelope
→ creates span "messaging.process" (kind: CONSUMER)
→ linked to producer span via Span Links (not as direct child)
→ when producing new messages, repeats the cycle
```
### Correlation IDs
Store a unique identifier (e.g., `order.transaction_id`) and propagate it through the
entire pipeline. This ties all traces together directly without walking the tree in reverse.
This shifts from one trace with a single root and many children into a **timeline of
traces** linked by a shared correlation ID. You can pull all relevant spans into a
result set by filtering on the correlation ID.
### Fleet-Level Alerting with Metrics
Span-level tracing provides excellent visibility into a single message's journey, but
generalizing across the fleet requires metrics:
- **Counters per stage**: Emit success/failure counts with attributes mirroring span data
(but at lower cardinality — no transaction IDs on metric points)
- **Histograms for dwell time**: Track time from publish to process per stage/type/region
- **Exemplars**: Annotate histogram data points with trace and span IDs to link outlier
metrics directly to the traces that produced them
- **Alert on ratios**: `failed_total / processed_total` per stage, with histograms for
proactive alerting on unusual queue behavior
### Example Pipeline
```
orders-api (produces OrderPlaced)
→ topic: orders.events
├─ payment-svc (consumes OrderPlaced → produces PaymentAuthorized/Declined)
├─ inventory-svc (consumes PaymentAuthorized → produces InventoryReserved)
├─ shipping-svc (consumes InventoryReserved → produces ShipmentCreated)
└─ reconcile-job (nightly batch over events store)
```
Each stage: extract context → create consumer span → link to producer → add business
attributes → produce new messages with propagated context.
## Async Jobs (Fan-Out / Fan-In)
For scatter-gather patterns like video processing or parallel task execution.
### Pattern
```
POST /render → orchestrator (ROOT span)
├─ publish task.transcode ─→ worker processes
├─ publish task.thumbnail ─→ worker processes
└─ publish task.caption ─→ worker processes
workers process → orchestrator joins when N succeed or deadline
```
The root span is the entrypoint (the POST). Child processes link back via parent/child
or producer/consumer relationships. The root span should contain **links to each child
process** and a **final summarization span** that rolls up results.
### Avoiding the "Million Children" Hazard
Fan-out patterns can produce traces with enormous numbers of children, making them
expensive to store and unusable to visualize. Strategies:
- Use Span Links instead of parent/child for loosely coupled work
- Add summarization attributes on the root span (`tasks.total`, `tasks.succeeded`,
`tasks.failed`, `tasks.duration_ms`)
- Alert on the entrypoint span or specific child tasks depending on ownership
### Alerting
Metrics may be less useful than spans here because there's no shared infrastructure to
optimize beyond network links. Focus alerting on the orchestrator span.
## Long-Running Jobs (ETL / Batch Processing)
Jobs that run for hours or days (e.g., nightly ETL compiling inventory from thousands
of suppliers) don't fit the traditional trace model.
### The Problem
A single root span lasting hours creates issues:
- Most tracing backends don't handle updates to existing spans
- A trace with thousands of child spans per stage is expensive and hard to visualize
- The "million children trace" hazard applies here too
### Recommended Pattern: Separate Traces Per Stage
Model each stage as its own trace, related by links and correlation identifiers:
```
scheduler → [job.run nightly_etl] (correlation: job_id=abc123)
├─ etl.extract (own trace, linked by job_id)
├─ etl.transform (own trace, linked by job_id, may run for hours)
├─ etl.load (own trace, linked by job_id)
└─ report.write (own trace, linked by job_id)
```
Correlate all stages by querying on the shared `job_id`.
### Alternative: Wide-Event Summarization
If your primary goal is a safety net rather than fine-grained optimization, emit a
single **summarization span** (or log event) at job completion with rich metadata:
- Hundreds of attributes capturing what succeeded, what failed, and why
- Stage durations, record counts, error summaries
- Useful for tracking changes over time without the overhead of a well-formed trace
### Intermediate Visibility
Use metrics or logs for intermediate status reports you can monitor while the job runs,
without breaking the span/trace data model. As long as trace context is preserved through
child processes, metrics associate back to their underlying traces.
### Which Approach to Choose
Ask yourself: are you checking a box for requirements, or trying to understand and
optimize individual job performance?
- **Safety net**: Wide-event summarization approach
- **Performance optimization**: Separate traces per stage with links
- **Both**: Combine both approaches — separate traces for stages plus a summarization
event at completion
## Serverless (Lambda / Cloud Functions)
Serverless constraints change the tradeoffs for instrumentation.
### Favor Custom Over Auto-Instrumentation
Resource constraints in serverless environments mean auto-instrumentation's overhead
(cold-start time, memory) may not be worth the generic spans it produces. A single
well-designed span per invocation, rich with attributes, often beats a sprawling hierarchy.
### Design Pattern
```
Lambda invocation → single span with rich attributes:
- function.name, function.version
- trigger.type (API Gateway, SQS, S3, etc.)
- invocation.id (for post-hoc correlation)
- Business context attributes
- Duration breakdowns as timing attributes
- Error details with exception.slug
```
### Stateless Tracing Strategy (AWS-Specific)
An advanced pattern for performance-critical Lambda functions:
1. Emit lightweight events from the function runtime
2. A stateful OpenTelemetry Collector (in an extension layer) assembles events into spans
3. Reduces statefulness requirements in the function, minimizing cold-start impact
This trades simpler function code for more complex Collector-side logic.
### Cross-Invocation Correlation
Different serverless platforms offer different hooks for post-execution telemetry, often
not in native OTel formats. Correlate data post-hoc through:
- A shared invocation ID or UUID
- Telemetry pipelines that normalize and forward events together
- Federated queries across multiple data stores
### Key Principles
- **Minimize cold-start impact**: Keep instrumentation lightweight in the function itself
- **Rich attributes over deep span trees**: One wide span beats ten narrow ones
- **Timing attributes**: Track sub-operation durations as attributes, not child spans
- **Correlation IDs**: Essential for connecting invocations across event-driven chains
references/collector-config.md
# OpenTelemetry Collector Configuration
The OTel Collector receives, processes, and exports telemetry data. Use it between
your application and Honeycomb for format conversion, processing, sampling, and routing.
## When to Use a Collector
- Converting from other formats (Zipkin, Jaeger, OpenTracing) to OTLP
- Adding common attributes across all services (e.g., deployment info)
- Tail sampling at the infrastructure level
- Routing to multiple backends
- Gateway/proxy designation for Service Map
## Basic Honeycomb Configuration
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
timeout: 1s
send_batch_size: 1024
exporters:
otlp/honeycomb:
endpoint: "api.honeycomb.io:443"
headers:
x-honeycomb-team: "${HONEYCOMB_API_KEY}"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/honeycomb]
```
## Adding Attributes via Processor
Tag all spans with deployment info:
```yaml
processors:
attributes:
actions:
- key: deployment.environment
value: "production"
action: upsert
- key: net.component
value: "proxy"
action: upsert # For gateway/proxy services in Service Map
```
## Tail Sampling Configuration
Sample based on trace characteristics (requires seeing complete traces):
```yaml
processors:
tail_sampling:
decision_wait: 10s
num_traces: 100000
policies:
# Always keep error traces
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
# Always keep slow traces
- name: slow-traces
type: latency
latency:
threshold_ms: 5000
# Sample 10% of everything else
- name: probabilistic
type: probabilistic
probabilistic:
sampling_percentage: 10
```
For production tail sampling, consider Honeycomb's **Refinery** which is purpose-built
for trace-aware sampling decisions and integrates natively with Honeycomb.
## Converting from Other Formats
### Zipkin to Honeycomb
```yaml
receivers:
zipkin:
endpoint: "0.0.0.0:9411"
```
### Jaeger to Honeycomb
```yaml
receivers:
jaeger:
protocols:
grpc:
endpoint: "0.0.0.0:14250"
thrift_http:
endpoint: "0.0.0.0:14268"
```
## Gateway for Service Map
To show a proxy/gateway in Honeycomb's Service Map, add `net.component: proxy`:
```yaml
processors:
attributes:
actions:
- key: net.component
value: "proxy"
action: upsert
```
references/custom-instrumentation.md
# Custom Instrumentation Patterns
Detailed patterns for adding custom instrumentation beyond auto-instrumentation.
## When to Add Custom Instrumentation
Auto-instrumentation covers:
- HTTP server/client requests
- Database queries
- gRPC calls
- Message queue operations
Add custom instrumentation for:
- Business logic (checkout flow, payment processing)
- Cache operations
- Internal function calls that matter
- Custom attributes with business context
## Pattern: Adding Context to Auto-Instrumented Spans
The most impactful custom instrumentation. No new spans needed — just add
attributes to existing spans.
### Go
```go
func handleCheckout(w http.ResponseWriter, r *http.Request) {
span := trace.SpanFromContext(r.Context())
span.SetAttributes(
attribute.String("user.id", getUserID(r)),
attribute.Float64("cart.total", cart.Total()),
attribute.Int("cart.items", cart.ItemCount()),
attribute.String("payment.method", cart.PaymentMethod()),
)
// ... rest of handler
}
```
### Python
```python
@app.route("/checkout", methods=["POST"])
def handle_checkout():
span = trace.get_current_span()
span.set_attribute("user.id", get_user_id())
span.set_attribute("cart.total", cart.total)
span.set_attribute("cart.items", cart.item_count)
span.set_attribute("payment.method", cart.payment_method)
# ... rest of handler
```
### Node.js
```javascript
app.post("/checkout", (req, res) => {
const span = trace.getActiveSpan();
span.setAttribute("user.id", req.user.id);
span.setAttribute("cart.total", cart.total);
span.setAttribute("cart.items", cart.itemCount);
span.setAttribute("payment.method", cart.paymentMethod);
// ... rest of handler
});
```
## Pattern: Wrapping Business Logic in Custom Spans
Create spans around operations you want to see in the trace waterfall.
> The `RecordError`/`record_exception` calls in the compatibility snippets below show the
> legacy span-event exception API. For new code, prefer the **Exception Events with the Logs API**
> pattern below; retain the legacy call only when required by an existing SDK, query, or migration.
### Go
```go
func processPayment(ctx context.Context, order *Order) error {
tracer := otel.Tracer("checkout-service")
ctx, span := tracer.Start(ctx, "process-payment")
defer span.End()
span.SetAttributes(
attribute.String("order.id", order.ID),
attribute.Float64("order.total", order.Total),
attribute.String("payment.provider", order.PaymentProvider),
)
result, err := paymentGateway.Charge(ctx, order)
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return err
}
span.SetAttributes(attribute.String("payment.transaction_id", result.TransactionID))
return nil
}
```
### Python
```python
def process_payment(order):
tracer = trace.get_tracer("checkout-service")
with tracer.start_as_current_span("process-payment") as span:
span.set_attribute("order.id", order.id)
span.set_attribute("order.total", order.total)
span.set_attribute("payment.provider", order.payment_provider)
try:
result = payment_gateway.charge(order)
span.set_attribute("payment.transaction_id", result.transaction_id)
except Exception as e:
span.record_exception(e)
span.set_status(StatusCode.ERROR, str(e))
raise
```
## Pattern: Exception Events with the Logs API
For new exception instrumentation, emit a Logs API record while the relevant span is active.
Use the standard `exception.*` fields, ERROR severity, and `event.name="exception"`. Set span
status and low-cardinality span dimensions separately so humans can aggregate operation failures
without copying stack traces onto spans.
```text
log event (while span is active):
event.name = "exception"
body = "exception"
severity = ERROR
exception.type, exception.message, exception.stacktrace, exception.escaped
containing span:
status = ERROR
error = true
exception.slug = "err-payment-provider-timeout" # static, optional, low-cardinality
```
Honeycomb correlates the log to the active trace and renders it as a `span_event` annotation.
The event row carries `trace.trace_id` and `trace.parent_id`, but Logs API exception fields are
not hoisted onto the containing span. Query the event row for the diagnostic payload, then use
its `trace.trace_id` with `get_trace(show_events=true)` to inspect the surrounding trace.
The legacy `record_exception` / `RecordError` APIs remain useful for compatibility and existing
instrumentation. Do not assume their `name=exception` shape applies to Logs API events: Logs API
uses `event.name=exception` (and often `body=exception`) with `meta.signal_type=log`.
**SDK implementation note:** use the language's supported Logs API or logging bridge, and verify
that the record carries the active context. Python's current app-facing path is the stdlib
`logging` bridge; Go passes the active context explicitly to `logger.Emit(ctx, record)`; Node
requires an installed context manager and an explicit active context; Java attaches context from
`span.makeCurrent()`. Do not emit the exception after the span scope closes.
### Optional compatibility pattern: exception-promoting LogRecordProcessor
Honeycomb's historical span-event path can promote exception fields onto the parent span. A
trace-correlated Logs API exception does not receive that promotion automatically. If an existing
span-oriented query surface must be preserved, implement a custom **LogRecordProcessor**, registered
before the batch/export processor:
```text
on_emit(log_record, resolved_context):
if log_record.event_name != "exception":
return
span = span_from_context(resolved_context)
if span is absent or not recording:
return
# Promote only fields needed by span-level queries.
span.set_attribute("error", true)
span.set_attribute("error.type", log_record["exception.type"])
span.set_attribute("exception.type", log_record["exception.type"])
if log_record has "exception.slug":
span.set_attribute("exception.slug", log_record["exception.slug"])
if configured for legacy compatibility:
span.set_attribute("exception.message", log_record["exception.message"])
span.set_attribute("exception.stacktrace", log_record["exception.stacktrace"])
# Leave the log record unchanged; it remains the diagnostic source of truth.
```
The processor should be synchronous and must run while the span is still mutable. It should no-op
when there is no valid recording span, preserve the original log record, and never invent fields
that were not emitted. Treat `exception.message` and especially `exception.stacktrace` promotion
as an explicit compatibility option because they are high-cardinality and potentially large.
This is not a standalone `SpanProcessor`: span processors do not receive log records. A span
processor could only participate through a separate shared registry, which adds races, cleanup,
and lifecycle complexity. The exact processor and context APIs are language-specific:
- Go: use the explicit context passed to `logger.Emit(ctx, record)`.
- Java: use the resolved log context; do not reconstruct context after async queueing.
- Node.js: ensure the context manager is installed and use the record's explicit context.
- Python: preserve the active context through the stdlib logging bridge and Logs SDK processor.
Use this only as a migration aid. Agents should still query Logs API exception rows for full
`exception.*` diagnostics and treat promoted span fields as instrumentation-dependent.
## Pattern: Recording Events Within a Span
For non-exception milestones and state changes, use the Logs API for new point-in-time events
when the language SDK supports it. Keep legacy span events where the SDK has no usable Logs API
or compatibility requires them:
```python
with tracer.start_as_current_span("process-order") as span:
span.add_event("validating_order", {"order.id": order.id})
if not validate(order):
span.add_event("validation_failed", {"reason": "invalid_address"})
raise ValidationError()
span.add_event("charging_payment", {"amount": order.total})
charge(order)
span.add_event("order_completed", {"order.id": order.id})
```
## Pattern: Linking Related Traces
When an async job is triggered by a request, link them:
```python
# In the message consumer:
from opentelemetry.trace import Link
def process_message(message):
# Extract the producing span's context from the message
producer_context = extract_context(message.headers)
with tracer.start_as_current_span(
"process-message",
links=[Link(producer_context, {"link.reason": "triggered_by"})],
) as span:
span.set_attribute("message.id", message.id)
# ... process message
```
## Pattern: Timing Attributes on Parent Spans
Put important sub-operation durations as attributes on the parent span instead of
creating child spans for everything.
> **Anti-pattern warning:** Wrapping absolutely everything in its own span is the most
> common failure mode when engineers first get access to tracing tools. You have to
> design the structure of your data for the way you want to query it.
Child spans are helpful for waterfall visualization of a single request, but they're
difficult to query across *all* requests. Timing attributes on a single span are
easier to query and work directly with tools like BubbleUp — which can immediately
surface "that group of requests was slow because authentication took 10 seconds."
### Go
```go
func handleRequest(w http.ResponseWriter, r *http.Request) {
span := trace.SpanFromContext(r.Context())
// Time authentication
authStart := time.Now()
user, err := authenticate(r)
authDur := time.Since(authStart)
span.SetAttributes(attribute.Float64("auth.duration_ms", float64(authDur.Milliseconds())))
// Time payload parsing
parseStart := time.Now()
payload, err := parsePayload(r)
parseDur := time.Since(parseStart)
span.SetAttributes(attribute.Float64("payload_parse.duration_ms", float64(parseDur.Milliseconds())))
// ... rest of handler
}
```
### Python
```python
@app.route("/api/resource", methods=["POST"])
def handle_request():
span = trace.get_current_span()
# Time authentication
auth_start = time.monotonic()
user = authenticate(request)
span.set_attribute("auth.duration_ms", (time.monotonic() - auth_start) * 1000)
# Time payload parsing
parse_start = time.monotonic()
payload = parse_payload(request)
span.set_attribute("payload_parse.duration_ms", (time.monotonic() - parse_start) * 1000)
# ... rest of handler
```
### Node.js
```javascript
app.post("/api/resource", async (req, res) => {
const span = trace.getActiveSpan();
// Time authentication
const authStart = performance.now();
const user = await authenticate(req);
span.setAttribute("auth.duration_ms", performance.now() - authStart);
// Time payload parsing
const parseStart = performance.now();
const payload = await parsePayload(req);
span.setAttribute("payload_parse.duration_ms", performance.now() - parseStart);
// ... rest of handler
});
```
**When to use this pattern:**
- The operation is important to understanding request latency
- You want to GROUP BY or BubbleUp on the timing alongside other parent span attributes
- The alternative (a child span) would require JOINs for cross-request analysis
**When a child span is still better:**
- The operation makes downstream calls you also want to trace
- You need to see the operation in the waterfall view for single-request debugging
- The operation has its own rich set of attributes worth capturing
## Pattern: Exception Slugs
Tag each error throw site with a unique static string (`exception.slug`). This creates
a low-cardinality, greppable identifier that connects dashboards directly to code. Keep this
attribute on the operation span even when full exception details are emitted as a Logs API event.
The legacy `RecordError`/`record_exception` calls shown in this section are compatibility examples;
they are not a requirement for new Logs API instrumentation.
### Go
```go
func processPayment(ctx context.Context, order *Order) error {
span := trace.SpanFromContext(ctx)
result, err := stripe.Charge(ctx, order)
if err != nil {
// Static string — not dynamically generated
// Consider enforcing this with custom lint rules
span.SetAttributes(
attribute.String("exception.slug", "err-stripe-charge-failed"),
attribute.Bool("error", true),
)
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return err
}
if !result.Approved {
span.SetAttributes(
attribute.String("exception.slug", "err-payment-declined"),
attribute.Bool("error", true),
)
return ErrPaymentDeclined
}
return nil
}
```
### Python
```python
def process_payment(order):
span = trace.get_current_span()
try:
result = stripe.charge(order)
except stripe.CardError as e:
span.set_attribute("exception.slug", "err-stripe-card-error")
span.set_attribute("error", True)
span.record_exception(e)
span.set_status(StatusCode.ERROR, str(e))
raise
except stripe.APIError as e:
span.set_attribute("exception.slug", "err-stripe-api-unavailable")
span.set_attribute("error", True)
span.record_exception(e)
span.set_status(StatusCode.ERROR, str(e))
raise
```
### Node.js
```javascript
async function processPayment(order) {
const span = trace.getActiveSpan();
try {
const result = await stripe.charges.create(order);
if (!result.approved) {
span.setAttribute("exception.slug", "err-payment-declined");
span.setAttribute("error", true);
throw new PaymentDeclinedError();
}
} catch (err) {
if (!span.attributes?.["exception.slug"]) {
span.setAttribute("exception.slug", "err-stripe-call-failed");
}
span.setAttribute("error", true);
span.recordException(err);
span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
throw err;
}
}
```
**Why this pattern matters:**
- **Greppable:** Search your codebase for the exact slug string to find the throw site
- **Low-cardinality GROUP BY:** Safe to use in `GROUP BY exception.slug` queries
- **Gap detection:** Any failed request *without* an `exception.slug` reveals places
where your error handling could be improved — it's easy to find errors you didn't
anticipate
**Query — find unhandled errors (missing slugs):**
```
VISUALIZE COUNT
WHERE error = true AND exception.slug = NULL
GROUP BY http.route
```
## Pattern: Async Request Summaries
Roll up child operation statistics onto the parent span to identify outlier requests
without needing to count child spans manually.
### Go
```go
type RequestStats struct {
mu sync.Mutex
pgQueryCount int
pgQueryDurMs float64
httpReqCount int
httpReqDurMs float64
}
func (s *RequestStats) RecordPgQuery(dur time.Duration) {
s.mu.Lock()
defer s.mu.Unlock()
s.pgQueryCount++
s.pgQueryDurMs += float64(dur.Milliseconds())
}
func (s *RequestStats) RecordHTTPReq(dur time.Duration) {
s.mu.Lock()
defer s.mu.Unlock()
s.httpReqCount++
s.httpReqDurMs += float64(dur.Milliseconds())
}
func (s *RequestStats) SetOnSpan(span trace.Span) {
s.mu.Lock()
defer s.mu.Unlock()
span.SetAttributes(
attribute.Int("stats.postgres_query_count", s.pgQueryCount),
attribute.Float64("stats.postgres_query_duration_ms", s.pgQueryDurMs),
attribute.Int("stats.http_requests_count", s.httpReqCount),
attribute.Float64("stats.http_requests_duration_ms", s.httpReqDurMs),
)
}
```
### Python
```python
class RequestStats:
def __init__(self):
self.pg_query_count = 0
self.pg_query_duration_ms = 0.0
self.http_req_count = 0
self.http_req_duration_ms = 0.0
def record_pg_query(self, duration_ms):
self.pg_query_count += 1
self.pg_query_duration_ms += duration_ms
def record_http_request(self, duration_ms):
self.http_req_count += 1
self.http_req_duration_ms += duration_ms
def set_on_span(self, span):
span.set_attribute("stats.postgres_query_count", self.pg_query_count)
span.set_attribute("stats.postgres_query_duration_ms", self.pg_query_duration_ms)
span.set_attribute("stats.http_requests_count", self.http_req_count)
span.set_attribute("stats.http_requests_duration_ms", self.http_req_duration_ms)
# Usage in a request handler:
@app.route("/api/resource")
def handle():
stats = RequestStats()
# Pass stats to DB and HTTP client wrappers...
# At end of request:
stats.set_on_span(trace.get_current_span())
```
**Why this pattern matters:**
- A request that makes 742 database queries is almost certainly doing something wrong
- Without summary stats, these outliers are invisible — you'd need to count child spans
per trace manually
- HEATMAP of `stats.postgres_query_count` instantly reveals bimodal distributions
and outliers
**Query — database queries per request:**
```
VISUALIZE HEATMAP(stats.postgres_query_count)
WHERE service.name = "api-service"
```
## Attribute Naming Best Practices
- Use dot-separated namespaces: `user.id`, `order.total`, `cache.hit`
- Follow OTel semantic conventions where they exist
- Create your own namespace for custom attributes: `app.`, `mycompany.`
- Keep attribute values low-cardinality where possible (for GROUP BY)
- High-cardinality is fine for debugging (trace IDs, user IDs, order IDs)
references/lambda.md
# Lambda Instrumentation Patterns (Node.js / TypeScript)
Patterns for instrumenting AWS Lambda functions with OpenTelemetry. Lambda's
execution model introduces constraints that don't apply to long-running servers:
execution freezes when the handler promise resolves, cold starts are expensive,
and there are no HTTP headers to propagate context over direct Lambda invocations.
## Choosing an Approach: OTel Layer vs Manual SDK Setup
There are two fundamentally different ways to instrument a Lambda function. The
choice comes down to where you want to pay the latency cost.
| | AWS Managed OTel Layer | Manual SDK Setup |
| :--- | :--- | :--- |
| Cold start overhead | Higher (~2–5 s extra) | Lower |
| Per-request latency | None — export happens after response | Added — `forceFlush()` blocks the response |
| Code complexity | Less — layer handles SDK init and export | More — full SDK wiring in your code |
| Control over SDK config | Limited | Full |
**When to prefer the OTel Layer:**
- Cold starts are infrequent or not important (async workflows, scheduled jobs, or when using provisioned concurrency)
- Per-request latency budget is tight (user-facing APIs)
- You want minimal instrumentation code
- The OTLP endpoint is outside your Lambda's AWS region and you can't or don't want to run an Open Telemetry Collector in the region. — `forceFlush()` in the manual SDK setup incurs a full cross-region HTTPS round-trip on every invocation before the response can be returned to the client, which can add hundreds of milliseconds per request
**When to prefer Manual SDK Setup:**
- Cold start latency is critical (latency-sensitive functions hit often)
- You need fine-grained control over sampling, resource attributes, or processors
**Ask the developer which trade-off matters more before choosing an approach.**
---
## Approach 1: AWS Managed OTel Layer
The [AWS Distro for OpenTelemetry (ADOT) Lambda layer](https://github.com/aws-observability/aws-otel-lambda)
bundles an OTel Collector sidecar that runs in the Lambda execution environment.
It intercepts OTLP exports from the SDK and forwards them to the OTLP endpoint
*after* the function response has been returned to the client — so telemetry
export does not add to request latency.
### Add the Layer and Configure Environment Variables
Find the correct layer ARN for your region and runtime at
<https://github.com/aws-observability/aws-otel-lambda/releases>.
In CDK:
```typescript
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as cdk from 'aws-cdk-lib';
const fn = new NodejsFunction(this, 'Fn', {
runtime: lambda.Runtime.NODEJS_20_X,
timeout: cdk.Duration.seconds(15),
layers: [
lambda.LayerVersion.fromLayerVersionArn(this, 'OtelLayer',
'arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-30-1:1'
// ↑ replace with the latest ARN for your region and architecture
),
],
environment: {
AWS_LAMBDA_EXEC_WRAPPER: '/opt/otel-handler',
OTEL_EXPORTER_OTLP_ENDPOINT: 'https://your-otlp-endpoint', // Honeycomb: https://api.honeycomb.io, or your OTel Collector URL
OTEL_EXPORTER_OTLP_HEADERS: 'x-honeycomb-team=YOUR_API_KEY', // omit if using a Collector that handles auth
OTEL_SERVICE_NAME: 'my-function',
OTEL_PROPAGATORS: 'tracecontext',
},
bundling: {
externalModules: ['@aws-sdk/*'],
sourceMap: true,
},
});
```
`AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler` tells the layer to wrap the Node.js
handler — it initialises the OTel SDK automatically before your code runs.
### Dependencies
With the layer handling SDK init and export, you only need the API package for
custom spans and attributes:
```bash
npm install @opentelemetry/api
```
You do **not** need `sdk-trace-node`, `sdk-trace-base`, or the OTLP exporter —
those are provided by the layer.
### Handler Code (no forceFlush needed)
This is just an example of how to create a span if needed. By default the layer creates a span already, so it isn't needed to create a span for the entire handler method.
```typescript
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('my-function');
export async function handler(event: any) {
return tracer.startActiveSpan('my-operation', async (span) => {
try {
// … do work …
return result;
} catch (err) {
span.recordException(err as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end();
// No forceFlush() needed — the layer flushes after the response is returned
}
});
}
```
### Cold Start Impact
The layer adds approximately 2–5+ seconds to cold starts. For latency-sensitive
functions that are invoked frequently (so cold starts are rare), this is usually
acceptable. For functions where cold starts happen on every invocation (e.g.,
low-traffic functions with short TTLs), evaluate whether that overhead is tolerable.
Provisioned Concurrency eliminates cold starts entirely if the overhead is not
acceptable.
---
## Approach 2: Manual SDK Setup
Set up the OTel SDK directly in your function code. Gives full control but
requires calling `forceFlush()` before the handler returns, which adds latency
on every invocation.
### Manual SDK Dependencies
```bash
npm install @opentelemetry/sdk-trace-node \
@opentelemetry/sdk-trace-base \
@opentelemetry/exporter-trace-otlp-proto \
@opentelemetry/resources \
@opentelemetry/core \
@opentelemetry/api
```
Use `@opentelemetry/exporter-trace-otlp-proto` (protobuf over HTTP) rather
than the gRPC exporter — gRPC adds significant cold-start overhead in Lambda.
### SDK 2.x API (breaking changes from 1.x)
OTel Node SDK 2.x changed two construction APIs that matter for Lambda:
```typescript
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { resourceFromAttributes } from '@opentelemetry/resources'; // not new Resource()
import { W3CTraceContextPropagator } from '@opentelemetry/core';
const provider = new NodeTracerProvider({
resource: resourceFromAttributes({ 'service.name': 'my-function' }), // not new Resource()
spanProcessors: [new BatchSpanProcessor(new OTLPTraceExporter())], // in constructor, not addSpanProcessor()
});
provider.register({ propagator: new W3CTraceContextPropagator() });
```
In SDK 1.x, `spanProcessors` was not a constructor option — you called
`provider.addSpanProcessor()` after construction. In 2.x, processors passed in
the constructor are the only supported path.
### Singleton Init Pattern
Call `initTelemetry()` at **module scope**, not inside the handler. Lambda reuses
the execution environment across warm invocations — re-initialising on every call
leaks providers and re-registers propagators.
```typescript
// ✅ module-level: runs once on cold start, no-op on warm invocations
let provider: NodeTracerProvider | null = null;
export function initTelemetry(): void {
if (provider) return; // guard: no-op on warm invocations
provider = new NodeTracerProvider({ … });
provider.register({ propagator: new W3CTraceContextPropagator() });
}
initTelemetry(); // ← called at module load, outside the handler
export async function handler(event, ctx) { … }
```
### Resource Attributes from Lambda Env Vars
AWS injects these env vars automatically — wire them to `faas.*` semantic conventions:
```typescript
resourceFromAttributes({
'service.name': process.env.OTEL_SERVICE_NAME ?? 'unknown',
'cloud.provider': 'aws',
'cloud.region': process.env.AWS_REGION ?? 'unknown',
'faas.name': process.env.AWS_LAMBDA_FUNCTION_NAME ?? 'unknown',
'faas.version': process.env.AWS_LAMBDA_FUNCTION_VERSION ?? '$LATEST',
})
```
### Critical: forceFlush Before Returning
`BatchSpanProcessor` queues spans in memory and flushes on a background timer.
When the handler promise resolves, Lambda **freezes the process** — the timer never
fires and any queued spans are silently dropped.
Call `provider.forceFlush()` in the handler's `finally` block before returning:
```typescript
export async function handler(event, ctx) {
return tracer.startActiveSpan('my-operation', async (span) => {
try {
// … do work …
return result;
} catch (err) {
span.recordException(err);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end();
await provider.forceFlush(); // ← must come before the function returns
}
});
}
```
This is the single most common cause of missing spans in Lambda. There is no
error — the function succeeds, the spans are created, but they never leave the
process.
**Latency note:** `forceFlush()` performs a synchronous HTTPS POST to the OTLP
endpoint before the handler returns. The round-trip cost depends entirely on
where the endpoint is relative to the Lambda:
- **OTel Collector in the same AWS region** (e.g. running on ECS or EC2 in the
same VPC): typically 1–5 ms — negligible.
- **OTLP endpoint in the same AWS region but outside the VPC** (e.g. a managed
service in the same region): typically 20–50 ms per invocation.
- **OTLP endpoint in a different region or continent**: the round-trip can add
200–500 ms or more to every request.
Running a regional OTel Collector sidecar is the most effective way to keep
`forceFlush()` latency low while retaining the Manual SDK approach. Factor
the expected round-trip into your timeout sizing.
### Timeout Sizing
The 3-second default timeout is too short once OTel is added. Two costs compound:
- **Cold start**: the OTLP exporter opens an HTTPS connection to the endpoint
- **Per-invocation**: `forceFlush()` awaits a real HTTPS POST before returning
Practical minimums with a remote OTLP endpoint (adjust down if using a same-region Collector):
| Function role | Recommended timeout |
| :--- | :--- |
| Leaf function (DynamoDB / simple logic only) | **15s** |
| Orchestrator (invokes another Lambda + flushes) | **30s** |
| Authorizer | **15s** |
In CDK:
```typescript
import * as cdk from 'aws-cdk-lib';
new NodejsFunction(this, 'Fn', {
timeout: cdk.Duration.seconds(30),
// …
});
```
### Bundling — Exclude AWS SDK
Lambda Node.js 18+ runtimes include `@aws-sdk/*` v3. Mark it as external to keep
bundle size small and cold-start fast:
```typescript
// CDK NodejsFunction (esbuild under the hood)
bundling: {
externalModules: ['@aws-sdk/*'],
sourceMap: true,
}
```
Do **not** mark `@opentelemetry/*` as external — it must be bundled because the
Lambda runtime does not include it (unlike the OTel Layer approach).
---
## Common Patterns (Both Approaches)
### Header Normalisation (SAM Local vs API Gateway)
API Gateway v1 lowercases all HTTP headers in production (`traceparent`,
`authorization`). SAM Local uses a Flask dev server that title-cases them
(`Traceparent`, `Authorization`). The W3CTraceContextPropagator always looks
for the lowercase `traceparent` key.
**Always normalise header keys to lowercase** before calling
`propagation.extract()`:
```typescript
import { propagation, ROOT_CONTEXT, context } from '@opentelemetry/api';
export function extractTraceFromHeaders(
headers: Record<string, string | undefined> | null | undefined,
): ReturnType<typeof context.active> {
if (!headers) return ROOT_CONTEXT;
const normalized: Record<string, string> = {};
for (const [k, v] of Object.entries(headers)) {
if (v !== undefined) normalized[k.toLowerCase()] = v;
}
return propagation.extract(ROOT_CONTEXT, normalized);
}
// In the handler:
const parentCtx = extractTraceFromHeaders(event.headers);
return otelContext.with(parentCtx, () => tracer.startActiveSpan(…));
```
Without this, traces connect correctly in production but appear as disconnected
roots when testing under SAM Local — a confusing local/prod discrepancy.
### Cross-Lambda Trace Propagation (Direct Invoke)
When invoking a Lambda function via the AWS SDK (`InvokeCommand`), there are no
HTTP headers — the payload is a raw JSON blob. Inject W3C context into a custom
field in the payload and extract it on the receiving side.
#### Caller (e.g. middleware Lambda)
```typescript
import { LambdaClient, InvokeCommand } from '@aws-sdk/client-lambda';
import { propagation, context } from '@opentelemetry/api';
const client = new LambdaClient({});
// Serialise the active trace context into the request payload
const carrier: Record<string, string> = {};
propagation.inject(context.active(), carrier);
const payload = { ...myRequest, traceContext: carrier };
const result = await client.send(new InvokeCommand({
FunctionName: process.env.BACKEND_FUNCTION_NAME,
InvocationType: 'RequestResponse',
Payload: Buffer.from(JSON.stringify(payload)),
}));
```
#### Callee (e.g. backend Lambda)
```typescript
import { propagation, context, ROOT_CONTEXT } from '@opentelemetry/api';
export async function handler(event: MyRequest) {
const parentCtx = event.traceContext
? propagation.extract(ROOT_CONTEXT, event.traceContext)
: ROOT_CONTEXT;
return context.with(parentCtx, () =>
tracer.startActiveSpan('backend.invoke', async (span) => {
// … this span is now a child of the middleware span …
})
);
}
```
### API Gateway Authorizer — Use REQUEST Type
`TOKEN` type authorizers receive only the `authorizationToken` value — no headers reach the function, so `traceparent` is unavailable and the authorizer span cannot join the caller's trace.
**Use `REQUEST` type authorizers** to receive the full header map:
```typescript
// AWS CDK
import { RequestAuthorizer, IdentitySource } from 'aws-cdk-lib/aws-apigateway';
new RequestAuthorizer(this, 'Auth', {
handler: authorizerFn,
identitySources: [IdentitySource.header('Authorization')],
// identitySources controls the cache key — traceparent doesn't need to be here
resultsCacheTtl: Duration.minutes(5),
});
```
In the authorizer handler, extract the token from `event.headers.Authorization`
(or the lowercase variant — apply the normalisation helper above):
```typescript
const token = event.headers?.Authorization?.replace(/^Bearer\s+/i, '')
?? event.headers?.authorization?.replace(/^Bearer\s+/i, '');
```
references/local-collector-debug-test.md
# Local OTel Collector for Migration Verification
Running a local OTel Collector during migration lets you verify that spans are being
produced and structured correctly without needing a live Honeycomb account or sending
data to a remote backend. Useful for the early phases of migration (SDK init, middleware,
context propagation) where you just need to confirm telemetry is flowing.
The skill ships a script at `${CLAUDE_PLUGIN_ROOT}/scripts/start-collector.sh` that starts
the collector via Docker with a pre-built config.
## Starting the Collector
**Without Honeycomb (local verification only):**
```bash
./scripts/start-collector.sh --no-honeycomb
```
No API key required. Spans are printed to stdout and written to `./otelcol-spans.ndjson`.
**With Honeycomb (verify locally and forward to backend):**
```bash
./scripts/start-collector.sh --api-key YOUR_API_KEY
# or
HONEYCOMB_API_KEY=YOUR_API_KEY ./scripts/start-collector.sh
```
**Custom log file location:**
```bash
./scripts/start-collector.sh --no-honeycomb --log-file /tmp/my-service-spans.ndjson
```
**Custom collector config (bypasses all default flags):**
```bash
./scripts/start-collector.sh --config ./my-collector-config.yaml
```
The collector listens on `localhost:4317` (gRPC) and `localhost:4318` (HTTP). Point your
SDK at either:
```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 # HTTP
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 # gRPC
```
## Reading the Debug Output
The `debug` exporter prints a human-readable block for each span to stdout as it arrives.
Key fields to check during migration:
```
ResourceSpans #0
Resource attributes:
-> service.name: Str(my-service) ← confirms SDK is setting service name
ScopeSpans #0
Span #0
Trace ID : 0af7651916cd43dd8448eb211c80319c
Parent ID : b7ad6b7169203331 ← non-empty = span is connected to a parent
ID : c5e2f3a1b4d67890
Name : GET /api/orders
Kind : Server
Start time : ...
End time : ...
Status code : Ok
Attributes:
-> http.method: Str(GET)
-> http.route: Str(/api/orders)
-> http.status_code: Int(200)
```
**What to look for:**
| Field | What it tells you |
| :--- | :--- |
| `service.name` present | SDK resource is configured correctly |
| `Parent ID` non-empty | Context is propagating; span is connected to a parent |
| `Parent ID` empty | Span is a root — expected for entry points, a bug for internal spans |
| `http.route` present | Framework middleware is installed and working |
| `Status code: Error` | `span.SetStatus` is being called on error paths |
## Reading the NDJSON Log File
Each line in `otelcol-spans.ndjson` is one batch of spans serialised as OTLP JSON. To
inspect individual spans, use `jq`:
```bash
# Pretty-print all spans
jq . otelcol-spans.ndjson
# List all span names received
jq -r '.resourceSpans[].scopeSpans[].spans[].name' otelcol-spans.ndjson
# Check for disconnected spans (missing parentSpanId)
jq '.resourceSpans[].scopeSpans[].spans[] | select(.parentSpanId == null or .parentSpanId == "") | .name' \
otelcol-spans.ndjson
# List all attribute keys seen across all spans
jq -r '.resourceSpans[].scopeSpans[].spans[].attributes[].key' otelcol-spans.ndjson | sort -u
```
## Stopping the Collector
`Ctrl+C` — the script traps the signal, stops the container, and removes the temporary
config file. The NDJSON log file is kept so you can inspect it after shutdown.
references/python.md
# Python OpenTelemetry — In-Depth Guide
Additional detail for Python instrumentation beyond the basics in
`sdk-setup-by-language.md`. Covers framework-specific packages, async patterns,
programmatic SDK setup for ASGI apps, and attribute enrichment.
---
## Critical: async SQLAlchemy requires `.sync_engine`
**Read this before writing any SQLAlchemy instrumentation.** `SQLAlchemyInstrumentor`
does not support async engines directly. Passing the async engine raises
`NotImplementedError: asynchronous events are not implemented at this time`.
Always pass the underlying sync engine:
```python
# WRONG — raises NotImplementedError at startup
SQLAlchemyInstrumentor().instrument(engine=async_engine)
# CORRECT
SQLAlchemyInstrumentor().instrument(engine=async_engine.sync_engine)
```
This applies to `create_async_engine(...)` from `sqlalchemy.ext.asyncio`.
**After writing the call, verify the argument ends in `.sync_engine`.** If it reads
`engine=some_engine` without `.sync_engine`, the app will crash at startup with
`NotImplementedError: asynchronous events are not implemented at this time` — the
fix is always to append `.sync_engine` to the engine argument.
---
## Choosing Your Setup Approach
| Approach | When to use |
| :--- | :--- |
| `opentelemetry-instrument` CLI | Simple WSGI apps (Flask, Django); no code changes needed |
| Programmatic SDK init | ASGI apps (FastAPI, Starlette); gives full control over lifecycle |
For **FastAPI / NiceGUI / Starlette** always use programmatic setup. The CLI runner
doesn't integrate cleanly with ASGI lifespans and may miss startup instrumentation.
---
## Instrumentation Package Reference
Install only what the app actually uses. Each package auto-instruments its library
when `.instrument()` is called.
### Web frameworks
```bash
pip install opentelemetry-instrumentation-fastapi # FastAPI + Starlette
pip install opentelemetry-instrumentation-flask # Flask
pip install opentelemetry-instrumentation-django # Django
pip install opentelemetry-instrumentation-aiohttp-server # aiohttp server
```
### HTTP clients
```bash
pip install opentelemetry-instrumentation-httpx # httpx (sync + async)
pip install opentelemetry-instrumentation-requests # requests
pip install opentelemetry-instrumentation-aiohttp-client # aiohttp client
pip install opentelemetry-instrumentation-urllib3 # urllib3
```
### Databases
```bash
pip install opentelemetry-instrumentation-sqlalchemy # SQLAlchemy (sync + async)
pip install opentelemetry-instrumentation-asyncpg # asyncpg (raw driver)
pip install opentelemetry-instrumentation-psycopg2 # psycopg2
pip install opentelemetry-instrumentation-sqlite3 # sqlite3 (stdlib)
pip install opentelemetry-instrumentation-redis # redis-py
```
### Other
```bash
pip install opentelemetry-instrumentation-celery # Celery tasks
pip install opentelemetry-instrumentation-logging # stdlib logging bridge
pip install opentelemetry-instrumentation-system-metrics # CPU, memory, GC
```
With `uv`:
```bash
uv add opentelemetry-sdk opentelemetry-exporter-otlp-proto-http \
opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-sqlalchemy
```
---
## Programmatic Setup for ASGI Apps (FastAPI / Starlette)
Create a `telemetry.py` module. The SDK wiring is always the same; the
auto-instrumentation calls depend on what the app actually uses — inspect the
codebase and choose from the **Instrumentation Package Reference** above.
```python
import os
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
# Import only the instrumentors that match the libraries this app uses.
# Check pyproject.toml / requirements.txt, then see Package Reference above.
def configure_opentelemetry(**kwargs):
endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")
if not endpoint:
return # no-op when unconfigured
resource = Resource.create() # reads OTEL_SERVICE_NAME + OTEL_RESOURCE_ATTRIBUTES
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
# Call .instrument() for each library the app uses.
# SQLAlchemy example — note .sync_engine (async engines crash without it):
# SQLAlchemyInstrumentor().instrument(engine=async_engine.sync_engine)
# httpx example:
# HTTPXClientInstrumentor().instrument()
# See Package Reference above for the full list.
def instrument_app(app):
FastAPIInstrumentor.instrument_app(app)
```
Call order in `main.py`:
```python
# 1. Configure SDK and library auto-instrumentation (before app is created)
configure_opentelemetry(...)
app = FastAPI(lifespan=lifespan)
# 2. Mount all routers
init_api_routes(app)
init_gui_routes(app)
# 3. Instrument the fully-wired app
instrument_app(app)
# 4. NiceGUI only: if this app uses @ui.page() decorators, add the http.route
# recovery middleware (see "NiceGUI / custom router" section below).
# Skip this step for non-NiceGUI apps.
```
**Why order matters:** `FastAPIInstrumentor.instrument_app()` wraps the router list
at call time. If called before routes are mounted, some routes won't be captured.
---
## Adding Attributes to Existing Spans
Get the current span from context and annotate it — no new span needed:
```python
from opentelemetry import trace
span = trace.get_current_span()
span.set_attribute("user.id", str(user.id))
span.set_attribute("habit.id", habit_id)
span.set_attribute("habit.name", habit.name)
```
**In FastAPI request handlers**, the current span is the auto-instrumented HTTP span.
Adding attributes here enriches every request trace with business context.
---
## Middleware for Per-Request Attributes
For attributes that come from session/auth context (user ID, tenant), a middleware
runs inside the auto-instrumented HTTP span and can annotate it:
```python
from starlette.middleware.base import BaseHTTPMiddleware
from opentelemetry import trace
class OtelAttributeMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request, call_next):
span = trace.get_current_span()
# Read from session / auth token already attached by auth middleware
if user_id := request.state.__dict__.get("user_id"):
span.set_attribute("user.id", str(user_id))
return await call_next(request)
```
Add it **after** `FastAPIInstrumentor.instrument_app()` and **after** auth middleware:
```python
app.add_middleware(OtelAttributeMiddleware)
```
### NiceGUI / custom router: recovering `http.route`
A common mistake: assuming that because `FastAPIInstrumentor` wraps the ASGI layer,
NiceGUI requests are fully instrumented. Spans *are* created — but `http.route` is
**not** populated. `FastAPIInstrumentor` reads `http.route` from FastAPI's route
registry, and NiceGUI's `@ui.page()` routes are never registered there. The result is
spans with no route, making every `http.route` breakdown in Honeycomb empty.
**Another common mistake: using `server_request_hook` to read `scope["route"]`.**
This looks correct but silently fails for NiceGUI — the route is not yet matched
in the scope when the hook fires (span creation). It is only populated *after*
routing completes. The only working fix is the middleware below, which runs after
`call_next` when routing has already happened.
If the app uses NiceGUI, add this after `instrument_app(app)` in `main.py`:
```python
from starlette.middleware.base import BaseHTTPMiddleware
class _RouteAttributeMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request, call_next):
response = await call_next(request)
span = trace.get_current_span()
if route := request.scope.get("route"):
span.set_attribute("http.route", route.path)
else:
span.set_attribute("http.route", request.url.path)
return response
app.add_middleware(_RouteAttributeMiddleware)
```
---
## Creating Custom Spans
Wrap business logic in a span to make it visible in the trace waterfall:
```python
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
async def process_habit_completion(habit_id: str, done: bool):
with tracer.start_as_current_span("habit.complete") as span:
span.set_attribute("habit.id", habit_id)
span.set_attribute("completion.done", done)
# ... business logic ...
```
For async code, context propagates automatically through `async with` and
`start_as_current_span` — no manual context passing needed within a single
async task.
---
## Async Caveats
### asyncpg raw driver
If using `asyncpg` directly (no SQLAlchemy), use
`opentelemetry-instrumentation-asyncpg` and call
`AsyncPGInstrumentor().instrument()` before creating any connection pools.
### Background tasks / workers
Spans created in background `asyncio.Task`s are attached to the task's context,
not the request context. They arrive in Honeycomb as separate traces (expected).
Use `trace.use_span()` or `copy_context()` if you need to link them to the
originating request.
---
## Resource Attributes
Set custom resource attributes (shown on every span) via env var:
```bash
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production,service.version=1.2.3"
```
Or programmatically:
```python
from opentelemetry.sdk.resources import Resource
resource = Resource.create({
"service.name": "my-app",
"deployment.environment": "production",
"service.version": "1.2.3",
})
provider = TracerProvider(resource=resource)
```
`Resource.create()` with no arguments reads `OTEL_SERVICE_NAME` and
`OTEL_RESOURCE_ATTRIBUTES` from the environment automatically.
---
## Exception Slugs
Tag each error site with a static identifier so errors are greppable in code and
queryable by slug in Honeycomb:
```python
from opentelemetry import trace
span = trace.get_current_span()
try:
result = await do_something()
except ValueError as e:
span.set_attribute("exception.slug", "err-invalid-habit-data")
span.set_attribute("error", True)
span.record_exception(e)
raise
```
Query: `WHERE error = true AND exception.slug does-not-exist` — finds untagged error
paths that still need slugs.
references/sdk-setup-by-language.md
# SDK Setup by Language
Complete OpenTelemetry SDK setup instructions for each language, configured to send
traces to Honeycomb.
## Environment Variables (All Languages)
### Required
```bash
export OTEL_SERVICE_NAME="your-service-name"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY"
```
EU endpoint: `https://api.eu1.honeycomb.io`
### Optional (Recommended)
```bash
# Protocol selection (default: http/protobuf)
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" # or "grpc"
# Signal-specific endpoints (override base endpoint)
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.honeycomb.io/v1/traces"
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="https://api.honeycomb.io/v1/metrics"
```
### For Metrics (Preferred)
Prefer modern OTLP metrics and native datapoints. Use dataset hints to confirm the
destination type (`metrics` or `events`). Authenticate with:
```bash
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="x-honeycomb-team=YOUR_API_KEY"
```
### Honeycomb Authentication Pitfall
The `x-honeycomb-team` header in `OTEL_EXPORTER_OTLP_HEADERS` is **required** for
Honeycomb to accept OTLP data. Without it, Honeycomb **silently rejects** requests — no
error is returned, data simply never appears.
A common mistake: the app has `HONEYCOMB_API_KEY` in `.env` but never sets
`OTEL_EXPORTER_OTLP_HEADERS`. The OTel SDK does NOT automatically read
`HONEYCOMB_API_KEY` — you must either:
1. Set `OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_KEY"` explicitly, **or**
2. Pass headers programmatically when constructing exporters:
```typescript
const headers = { "x-honeycomb-team": process.env.HONEYCOMB_API_KEY };
new OTLPTraceExporter({ headers });
new OTLPMetricExporter({ headers });
```
Also ensure `.env` is loaded (e.g., `import "dotenv/config"`) **before** the OTel SDK
initializes. In ESM/TypeScript, all imports resolve before module body code runs, so
`dotenv.config()` in the main file may execute too late.
### Legacy Honeycomb Dataset Routing
Do not add `x-honeycomb-dataset` by default for modern OTLP metrics. Dataset hints identify
the destination type (`metrics` or `events`). Use the header only when hints or
configuration require legacy routing to a named event dataset:
```bash
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="x-honeycomb-team=YOUR_API_KEY,x-honeycomb-dataset=YOUR_METRICS_DATASET"
```
Traces do not need it; they route by `service.name`.
## Go
### Dependencies
```bash
go get go.opentelemetry.io/otel \
go.opentelemetry.io/otel/sdk/trace \
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
```
### Auto-instrumentation libraries
```bash
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
```
### Notes
- Use `otelhttp.NewHandler()` to wrap HTTP handlers
- Use `otelgrpc.UnaryServerInterceptor()` for gRPC
- SDK reads env vars automatically
## Python
See `${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/python.md` for the
full Python guide: package catalogue, ASGI programmatic setup, async SQLAlchemy,
middleware enrichment, and resource attributes.
### Quick start (WSGI apps only — Flask, Django)
```bash
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http \
opentelemetry-distro
opentelemetry-instrument python app.py
```
**For ASGI apps (FastAPI, Starlette, NiceGUI) always use programmatic setup** — the
CLI runner doesn't integrate cleanly with ASGI lifespans. See the Python guide above.
## Node.js
### Dependencies
```bash
npm install @opentelemetry/sdk-node \
@opentelemetry/exporter-trace-otlp-http \
@opentelemetry/auto-instrumentations-node
```
### Setup (tracing.js — require before app)
```javascript
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter(),
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
```
### Run
```bash
node --require ./tracing.js app.js
```
## Java
### Java Agent (recommended — zero code changes)
```bash
# Download agent jar
curl -L -o opentelemetry-javaagent.jar \
https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
# Run with agent
java -javaagent:opentelemetry-javaagent.jar \
-Dotel.exporter.otlp.endpoint=https://api.honeycomb.io \
-Dotel.exporter.otlp.headers=x-honeycomb-team=YOUR_API_KEY \
-Dotel.service.name=your-service \
-jar your-app.jar
```
### Notes
- Java agent auto-instruments most frameworks (Spring, Servlet, JDBC, etc.)
- No code changes required for basic tracing
- Add custom spans via OTel API for business logic
## Ruby
### Dependencies
```ruby
# Gemfile
gem "opentelemetry-sdk"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-all"
```
### Setup
```ruby
require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/instrumentation/all"
OpenTelemetry::SDK.configure do |c|
c.service_name = "your-service"
c.use_all # auto-instrument all supported libraries
end
```
## .NET
### Dependencies
```bash
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
```
### Setup (Program.cs)
```csharp
builder.Services.AddOpenTelemetry()
.WithTracing(tracing => tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddOtlpExporter());
```
## Rust
### Dependencies (Cargo.toml)
```toml
[dependencies]
opentelemetry = "0.32"
# reqwest-rustls is not optional — without it there's no TLS backend, and exports
# to https:// endpoints (like Honeycomb) fail silently. See Notes below.
opentelemetry-otlp = { version = "0.32", default-features = false, features = ["http-proto", "reqwest-blocking-client", "reqwest-rustls"] }
opentelemetry_sdk = "0.32"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-opentelemetry = "0.33"
```
Verify current versions with `cargo add` — this crate family moves fast and pins go stale.
### Setup
```rust
use opentelemetry::trace::TracerProvider as _;
use opentelemetry_sdk::trace::SdkTracerProvider;
use opentelemetry_sdk::Resource;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
use tracing_subscriber::EnvFilter;
fn init_telemetry(service_name: &str) -> Option<SdkTracerProvider> {
let provider = opentelemetry_otlp::SpanExporter::builder()
.with_http() // reads OTEL_EXPORTER_OTLP_ENDPOINT / _HEADERS / _PROTOCOL from env
.build()
.ok()
.map(|exporter| {
let resource = Resource::builder().with_service_name(service_name.to_string()).build();
SdkTracerProvider::builder().with_batch_exporter(exporter).with_resource(resource).build()
});
let otel_layer = provider.as_ref().map(|p| tracing_opentelemetry::layer().with_tracer(p.tracer(service_name.to_string())));
tracing_subscriber::registry()
.with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
.with(otel_layer)
// Don't skip this: batch-exporter failures (TLS, network, auth) log via
// tracing::error!, and with no fmt layer they vanish with no trace at all.
.with(tracing_subscriber::fmt::layer().with_writer(std::io::stderr))
.init();
provider
}
// Flush before exit: if let Some(p) = provider { let _ = p.shutdown(); }
```
### Notes
- Rust uses OTLP exporter directly; no auto-instrumentation, all spans are manual
- Prefer the `tracing` crate with `tracing-opentelemetry` for ergonomic instrumentation
(`#[tracing::instrument]`, `tracing::info_span!`) over calling `opentelemetry::trace::Tracer` directly
- **Tokio apps:** the plain `with_batch_exporter()` runs its export loop on a dedicated
OS thread, not a tokio task. Pairing it with the exporter's default async-reqwest client
panics at runtime ("no reactor running") because that thread has no tokio reactor. Use
the `reqwest-blocking-client` feature shown above, or if you need to stay on async
reqwest, use `opentelemetry_sdk`'s `rt-tokio` feature with
`span_processor_with_async_runtime::BatchSpanProcessor` instead.
- **Silent export failures:** the SDK reports export errors (TLS, network, auth) via
`tracing::error!`, not panics. With no `fmt` (or other output) layer in the subscriber,
those errors vanish and the process exits 0 having sent nothing. Keep a `fmt` layer
wired up, at least during setup.
- The local collector setup below runs over plain `http://`, so it proves span
structure/wiring but not that TLS export to a real `https://` endpoint works — do one
live check against Honeycomb (or an `https://` collector) before calling it done.
- Verify locally with the collector setup below before pointing at Honeycomb.
## Testing Locally Without Honeycomb
Before pointing your SDK at Honeycomb, verify that spans are being produced and
structured correctly using a local OTel Collector. Point your SDK at the local
collector instead of Honeycomb:
```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_SERVICE_NAME="your-service"
# No OTEL_EXPORTER_OTLP_HEADERS needed — the local collector has no auth
```
Then start the collector:
```bash
./scripts/start-collector.sh --no-honeycomb
```
Spans appear in the debug output (stdout) and are written to `./otelcol-traces.ndjson`,
`./otelcol-logs.ndjson`, and `./otelcol-metrics.ndjson` on the host.
For full setup instructions, available flags, and `jq` commands for inspecting the
NDJSON output, see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/local-collector-debug-test.md`.
references/wide-event-attributes.md
# Wide Event Attribute Catalog
**This is the canonical attribute catalog.** Other skills and agents reference this
file rather than maintaining their own attribute lists.
Attributes to add to your spans, organized by category. Each attribute enriches your
events with context that enables BubbleUp and investigation workflows. The wider your
events, the more questions you can answer without re-deploying.
Drawn from Chapter 6 of *Observability Engineering* (2nd edition) by Charity Majors,
Liz Fong-Jones, George Miranda, and Austin Parker, with contributions from Jeremy Morrell.
## Service Metadata
Connect services to their owners and understand which team is responsible.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `service.name` | `api`, `shoppingcart` | The name of this service |
| `service.environment` | `production`, `staging` | The environment where this service is running |
| `service.team` | `web-services`, `dev-ex` | The team that owns this service — useful for knowing who to page |
| `service.slack_channel` | `#web-services` | Where to reach out if you discover an issue |
**Why it matters:** During an incident, the first question is often "who owns this?" These
attributes let you answer that from your telemetry without switching to a service catalog.
**Example query — how many services does each team run?**
```
VISUALIZE COUNT_DISTINCT(service.name)
WHERE service.environment = "production"
GROUP BY service.team
```
## Infrastructure
Understand the physical or virtual resources backing each service instance.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `instance.id` | `656993bd-40e1...` | An ID mapping to this one instance of the service |
| `instance.memory_mb` | `12336` | RAM available to this service |
| `instance.cpu_count` | `4`, `8`, `196` | Number of cores available |
| `instance.type` | `m6i.xlarge` | Vendor name for this instance type |
**Why it matters:** Correlating performance with instance resources answers questions like
"is this service under-provisioned?" or "are the larger instances actually faster?"
**Example query — which services use the most memory?**
```
VISUALIZE MAX(instance.memory_mb)
GROUP BY service.name, instance.type
ORDER BY instance.memory_mb DESC
LIMIT 10
```
## Orchestration
Capture container and cluster context so you can correlate issues with infrastructure topology.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `container.id` | `a3bf90e006b2` | Docker container ID |
| `container.name` | `nginx-proxy` | Container name used by runtime |
| `k8s.cluster.name` | `api-cluster` | Kubernetes cluster name |
| `k8s.pod.name` | `nginx-2723453542-065rx` | Kubernetes pod name |
| `cloud.availability_zone` | `us-east-1c` | AZ where the service runs |
| `cloud.region` | `us-east-1` | Region where the service runs |
**Why it matters:** Infrastructure issues often manifest per-AZ or per-node. These attributes
let BubbleUp surface "all slow requests are from `us-east-1c`" automatically.
**Example query — request distribution across AZs:**
```
VISUALIZE COUNT
WHERE service.name = "api-service"
GROUP BY cloud.availability_zone
```
## Build and Deploy
Answer "what changed?" during incidents without leaving your observability tool.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `service.version` | `v123`, `9731945...` | Version string or image hash |
| `service.build.git_hash` | `6f6466b0e693...` | Git SHA of the deployed commit |
| `service.build.deployment.age_minutes` | `1`, `10230` | How long ago this version was deployed |
| `service.build.deployment.trigger` | `merge-to-main`, `slack-bot` | What triggered this deployment |
| `service.build.deployment.user` | `keanu@company.com` | Who kicked off the build |
**Why it matters:** "Did something just get deployed?" is one of the most frequent incident
questions. With `deployment.age_minutes < 20` you can instantly find recent deploys, and
with `service.version` you can compare error rates between versions.
**Example query — what was recently deployed?**
```
VISUALIZE MIN(service.build.deployment.age_minutes) AS age
WHERE service.build.deployment.age_minutes < 20
GROUP BY service.name
ORDER BY age ASC
LIMIT 10
```
**Example query — 500s correlated with deploy versions:**
```
VISUALIZE COUNT
WHERE service.name = "api-service"
GROUP BY http.response.status_code, service.version
```
## Feature Flags
Correlate issues with feature rollouts by tracking which flags are active per-request.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `feature_flag.<flag_name>` | `true`, `false` | The value of a particular feature flag for this request |
Use one attribute per flag (e.g., `feature_flag.auth_v2`, `feature_flag.new_checkout_flow`).
**Why it matters:** Feature flags are a developer superpower for testing in production, but
only if you can compare performance between flag states. BubbleUp can instantly show that
errors correlate with `feature_flag.auth_v2 = true`.
**Example query — errors by feature flag state:**
```
VISUALIZE COUNT
WHERE service.name = "api-service" AND error = true
GROUP BY feature_flag.auth_v2, exception.slug
```
## Runtime Versions
Track versions of languages, frameworks, and datastores to correlate issues with upgrades.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `go.version` | `go1.23.2` | Language runtime version |
| `rails.version` | `7.2.1.1` | Web framework version |
| `postgres.version` | `16.4` | Datastore version |
**Why it matters:** "Didn't we upgrade Go versions recently? Does that correlate with the
memory increase?" You can't answer this without version attributes.
**Example query — memory usage by Go version:**
```
VISUALIZE HEATMAP(metrics.memory_mb)
WHERE service.name = "api-service"
GROUP BY go.version
```
## HTTP Information (Beyond Auto-Instrumentation)
Auto-instrumentation captures basics, but parsing and enriching HTTP context unlocks
deeper analysis.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `http.route` | `/team/{team_id}/user/{user_id}` | The route pattern the URL matched |
| `http.route.param.team_id` | `14739` | Extracted route parameter value |
| `http.route.query.sort_dir` | `asc` | Relevant query parameters |
| `user_agent.device` | `computer`, `phone` | Device type parsed from User-Agent |
| `user_agent.browser` | `Chrome`, `Safari` | Browser parsed from User-Agent |
| `user_agent.browser_version` | `129` | Browser version parsed from User-Agent |
**Why it matters:** Without `http.route`, a latency spike just shows "some requests are slow."
With it, you see "only `POST /checkout` and `POST /signup` are slow." Parsed user-agent
fields let you find client-specific issues without regex.
**Example query — P99 latency by route:**
```
VISUALIZE P99(duration_ms)
WHERE service.name = "api-service"
GROUP BY http.route
```
## Timing Breakdowns
Put important sub-operation durations as attributes on the parent span rather than
creating child spans for everything. This enables direct querying without JOINs.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `auth.duration_ms` | `52.2`, `0.2` | Time spent in authentication |
| `payload_parse.duration_ms` | `22.1`, `0.1` | Time spent parsing the request payload |
**Why it matters:** Child spans require JOINs to correlate with parent attributes. Timing
attributes on the parent span let BubbleUp immediately tell you "that group of requests
was slow because authentication took 10 seconds." See the
[Timing Attributes pattern](${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/custom-instrumentation.md)
for implementation guidance.
**Example query — payload parse P99 by user type and region:**
```
VISUALIZE P99(payload_parse.duration_ms)
WHERE service.name = "api-service"
GROUP BY user.type, cloud.region
```
## Async Request Summaries
Roll up child operation statistics onto the parent span to identify outlier requests.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `stats.http_requests_count` | `1`, `140` | HTTP requests triggered during this request |
| `stats.http_requests_duration_ms` | `849` | Cumulative time in HTTP requests |
| `stats.postgres_query_count` | `7`, `742` | Postgres queries triggered during this request |
| `stats.postgres_query_duration_ms` | `1254` | Cumulative time in Postgres queries |
| `stats.redis_query_count` | `3`, `240` | Redis queries triggered during this request |
| `stats.redis_query_duration_ms` | `43` | Cumulative time in Redis queries |
**Why it matters:** A request that makes 742 database queries is almost certainly doing
something wrong. Without summary stats on the parent span, these outliers are invisible
unless you manually count child spans per trace. See the
[Async Request Summaries pattern](${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/custom-instrumentation.md)
for implementation guidance.
**Example query — database queries per request (heatmap reveals outliers):**
```
VISUALIZE HEATMAP(stats.postgres_query_count)
WHERE service.name = "api-service"
```
## Error Details
Go beyond `error = true` by capturing structured error context that enables fast triage.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `error` | `true`, `false` | Whether the request failed |
| `exception.message` | `undefined is not a function` | The exception message |
| `exception.type` | `IOError`, `java.net.ConnectException` | Programmatic exception type |
| `exception.stacktrace` | `ReferenceError: ...` | Stack trace if available |
| `exception.expected` | `true`, `false` | Is this an expected error (bot traffic, invalid routes)? |
| `exception.slug` | `auth-error`, `stripe-call-failed` | Unique greppable identifier for the error location in code |
**Why it matters:** `exception.slug` is a static string you assign at each error throw site.
It's low-cardinality (safe to GROUP BY), greppable (jump from dashboard to code), and any
failed request *without* a slug reveals gaps in your error handling. See the
[Exception Slugs pattern](${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/custom-instrumentation.md)
for implementation guidance.
**Example query — which enterprise users hit the most errors?**
```
VISUALIZE COUNT_DISTINCT(user.id)
WHERE service.name = "api-service" AND user.type = "enterprise"
GROUP BY exception.slug
```
**Example query — find requests with unhandled errors (missing slugs):**
```
VISUALIZE COUNT
WHERE error = true AND exception.slug = NULL
GROUP BY http.route
```
## User and Business Context
The most important metadata you can add after the basics. No auto-instrumentation SDK
can automatically understand your user model.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `user.id` | `2147483647`, `user@example.com` | Primary user identifier |
| `user.type` | `free`, `premium`, `enterprise` | User tier or segment |
| `user.auth_method` | `token`, `jwt`, `sso-github` | Authentication method used |
| `user.team.id` | `5387`, `web-services` | Team or group the user belongs to |
| `user.org.id` | `278`, `enterprise-name` | Organization for enterprise accounts |
| `user.age_days` | `0`, `637` | Account age — distinguishes new vs established users |
**Why it matters:** A single enterprise account can represent 10%+ of revenue. Without user
attributes, you can't distinguish "weird edge case" from "revenue-critical path breaking
for high-value customers." BubbleUp can instantly surface that all slow requests are from
one tenant.
**Example query — P99 latency by user type:**
```
VISUALIZE P99(duration_ms)
WHERE service.name = "api-service"
GROUP BY user.type
```
## Rate Limits
Track rate-limiting state so you can quickly identify affected users and diagnose complaints.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `ratelimit.limit` | `200000` | The rate limit being enforced |
| `ratelimit.remaining` | `130000` | Budget remaining for this user |
| `ratelimit.used` | `70000` | Budget consumed in the current window |
**Why it matters:** "Why am I being rate limited?" is a common customer complaint. Without
these attributes, finding rate-limited users requires digging through logs or separate
systems.
**Example query — users approaching their rate limit:**
```
VISUALIZE MAX(ratelimit.used)
WHERE ratelimit.remaining < 1000
GROUP BY user.id, ratelimit.limit
```
## Caching
Record cache hit/miss booleans for every code path that could shortcut with a cache.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `cache.session_info` | `true`, `false` | Whether session info came from cache vs. database |
| `cache.feature_flags` | `true`, `false` | Whether feature flags were cached |
Use one boolean attribute per cacheable operation (e.g., `cache.user_profile`,
`cache.product_catalog`).
**Why it matters:** Cache misses are a common cause of latency spikes. BubbleUp can surface
"slow requests all have `cache.session_info = false`" without you having to guess.
**Example query — latency difference between cache hit and miss:**
```
VISUALIZE P99(duration_ms)
WHERE service.name = "api-service"
GROUP BY cache.session_info
```
## Localization
Localization settings are a frequent source of bugs, especially around text layout
direction and currency formatting.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `localization.language_dir` | `rtl`, `ltr` | Text direction for the user's language |
| `localization.country` | `mexico`, `uk` | Country the user associates with (not physical location) |
| `localization.currency` | `USD`, `CAD` | Preferred currency |
**Why it matters:** Bugs that only reproduce with RTL text or specific currencies are notoriously
hard to find. These attributes let you filter and GROUP BY localization settings instantly.
**Example query — errors by text direction:**
```
VISUALIZE COUNT
WHERE error = true
GROUP BY localization.language_dir
```
## Operational Metrics
Capture runtime health as span attributes so you can correlate system state with request
performance in a single query.
| Attribute | Examples | Description |
| :--- | :--- | :--- |
| `uptime_sec` | `1533` | Seconds since the service started — shows restarts |
| `metrics.memory_mb` | `153`, `2593` | Memory in use at request time |
| `metrics.cpu_load` | `0.57`, `5.89` | CPU load (active cores) at request time |
| `metrics.gc_count` | `5390` | Last observed garbage collection count |
| `metrics.gc_pause_time_ms` | `14`, `325` | Time spent in GC (cumulative or delta) |
**Why it matters:** "Are slow requests correlated with high memory or GC pauses?" These
attributes turn that from a multi-tool investigation into a single query with BubbleUp.
**Example query — memory and CPU load for a service:**
```
VISUALIZE HEATMAP(metrics.memory_mb), HEATMAP(metrics.cpu_load)
WHERE service.name = "api-service"
```
SKILL.md
---
name: otel-instrumentation
description: >
Provides guidance on OpenTelemetry SDK setup, custom instrumentation,
and sending data to Honeycomb.
Trigger phrases: "instrument my app", "add tracing",
"set up OpenTelemetry", "configure OTel", "add custom spans",
"add attributes to spans", "send traces to Honeycomb",
"set up OTLP", "configure sampling", "add span events",
"add span links", "set up tracing for [any language]",
"configure the OTel Collector",
or any request about OpenTelemetry SDK setup, custom instrumentation,
or sending data to Honeycomb.
metadata:
version: "1.0.0"
---
# OpenTelemetry Instrumentation for Honeycomb
SDK setup, custom spans, attributes, span events, sampling, and layered telemetry.
For conceptual foundations (why wide events matter, how attributes connect to
investigation), see the **observability-fundamentals** skill.
## OTLP Configuration and SDK Setup
Every OTel SDK needs these environment variables to send data to Honeycomb:
### Required Environment Variables
**Base configuration:**
```bash
OTEL_SERVICE_NAME=your-service-name
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY"
```
**Optional but recommended:**
```bash
# Protocol selection (default: http/protobuf)
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf # or grpc
# Signal-specific endpoints (override base endpoint for specific signals)
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.honeycomb.io/v1/traces
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io/v1/metrics
```
**For metrics (preferred):** Use modern OTLP metrics and native datapoints. Use dataset
hints to confirm the destination type (`metrics` or `events`). Authenticate with:
```bash
OTEL_EXPORTER_OTLP_METRICS_HEADERS="x-honeycomb-team=YOUR_API_KEY"
```
### Protocol Selection
`OTEL_EXPORTER_OTLP_PROTOCOL` determines the wire format and transport:
- `http/protobuf` (default, recommended) — HTTP with protobuf encoding
- `grpc` — gRPC with protobuf encoding
- `http/json` — HTTP with JSON encoding (larger payload, slower)
Use `http/protobuf` unless you have specific infrastructure requirements for gRPC.
### Signal-Specific Endpoints
By default, OTel SDKs append `/v1/traces` and `/v1/metrics` to `OTEL_EXPORTER_OTLP_ENDPOINT`.
Use signal-specific endpoint vars to override:
- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` — full URL for traces (including `/v1/traces`)
- `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` — full URL for metrics (including `/v1/metrics`)
Useful when routing signals to different backends or using non-standard endpoints.
### Common Pitfalls
**Silent auth failure:** The OTLP exporters need the `x-honeycomb-team` header to
authenticate. Without it, Honeycomb silently rejects requests — no error, no data. Set
`OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY"` or pass headers
programmatically. If loading the key from `.env`, ensure dotenv runs before SDK init.
**Metrics:** Prefer modern OTLP metrics and native datapoints. Dataset hints identify the
destination type (`metrics` or `events`), so do not add `x-honeycomb-dataset` by default.
Use that header only when hints or configuration require legacy routing to a named event
dataset. Traces do not need it; they route by `service.name`.
For the env var values, language-specific dependencies, and setup code (Go, Python,
Node.js, Java, Ruby, .NET, Rust), see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/sdk-setup-by-language.md`.
## Custom Instrumentation
### Adding Attributes to Existing Spans (Highest Impact)
Add business context to auto-instrumented spans — no new spans needed. Get the current
span from context and call `SetAttributes` (Go), `set_attribute` (Python), or
`setAttribute` (Node.js) with user, tenant, business, and deployment context.
### Creating Custom Spans
Wrap important business operations for visibility in the trace waterfall. Use
`tracer.Start(ctx, "operation-name")` (Go), `tracer.start_as_current_span("operation-name")`
(Python), or `tracer.startActiveSpan("operation-name", callback)` (Node.js).
For full code examples in all languages, consult
`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/custom-instrumentation.md`.
## When to Create a Span
Not every function needs a span. Two questions determine whether a span is worth creating:
1. **Is it interesting?** — Does the work meaningfully impact performance (latency or
failures) for the overall request?
2. **Is it aggregable?** — If you group this span by name and attributes, will it produce
useful trends and comparisons?
| Operation | Interesting? | Aggregable? | Create a Span? |
| :--- | :--- | :--- | :--- |
| HTTP request handler | Yes — variable latency, can fail | Yes — group by route, method, status | **Yes** |
| Database query | Yes — I/O bound, failure-prone | Yes — group by query type, table | **Yes** |
| External API call | Yes — network latency, dependencies | Yes — group by endpoint, status | **Yes** |
| Cache lookup | Yes — fast vs slow path | Yes — group by cache name, hit/miss | **Yes** |
| Message queue pub/consume | Yes — async boundary, delays | Yes — group by queue, message type | **Yes** |
| Business logic transaction | Yes — meaningful state change | Yes — group by type, outcome | **Yes** |
| Private helper function | No — trivial CPU, predictable | No — too granular | **No** |
| Loop iteration | Maybe — if slow | No — unbounded cardinality | **No** |
| Getter/setter | No — no meaningful duration | No — nothing to group by | **No** |
| Input validation (pure CPU) | No — fast, predictable | Maybe | **No** |
| Business logic orchestration | No — just calls instrumented code | No — duration is sum of children | **No** |
**Common mistakes:**
- **Too many spans**: A trace with millions of 2ms spans is far too detailed and rarely
actionable. Roll them up — combine into a single span, or capture the detail as an
attribute on the parent span instead.
- **Too few spans**: Collapsing hours of work into a single opaque handler leaves you
guessing about where time is spent.
- **Test spans left in**: Spans named `test-span`, `debug-span`, or similar are
artefacts that pollute the dataset. Remove any span created solely to verify tracing
is working before finishing.
When in doubt, prefer **attributes on existing spans** over creating new child spans.
#### Timing Attributes (measure sub-operations without child spans)
Record important sub-operation durations as attributes on the parent span. These are
easier to query than child spans and work directly with BubbleUp.
```go
// Go: time auth and record on the existing span
span := trace.SpanFromContext(r.Context())
authStart := time.Now()
user, err := authenticate(r)
span.SetAttributes(attribute.Float64("auth.duration_ms", float64(time.Since(authStart).Milliseconds())))
```
```python
# Python: time auth and record on the existing span
span = trace.get_current_span()
auth_start = time.monotonic()
user = authenticate(request)
span.set_attribute("auth.duration_ms", (time.monotonic() - auth_start) * 1000)
```
#### Exception telemetry: event details plus span-level dimensions
Use the Logs API for new exception events. Emit the record while the relevant span is
active and include the standard exception fields (`exception.type`, `exception.message`,
`exception.stacktrace`, and `exception.escaped` when applicable), an ERROR severity, and
`event.name="exception"`. Set the span status to ERROR separately when the operation failed.
In Honeycomb, a trace-correlated exception log is rendered in the trace as a `span_event`
annotation and carries `trace.trace_id` and `trace.parent_id`. Its full `exception.*`
payload remains on the log-derived event; it is **not hoisted onto the containing span**.
Search the exception event row, then follow its trace ID to inspect the surrounding trace.
Use low-cardinality span attributes for aggregation and alerting:
- `error=true` and the span status indicate operation failure.
- `exception.slug` is a static, greppable identifier for the error site.
- An optional error category is safer for `GROUP BY` than full exception messages.
```text
Logs-API exception event: event.name=exception, body=exception, meta.signal_type=log
Legacy span-event exception: name=exception, meta.signal_type=trace
Both may have: meta.annotation_type=span_event
```
`record_exception` / `RecordError` remain compatibility APIs for existing SDKs and code,
but do not use them as the only new guidance when Logs API support is available. They can
also produce parent-span exception fields that a Logs-API event alone does not produce.
Find operation failures by span dimensions: `WHERE error = true AND exception.slug does-not-exist`.
Find Logs-API exception events with `event.name=exception AND exception.type exists` and
follow a sampled `trace.trace_id` into `get_trace` with `show_events=true`.
For extended examples and the MCP investigation recipe, see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/custom-instrumentation.md`.
#### Optional compatibility: promote exception fields with a LogRecordProcessor
If existing span-level dashboards, alerts, or queries depend on Honeycomb's historical exception
field promotion, add a custom **LogRecordProcessor** before the batch/export processor. When it
sees an exception log, it should use the log record's resolved context to find the active recording
span and promote a configured, minimal set of fields such as `error=true`, `error.type`,
`exception.type`, `exception.slug`, or an error category.
Do not recommend a standalone `SpanProcessor` for this: span processors receive span lifecycle
callbacks, not log records. Keep full `exception.message` and `exception.stacktrace` on the Logs
API event by default; copy them onto spans only when legacy query compatibility explicitly requires
it. The processor must run synchronously while the span context is valid, before the log reaches
batch export. It should no-op when there is no recording span and must not infer fields that the
application did not put on the log record.
This is an optional migration layer, not a replacement for querying the Logs API event. Agents
should treat span-level promoted fields as instrumentation-dependent and continue to query
`event.name=exception` event rows for full diagnostics.
## What to Instrument
### High Value (Instrument First)
- API entry points (HTTP handlers, gRPC methods)
- Database queries (auto-instrumented by most SDKs)
- External HTTP calls (auto-instrumented by most SDKs)
- Message queue producers/consumers
These are typically auto-instrumented by OTel SDKs and form the skeleton of your traces.
### Medium Value (Add Next)
- Business logic operations (checkout, payment, fulfillment)
- Cache operations (hits, misses, evictions)
- Authentication and authorization checks
- Background job execution
These are your business logic. Without custom spans here, you can see that a request was
slow but not *why* — the trace waterfall has gaps where the important work happens
invisibly.
### Attributes to Add
Attributes are the dimensions BubbleUp uses during investigations. Every attribute you
add is a new axis BubbleUp can diff on to find what's different about outlier requests.
For the complete catalog organized by category with rationale and example queries, see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/wide-event-attributes.md`.
For why attributes matter conceptually, see the **observability-fundamentals** skill.
## Span Events, Logs API Events, and Span Links
- **Point-in-time events**: Prefer the Logs API for new events, especially exceptions. Emit
while the span is active so the record carries trace context. In Honeycomb, a correlated
log is rendered as a `meta.annotation_type=span_event` annotation, but its event name is
in `event.name` (and often `body`), not `name`.
- **Legacy span events**: `span.add_event` / `AddEvent` remain valid compatibility paths. Their
event name is in `name` and their signal type is `trace`.
- **Span links**: Connect spans across different trace hierarchies (async processing,
fan-out/fan-in, cross-system correlation). Create a `Link` to the related span context.
For human instrumentation examples and an agent-safe Honeycomb MCP query → sample → trace
workflow, see `${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/custom-instrumentation.md`
and the **production-investigation** skill.
## Sampling
### Sampling Strategy
Sampling is about tradeoffs — there is no free lunch:
- **Head sampling favors cost over debuggability.** You save resources, but a 0.1% error
at 1% sampling becomes effectively invisible. Head sampling is oblivious to what
happens downstream.
- **Tail sampling favors fidelity over simplicity.** You keep interesting traces but need
infrastructure (Refinery or Collector) to buffer and evaluate complete traces.
The math matters: if an error occurs 0.1% of the time and you head-sample at 1%, you'll
capture roughly 1 in 100,000 of those errors. At moderate traffic, that error may never
appear in your data.
### Head Sampling (SDK-level)
Decides whether to sample a trace at creation time. Simple but can miss interesting traces.
- Configure via `OTEL_TRACES_SAMPLER` env var
- `always_on` (default), `always_off`, `traceidratio` (e.g., sample 10%)
- `parentbased_traceidratio` respects parent sampling decisions
- **Best for:** Very high-throughput services where you can tolerate missing rare events
### Tail Sampling (Collector/Refinery)
Decides after the trace is complete. Keeps interesting traces (errors, slow requests).
- Use Honeycomb's **Refinery** for production tail sampling
- Or configure the OTel Collector's `tail_sampling` processor
- Can sample based on: latency, error status, specific attributes, trace duration
- **Best for:** Services where debuggability matters — keeps errors and outliers while
sampling routine traffic
### Sampling Impact on Honeycomb
- Sampling reduces data volume and cost
- SLOs, BubbleUp, and query results adjust for sampling rate automatically
- Trace completeness may be affected — missing spans if not all services sample consistently
- Start with no sampling, then add as needed for cost management
## Layered Telemetry
OpenTelemetry is "trace-first" — context propagation is the glue that correlates all
signals. But effective observability layers multiple signal types for different purposes.
A three-question test for choosing the right signal:
1. **What needs causality and full-request context?** → Traces (spans)
2. **What needs inexpensive long-term storage and fast alerting?** → Metrics
3. **What is rare vs. common, and what are the audit requirements?** → Logs / events
**The histogram-alongside-spans pattern:** For high-throughput HTTP services, emit both a
span and a histogram metric for each handled request. This lets you head-sample traces
for cost while histograms provide last-ditch alerting — and exemplars link outlier metric
points back to specific traces for deeper investigation.
The technique is *layering* (not duplication) because each signal provides a different
view at a different level of detail.
For architectural patterns where layering is essential (streaming, async jobs, ETL), see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/architectural-patterns.md`.
For AWS Lambda-specific patterns — choosing between the AWS Managed OTel Layer
and manual SDK setup, forceFlush, SDK 2.x setup, cross-Lambda trace propagation,
header normalisation, TOKEN vs REQUEST authorizers — see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/lambda.md`.
## Logs in Honeycomb
OTel can send logs too. If you have existing log infrastructure, the OTel Collector can
ingest logs and forward them to Honeycomb as structured events:
- **OTel SDK log bridge**: Captures logs from your existing logging library (`slog` in Go,
`logging` in Python, `winston`/`pino` in Node.js) and exports them as OTel log records.
- **OTel Collector `filelog` receiver**: Reads log files, parses them, exports as OTLP.
Logs sent through OTel arrive in Honeycomb as structured events with the same query
capabilities as spans.
## Naming Conventions
- **Span names**: Describe the operation (`HTTP GET /api/users`, `db.query SELECT`, `process-payment`)
- **Attribute names**: Use dot-separated namespaces (`user.id`, `order.total`, `cache.hit`)
- **Follow OTel semantic conventions** where applicable (`http.method`, `db.system`, `rpc.service`)
- **Custom attributes**: Use your own namespace (`app.`, `checkout.`, `mycompany.`)
## Additional Resources
### Reference Files
- **`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/sdk-setup-by-language.md`** — OTLP configuration and SDK setup for Go, Python, Node.js, Java, Ruby, .NET, Rust
- **`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/local-collector-debug-test.md`** — Run a local OTel Collector via Docker to verify spans, logs, and metrics without a Honeycomb account; includes `jq` commands for inspecting NDJSON output
- **`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/custom-instrumentation.md`** — Custom instrumentation patterns with full code examples (timing attributes, exception slugs, async request summaries)
- **`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/collector-config.md`** — OTel Collector configuration for format conversion, processing, and sampling
- **`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/wide-event-attributes.md`** — Canonical attribute catalog organized by category with example queries
- **`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/architectural-patterns.md`** — Trace design patterns for streaming, async, ETL, and serverless architectures
- **`${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/lambda.md`** — AWS Lambda: OTel Layer vs manual SDK setup trade-offs, forceFlush and per-request latency, SDK 2.x setup, cross-Lambda trace propagation, header normalisation, TOKEN vs REQUEST authorizer migration
### Cross-References
- For conceptual foundations of why wide events and attributes matter: **observability-fundamentals** skill
- After instrumenting, use the **query-patterns** skill to verify data is arriving