Your distributed tracing captures every microservice call. Your APM dashboards show green. Yet, your compliance team just found three policy violations in a production AI workflow before lunch.
This checklist helps you determine whether your AI observability instrumentation meets the minimum threshold for governance oversight. If you're deploying multi-agent systems, retrieval-augmented generation, or any AI workflow that chains multiple model calls, you need telemetry that captures decision lineage, not just latency percentiles.
What This Checklist Covers
This checklist evaluates whether your AI telemetry layer provides the structured visibility required for audit readiness and incident investigation. It focuses on OpenTelemetry as the instrumentation standard because it's vendor-neutral, supports distributed tracing across agent hierarchies, and provides semantic conventions specifically for AI workloads. You'll verify that your telemetry captures AI-specific signals, integrates with your evaluation layer, and handles data governance constraints.
Prerequisites
Before starting this checklist, confirm:
- You're running AI workloads in production (LLM calls, agent orchestration, or retrieval systems).
- You have access to your observability backend configuration.
- You can modify instrumentation in at least one AI application.
- You understand your organization's data retention and PII handling requirements.
Checklist Items
1. Vendor-neutral instrumentation is deployed
Verify that your AI applications use OpenTelemetry instrumentation libraries, not proprietary SDKs from your observability vendor. Check for imports from opentelemetry, openlit, or framework-native OTel support in LangGraph or CrewAI.
Good looks like: Switching observability backends requires configuration changes only, not code rewrites.
2. GenAI semantic conventions are applied to LLM calls
Inspect spans for AI-specific attributes: gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. These attributes must appear on every LLM invocation span.
Good looks like: You can query your backend for all traces where gen_ai.request.model = "gpt-4o" and get accurate token consumption across providers.
3. Full agentic hierarchy is captured in distributed traces
For a multi-agent request, verify that the trace shows the complete call graph: orchestrator → sub-agents → tool calls → LLM invocations. Each step must appear as a child span with parent-child relationships intact.
Good looks like: Given a user request ID, you can reconstruct every agent decision and tool invocation in sequence without correlating timestamps manually.
4. Multi-turn conversation context is explicitly propagated
If your AI system handles multi-turn conversations, confirm that session-level context propagates across turns. Check whether turn 2 of a conversation appears as a sibling or child of turn 1 in the trace hierarchy.
Good looks like: A five-turn conversation appears as a single trace with five top-level spans, not five disconnected traces.
5. Prompt and completion capture aligns with data governance policy
Check the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT environment variable. If set to true, verify that your OTel Collector applies redaction processors before exporting to your backend. If set to false, confirm you have an alternative mechanism for debugging production issues.
Good looks like: You can review full prompts for debugging without exposing PII to your observability vendor, or you've documented why content capture is disabled and how you'll troubleshoot without it.
6. Span volume is controlled with sampling policies
For high-volume endpoints, verify that head-based sampling is configured (start with 10%) and tail-based sampling captures 100% of error traces. Check your collector configuration for sampling processor rules.
Good looks like: Your observability backend costs scale linearly with traffic, and you retain complete traces for every error condition.
7. Evaluation results are linked to their originating traces
Confirm that evaluation scores (faithfulness, toxicity, policy compliance) appear as span attributes or events within the same trace that generated the output being evaluated. This requires integration between your telemetry layer and your evaluation layer.
Good looks like: When an output fails a safety check, the trace shows both the LLM call and the evaluation score that flagged it, with a clear parent-child or span-event relationship.
8. Evaluation latency is measured separately from request latency
Verify that evaluation spans are instrumented with their own duration metrics. If evaluation runs synchronously, confirm it completes in under 100ms. If asynchronous, confirm evaluation results are correlated back to the original trace.
Good looks like: You can calculate the Evaluation Trust Tax (the per-query cost of running LLM judges or external evaluation APIs) from your telemetry data.
9. Token costs are attributed to business units or applications
Check whether spans include attributes that map to cost centers: application name, team identifier, or customer ID. Combine these with gen_ai.usage.input_tokens and gen_ai.usage.output_tokens to calculate cost attribution.
Good looks like: You can generate a monthly report showing token consumption and estimated costs per application without manual spreadsheet work.
10. Trace retention policy matches your audit requirements
Verify that your observability backend retains traces for the period required by your compliance framework. For financial services under SR 11-7, this typically means 12 months minimum. For EU AI Act high-risk systems, consider whether your retention supports Post-Market Monitoring obligations.
Good looks like: When an auditor requests evidence of model behavior from six months ago, you can retrieve the full trace without restoring from archive.
Common Mistakes
Treating OpenTelemetry as a complete observability solution. OpenTelemetry captures telemetry. It doesn't evaluate whether outputs are faithful, safe, or compliant. You need a separate evaluation layer.
Enabling full content capture without redaction. Setting OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true in production without collector-side redaction exposes you to data governance violations.
Assuming trace context propagates automatically in agentic workflows. Multi-turn conversations and long-running agent sessions require explicit session management. Default request-response tracing loses context.
Ignoring span explosion in complex agent hierarchies. A single user request can generate hundreds of spans. Without sampling, you'll overwhelm your backend and your budget.
Next Steps
If you checked fewer than eight items, your telemetry layer has gaps that will surface during the next audit or incident investigation. Prioritize items 1, 2, and 3 first. They provide the foundation for everything else.
If you checked all ten, verify that your evaluation layer integrates cleanly with your telemetry. The convergence of structured telemetry and in-environment evaluation is what makes AI observability audit-ready. Without it, you're collecting data you can't act on.
