Your model works in the demo but fails in production. The issue isn't your prompt engineering, it's your harness. An agent harness is the operational framework that turns a stateless language model into a production-ready agent. Without it, you're just running inference calls, not agents.
This checklist outlines the five core components that distinguish a functional agent from a potential liability. Each item includes the specific control you're implementing and what success looks like in practice.
Prerequisites
Before you start this checklist, ensure:
- You have a defined agent task scope with clear success criteria.
- You've identified all external systems the agent must interact with (APIs, databases, file systems).
- You have a staging environment that mirrors production constraints.
- You've documented your current context window limits and average task duration.
If you're missing any of these, stop. You can't validate harness controls without a stable baseline to test against.
Harness Readiness Checklist
Sandboxed Execution
1. Agent code executes in isolated containers with restricted permissions
Your agent should never run with the same privileges as your application server. Use containerized environments with mounted volumes for workspace persistence and process-level isolation.
Success looks like: A file system access attempt outside the mounted workspace returns a permission error that the agent can detect and handle. Network calls to non-allowlisted endpoints fail immediately with a clear error message.
2. File system access is constrained to explicit allowlisted paths
Default deny. The agent should only read from and write to directories you've explicitly permitted for its task scope.
Success looks like: Your allowlist is version-controlled alongside agent configuration. When a new task type requires broader file access, you update the allowlist through your change management process, not by granting blanket permissions.
State and Memory Management
3. Context compaction triggers automatically before window exhaustion
The most common agent failure in production is context window exhaustion. Set a hard limit at 75-80% of your model's context capacity and implement summarization or offloading when you hit it.
Success looks like: Your monitoring dashboard shows context consumption per step. You have alert thresholds set at 70% and 85%. When an agent hits the 85% mark, the harness automatically summarizes older conversation turns and persists the summary to external storage.
4. Task state persists across agent restarts
When an agent crashes or times out mid-task, the next instance should resume from the last known good state, not start over.
Success looks like: Your harness maintains a task journal with timestamped checkpoints. Each checkpoint includes conversation history, file modifications, intermediate results, and tool call outcomes. A restarted agent loads the most recent checkpoint and continues from there.
Tool Execution and Control
5. Tool call preconditions are validated before execution
An agent that attempts to read a file before it exists or calls an API with malformed parameters is wasting resources and accumulating technical debt. Validate preconditions before the tool fires.
Success looks like: Your tool registry includes a precondition schema for each tool. The harness checks parameters, required state, and dependency satisfaction before routing the call. Failed preconditions return structured error messages that the agent can reason about.
6. Rate limits are enforced at the harness layer
API providers impose rate limits. Your harness should enforce them before the provider does.
Success looks like: You've configured per-tool rate limits based on provider documentation. When the agent attempts to exceed a limit, the harness queues the request or returns a retry-after signal. Your monitoring tracks rate limit proximity as a first-class metric.
7. High-risk operations require explicit approval gates
Not every tool call should execute automatically. Destructive operations, external communications, and financial transactions need human review.
Success looks like: Your tool registry tags high-risk operations with an approval requirement. When the agent attempts one, the harness pauses execution, logs the request with full context, and waits for operator approval. The approval workflow includes a timeout, if no human responds within your SLA, the task fails safely.
Context Engineering
8. Context assembly is dynamic and task-specific
Don't dump your entire knowledge base into the prompt. Curate what enters the context window based on the current task state.
Success looks like: Your harness maintains a context budget per step. It prioritizes system prompts, active task instructions, and recent conversation history. Retrieval results are ranked by relevance and truncated to fit the remaining budget. Older context gets summarized or offloaded, not deleted.
9. System prompts and skill definitions are version-controlled
Your prompt is infrastructure. Treat it like code.
Success looks like: System prompts live in version control alongside your harness configuration. Changes go through pull requests with peer review. You can roll back to a previous prompt version if a change degrades agent performance.
Feedback Loops and Self-Correction
10. Computational checks run after every agent output
Linters, test suites, and schema validators are deterministic and fast. Use them.
Success looks like: For code generation tasks, a linter runs automatically after each generation step. For API responses, a schema validator confirms the structure matches expectations. Failed checks return structured feedback to the agent with correction guidance.
11. Inferential checks evaluate semantic quality
Some failures can't be caught with deterministic rules. LLM-as-a-Judge evaluations assess output relevance, safety, and task alignment.
Success looks like: You've defined evaluation criteria for your task category (code correctness, answer relevance, safety compliance). An evaluation model scores each agent output against these criteria. Scores below your threshold trigger a revision loop with specific feedback.
12. Drift detection monitors long-running tasks
Agents that run for hours can gradually drift from their objective without any single step being obviously wrong.
Success looks like: Your harness sets periodic checkpoints (every 10 steps or every 15 minutes, whichever comes first). At each checkpoint, it evaluates whether the agent is still making progress toward the original objective. If drift is detected, the harness can pause for human review or reset to the last good checkpoint.
Common Mistakes
Treating the harness as a one-time setup: Your harness evolves with your agent. When a new failure mode appears in production, trace it back to the harness layer where it should have been caught and add the control there.
Over-relying on prompt engineering: If an agent consistently fails at a task category, the problem is your harness, not your instructions. Add feedforward controls (allowlists, context curation rules) or feedback controls (validators, evaluators) instead of rewriting prompts.
Granting permissions reactively: Start with default deny. Explicitly allowlist every tool, file path, and network endpoint. Widen permissions based on observed need during controlled testing, not in response to production failures.
Next Steps
Your harness is operational infrastructure, not a feature. Schedule quarterly reviews to:
- Audit your tool allowlist against actual usage patterns.
- Review context consumption metrics and adjust compaction thresholds.
- Analyze failure logs to identify recurring patterns that need new controls.
- Update your approval gate criteria based on observed agent behavior.
The shift from model-centric to harness-centric thinking defines how production AI engineering matures. Teams that treat harness engineering as an ongoing discipline build agents that scale.
