An agent is a model inside a loop with tools, memory, and some authority to act. The final answer is only one part of the product. You must inspect the actions, evidence, approvals, retries, and stopping behavior that produced it.

Begin with the smallest useful authority. Let the agent propose before it executes, read before it writes, and work in a sandbox before it reaches production systems.

Define authority and success before testing

List the tools the agent may call, the data each tool can expose, the actions that require approval, and the actions that are forbidden. Put these rules in one policy rather than scattering them through prompts. Define task success in terms of changed state and evidence, not a confident final message.

Give every action an idempotency or recovery strategy. A retry must not send the same email twice, create duplicate records, or repeat a purchase. Set budgets for turns, tool calls, tokens, elapsed time, and monetary cost.

Build scenarios around failure paths

Test successful work, missing data, stale data, tool errors, timeouts, rate limits, permission denial, conflicting instructions, malicious content returned by a tool, and a task whose correct result is to stop. Include partial success where some state changed before the failure.

Score the trace for unnecessary actions, boundary violations, evidence quality, approval compliance, recovery, and whether the final state matches the requested outcome. Preserve the trace and tool results for every failure.

  • Did the agent select the right tool and arguments?
  • Did it verify that the action produced the expected state?
  • Did it request approval at the actual boundary?
  • Did it stop when another attempt was unlikely to help?
  • Can a reviewer explain the decision from the trace?

Release with gates and a kill path

Turn every meaningful failure into a regression case. Block release when a known safety or correctness case reappears. Start with a limited user group, restricted tools, small budgets, and visible monitoring.

Provide a way to pause the agent, revoke credentials, inspect queued actions, and reverse changes where possible. Model quality cannot compensate for an operational system that lacks control.

If the agent cannot explain what changed, it is not ready for more authority.

Primary sources

Verify before you commit money or architecture