Executive Summary
- A 90% success rate on public benchmarks often masks severe step variance and cost unpredictability.
- Step count variance measures the deterministic repeatability of the agent's reasoning DAG.
- Tool invocation efficiency evaluates whether the agent calls tools with minimal retries and zero hallucinated parameters.
- Cost predictability tests calculate the standard deviation of token spend across identical task variations.
- Safety boundary compliance verifies that the agent refuses unauthorized actions 100% of the time under adversarial evaluation.
The fundamental flaw of aggregate success rates
In AI benchmarking literature (such as SWE-bench or GAIA), models are ranked by a single percentage: 'Task Success Rate: 84%'.
In enterprise software engineering, this number is dangerously deceptive. An agent that succeeds 84% of the time by brute-forcing 45 tool retries and spending $12.00 per task is fundamentally unusable in production compared to an agent that succeeds 82% of the time in 3 deterministic steps costing $0.05.
Production readiness requires evaluating the stability, variance, cost, and safety boundaries of the execution trajectory.
The Trajectory Principle
In production agent systems, how an agent reaches a solution is just as critical as whether it reaches the solution.
The four dimensions of agent reliability
A rigorous reliability framework evaluates four distinct vectors:
1. Trajectory Variance: Running the identical prompt 20 times to measure the standard deviation of execution steps and plan structure.
2. Tool Invocation Precision: The percentage of tool calls executed with zero schema validation errors on first attempt.
3. Cost and Token Predictability: Ensuring p99 token spend remains within a narrow budget envelope without exponential outlier spikes.
4. Adversarial Invariance: Verifying that adversarial perturbations and distractor documents do not induce prompt injection or plan corruption.
Academic benchmarks vs SRE evaluation harnesses
Comparing the metrics, test depth, and operational validity of evaluation approaches.
Evaluation paradigms compared
| Feature | Dimension | Academic Benchmarking (Single-Pass) | SRE Reliability Harness (Multi-Run Stochastic) |
|---|---|---|---|
| Execution Iterations | Single execution per task prompt | 10-20 repeated runs with temperature seeds | |
| Metric Focus | Binary Success / Failure | Step count standard deviation, tool retries, & cost p99 | |
| Tool Efficiency | Ignored (Unlimited tool loops permitted) | Penalizes redundant queries and validation errors | |
| Cost Attribution | Not tracked | Exact dollar cost measured per successful run | |
| Production Predictability | Poor (High variance in production) | High (Guarantees bounded latency and spend) |
Automated multi-run evaluation harness in TypeScript
Below is a TypeScript implementation of a multi-run evaluation runner calculating trajectory variance and cost metrics.
Calculating step variance and entropy across runs
When an agent produces wildly different execution paths for identical inputs (e.g. 3 steps on run 1, 14 steps on run 2), it indicates high reasoning entropy.
High entropy causes unpredictable latency spikes and makes capacity planning impossible. Production systems require prompt grounding and structured planning to constrain step variance to +/- 1 step.
Continuous CI/CD reliability regression testing
Every change to system prompts, tool schemas, or model versions must trigger automated evaluation suites in GitHub Actions.
If a prompt update increases step variance by more than 15% or introduces tool validation retries, the pull request is automatically blocked.
Agent reliability evaluation checklist
Ensure your AI testing pipeline incorporates these reliability evaluation standards.
Evaluation readiness checklist
1Multi-Run Testing
- Evaluation suites run tasks at least 10 times across different seeds
- Step count standard deviation is tracked and alerted on in CI/CD
- Tool retry frequencies and validation error rates are monitored
2Cost & Safety Metrics
- p95 and p99 token spend envelopes are established per task type
- Adversarial injection test suites run on every prompt update
- Regression thresholds block pull requests that increase trajectory entropy