Executive Summary
- Superficial text diff metrics (BLEU, string distance) fail because there are infinite correct ways to write software.
- SWE-bench execution evaluation tests agents against real-world GitHub issues with ground-truth test assertions.
- Evaluation harnesses spin up clean, ephemeral Docker sandboxes (10-15s lifecycle) for every test sample.
- Pass/Fail criteria are binary and empirical: does the code compile, pass existing tests, and resolve the issue test?
- Continuous evaluation tracking in CI/CD prevents model degradation during prompt engineering or framework updates.
The failure of static diff and string matching evaluation
In early academic NLP research, code generation was evaluated using text similarity metrics like BLEU or exact token match against a reference solution written by a human.
In real-world software engineering, this is catastrophic. If an AI agent refactors a function to use modern functional composition or introduces cleaner variable names, a text diff will mark it as a 90% failure, even if the code is objectively superior and 100% bug-free.
Conversely, an agent that reproduces the exact variable names of the prompt but introduces a silent memory leak will receive a high text similarity score.
The only valid test of software correctness is execution.
The Empirical Standard
Software is not literature; it is a running machine. You cannot judge software by comparing words; you must run the unit tests and inspect the exit code.
The SWE-bench execution model: Real tests in real sandboxes
The SWE-bench paradigm pioneered execution-based evaluation:
1. Problem Instance: A real GitHub issue description and a clean Git checkout at the commit prior to the fix.
2. Agent Trajectory: The agent is given shell tools, file editing capabilities, and a terminal.
3. Verification Harness: The harness applies a hidden verification test patch and runs the test suite. If all existing regression tests pass AND the new issue test passes, the task is marked as `RESOLVED`.
Text Diff Matching vs Execution-Based Sandbox Evaluation
Evaluating metric fidelity, handling of refactorings, and infrastructure complexity.
Evaluation approaches compared
| Feature | Dimension | Static Diff / String Matching | Execution-Based Sandbox (SWE-bench) |
|---|---|---|---|
| Fidelity to Real-World Value | Near Zero (Misleading surface similarity) | Maximum (True ground-truth functionality) | |
| Handling of Clean Refactoring | Fails (Scores refactored code as wrong) | Flawless (Tests pass regardless of syntax style) | |
| Catching Syntax & Import Errors | Poor | 100% (Compiler immediately fails the build) | |
| Infrastructure Complexity | Trivial (Simple string comparison script) | Moderate (Requires Docker/MicroVM orchestration) | |
| Industry Adoption | Deprecated | Gold standard across frontier AI research labs |
Ephemeral Docker evaluation runner in TypeScript
Below is a TypeScript implementation of an execution-based evaluation harness executing coding agent tasks in Docker.
Managing sandbox isolation, timeouts, and parallel execution
Running hundreds of evaluations in parallel requires strict resource budgeting. Containers must be capped at 4GB RAM and 2 CPU cores, with a hard 10-minute timeout.
Containers run with `networkMode: 'none'` to guarantee that untrusted code written during agent exploration cannot access corporate networks or exfiltrate environment data.
Curating internal enterprise coding benchmarks from Git history
Organizations should curate proprietary evaluation benchmarks by mining their own Git history: extracting past closed bug tickets, git commits, and corresponding test cases.
This ensures coding agents are benchmarked against the company's actual architecture patterns and business domains.
Coding agent evaluation harness checklist
Audit your AI evaluation infrastructure against these empirical standards.
Evaluation harness readiness checklist
1Execution & Sandboxes
- Evaluation runs tests in isolated ephemeral Docker or MicroVM containers
- Containers enforce strict CPU, RAM, and network isolation boundaries
- Static text diff metrics are discarded in favor of binary test exit codes
2Dataset & CI/CD Tracking
- Internal evaluation suites cover high-frequency domain workflows and bug patterns
- Automated nightly benchmarks track coding agent resolution rates over time
- Failed trajectories are captured with full shell logs for prompt and tool debugging