Module 4: Measure and Improve the Factory

Measure what your factory ships

The number of features an engineer ships is a failure metric. Judge the factory by its efficiency instead:

Count human touches per PR

Track human touches per PR, and push the number down over time. Here is how the touches add up on a relatively complex feature, such as search, sort and filter for a database-driven view:

For the outside view, track the standard DORA metrics: PR merge rate, cycle time, defect rate and time to fix errors. If they all improve as agent use grows, your agents deliver value.

Build a small eval suite from real failures

Without evals, you debug reactively: you wait for complaints, reproduce by hand and hope nothing else regressed. Teams with evals also adopt new models in days, while teams without them face weeks of tests, according to Anthropic's engineers.

Start with 20 to 50 tasks

Draw 20 to 50 simple tasks from real failures in your bug tracker and support queue. For coding agents, unit tests check correctness and an LLM rubric assesses code quality. Use deterministic graders where you can, and LLM graders only where you must.

Agent output varies between runs, so run several trials of each task. One pass tells you little. At a 75% success rate per trial, the chance that all three trials pass is (0.75)³, or about 42%. Measure that consistency for agents that users expect to work every time.

Split capability evals from regression evals

Capability evals ask what the agent can do well. They start at a low pass rate and give you a hill to climb. Regression evals check that the agent still handles everything it used to, and they should pass nearly every time:

Keep the suite trustworthy

• Write unambiguous tasks. Two domain experts should independently reach the same verdict, and each task needs a reference solution that passes.
• Isolate every trial. Start each one from a clean environment, because shared state can inflate scores.
• Grade the outcome, not the path. Agents find valid approaches that eval designers didn't anticipate.
• Name owners. Let the people closest to the product and its users add tasks.

Score agent sessions with an LLM judge

Agents leave a complete record of their work, so you can grade past sessions with an LLM as the judge. Set it up in four steps:

1. Store complete traces behind an API: every prompt, tool call and MCP result, plus every PR, spec and screenshot the agent created.
2. Define scoring agents. Each one grades a single dimension with a prompt, classification instructions and a judge model.
3. Sample runs by percentage, or by a rule such as "score all my front-end tasks." In Warp's own factory, scoring takes about 3% of total token costs.
4. Schedule the scorers with a cron-based cloud agent.

What to score

Worked example: a scorer for redundant tests

Redundant tests were a common failure in Warp's factory, so the team built a scorer for them. The scorer has three parts:

• A judging prompt: a rubric for what counts as a redundant test.
• Output classifications: the results that count as a pass.
• A sampling rate: the percentage of runs to score.

When it runs, the scorer loads an agent trace, brings all its inputs and outputs into context, and prompts an LLM to judge the run. The pass rate showed that agents mostly avoided redundant tests, with a few failing runs each day.

Open failures like these, and check what the coding agent did. Check the scorer's own run too, since the scorer is just another agent. Look for patterns, then adjust your skills so agents write tests more sparingly.

Keep your judges honest

• Calibrate against humans. Compare the judge with human experts until little divergence remains.
• Give the judge a way out. Tell it to return "Unknown" when it lacks enough information.
• Read the transcripts. Grading bugs can produce low scores even when the agent performs well:

Build a self-improvement loop

You can automate the outer loop with agents too. Ben Holmes, a software developer at Warp, runs one on his team's factory to improve agent skills. The team has merged quite a few skill changes from it. For more patterns, see our loop engineering guide.

Feed the loop more than scores

• Every human intervention. Record it, and have self-improvement agents change the flow so a human is less likely to step in again.
• Token waste. Run agents over factory conversations to find patterns of waste and suggest efficiency gains.
• Harnesses and prompts. Eval them, and note which work best for the next run.

Point outer-loop agents at code architecture too, and have them add static analysis checks. These checks are even better guardrails than skills, because they stop bad code before it spreads. Before you roll out a new model configuration, benchmark it against your scorers.