Warp Factories adds LLM-as-a-judge scorers for agent evaluation

Warp has published a new software-factory workflow built around scorers: agents that grade previous coding-agent sessions with an LLM-as-a-judge. The September 18, 2026 post describes scoring as a way to measure and improve agent performance instead of relying only on public benchmarks or intuition.

Score the work an agent actually did

Warp’s approach starts with a record of prior agent traces. A useful trace includes the agent’s inputs and outputs, conversation, and the surrounding work context so a scorer can judge the complete run rather than a final answer alone.

A scoring agent takes a prior trace and returns a grade. Each scorer can focus on one dimension, such as quality, efficiency, verbosity, cost, compliance, or another criterion a team wants to track. The scorer is defined by its prompt, classification instructions, and judge model.

The resulting scores can be stored and viewed in aggregate. Over time, teams can use them as a measurement layer for the software factory, similar to tracking delivery metrics while keeping the evaluation tied to actual agent work.

From scoring to improvement

Warp describes a loop in which scheduled scorers grade past sessions, then observer agents examine the results and suggest changes to prompts, skills, model choices, or context. Scorers can also supply the evaluation layer for benchmarks that compare configurations on the same set of tasks.

This is useful only when the rubric is stable and reviewed. LLM-as-a-judge can drift or reward surface patterns, so validate scorers against human judgments, monitor disagreement, and keep high-impact changes behind an approval step.

Relation to Factory Benchmarks

Warp’s earlier Factory Benchmarks release uses team coding tasks to compare models, harnesses, and scorers. The September scoring workflow explains the measurement primitive behind that evaluation: traces are graded along explicit dimensions, and those scores can inform a configuration or router.

Teams should version the task set, scorer prompt, judge model, and acceptance thresholds together. A configuration that wins on historical traces may not be best for a new repository, so watch for drift when routing live work based on benchmark results.

Sources

• Warp — “Using LLM-as-a-judge scoring to measure your software factory” (September 18, 2026): https://www.warp.dev/blog/using-llm-as-a-judge-scoring-to-measure-your-software-factory
• Warp — “Launch Factory Benchmarks” (September 3, 2026): https://www.warp.dev/blog/warp-factory-benchmarks
• Warp Docs — 2026 changelog: https://docs.warp.dev/changelog/2026/
• Warp on X — official product updates: https://x.com/warpdotdev