Warp Factories adds benchmarks built from your coding tasks

Warp has added Benchmarks to Warp Factories Early Access. The September 3, 2026 feature generates a model benchmark from a team’s own coding tasks instead of relying only on public suites such as SWE-bench or Terminal-Bench.

Evaluate the work your team actually does

A benchmark contains a set of agent tasks, factory configurations, and scorers. Tasks can come from prior runs or be created from scratch. Configurations can vary the model, harness, and repetition count, while scorers evaluate dimensions such as cost, quality, correctness, efficiency, or verbosity.

Warp says the system replays tasks in their original context and compiles results across the matrix of tasks and configurations. That makes the comparison more relevant to a team’s repository, prompts, skills, MCP servers, and review expectations than a score from an unrelated public benchmark.

From traces to a recommendation

Warp Factories stores agent traces and metadata, including the prompt, conversation, Git state, pull request, and generated artifacts. Enterprise customers can keep those records within their own security boundary, according to Warp’s announcement.

Built-in scorers use an LLM-as-judge loop against a rubric the user defines. Results are shown across tradeoffs such as cost and quality, with Pareto-style views to help teams choose a configuration rather than chase one undifferentiated score.

Use the result to tune routing

Benchmarks can compare models and harnesses, including Warp, Claude Code, and Codex as supported or planned options described in Warp’s launch post. Factory definitions remain code, so a team can version the benchmark and configuration it uses for an evaluation.

Warp also describes custom model routers that can route task classes to the best-performing configuration from a benchmark. That can turn an evaluation into a production policy, but routing should be monitored for drift: a configuration that wins on historical tasks may not be best for new repositories or changing requirements.

Availability and practical limits

Benchmarks are in Warp Factories Early Access. Warp says its internal WarpBench helped reduce cost per pull request by about 63% on certain tasks without impacting quality. That is Warp’s reported internal result, not a guarantee for another team.

Benchmark runs are not free. Warp recommends using them when a new model arrives or when a team is considering changes to prompts, skills, or agent context, rather than running a large matrix automatically for every small edit.

Sources

• Warp — “Launch Factory Benchmarks” (September 3, 2026): https://www.warp.dev/blog/warp-factory-benchmarks
• Warp Factories — “WarpBench”: https://www.warp.dev/factories/benchmarks/warpbench
• Warp Docs — 2026 changelog: https://docs.warp.dev/changelog/2026/
• Warp on X — official product updates: https://x.com/warpdotdev