Octobench Finds a 5-Task Harness Swing • The Weighted Average

The same GLM-5.2 model solved 24 of 25 fresh pull-request tasks inside Octomind and 19 inside OpenCode, while the winning harness cost 51% less. For engineering leaders, the decision is no longer which model tops a leaderboard; it is which model–harness pair buys the fewest expensive failures on their own repository.

The five-task gap around the same weights

Muvon’s real-commit Octobench gives the harness argument an unusually clean comparison. Its pinned benchmark artifact records four stock single-agent configurations: Octomind with GLM-5.2 solved 24 tasks for $63.43 in 3.6 hours; Claude Code with Opus 5 solved 23 for $81.79 in 6.7 hours; Codex with GPT-5.6 Sol solved 21 for $14.86 in one hour; and OpenCode with GLM-5.2 solved 19 for $129.54 in 3.3 hours.

The cross-model ranking is interesting. The controlled pair is the news. Octomind and OpenCode called the same GLM-5.2 endpoint at the same prices, yet the higher-scoring Octomind configuration delivered +5 passing patches—a 20-point absolute or 26.3% relative lift—and spent $66.11 less. Wall time barely helps explain it: Octomind was 18 minutes slower. Something between the model and the task changed what got solved.

Normalize the bill by successful work and the gap widens. Octomind cost $2.64 per solved task; Claude Code, $3.56; Codex, $0.71; OpenCode, $6.82. OpenCode therefore spent 2.58 times as much per success as the other GLM harness. Those figures are descriptive, not forecasts, but they demolish the habit of pricing an agent from token rates alone.

The same GLM model solved more for half the harness cost

Tasks passing held-out tests versus total run cost; one run per configuration

Muvon Octobench pinned results · Jul 2026

Codex defines the opposite frontier. It was 76.6% cheaper and 72.2% faster than Octomind, but missed three additional tasks. The extra $48.57 spent by Octomind divided by three net additional passing tasks in this run is $16.19—a descriptive ratio, not the causal price of buying a solve— while comparing Octomind’s five-solve advantage over OpenCode yields savings rather than a marginal cost. More useful operationally: a cheap Codex attempt can dominate when review and retries are inexpensive; a completion-oriented harness can win when one subtly incomplete patch creates a production incident. The buyer must price rework, not merely inference.

That is why this edition pairs the benchmark with Supabase’s outcome-based agent regression loop and India’s mobile subscription economics. All three expose the same accounting error: the advertised unit—the model, the download, the token—does not equal the completed outcome.

The benchmark’s construction deserves credit. Muvon’s methodology uses 25 pull requests merged in 2026 across Python, PHP, Rust, C++, and JavaScript. Each task checks out the commit immediately before the fix as a shallow clone, removes the remote, and keeps the answer out of the object store. Twenty prompts were reconstructed from merged changes and hand-curated; five use original issue text. Every case had to prove that hidden tests fail before the fix and pass after it.

After an agent finishes, the harness fetches only the maintainers’ held-out test paths, overwrites agent changes to those tests, and runs the project’s own checks. This is closer to an engineering contract than judging whether a diff resembles the original patch. It extends the outcome-first distinction in the earlier Claude-versus-Codex analysis: passing the running system matters more than narrating a plausible edit.

Retrieval and stopping rules are part of intelligence

The vendor attributes much of the same-model gap to context discipline. In one reported task, OpenCode pushed 30 million input tokens while Octomind used 8 million—22 million fewer, or a 73.3% reduction. Octomind bundles Octocode structural search, allowing the agent to seek symbols, signatures, and patterns instead of repeatedly grepping and rereading broad files. The blog does not name that task in the sentence, so treat the ratio as a vendor-reported mechanism, not an independently isolated causal result.

Independent research nevertheless points in the same direction. Agent Retrieval Bench covers 427 workflow-derived samples across 25 repositories and finds that logged agents miss every gold file on 27% to 35% of samples. No retrieval family wins everywhere: one embedding leads mean reciprocal rank, another leads Recall@20, and RepoMap packs the best useful context into an 8,000-token budget. Retrieval is not plumbing. It determines what the model can reason about and what context the customer repurchases every turn.

The second mechanism is stopping. During dry runs, an agent announced that it would plan and implement a fix, then ended with a zero diff. Octomind added a deterministic guard: if a turn ends without action while the agent still reports work in progress, send it back. This resembles Anthropic’s long-running-agent harness, which uses explicit feature state, progress artifacts, git history, and end-to-end verification to prevent agents from one-shotting a project or declaring victory early.

The guard is also the benchmark’s most important fairness disclosure. The blog calls the four invocations stock and untuned, while the pinned reproduction notes say Octomind used a staged tap plus a binary override containing the unfinished-handback pre-gate. That does not invalidate the thesis—the guard is the harness difference—but it means this is not a neutral model contest. Muvon built the test, supplied one contestant, curated the prompts, and used an Octomind judge for qualitative scoring. Hidden tests reduce that conflict; they cannot erase it.

A broader Scaffold Effect study held models constant across three open-source harnesses and found up to a 40-fold difference in tokens per solved task, even when paired pass-rate gaps were usually modest. Failure fingerprints followed harnesses: reasoning failures clustered in one, verification and max-turn failures in another, idle loops in a third. Harness-Bench’s 106 sandboxed tasks and 5,194 trajectories reaches the same reporting principle: capability belongs to the configuration, not the model label.

Caching complicates the dollar chart. The two GLM runs used an endpoint without prompt caching and rebought context at $1.40 per million input tokens. The benchmark says about 97% of Opus rereads received a one-tenth cache rate; Codex also benefits from implicit caching. Anthropic’s official cache pricing makes hits 0.1 times base input, while OpenAI documents automatic prefix caching. The same-GLM cost comparison is controlled. The cross-family one is a real invoice, but not normalized inference efficiency.

The practical architecture is therefore layered. Give the model targeted repository retrieval. Persist task state outside the conversation. Refuse an empty handback when acceptance remains unmet. Grade the running artifact with tests the agent cannot rewrite. Then log token mix, cache hits, elapsed time, retries, review minutes, and escaped defects. A model card tells you almost none of that.

Twenty-five tasks cannot crown a procurement winner

The sample is too small for the swagger of a league table. One task moves pass rate by four points, and each configuration ran once. There are no confidence intervals, randomized repetitions, retrieval ablations, or guard-on/guard-off results. Octomind’s 24 versus Claude’s 23 could reverse on the next seed. Even the five-task same-model gap needs repetition before it becomes procurement-grade.

Task selection can flatter a product. Muvon chose recent projects and scenarios, reverse-engineered most prompts, and audited derivability. Those are sensible defenses against contamination and taste-grading. They also leave the vendor in control of what “normal Tuesday” means. A fintech monorepo with generated code, private build tools, and hour-long integration tests may produce a different harness ranking.

Benchmarks can fail while appearing precise. SWE-smith constructs repositories for executable agent training and evaluation, while the SWE-bench Multilingual dataset spans 300 tasks across nine languages; both make environment and task construction visible instead of treating the grader as infallible. Octobench caught defects during its own review. That makes its process more credible—and reminds us that a passing benchmark still measures the quality of the ruler.

The best counterpoint is economic. Codex solved 84% of tasks at $0.71 per success and finished the entire set in one hour. An organization with cheap automated retries, strong CI, and mandatory review may rationally choose the fast system, accept three more misses, and still clear more work per dollar. The “best” harness depends on the cost of a false finish, not an abstract preference for maximum pass rate.

The thesis breaks if repeated runs collapse the GLM gap. Run each task at least three times, randomize execution order, publish cache and token traces, and switch Octocode retrieval and the handback guard on and off. If the 24-versus-19 difference vanishes, model stochasticity or task selection carried the result. If it persists across neutral task sets and private repositories, harness choice becomes a board-level productivity variable.

There is also a portability risk. Structural search requires indexing and language support. Deterministic stopping rules can trap an agent in costly loops or reject a legitimate “cannot proceed” handoff. More context is not always waste: the extra file can reveal an integration boundary the gold patch ignored. The right harness should abstain and escalate as deliberately as it persists.

Finally, benchmark freshness decays. Every task was merged in 2026 and mostly after model cutoffs, but publication dates do not prove weights never saw the code. A maintained benchmark must keep harvesting, quarantine traces, and retire contaminated cases. That maintenance bill belongs in the agent program’s total cost.

Buy the measured system, not the famous model

The next-quarter move is an internal bake-off, not an enterprise-wide migration. Select 30 to 50 recently merged tasks that represent your repositories, redact the solutions, and recreate the pre-fix state. Include small bugs, multi-file changes, build-system work, and tasks whose failure cost is high. Keep the model fixed first; vary retrieval, context policy, stopping, and verification. Only then compare models.

Measure the entire funnel: first-attempt pass rate, cost per passing task, wall time, human review minutes, retries, revert rate, and defects found after merge. A system that saves $2 in tokens but adds 20 minutes of senior review is not cheaper. A harness that wins tests by overfitting narrow assertions is not safer. This is the operating lesson behind the agentic engineer’s shift from typing to orchestration.

Use three explicit gates:

Platform teams should A/B the harness before the model. Hold weights and endpoint constant, then test structural retrieval, progress state, maximum-turn policy, and unfinished-work guards on recent private tasks.
Engineering leaders should price completed work. Record inference, cache, CI, retry, and review costs per accepted patch; compare Codex’s fast frontier with a higher-completion harness according to failure severity.
Security and reliability teams should own the stop conditions. Define when the agent persists, abstains, escalates, or may declare completion; never let self-reported confidence substitute for executable acceptance.
Procurement should demand repeatability. Require multiple seeds, pinned versions, raw traces, confidence intervals, and a neutral held-out set before signing a volume commitment.
Developers should preserve exit routes. Keep tests, task state, and repository indexes portable so a harness swap does not erase the organization’s evaluation history.

Evidence that would change the verdict is straightforward: an independent rerun, guard and retrieval ablations, and private-task results that include human rework. Until then, Muvon’s benchmark proves neither that GLM beats Opus nor that Octomind is universally best. It proves something more durable: a coding model never arrives alone. Retrieval, memory, stopping rules, tools, and tests walk into production with it—and they appear on the bill.