OfficeQA Makes the Harness the Enterprise Moat • The Weighted Average

Databricks’ new OfficeQA Pro V2 results moved matched frontier models from 37.5% accuracy to 52.8% by changing the harness around them. That is a derived 40.8% relative lift—(52.8 − 37.5) ÷ 37.5—so teams buying document agents should stop treating the model API as the complete product and benchmark parsing, retrieval, tools, and stopping policy as one system.

A harder corpus exposes the orchestration gap

Databricks released OfficeQA Pro V2 with 90 questions grounded in roughly 120,000 pages of U.S. federal accounts. Out-of-the-box frontier agents using provider harnesses averaged 37.5% across four matched comparisons in the headline analysis. The same underlying models inside Databricks Genie averaged 52.8%, a 15.3-point absolute and 40.8% relative improvement.

The larger evaluation reports an even wider average across five baseline models: provider harnesses reached 26.0%, and Genie improved matched configurations by 24.0 percentage points, or 92% relative. The two summaries use different matched sets, so they should not be blended. Their common message is sturdy: substantial capability was trapped outside the weights.

The corpus makes that plausible. The public V2 dataset card lists 1,435 PDFs spanning 1793–2024, totaling 13.3 GB in original form and about 794 MB as parsed JSON. A question needs a median 5.5 source documents and as many as 24; Databricks’ release reports a 6.7-document average. The separate OfficeQA repository lists 697 Treasury Bulletin issues in the original corpus. Dividing the two source-document counts gives a cross-source 2.06× corpus expansion—1,435 ÷ 697. That is not a per-question difficulty multiplier, but it expands the retrieval surface before reasoning begins.

This is not ordinary question answering. The records change typography, table structure, terminology, fiscal conventions, and institutional scope across 232 years. Some answers require revised values rather than the first reported number; others require chart interpretation, web context, or calculations across many statements. A frontier model can reason correctly over the wrong cell and still fail exact match.

The repository also makes the benchmark split explicit: the original Pro has 133 hard questions, while V2 uses a separate corpus to test generalization. The original OfficeQA Pro technical report found frontier agents averaging 34.1%, establishing the older baseline without pretending its corpus is interchangeable with V2. The answer keys are gated so browsing agents cannot casually harvest them. That design is more useful for procurement than another public leaderboard whose test set has become part of the web.

Databricks attributes much of its gain to ai_parse_document, which turns PDFs into typed elements for text, tables, figures, headers, footnotes, and bounding boxes. The official function documentation supports PDF, Office, and image inputs, requires Runtime 17.3 or newer, and caps a single parse request at 500 selected pages. Those constraints belong in the implementation estimate. So does the blog’s caveat that plotted rollout costs exclude the one-time corpus-parsing bill.

The operator decision extends today’s Claude Code control-plane lead. In both cases, a classifier or parser outside the principal model changes usable behavior. It also sharpens the earlier Octobench finding that the same coding model gained five solved tasks under another harness. A model never arrives alone; document representation, retrieval, tool use, and stopping rules arrive on the invoice.

Buy a benchmarkable stack, not a model logo

Teams with dense financial, legal, clinical, or policy archives should run a harness bake-off before changing frontier providers. Assemble 50 to 100 production-shaped questions whose answers require multiple documents, revised values, tables, and entity-scope decisions. Freeze the corpus snapshot and ground truth. Then vary one layer at a time: parser, chunking, retrieval, reranking, model, tool set, and verification.

Measure exact answer accuracy, citation fidelity, source recall, latency, rollout cost, parse cost, and human repair. Databricks reports that Claude Fable 5 gained 14.4 points inside Genie while rollout cost fell about ninefold because the provider harness looped over document parsing. That is a vendor-run result and excludes preprocessing, but it shows why a cheap parse index can dominate an expensive agent loop.

The benchmark also reveals a practical ceiling. The strongest configuration cited in the release reached 60%, while the live competition winner reached 63.3%. Neither is safe for unattended high-stakes decisions. A 60% score implies 36 misses across 90 questions if the rate transfers exactly; it will not transfer exactly, but the arithmetic makes the reliability gap visceral. Teams should route answers with weak evidence, conflicting values, or unsupported calculations to review rather than treating retrieval as solved.

Cost needs two ledgers. The first records one-time ingestion: parsing, OCR, figure descriptions, indexing, storage, and ground-truth construction. The second records each answer: model tokens, tool calls, retries, latency, and review. A harness that looks cheap per rollout can be expensive to refresh; a costly parser can amortize well across millions of questions. The public results do not disclose enough pricing to calculate a universal break-even point.

The strongest counterpoint is vendor entanglement. Databricks built the benchmark, entered its own harness, chose the parser, and published the comparison. Genie’s advantages may partly reflect tight integration with its own data platform. Exact-match scoring can also mark semantically acceptable answers wrong, while synthetic question generation may encode the builder’s preferred workflow. Independent reproduction on private documents matters more than the headline.

Generality is another risk. Treasury statements are unusually tabular, historical, and numerical. A support knowledge base with short current articles may benefit less from sophisticated parsing. Conversely, contracts with footnotes and amendments may be harder. OfficeQA V2 is a stress test, not a workload census.

Evidence that would change the verdict is straightforward: parser ablations, equalized preprocessing costs, repeated runs, independent harness implementations, and private-corpus results. If matched models converge after those controls, the headline overstates harness value. If the gap persists across domains, model procurement becomes secondary to document-system engineering.

The quarter-level move is therefore narrow:

Document-agent teams should switch evaluation units from model response to end-to-end grounded answer, including parse, retrieval, reasoning, citation, and review.
Platform teams should budget both ledgers and pin parser schema versions; the documentation warns major versions can break representations.
Risk owners should require abstention when evidence conflicts or source coverage is incomplete; 60% benchmark accuracy is not an autonomous-decision mandate.
Procurement should demand layer-level traces so a vendor can show whether failures came from parsing, retrieval, reasoning, or verification.
Change the verdict with private evidence: a representative corpus, held-out questions, multiple runs, and human repair time.

The adjacent Supabase evaluation-loop analysis argued that production support outcomes should become regression tests. OfficeQA adds the missing anatomy. When a document agent fails, “the model was wrong” is often too coarse to fix. The competitive moat is the system that can locate the failure layer—and improve it without buying a new logo.