Agentic Engineering
Supabase Evals Turns Agent Support Into Tests
Supabase's live eval gives Codex 100%, but its real lesson is a containerized regression loop for agents, skills, and docs.
Supabase’s new live benchmark gives Codex with GPT-5.6 Sol a 100% score, while its launch snapshot implies an 80% reduction in mean Build-stage error among the three reported models that improved with skills. That illustrative result is less durable than the containerized regression loop teams can copy.
Compatibility becomes a product surface
The Supabase Evals launch turns agent support into something a vendor can test every day. Scenarios ask Claude Code, Codex, and OpenCode to build schemas, debug Edge Functions, repair row-level security, investigate failures, and use the same CLI, MCP server, and documentation paths available to customers. The open-source repository includes prompts, scorers, starting state, experiment definitions, and local results, so a failure can become a durable regression instead of a support anecdote.
Supabase separates two suites. The public benchmark is a small breadth sample spanning products, topics, and stages of the builder journey. The private-facing regression suite goes deep on known failures, refreshes daily, and does not inflate the public score; mature cases can graduate later. That seam matters. A leaderboard should remain interpretable while a regression corpus grows with every incident.
The live page currently reports 100% for Codex/GPT-5.6 Sol, 95% for Claude Code with Opus 5, Claude Code with Sonnet 5, and OpenCode with Kimi K3, and 79% for Codex/GPT-5.4 mini. Those totals are mutable snapshots. They carry no visible sample count, confidence interval, cost, latency, or first-attempt split, so five points between first and second is not a statistically established crown.
The execution model is more durable than the ranking. According to the repository’s reproducibility guide, each scenario combines remote project state with local working files. MCP experiments use a management-API-compatible lightweight project so the real Supabase MCP server can run; local-stack experiments place the agent in a fresh Docker sandbox with the real CLI and sibling service containers. Scorers inspect SQL state, call applications as actual users, test function responses, and read generated files. An LLM judge handles semantic questions that deterministic checks cannot.
Agents receive one retry after a failure. That lowers false negatives and raises the practical question the leaderboard hides: is 100% a first-pass outcome or success within two attempts? An organization buying autonomy cares about both. A retry that takes ten minutes and fixes itself may be cheap; one that burns an approval window or mutates production twice may not be.
The launch report says skills changed the Build-stage snapshot unevenly. Opus 5 and Kimi K3 already scored 100% without one. Sonnet 5 moved from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89%. Convert those scores to error rates and average only the three models that moved: failure falls from 18.3% to 3.7%, an 80% relative reduction. That is this article’s derived figure, not a vendor claim, and it applies only to those reported Build observations—not the entire current suite.
The design extends the outcome-verification argument in today’s coding-harness lead. Muvon restores hidden maintainer tests after an agent works; Supabase interrogates a running database and user permissions. Both reject “the diff looks plausible” as an acceptance criterion. The same principle informed the earlier Claude-versus-Codex grader-loop analysis: agents improve when success is executable and external to their narration.
The skill’s job is to route attention
The same launch report tracks documentation behavior: Codex/GPT-5.6 reads about 8 documentation pages per scenario, versus about 2 for Claude Code—a 4× ratio. Claude Code consults the docs in under 40% of scenarios even when skills are available. The skill’s value is therefore not merely stuffing more prose into context; it is steering an agent toward current evidence when pretraining is stale.
Trigger wording changes that routing. Supabase reports its main skill loaded whenever eligible, but its Postgres best-practices skill initially activated in about one of ten sessions. Rewriting the description to lead with clearer triggers lifted activation to 60%, a sixfold increase. The official skills documentation describes progressive disclosure: the prompt gets compact metadata first, while the agent fetches full instructions only when relevant. Skill descriptions are executable interface design.
This also clarifies the cost. Installing an open skill is trivial; maintaining trustworthy compatibility is not. A team needs provider tokens, Docker capacity, stable fixtures, judge calls, version pins, and owners who convert production escapes into scenarios. A failing case can consume two attempts. LLM judgments add nondeterminism. The hosted side is compatible and realistic, but not Supabase’s full production control plane.
The operator move is narrow. Supabase users with recent escaped failures in newer APIs, RLS, declarative schemas, or Edge Functions should install the official skill and encode those failures with deterministic scorers before changing model or harness. Report first-pass and retry-adjusted results separately, alongside cost and latency. Platform vendors should maintain one broad public benchmark and one deep regression inventory. Teams using both MCP and CLI should test both, rather than infer tool quality from documentation coverage; the archive’s MCP-versus-CLI analysis explains why the two surfaces carry different context and failure costs.
The strongest counterpoint is saturation. Two models were already perfect on Build cases without a skill, and the published percentages may represent few scenarios. A skill can add tokens and confidently route an agent toward advice that no longer applies. A vendor-authored suite can also reward the workflow the vendor prefers. This is why OpenAI’s skill-evaluation guidance recommends scoring trigger behavior, traces, artifacts, and deterministic outcomes, not just the final answer.
Evidence that would change the verdict is concrete: publish run counts, multiple seeds, confidence intervals, first-pass versus retry-adjusted results, tokens, cost, and latency. Reverse the “install by default” recommendation if a preregistered test shows no meaningful first-pass improvement while cost or latency rises materially. Strengthen it if gains persist outside Build tasks and across pinned agent versions.
Until then, 100% is a headline, not a service-level agreement. Supabase’s real contribution is making every agent compatibility claim falsifiable against a live stack—and giving yesterday’s escaped bug somewhere permanent to live.