Article
AI Engineering
Which LLM Would I Use for Backend Development?
I gave several LLMs the same Go backend assessment to see which one is actually useful in day-to-day software development. The result was less about who writes the most code and more about who survives verification.
I started this out of curiosity, not as a formal research paper. I wanted to know something very practical: if I am building backend software every day, which LLM actually helps me more? Not which one sounds smartest in a chat window. Not which one writes the longest README. Which one can take a backend assignment, make reasonable engineering decisions, generate maintainable Go code, and survive the kind of verification I would apply before trusting anything in a real codebase.
So I gave multiple models the same task: implement a notification orchestration service in Go. The service had to expose an API, validate notifications, process multiple recipients, handle unreliable providers, use concurrency safely, remain extensible, and include tests and documentation. Then I evaluated the outputs with a backend-engineering benchmark scored out of 280 points.
The full benchmark will live here:
Repositoryaikssen/llm-benchmark-for-devThe backend assessment, scoring rubric, model submissions and evaluation reports behind this article.The most interesting result was not the leaderboard itself. It was the shape of the failures.
The evaluated models are: Opus5, Fable5, Opus-4.8, Sonnet5, GPT-5.6-Sol, GPT-5.6-Terra, GPT-5.5, Kimi-K3, Grok-4.5, Gemini-3.6-flash, Gemini-3.5-Flash, GLM-5.2, MiniMax-M3, DeepSeek-V4-Pro, Qwen3.7-Max, MiMo-V2.5-Pro and Kimi-K2.7-Code
The problem: demos are too forgiving
That is why I did not want a toy prompt. A notification service is small enough to implement in one session, but it contains real backend pressure points:
- API boundaries and request validation.
- Domain modeling and status transitions.
- Provider abstractions for email, SMS and push.
- Concurrent processing under bursty traffic.
- Retries, timeouts, backpressure and graceful shutdown.
- Tests that need to prove behavior, not just improve coverage numbers.
It is the kind of assignment where a model can look good at first glance and still ship a data race.
The mental model: an LLM is not the engineer
The model is not the engineer. The loop is the engineer.
For day-to-day development, I care about the complete loop:
- The model receives the task.
- An agent or harness turns that task into file edits.
- The generated project is built and tested.
- The result is reviewed with engineering judgment.
- The cost and friction decide whether I would actually keep using that setup every day.
That last point matters. A model can be excellent and still be impractical if it is too expensive, unavailable, awkward to run, or hidden behind a harness that does not fit my workflow. In real development, usefulness is not just score. It is score divided by friction.
DIAGRAM
The assessment
The task was to implement a backend service responsible for orchestrating notification delivery across email, SMS and push notifications.
The assignment intentionally left room for engineering judgment. The exact API design was open. External providers could be simulated. The service had to be runnable locally, tested, documented and designed as the first production-ready version of a system a team might maintain over time.
The benchmark scored each submission across twelve categories:
- Correctness.
- Architecture and project organization.
- Dependency injection and dependency management.
- Idiomatic Go.
- Concurrency.
- Error handling.
- Code quality and readability.
- Testing.
- Design patterns and extensibility.
- Production readiness.
- Documentation.
- Engineering judgment.
The important part is that the evaluation was evidence-based. Every project was built. Tests were run. go test -race was run. In some cases, targeted probes were added when the submitted tests did not exercise the risky path.
Every model was run with its default thinking level. I did not raise, lower or otherwise tune the reasoning-effort parameter for any of them. The goal was to see how each model behaves out of the box, the way most people actually use it day to day, not how far it can be pushed with hand-tuned reasoning settings.
The harness matters as much as the model
There is one more variable I want to make explicit, because it is easy to forget and it changes the result: the harness. A model never touches the codebase directly. It is driven by an agent or CLI that decides how tools are called, how files are edited, how tests are run and how much context the model actually sees. The same model can produce very different code depending on that layer.
Each model was driven by its native agent, and in every case I used the CLI version of the harness, not an IDE or web UI:
| Harness (CLI) | Models it drove |
|---|---|
| Claude Code | Opus5, Fable5, Sonnet5, Opus-4.8 |
| Codex | GPT-5.6-Sol, GPT-5.6-Terra, GPT-5.5 |
| Antigravity | Gemini-3.6-flash, Gemini-3.5-Flash |
| OpenCode | Kimi-K3, Grok-4.5, GLM-5.2, MiniMax-M3, DeepSeek-V4-Pro, Qwen3.7-Max, MiMo-V2.5-Pro, Kimi-K2.7-Code |
The harness is recorded per model, but the scores reflect the submitted code, not the tool. Still, keep it in mind when reading the leaderboard: what you are comparing is a model plus its agent, not a model in a vacuum.
One more thing worth stating plainly: none of this ran on metered API tokens. Every model was driven through a flat subscription — OpenCode (Go plan), Codex on ChatGPT Plus, and Claude Code on the Max (5x) plan. That is exactly how most people actually use these tools day to day, and it changes what “cost” means: the real constraint is not a per-call dollar figure, it is how much of the plan’s usage a model burns before you hit its limits.
The leaderboard
Benchmark
Backend engineering score
Same Go notification-orchestration assessment, scored against a 280-point backend benchmark.
- Strong Hire
- Hire
- Lean Hire
- Lean No Hire
ComparisonResult by tier
| Tier | Models | What separated them |
|---|---|---|
| Strong Hire | Opus5, Fable5, Sonnet5, Opus-4.8, GPT-5.6-Sol | Clean architecture, bounded concurrency, race-clean execution, useful tests and strong documentation. Opus5 sits twelve points clear of the rest by closing the operational and error-handling gaps the others left open. |
| Hire | Kimi-K3, GLM-5.2, Gemini-3.6-flash, Gemini-3.5-Flash, MiniMax-M3 | Correct, race-clean and cleanly layered with genuinely good ideas (batch admission, provider middleware, circuit breakers, provider failover), but each pulled down by a rigor or shutdown gap. |
| Lean Hire | GPT-5.5, Grok-4.5, GPT-5.6-Terra, DeepSeek-V4-Pro, Qwen3.7-Max | Light on the production/rigor axes, shutdowns that drop accepted work, or solid-looking architecture where real data races surfaced under inspection. |
| Lean No Hire | MiMo-V2.5-Pro, Kimi-K2.7-Code | Dangerous concurrency behavior: data race, unbounded fan-out or deterministic deadlock. |
Kimi-K3 and Kimi-K2.7-Code are distinct models, so they appear as separate entries. Gemini appears twice on purpose: Gemini-3.6-flash is a successor run to Gemini-3.5-Flash, and both are kept in the results. Together they stand in for Google’s current Flash line, which replaced the earlier Gemini 3 preview. (That preview, an aborted Gemini3-Pro run, never produced a runnable service and is not included.)
Opus5 is the exception to the “do not overread the order” rule. At 268 it clears the previous leader by twelve points and its own prior generation by sixteen, and the margin is not noise: it wins the two categories this problem turns on and is the only submission whose production surface was verified running, not just read.
Below it, the rest of the top tier was close enough that I would not overread the exact order. Fable5 scored 256, Sonnet5 254, Opus-4.8 252 and GPT-5.6-Sol 247. That is a nine-point spread on a 280-point rubric. In practical terms, all four were strong.
The more useful observation is what happened below that top tier.
What separated the models
Concurrency was the great separator.
Most submissions could produce a plausible layered backend. Most could write a README that sounded mature. Many could implement providers, validators, repositories and handlers with reasonable names. But the assessment centered on safe concurrent delivery under bursts, and that is where the illusion broke.
The strongest submissions had a coherent concurrency story:
- Bounded worker pools.
- Explicit backpressure.
- Per-attempt timeouts.
- Retry policies that distinguish transient and permanent errors.
- Defensive copies around in-memory state.
- Tests that actually exercised concurrent paths.
The weaker submissions often failed in subtle ways. The most common bug was a shared-pointer or shared-slice trap: the in-memory store returned data that background workers could still mutate. The code looked clean. The tests passed. But concurrent reads and writes raced because the copy boundary was incomplete.
That is the kind of bug that matters in backend work. It does not show up in a screenshot. It shows up when production traffic hits a path the model’s own tests never drove.
The top results
Opus5 produced the best submission in the entire benchmark at 268, and it is the only one that combines every top-tier property in a single codebase. The architecture is textbook ports-and-adapters around a dependency-free domain, with a genuine transport-agnostic application layer — the exact gap that kept Kimi-K3 out of the top tier. Its concurrency model is the most complete in the field: a bounded worker pool fed by a bounded queue with atomic all-or-nothing batch admission (Queue.Reserve(n) claims all N slots or none), a graceful drain that is correct and tested — Stop closes the queue first and only cancels in-flight work if the deadline expires — a structural guard against the send-on-closed-channel panic, and per-job panic recovery. All of it race-clean, with tests that actually drive those paths rather than merely coexisting with them.
Its error handling is where it separates itself for good. It is one of only two submissions that does not swallow terminal store-write errors, and the only one that writes the delivery outcome through context.WithoutCancel, so a result is still recorded when the surrounding context is already cancelled at shutdown. That is a durability property nobody else in the field has, and it is the kind of detail you normally only see after someone has lost data to it once.
But the thing that genuinely surprised me happened during the run, not in the review. Opus5 is the only model that verified its own container: it opened Docker Desktop on my Mac, built its image, started the container, checked that /healthz and /readyz came back green, submitted a notification and watched it reach delivered — and then tore the container down and cleaned up after itself. No other model did anything close. Six of the seventeen submissions shipped no Dockerfile at all, and several of the ones that did pin a Go version their own module cannot build with. Opus5 is the only submission whose Dockerfile I did not have to take on faith: an 8.99 MB static binary on scratch, non-root, confirmed running end to end. Its only defect is mechanical and small — five unused exported symbols, deadcode-confirmed, for −2.
Fable5 produced the second-best score. Its solution read like a first production version: clean inward-pointing architecture, bounded queue, fixed worker pool, fail-fast validation, retry/backoff, good tests and excellent documentation. It also made a strong modeling decision: the unit of work was not just a notification, but the delivery of a notification to a recipient.
Sonnet5 was almost tied. It had mature architecture, strong tests, circuit breakers and excellent design documentation. The main caveats were operational: no Dockerfile or Makefile, a shutdown-drain wrinkle, and unbounded in-memory maps. These are not trivial, but they are not the same as a race in the core path. It is also worth mentioning that Sonnet5 works better when you give it a well-structured requirement and tell it clearly what to do.
Opus-4.8 was also excellent, though it is worth saying up front that it is no longer selectable in Claude Code — more on that below. It used a restrained dependency set, strong production-minded defaults, a bounded worker pool, backpressure, retry with jitter, circuit breakers, structured logging and good coverage. The main gaps were mostly around Docker and observability.
GPT-5.6-Sol, driven by Codex, was the surprise of the round at 247, joining the Strong Hire tier right at the doorstep of the Anthropic block. It read like a genuinely production-minded service: clean inward-pointing packages, race-clean concurrency with atomic whole-batch admission and a graceful shutdown that drains the queue, and a real reliability stack — retryable-vs-permanent error classification, capped backoff with deterministic jitter, idempotency keys that return 409 on conflict, validated env-based config, Docker, health and readiness endpoints, and even an OpenAPI contract. The only thing keeping it out of the top three was discarded terminal store-write errors (−5) and moderate test coverage. This is the strongest of all the GPT and Gemini submissions.
Kimi-K3, driven by OpenCode, was the strongest submission outside the top tier at 242 (Hire). It shipped the two properties that defined the Strong-Hire tier — atomic all-or-nothing batch admission and a graceful drain proven by a test — plus a resilience stack sized to the brief: decorator-composed per-attempt timeouts, retry with backoff and jitter, a circuit breaker, and temporary-vs-permanent error classification. It ran 44 property-pinning tests with injected clocks, carried zero third-party dependencies, and shipped the best documentation in the field. It was also one of only two submissions that never swallow a store-write error — Opus5 is the other — and one of the few whose rate limiter evicts idle buckets. What kept it out of the top tier was a single architectural gap — no application layer, with the submit use case living inside the HTTP handler and reachable only through HTTP — plus two dead exported methods (−2), no Dockerfile, and a compiled binary committed to the tree.
GLM-5.2, driven by OpenCode, reached 225 (Hire) with high-craft work: a provider middleware stack (circuit breaker ∘ retry ∘ timeout ∘ attempts) that was the cleanest answer to the unreliable-providers brief, errgroup used exactly where the benchmark rewards it and used correctly, and per-recipient results with structured error codes and a partial aggregate status — the most expressive delivery reporting in the field. It added real 503 backpressure, panic recovery, 44 tests and a thorough README. It sat below Kimi-K3 because of a shutdown that abandons most of its queued notifications — Stop() cancels before it drains, with a test that pins the very mechanism causing the loss — plus a discarded terminal write on the backpressure path (−5), two dead exported symbols and an unwired Jitter field (−2), an http.Server missing three of its four timeouts, and no Dockerfile.
Gemini-3.6-flash, also driven by Antigravity, edged just above its predecessor at 216 (Hire) — a successor run to 3.5 that widens the production surface considerably. It adds a circuit breaker, real non-blocking backpressure (Submit returns ErrQueueFull), externalized env config, a health endpoint, a Dockerfile with docker-compose, an OpenAPI spec, and sentinel-error HTTP classification via errors.Is — fixing exactly the string-matching that held 3.5 back. What keeps it mid-field is three rigor issues, two of them regressions against 3.5. Its graceful drain is cosmetic: Pool.Stop() cancels the pool context before draining, so the shutdown backlog is dequeued but force-failed under a cancelled context (verified 0/20 delivered, terminal failed, where 3.5 at least delivered its drain). Its validation pipeline fans out a goroutine per validator per item over four pure in-memory checks with no I/O (−3, where 3.5 was spared only because its validators simulated latency). And it repeats 3.5’s systematically ignored repo.Update errors (−5). Add three unformatted source files, a committed 9.1 MB binary, an untidy go.mod, and a Dockerfile that pins golang:1.22 against a go 1.26.3 module, and you get a model that improves on its predecessor’s operational surface while carrying its core rigor gaps forward unchanged.
Gemini-3.5-Flash, driven by Antigravity, was the strongest of the solid middle at 214 (Hire). It brought genuinely good ideas: provider failover with a primary and a fallback per channel, and a concurrent validation engine whose parallelism is actually justified because the validators carry simulated I/O latency. Coverage on the logic packages was high. What held it back was a cluster of avoidable rigor and hygiene problems: every repo.Update error discarded in the worker path (−5), HTTP status codes classified by string-matching the error message, no health endpoint, no external configuration, a List that quietly dropped its filters and pagination, four unformatted test files, and a compiled binary committed to the repo.
Grok-4.5, driven by OpenCode, landed at 206 (Lean Hire) and was the best-equipped of the lower-middle operationally: sentinel errors mapped to HTTP status via errors.Is rather than string-matching, real 503 backpressure, nine env knobs, a health endpoint and full timeouts. But its rigor gaps all land on the async delivery path the assessment centers on: the worker pool never closes its jobs channel and workers exit on cancellation, so a probe found 18 of 20 queued jobs silently abandoned on shutdown; all 12 store-update calls in the worker discard their error (−5); a mid-fan-out enqueue failure returns 503 with no IDs while earlier recipients are already in flight; and the suite exercises neither concurrency, backpressure nor shutdown.
GPT-5.6-Terra, also driven by Codex, landed at 203 (Lean Hire). Its core is correct and race-clean, with clean package layering, validated env-based config, a shutdown that genuinely drains with a deadline and guards the producer so a post-shutdown submit can never panic, and the field’s best idempotency (202/200/409). What holds it back is that it does not model the brief’s opening premise: its only provider always returns success, so the retry, per-attempt-timeout and permanent-error paths never execute at runtime; preferences and rate limiting ship as no-ops; the terminal delivered write discards its error (−5); an unreachable retry branch adds −2; and its six tests are the fewest in the field. Its README is the honest counterpoint — it names every one of these gaps rather than papering over them. Same harness and default settings as Sol, a different result: a good reminder that even one model family on one agent is not a single fixed quantity.
If this were only about the benchmark score, the answer is now unambiguous: Opus5 won, and not narrowly.
But for daily development, the answer is more nuanced.
Time and cost
There are two more dimensions I captured, at least in part: how long each run took, and what it cost.
Wall-clock time ranged from about three and a half minutes to nearly forty — and the striking part is how little that predicted quality.
| Model | Worked for | Score |
|---|---|---|
| Grok-4.5 | 3m 35s | 206 |
| Gemini-3.5-Flash | ~4m 10s | 214 |
| DeepSeek-V4-Pro | 5m 41s | 164 |
| Gemini-3.6-flash | ~6m 00s | 216 |
| MiniMax-M3 | 6m 43s | 211 |
| Kimi-K2.7-Code | 7m 01s | 153 |
| GPT-5.5 | 7m 20s | 209 |
| GPT-5.6-Terra | 8m 40s | 203 |
| Qwen3.7-Max | 9m 39s | 161 |
| GPT-5.6-Sol | 10m 23s | 247 |
| MiMo-V2.5-Pro | 10m 31s | 157 |
| GLM-5.2 | 11m 12s | 225 |
| Sonnet5 | ~15m | 254 |
| Fable5 | ~18m | 256 |
| Opus-4.8 | 19m 48s | 252 |
| Kimi-K3 | 25m 41s | 242 |
| Opus5 | 39m 11s | 268 |
The fastest run in the field (Grok-4.5, 3m 35s) and one of the slowest (Kimi-K3, 25m 41s) land a tier apart in the wrong direction — the slow one scored higher. Opus-4.8 took nearly twenty minutes for 252; Gemini-3.5-Flash took four for 214. Speed is a real ergonomic factor day to day, but it is not a proxy for engineering quality.
Opus5 is the clearest illustration of that, in both directions. It is by a wide margin the slowest run in the field at 39m 11s — more than eleven times Grok-4.5 — and it also produced the best code. Part of that time is not generation at all: it is the model building its own image, running the container and probing the endpoints before declaring itself done. That is time spent on verification rather than output, which is exactly the trade a careful engineer makes and exactly the trade a benchmark that only counts tokens per minute would score as a loss.
Cost is the dimension a flat subscription deliberately hides, and it is the one that most changes the recommendation. Because none of this ran on metered API tokens, I cannot put a clean per-model dollar figure across the board — but the economics underneath are not a secret, and they matter:
- Anthropic’s models are the most expensive to run — but Opus5 changed the shape of that. Fable5 is the extreme case; Opus5 lands at roughly half of it per run, which is the difference between a model you reach for occasionally and one you can actually work with. It is still a premium over everything else here, and it is still the longest run in the field, but it is no longer priced out of a normal workflow the way the previous top tier was.
- OpenAI’s GPT-5.6 line is the competitive answer. Sol and Terra reach comparable engineering quality at a meaningfully lower cost, which is exactly what makes them attractive as everyday tools rather than occasional ones.
- Open-source models are cheaper still, by a wide margin. For the models I ran metered through OpenCode, the whole task landed between roughly
$0.14and$1.50— DeepSeek-V4-Pro around$0.14, MiniMax-M3$0.20, MiMo-V2.5-Pro$0.23, Grok-4.5$0.29, Kimi-K2.7-Code$0.33, Qwen3.7-Max$0.83, Kimi-K3$1.35, GLM-5.2$1.50.
The top of the leaderboard is Anthropic. The best score-per-dollar is not — and that gap is the whole reason the practical answer differs from the raw ranking.
The practical recommendation
Before the recommendation, one piece of context that the leaderboard cannot show: Claude Code replaces its models rather than accumulating them. This is not like ChatGPT, where the previous generation stays selectable for a while after the new one lands. On the plan today there is Fable5, Opus5, Sonnet5 and Haiku 4.5 — and that is the whole list. Opus-4.8 scored 252 here and is simply not an option anymore. Any recommendation built on it expired the day its successor shipped, which is its own small lesson about how quickly this kind of comparison ages.
That reshapes the practical answer more than the new score does. Opus5 is the benchmark winner and, unexpectedly, the reasonable pick too. Its cost per run lands at roughly half of Fable5’s, which moves it out of the reach-for-it-sparingly category the previous Anthropic top tier lived in. It is still a premium over the open models, which remain dramatically cheaper, and it is still the slowest run in the field at nearly forty minutes. But “best result in the benchmark” and “affordable enough to actually use” have not overlapped in this comparison until now.
Fable5, the previous winner, remains on the plan and remains the expensive end of it. Its availability has been extended more than once with no announced decision on whether it stays permanently, and at roughly twice Opus5’s cost per run for twelve fewer points, it is now hard to argue for on anything but preference.
Sonnet5 is also very strong, but there are practical caveats depending on its use as an agent or sub-agent, the subscription, usage limits and the workflow around it. It works much better when you give it clear and precise instructions about what to do.
That leaves a practical recommendation that is different from the raw leaderboard:
ComparisonHow I would choose in practice
| Practical choice | Why I would consider it | Caveat |
|---|---|---|
| Opus5 | The best engineering result in the benchmark by a wide margin, the only model that verified its own container instead of asking me to trust its Dockerfile, and — at roughly half of Fable5’s cost per run — the first Anthropic top-tier result that is not painful to use regularly. | The slowest run in the field (39m 11s), and still a clear premium over the open models. On a Max-style plan the constraint is the plan’s usage limits, not a per-token price. |
| GPT-5.6-Sol | A genuine top-tier result driven by Codex, with a full reliability stack (error classification, jitter, idempotency, Docker, health/ready) and wide availability. | Still discards terminal store-write errors in the worker path, and test coverage is only moderate. |
| Kimi-K3 | Near-top engineering — atomic batch admission, a tested graceful drain, a full resilience stack, zero dependencies, and one of only two submissions that never swallow a store-write error. | No application layer (submit logic lives in the HTTP handler), no Dockerfile, and a binary committed to the tree. |
| GLM-5.2 | The cleanest provider-reliability design in the field (middleware stack plus errgroup) with expressive per-recipient results, at an open, low-cost profile. | A shutdown that drops most queued notifications, a discarded terminal write, and an http.Server missing most of its timeouts. |
| Gemini-3.6-flash | A successor run to 3.5 with a much wider production surface — circuit breaker, real backpressure, health endpoint, Docker, OpenAPI and errors.Is HTTP classification — at the same fast, cheap Flash profile. | A cosmetic drain that delivers nothing on shutdown, the ignored update errors carried over from 3.5, unnecessary per-validator goroutines, and hygiene slips (unformatted files, committed binary). |
| Gemini-3.5-Flash | Cleanly layered, race-clean result with nice ideas like provider failover, at a fast, cheap Flash profile. | Ignored update errors, HTTP status by string-matching, no health endpoint or external config, and hygiene slips. |
| MiniMax-M3 | Good enough architecture and race-clean execution, with a shutdown that genuinely drains and a TTL sweeper bounding memory. | Swallowed repo-update errors, a validation pipeline it never calls, a queue that can answer 202 for work it never accepted, and no Dockerfile. |
| GPT-5.5 | Compact, readable, pragmatic output with good architecture and no data races. | Thinner tests, a shutdown that drops accepted work, no Docker, and ignored repository update errors. |
A note on the two GPT frontier models: GPT-5.5 is still on the list and remains perfectly usable for real work. But GPT-5.6-Terra reaches roughly the same result while being lighter on the plan, so in practice Terra quietly takes 5.5’s slot — until 5.5 is eventually retired from the subscription altogether.
For daily backend work, I would rather use a model that is slightly less brilliant but fast, cheap and reliable enough to keep in the edit-test-review loop than one I only reach for sparingly because each iteration is expensive or leans hard on the plan. That was the trap the whole Anthropic top tier fell into: they won the benchmark, but their cost pushed them toward occasional use. Opus5 is the first one to partly escape it. At half of Fable5’s cost it is defensible as a regular tool, not just an emergency one — though the honest version is that its long runs still make it better suited to the piece of work that has to be right the first time than to rapid back-and-forth, and the open models remain an order of magnitude cheaper for everything that does not need this level of rigor.
Agents and harnesses are part of that decision. The same model can feel different inside Claude Code, OpenCode, Cursor, a custom CLI, or an API-driven evaluator. The harness decides how tools are called, how files are edited, how tests are run, how much context is available, and how expensive each loop becomes. That means the real comparison is not only model versus model. It is model plus workflow.
What surprised me
The first surprise was that documentation did not discriminate much. Almost every serious submission had a good README. Some had excellent architecture notes. But a well-written README can describe the system the model intended to build, not necessarily the system it actually built.
The second surprise was how often tests gave false confidence. Passing go test -race matters only if tests exercise the concurrent path. Qwen3.7-Max is a good example: the submitted suite passed under -race, but a targeted probe reproduced a race in the core path.
The third surprise was how valuable boring verification became. go build, go vet, gofmt, go test, go test -race, Docker build checks and small targeted tests were more revealing than reading the README. This is probably the most important daily-development lesson from the experiment.
The fourth surprise arrived with Opus5, and it is the one I did not anticipate at all: a model that ran that verification loop on itself. Sixteen models handed me a Dockerfile — or didn’t — and moved on. Opus5 opened Docker Desktop, built the image, ran the container, checked the health endpoints, drove a notification to delivered, then destroyed the container. It is a small thing and a large one at once. Small, because building your own image is not a hard idea. Large, because the single most common failure mode in this benchmark is a model asserting a property it never checked — and this is the only run where the model closed that gap itself instead of leaving it to me.
What I would do differently next time
I would make the harness metadata explicit from the beginning: model, provider, agent, version, cost, duration, subscription tier, retry count and whether the model had access to tool execution. I captured some of this informally, but not enough to make a clean cost-performance table.
I would also add a standard adversarial test pack. The submitted tests are useful, but they are not enough. A benchmark like this should include probes for:
- Concurrent reads while background workers mutate state.
- Queue saturation and partial enqueue behavior.
- Graceful shutdown with in-flight and queued jobs.
- Docker builds against the declared Go version.
- Multi-recipient validation edge cases.
- Persistence errors in terminal status updates.
That is where backend quality shows up.
Key takeaways
- The best benchmark score belonged to Opus5 at 268 — twelve points clear of Fable5 (256) and sixteen above its own prior generation — followed by Sonnet5, Opus-4.8 and GPT-5.6-Sol, which rounded out the Strong Hire tier at 247.
- Opus5 was also the only model that verified its own work end to end: it opened Docker, built and ran its container, checked the endpoints and then destroyed it. In a field where the most common defect is an unverified claim, that behavior mattered more than any single category score.
- Kimi-K3 (242) and GLM-5.2 (225) led the Hire tier, ahead of Gemini-3.6-flash (216), Gemini-3.5-Flash (214) and MiniMax-M3 (211); GPT-5.6-Terra landed at 203 despite sharing Sol’s model family, harness and default settings — a reminder that the model variant and the harness matter, not just the model name.
- Every model was driven through its native CLI harness at default thinking, so what the leaderboard really compares is a model plus its agent, not a model in isolation.
- For the first time the benchmark winner is also a practical answer: Opus5 costs roughly half of Fable5 per run, which makes it defensible as a regular tool rather than an occasional one — with GPT-5.6-Sol, Kimi-K3, GLM-5.2, Gemini-3.6-flash, Gemini-3.5-Flash, MiniMax-M3 or GPT-5.5 as the cheaper day-to-day options depending on cost, availability and preferred harness.
- Availability is part of the recommendation: Claude Code replaces its models instead of accumulating them, so Opus-4.8 — fourth on this leaderboard — is no longer on the plan at all. Today it is Fable5, Opus5, Sonnet5 and Haiku 4.5.
- Concurrency correctness separated the strong models from the polished-but-risky ones.
- Documentation quality was high across the board, so it was not enough to judge usefulness.
- Passing tests only matter when they exercise the paths where backend systems fail.
- Run time (from ~3.5 to ~39 minutes) did not predict score in either direction: fast runs landed mid-pack, and so did slow ones — though the single slowest run, Opus5 at 39m 11s, produced the best code, with much of that time spent verifying rather than writing.
- Cost is where the leaderboard and the recommendation used to diverge hardest, and that gap narrowed: Anthropic still tops the score and still costs the most, but Opus5 runs at roughly half of Fable5; OpenAI’s GPT-5.6 line matches much of that quality for less; and the open-source models finished the whole task for roughly
$0.14–$1.50. - The daily-development question is not “which model is smartest?” but “which model plus agent plus verification loop helps me ship safer code at a practical cost?”
Continue exploring
If you want to reproduce or challenge the conclusions, start with the benchmark repository. Read the assessment, then the rubric, then the per-model reports. The most useful exercise is not looking at who won. It is looking at why apparently good backend code failed.
Related ideas worth exploring next:
- LLM evaluation for software engineering tasks.
- Agent harness design and tool-use reliability.
- Race detection and concurrency testing in Go.
- Backend take-home assessments as model benchmarks.
- Cost-aware AI workflows for everyday development.
// continue exploring