{"data":{"items":[{"id":"0d665ef6-fffd-4581-bf11-3f52922e27b3","excerpt":"GPT-5.5 low vs medium vs high vs xhigh: the reasoning curve on 26 real tasks from an open source repo — # TL;DR\n\nI ran GPT-5.5 Codex at all reasoning effort settings (low, medium, high, and xhigh) on the same 26 tasks from an open source repo (GraphQL-go-tools, in Go).\n\nLow and medium tied on tests at 21/26, but medium","url":"https://www.reddit.com/r/codex/comments/1t7dqnc/gpt55_low_vs_medium_vs_high_vs_xhigh_the/","role":"demand","weight":1.5210675,"occurredAt":"2026-05-08T16:56:56.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"codex","intent":"tool_discovery","painScore":0.53256166,"sentiment":-0.49532712,"confidence":0.9925,"matchedPatterns":["looking_for","frustrating","missing_feature","manual_process"],"statement":"Medium handled much of alias serialization and missing-message validation, but missed the mutation mapping update and overloaded with response-key semantics.","title":"GPT-5.5 low vs medium vs high vs xhigh: the reasoning curve on 26 real tasks from an open source repo","body":"# TL;DR\n\nI ran GPT-5.5 Codex at all reasoning effort settings (low, medium, high, and xhigh) on the same 26 tasks from an open source repo (GraphQL-go-tools, in Go).\n\nLow and medium tied on tests at 21/26, but medium was much better on semantic equivalence with the original human PR, and posted higher review quality. High looked like the practical sweet spot. Xhigh produced the best equivalence/review scores, but was much more expensive.\n\nReasoning effort seems to change the *kind* of patch Codex produces, not just the pass rate of the tests.\n\nLow → medium: less heuristic/partial implementation, more repo/domain modeling.\n\nMedium → high: the practical jump. More tasks become complete, integrated, and reviewable without xhigh-level cost.\n\nHigh → xhigh: quality mode. Better on complex tasks, but expensive and slow.\n\nOne broader takeaway for me: this should not have to be a one-off manual benchmark. If reasoning level changes the kind of patch an agent writes, the natural next step is to let the agent test and improve its own setup on real repo work.\n\nData dump (will explore this later throughout the post):\n\n*For this post, “equivalent” means the patch matched the intent of the merged human PR; “code-review pass” means an AI reviewer judged it acceptable; craft/discipline is a 0-4 maintainability/style rubric; footprint risk is how much extra code the agent touched relative to the human patch.*\n\nI also made an interactive version with pretty charts and per-task drilldowns here: [https://stet.sh/blog/gpt-55-codex-graphql-reasoning-curve](https://stet.sh/blog/gpt-55-codex-graphql-reasoning-curve)\n\n|Metric|Low|Medium|High|Xhigh|\n|:-|:-|:-|:-|:-|\n|Tests pass|21/26, 80.8%|21/26, 80.8%|25/26, 96.2%|24/26, 92.3%|\n|Equivalent with human patch|4/26, 15.4%|11/26, 42.3%|18/26, 69.2%|23/26, 88.5%|\n|Code-review pass|3/26, 11.5%|5/26, 19.2%|10/26, 38.5%|18/26, 69.2%|\n|Footprint risk mean (lower better)|0.200|0.268|0.314|0.365|\n|Craft/Discipline avg|2.311|2.604|2.736|3.071|\n|Cost per task (avg)|$2.65|$3.13|$4.49|$9.77|\n|Cost per task (median)|$1.91|$2.87|$3.99|$6.39|\n|Tests passes per dollar|0.3051|0.2577|0.2144|0.0945|\n|Equivalent passes per dollar|0.0581|0.1350|0.1544|0.0905|\n|Mean agent duration|286.9s|411.0s|579.0s|753.3s|\n|Input tokens|61,109,728|109,323,987|159,919,731|217,865,624|\n|Output tokens|198,594|292,734|421,418|569,850|\n|Cached input tokens|58,907,856|105,313,408|154,584,448|189,416,832|\n|Uncached input tokens|2,201,872|4,010,579|5,335,283|28,448,792|\n\n|Delta|Tests|Equivalent|Code-review pass|Footprint risk mean|Cost/task|Mean duration|\n|:-|:-|:-|:-|:-|:-|:-|\n|Medium minus low|\\+0.0pp|\\+26.9pp|\\+7.7pp|\\+0.068|\\+$0.49, 1.18x|\\+124.1s|\n|High minus medium|\\+15.4pp|\\+26.9pp|\\+19.2pp|\\+0.046|\\+$1.35, 1.43x|\\+168.0s|\n|Xhigh minus high|\\-3.8pp|\\+19.2pp|\\+30.8pp|\\+0.051|\\+$5.29, 2.18x|\\+174.3s|\n|Xhigh minus low|\\+11.5pp|\\+73.1pp|\\+57.7pp|\\+0.165|\\+$7.12, 3.69x|\\+466.4s|\n\n# Why I Ran This\n\nAfter my last post comparing GPT-5.5 vs 5.4 vs Opus 4.7, I was curious how intra-model performance varied with reasoning effort. On X/Reddit/HN I had seen speculation around which reasoning effort level is optimal for GPT-5.5 (with some claiming that low/medium is better than high/xhigh due to \"overthinking\", a known failure mode for 5.4 and 5.3-codex).\n\nTo separate vibes from reality, and figure out where the cost/performance sweet spot is for GPT-5.5, I ran this experiment.\n\nThis is not meant to be a universal benchmark result - I don’t have the funds or time to generate statistically significant data. The purpose is closer to \"how should I choose the reasoning setting for real repo work?\", with `GraphQL-Go-Tools` as the example repo.\n\nPublic benchmarks flatten the reviewer question that most SWEs actually care about: would I actually merge the patch, and do I want to maintain it? That's why I ran this test - to gain more insight, at a small scale, into how coding agents perform on real-world tasks.\n\nTerminal-Bench is primarily esoteric coding questions, SWE-bench verified is contaminated (as in models already have answers baked in), and SWE-bench Pro is useful, but generic. That is not a knock on SWE-bench or Terminal-Bench. Standardized benchmarks are useful, but they mostly answer a binary task-outcome question.\n\nThe question I care about day to day is narrower and more annoying: did the agent make the same kind of change a human merged in my codebase, and would I want to own the patch afterward?\n\n# Experimental Setup\n\nEach task is derived from a real merged PR or commit. The model gets a frozen repo snapshot, a prompt describing the change, and one attempt to produce a patch in a Docker container. Stet then applies the patch and runs the task's tests in an isolated container to check if it passed/failed.\n\nThen Stet grades the result beyond pass/fail:\n\n* **Equivalence:** does the candidate patch accomplish the same behavioral change as the original human patch?\n* **Code review:** would a reviewer accept the patch, considering correctness, introduced-bug risk, maintainability, and edge cases?\n* **Footprint risk:** how much additional code did the agent touch when compared with the human patch?\n* **Craft/discipline rubrics:** attempt to capture non-correct aspects of code. Basically, would a reviewer want to maintain this code. The categories are clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality\n\nEvery model ran once per task with a single seed. The LLM-as-a-judge model was GPT-5.4. Each patch was scored independently - the judge sees the patch and the task, and was blinded to the model/effort that produced the patch. I also manually inspected representative examples as sanity checks. There was no human calibration pass on this task set, so I would trust the direction of the deltas more than any single absolute score.\n\nAs an aside, I've also been using these evaluations as an \"autoresearch\" optimization loop, not just a benchmark. I tell my agent something like \"make [`AGENTS.md`](http://AGENTS.md) better for this repo\"; it proposes an edit, runs Stet on historical tasks, figures out where the candidate was better / worse and why, and iterates to improve the evaluation numbers.\n\nDetails:\n\n* Model: GPT-5.5\n* Harness: Codex 0.128.0\n* Dataset: 26 matched real GraphQL-go-tools tasks.\n   * Yes this is small - however running even this used \\~50+% of my weekly 20x quota\n* Main metrics:\n   * test pass\n   * semantic equivalence\n   * code-review pass\n   * footprint risk\n   * craft/discipline custom graders\n   * cost and runtime\n\n# Low To Medium: From Heuristics To Domain Modeling\n\nLet's jump into the data!\n\n|Metric|Low|Medium|Δ|\n|:-|:-|:-|:-|\n|Tests pass|21/26, 80.8%|21/26, 80.8%|\\+0.0pp|\n|Equivalent|4/26, 15.4%|11/26, 42.3%|\\+26.9pp|\n|Code-review pass|3/26, 11.5%|5/26, 19.2%|\\+7.7pp|\n|Footprint risk mean|0.200|0.268|\\+0.068|\n|Craft/Discipline avg|2.311|2.604|\\+0.293|\n|Cost/task (mean)|$2.65|$3.13|\\+$0.49, 1.18x|\n|Mean duration|286.9s|411.0s|\\+124.1s|\n\nLow and medium both pass tests on 21/26 tasks. If tests were the only metric, low and medium would look tied.\n\nHowever, when we look at semantic equivalence, the jump from low to medium is 4/26 to 11/26. Similarly, code-review pass jumps from 3/26 to 5/26, and aggregate craft/discipline scores rise from 2.311 to 2.604.\n\n**In this slice, tests alone would have missed most of the reasoning-effort differences.**\n\nCoding-agent evals that only measure tests can flatten differences that matter to humans reviewing the patch.\n\nSpeaking from the perspective of a professional software engineer, the code I want AI to merge into my team's codebase doesn't just pass tests. It is also clear, maintainable, at the correct level of abstraction, and following the codebase's standards.\n\nExample: PR #1297 asks the agent to validate nullable external u/requires dependencies in GraphQL Federation. If a nullable required field comes back null with an error, dependent downstream fetches should not receive that tainted entity.\n\n* Task: model a subtle federation data-dependency rule, not just add a validation branch.\n* Lower-effort failure mode: low passed tests, but it was non-equivalent and review-failing because it used heuristic required-field/error matching and missed structured nullable u/requires metadata.\n* Higher-effort change: medium became equivalent, passed review, tracked tainted objects, filtered downstream fetch inputs, and improved craft/discipline quality from `1.350` to `3.225`.\n* Lesson: medium stops guessing and starts representing the actual federation behavior. High and xhigh stayed in the same quality band, so this is mainly a low-to-medium example.\n\n# High Looks Like The Practical Sweet Spot\n\nHigh vs medium:\n\n|Metric|Medium|High|Δ|\n|:-|:-|:-|:-|\n|Tests pass|21/26, 80.8%|25/26, 96.2%|\\+15.4pp|\n|Equivalent|11/26, 42.3%|18/26, 69.2%|\\+26.9pp|\n|Code-review pass|5/26, 19.2%|10/26, 38.5%|\\+19.2pp|\n|Footprint risk mean|0.268|0.314|\\+0.046|\n|Craft/Discipline avg|2.604|2.736|\\+0.132|\n|Cost/task (mean)|$3.13|$4.49|\\+$1.35, 1.43x|\n|Mean duration|411.0s|579.0s|\\+168.0s|\n\nHigh was the cleanest practical upgrade. It improved the obvious metrics and the semantic/review metrics, while cost rose meaningfully but not absurdly.\n\nHigh appears to be the point where the extra tokens pay off in terms of real gains - it’s the point where integration details are correct more often.\n\nLet’s look at some examples:\n\nPR #1209 asks the gRPC datasource to honor GraphQL aliases in response JSON, validate referenced protobuf message types up front, and update mapping coverage for union/interface mutation paths.\n\n* Task: carry alias/response-key semantics through planning, marshaling, and gRPC mapping coverage.\n* Lower-effort failure mode: low and medium both passed tests but stayed non-equivalent and review-failing. Medium handled much of alias serialization and missing-message validation, but missed the `createUser` mutation mapping update and overloaded `JSONPath` with response-key semantics.\n* Higher-effort change: high became the first strict pass. It introduced explicit response-key/alias handling, carried aliases through planning and JSON marshaling, and raised custom quality to `3.625`.\n* Lesson: high did not just add more code. It got the integration obligation exactly right. Xhigh also passed, but did not improve the task-level read and was much slower in the regenerated summary (`790.7s` agent duration versus `314.0s` for high).\n\nPR #1155 is a broad gRPC datasource hardening task: support repeated scalar fields, avoid null/invalid message panics, propagate gRPC status codes, allow disabling the datasource, and support dynamic clients.\n\n* Task: harden several production boundaries across gRPC datasource behavior.\n* Lower-effort failure mode: low and medium were test-green but non-equivalent. Medium improved robustness, but still serialized invalid repeated fields as empty arrays, missed aliased-root planning behavior, and had dynamic-client lifecycle risk.\n* Higher-effort change: high became equivalent and review-passing, with safer nil/invalid handling, status-code propagation, disabled-datasource behavior, and dynamic client-provider coverage.\n* Lesson: this is also a high-vs-xhigh reversal. Xhigh still passed tests, but became non-equivalent and review-failing because disabled datasource semantics and invalid-list behavior were wrong.\n\n# Xhigh Is Better Quality, Not Obviously A Better Default\n\nXhigh vs high:\n\n|Metric|High|Xhigh|Δ|\n|:-|:-|:-|:-|\n|Tests pass|25/26, 96.2%|24/26, 92.3%|\\-3.8pp|\n|Equivalent|18/26, 69.2%|23/26, 88.5%|\\+19.2pp|\n|Code-review pass|10/26, 38.5%|18/26, 69.2%|\\+30.8pp|\n|Footprint risk mean|0.314|0.365|\\+0.051|\n|Craft/Discipline avg|2.736|3.071|\\+0.335|\n|Cost/task (mean)|$4.49|$9.77|\\+$5.29, 2.18x|\n|Mean duration|579.0s|753.3s|\\+174.3s|\n\nXhigh seems to buy semantic and review quality, but it is not a simple \"turn the knob up and everything improves\" story. It is expensive, and tests are ","offTopic":true},{"id":"3d18ae89-7095-420f-aa6b-21a0e59da5d9","excerpt":"Opus 4.7 Low Vs Medium Vs High Vs Xhigh Vs Max: the Reasoning Curve on 29 Real Tasks from an Open Source Repo — # TL;DR\n\nI ran Opus 4.7 in Claude Code at all reasoning effort settings (low, medium, high, xhigh, and max) on the same 29 tasks from an open source repo (GraphQL-go-tools, in Go).\n\n**On this slice, Opus 4.7 ","url":"https://www.reddit.com/r/ClaudeCode/comments/1tc293h/opus_47_low_vs_medium_vs_high_vs_xhigh_vs_max_the/","role":"demand","weight":1.3235183,"occurredAt":"2026-05-13T15:02:01.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"ClaudeCode","intent":"tool_discovery","painScore":0.34041443,"sentiment":-0.042016808,"confidence":0.98739487,"matchedPatterns":["looking_for","frustrating","switching_from","manual_process"],"statement":"But equivalence moved from 10/28 on low to 16/28 on medium, 13/28 on high, and 19/28 on xhigh; code-review pass moved from 4/27 to 10/27, 10/27, and 11/27.","title":"Opus 4.7 Low Vs Medium Vs High Vs Xhigh Vs Max: the Reasoning Curve on 29 Real Tasks from an Open Source Repo","body":"# TL;DR\n\nI ran Opus 4.7 in Claude Code at all reasoning effort settings (low, medium, high, xhigh, and max) on the same 29 tasks from an open source repo (GraphQL-go-tools, in Go).\n\n**On this slice, Opus 4.7 did not behave like a model where more reasoning effort had a linear correlation with more intelligence. In fact, the curve appears to peak at medium.**\n\nIf you think this is weird, I agree! This was the follow-up to a Zod run where Opus also looked non-monotonic. I reran the question on GraphQL-go-tools because I wanted a more discriminating repo slice and didn’t trust the fact that more reasoning != better outcomes. Running on the GraphQL repo helped clarified the result: Opus still did not show a simple higher-reasoning-is-better curve.\n\nThe contrast is GPT-5.5 in Codex, which overall *did* show the intuitive curve: more reasoning bought more semantic/review quality. That post is here: [https://www.stet.sh/blog/gpt-55-codex-graphql-reasoning-curve](https://www.stet.sh/blog/gpt-55-codex-graphql-reasoning-curve)\n\nMedium has the best test pass rate, highest equivalence with the original human-authored changes, the best code-review pass rate, and the best aggregate craft/discipline rate. Low is cheaper and faster, but it drops too much correctness. High, xhigh, and max spend more time and money without beating medium on the metrics that matter.\n\nMore reasoning effort doesn't only cost more - it changes the way Claude works, but without reliably improving judgment. Xhigh inflates the test/fixture surface most. Max is busier overall and has the largest implementation-line footprint. But even though both are supposedly thinking more, neither produces \"better\" patches than medium.\n\nOne likely reason: Opus 4.7 uses adaptive thinking - the model already picks its own reasoning budget per task, so the effort knob biases an already-adaptive policy rather than buying more intelligence. More on this below.\n\nAn illuminating example is PR #1260. After retry, medium recovered into a real patch. High and xhigh used their extra reasoning budget to dig up commit hashes from prior PRs and confidently declare \"no work needed\" - voluntarily ending the turn with no patch. Medium and max read the literal control flow and made the fix.\n\nOne broader takeaway for me: this should not have to be a one-off manual benchmark. If reasoning level changes the kind of patch an agent writes, the natural next step is to let the agent test and improve its own setup on real repo work.\n\n*For this post, \"equivalent\" means the patch matched the intent of the merged human PR; \"code-review pass\" means an AI reviewer judged it acceptable; craft/discipline is a 0-4 maintainability/style rubric; footprint risk is how much extra code the agent touched relative to the human patch.*\n\nI also made an interactive version with pretty charts and per-task drilldowns here: [https://stet.sh/blog/opus-47-graphql-reasoning-curve](https://stet.sh/blog/opus-47-graphql-reasoning-curve)\n\nThe data:\n\n|Metric|Low|Medium|High|Xhigh|Max|\n|:-|:-|:-|:-|:-|:-|\n|All-task pass|23/29|28/29|26/29|25/29|27/29|\n|Equivalent|10/29|14/29|12/29|11/29|13/29|\n|Code-review pass|5/29|10/29|7/29|4/29|8/29|\n|Code-review rubric mean|2.426|2.716|2.509|2.482|2.431|\n|Footprint risk mean|0.155|0.189|0.206|0.238|0.227|\n|All custom graders|2.598|2.759|2.670|2.669|2.690|\n|Mean cost/task|$2.50|$3.15|$5.01|$6.51|$8.84|\n|Mean duration/task|383.8s|450.7s|716.4s|803.8s|996.9s|\n|Equivalent passes per dollar|0.138|0.153|0.083|0.058|0.051|\n\n# Why I Ran This\n\nAfter my last post comparing GPT-5.5 vs 5.4 vs Opus 4.7, I was curious how intra-model performance varied with reasoning effort. Doing research online, it's very very hard to gauge what *actual experience* is like when varying the reasoning levels, and how that applies to the work that I'm doing.\n\nI first ran this on Zod, and the result looked strange: tests were flat across low, medium, high, and xhigh, while the above-test quality signals moved around in mixed ways. Low, medium, high, and xhigh all landed at 12/28 test passes. But equivalence moved from 10/28 on low to 16/28 on medium, 13/28 on high, and 19/28 on xhigh; code-review pass moved from 4/27 to 10/27, 10/27, and 11/27. That was interesting, but not clean enough to make a default-setting claim. It could have been a Zod-specific artifact, or a sign that Opus 4.7 does not have a simple \"turn reasoning up\" curve.\n\nSo I reran the question on GraphQL-go-tools. To separate vibes from reality, and figure out where the cost/performance sweet spot is for Opus 4.7, I wanted the same reasoning-effort question on a more discriminating repo slice.\n\nThis is not meant to be a universal benchmark result - I don't have the funds or time to generate statistically significant data. The purpose is closer to \"how should I choose the reasoning setting for real repo work?\", with `GraphQL-Go-Tools` as the example repo.\n\nPublic benchmarks flatten the reviewer question that most SWEs actually care about: would I actually merge the patch, and do I want to maintain it? That's why I ran this test - to gain more insight, at a small scale, into how coding agents perform on real-world tasks.\n\nTerminal-Bench consists of esoteric problems that mostly aren't encountered in day-to-day software, SWE-bench verified is contaminated (as in models already have answers baked in), and SWE-bench Pro is useful, but generic. That is not a knock on SWE-bench or Terminal-Bench. Standardized benchmarks are useful, but they mostly answer a binary task-outcome question.\n\nThe question I care about day to day is narrower and more annoying: did the agent make the same kind of change a human merged in my codebase, and would I want to own the patch afterward?\n\n# Experimental Setup\n\nEach task is derived from a real merged PR or commit. The model gets a frozen repo snapshot, a prompt describing the change, and one attempt to produce a patch in a Docker container. Stet then applies the patch and runs the task's tests in an isolated container to check if it passed/failed.\n\nThen Stet grades the result beyond pass/fail:\n\n* **Equivalence:** does the candidate patch accomplish the same behavioral change as the original human patch?\n* **Code review:** would a reviewer accept the patch, considering correctness, introduced-bug risk, maintainability, and edge cases?\n* **Footprint risk:** how much additional code did the agent touch when compared with the human patch?\n* **Craft/discipline rubrics:** attempt to capture non-correct aspects of code. Basically, would a reviewer want to maintain this code. The categories are clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality\n\nThose metrics exist because tests alone do not answer the thing I actually care about: would this patch be something I want to merge and maintain?\n\nEvery model ran once per task with a single seed. The LLM-as-a-judge model was GPT-5.4. Each patch was scored independently - the judge sees the patch and the task, and was blinded to the model/effort that produced the patch. I also manually inspected representative examples as sanity checks. There was no human calibration pass on this task set, so I would trust the direction of the deltas more than any single absolute score.\n\nAs an aside, I've also been using these evaluations as an \"autoresearch\" optimization loop, not just a benchmark. I tell my agent something like \"make [`AGENTS.md`](http://AGENTS.md) better for this repo\"; it proposes an edit, runs Stet on historical tasks, figures out where the candidate was better / worse and why, and iterates to improve the evaluation numbers.\n\nDetails:\n\n* Model: Opus 4.7\n* Harness: Claude Code 2.1.126-2.1.138 (varied across arms by run date; npm-installed latest at each run)\n* Dataset: 29 real GraphQL-go-tools tasks.\n   * Yes this is small - however running even this used most of my weekly 20x quota\n* Main metrics:\n   * test pass\n   * semantic equivalence\n   * code-review pass\n   * footprint risk\n   * craft/discipline custom graders\n   * cost and runtime\n\n# Low: Cheaper, Shallower, and Incomplete\n\n|Metric|Low|Medium|Δ|\n|:-|:-|:-|:-|\n|All-task pass|23/29, 79.3%|28/29, 96.6%|\\-17.2pp|\n|Equivalent|10/29, 34.5%|14/29, 48.3%|\\-13.8pp|\n|Code-review pass|5/29, 17.2%|10/29, 34.5%|\\-17.2pp|\n|Footprint risk mean|0.155|0.189|\\-0.034|\n|Craft/Discipline avg|2.598|2.759|\\-0.161|\n|Cost/task (mean)|$2.50|$3.15|\\-$0.65, 0.79x|\n|Mean duration|383.8s|450.7s|\\-66.8s|\n\nLow appears to drive Opus 4.7 to work through most issues on a surface level. It is faster, cheaper, and lower-footprint (touching less files relative to the human-authored change), but misses important pieces of the task, leaving gaps in correctness.\n\nIn practice, low is superseded by medium, with just a \\~26% increase in cost ($2.50 → $3.15) and a noticeably better performance across the board.\n\nExample: PR #1230 fixes two GraphQL federation query-planner bugs and adds an empty-selection-set guard on the GraphQL datasource print path.\n\n* Task: tighten the planner's parent-chain selection and add the right-shape validation guard.\n* Lower-effort failure mode: low worked in the wrong boundary, inlining hand-rolled recursive AST helpers directly into `graphql_datasource.go` rather than registering a planner-scoped validation rule. The unique-node selection logic stayed eager, tests failed, and the patch was non-equivalent with the human PR.\n* Higher-effort change: medium did the same job at the right boundary - a dedicated validation rule wired into the planner's `printKitPool` \\- and matched the two-pass planner shape the human PR used.\n* Lesson: low does work, but at the wrong level of abstraction. It tends to inline behavior into the file it happens to be reading rather than picking the package boundary the task is actually about.\n\n# Medium: Balance of Restraint and Correctness\n\n|Metric|Medium|\n|:-|:-|\n|All-task pass|28/29, 96.6%|\n|Equivalent|14/29, 48.3%|\n|Code-review pass|10/29, 34.5%|\n|Footprint risk mean|0.189|\n|Craft/Discipline avg|2.759|\n|Cost/task (mean)|$3.15|\n|Mean duration|450.7s|\n\nMedium appears to be the level that does enough repo modeling without drifting into prior-PR rationalization, no-op stories, or oversized patch surface.\n\nIt has the best test pass count, is the most equivalent with the human patches, passes code review at a higher rate, and performs the best on the craft/discipline rubrics.\n\nWhen looking at the original Zod slice, medium improved over low, but the higher-effort signal was mixed: xhigh had the best equivalence rate, high had the best discipline average, and tests stayed flat. GraphQL is the cleaner medium-wins read.\n\nMedium spends its extra effort productively - looking at the agent trajectories, it runs more tests than high/xhigh while avoiding the bloated time/tokens from max. On this slice, medium looks like the local optimum: enough reasoning to execute the user's intent, without going down too many rabbit holes.\n\nExample: PR #1260 makes GraphQL subscription query plans include trigger metadata (subgraph name/ID, trigger query), and lets a `SkipLoader` query-plan introspection request return the plan for a subscription without opening the upstream stream.\n\n* Task: make the existing `SkipLoader` early-return reachable for plan-only requests, then surface trigger metadata in the printed plan. The repo already contained partial scaffolding from PR #1008, which is the trap.\n* Lower-effort failure mode: low got confused by the partial pre-existing code and asked the operator for the diff - \"I can't proceed without knowing what specifically PR #1260 changes.\" End of turn, no patch.\n* Higher-effort failure mode: high and xhigh used their extra reasoning budget to dig up commit hashes (`34cc4fa8`, `69485dfe`), conclude the feature had already been shipped in earlier PRs, and stop with `end_turn` and no patch. Not a timeout, not a refusal - a confidently-wrong no-op. Xhigh's final message: *\"This work was originally ad","offTopic":true},{"id":"77bb0038-d203-4610-8452-73d8e8024725","excerpt":"Opus 4.7 Low Vs Medium Vs High Vs Xhigh Vs Max: the Reasoning Curve on 29 Real Tasks from an Open Source Repo — [Original Reddit post](https://www.reddit.com/r/ClaudeCode/comments/1tc293h/opus_47_low_vs_medium_vs_high_vs_xhigh_vs_max_the/)\n\nTL;DR\nI ran Opus 4.7 in Claude Code at all reasoning effort settings (low, medi","url":"https://lemmy.world/post/46797783","role":"demand","weight":1.0972949,"occurredAt":"2026-05-13T15:28:32.989Z","sourceKey":"lemmy","sourceName":"Lemmy","credibility":0.58,"venue":"lemmy.world","intent":"tool_discovery","painScore":0.34041443,"sentiment":-0.042016808,"confidence":0.8186236,"matchedPatterns":["looking_for","frustrating","switching_from","manual_process","product:anthropic"],"statement":"But equivalence moved from 10/28 on low to 16/28 on medium, 13/28 on high, and 19/28 on xhigh; code-review pass moved from 4/27 to 10/27, 10/27, and 11/27.","title":"Opus 4.7 Low Vs Medium Vs High Vs Xhigh Vs Max: the Reasoning Curve on 29 Real Tasks from an Open Source Repo","body":"[Original Reddit post](https://www.reddit.com/r/ClaudeCode/comments/1tc293h/opus_47_low_vs_medium_vs_high_vs_xhigh_vs_max_the/)\n\nTL;DR\nI ran Opus 4.7 in Claude Code at all reasoning effort settings (low, medium, high, xhigh, and max) on the same 29 tasks from an open source repo (GraphQL-go-tools, in Go).\nOn this slice, Opus 4.7 did not behave like a model where more reasoning effort had a linear correlation with more intelligence. In fact, the curve appears to peak at medium.\nIf you think this is weird, I agree! This was the follow-up to a Zod run where Opus also looked non-monotonic. I reran the question on GraphQL-go-tools because I wanted a more discriminating repo slice and didn’t trust the fact that more reasoning != better outcomes. Running on the GraphQL repo helped clarified the result: Opus still did not show a simple higher-reasoning-is-better curve.\nThe contrast is GPT-5.5 in Codex, which overall\ndid\nshow the intuitive curve: more reasoning bought more semantic/review quality. That post is here:\nhttps://www.stet.sh/blog/gpt-55-codex-graphql-reasoning-curve\nMedium has the best test pass rate, highest equivalence with the original human-authored changes, the best code-review pass rate, and the best aggregate craft/discipline rate. Low is cheaper and faster, but it drops too much correctness. High, xhigh, and max spend more time and money without beating medium on the metrics that matter.\nMore reasoning effort doesn't only cost more - it changes the way Claude works, but without reliably improving judgment. Xhigh inflates the test/fixture surface most. Max is busier overall and has the largest implementation-line footprint. But even though both are supposedly thinking more, neither produces \"better\" patches than medium.\nOne likely reason: Opus 4.7 uses adaptive thinking - the model already picks its own reasoning budget per task, so the effort knob biases an already-adaptive policy rather than buying more intelligence. More on this below.\nAn illuminating example is PR #1260. After retry, medium recovered into a real patch. High and xhigh used their extra reasoning budget to dig up commit hashes from prior PRs and confidently declare \"no work needed\" - voluntarily ending the turn with no patch. Medium and max read the literal control flow and made the fix.\nOne broader takeaway for me: this should not have to be a one-off manual benchmark. If reasoning level changes the kind of patch an agent writes, the natural next step is to let the agent test and improve its own setup on real repo work.\nFor this post, \"equivalent\" means the patch matched the intent of the merged human PR; \"code-review pass\" means an AI reviewer judged it acceptable; craft/discipline is a 0-4 maintainability/style rubric; footprint risk is how much extra code the agent touched relative to the human patch.\nI also made an interactive version with pretty charts and per-task drilldowns here:\nhttps://stet.sh/blog/opus-47-graphql-reasoning-curve\nThe data:\nWhy I Ran This\nAfter my last post comparing GPT-5.5 vs 5.4 vs Opus 4.7, I was curious how intra-model performance varied with reasoning effort. Doing research online, it's very very hard to gauge what\nactual experience\nis like when varying the reasoning levels, and how that applies to the work that I'm doing.\nI first ran this on Zod, and the result looked strange: tests were flat across low, medium, high, and xhigh, while the above-test quality signals moved around in mixed ways. Low, medium, high, and xhigh all landed at 12/28 test passes. But equivalence moved from 10/28 on low to 16/28 on medium, 13/28 on high, and 19/28 on xhigh; code-review pass moved from 4/27 to 10/27, 10/27, and 11/27. That was interesting, but not clean enough to make a default-setting claim. It could have been a Zod-specific artifact, or a sign that Opus 4.7 does not have a simple \"turn reasoning up\" curve.\nSo I reran the question on GraphQL-go-tools. To separate vibes from reality, and figure out where the cost/performance sweet spot is for Opus 4.7, I wanted the same reasoning-effort question on a more discriminating repo slice.\nThis is not meant to be a universal benchmark result - I don't have the funds or time to generate statistically significant data. The purpose is closer to \"how should I choose the reasoning setting for real repo work?\", with\nGraphQL-Go-Tools\nas the example repo.\nPublic benchmarks flatten the reviewer question that most SWEs actually care about: would I actually merge the patch, and do I want to maintain it? That's why I ran this test - to gain more insight, at a small scale, into how coding agents perform on real-world tasks.\nTerminal-Bench consists of esoteric problems that mostly aren't encountered in day-to-day software, SWE-bench verified is contaminated (as in models already have answers baked in), and SWE-bench Pro is useful, but generic. That is not a knock on SWE-bench or Terminal-Bench. Standardized benchmarks are useful, but they mostly answer a binary task-outcome question.\nThe question I care about day to day is narrower and more annoying: did the agent make the same kind of change a human merged in my codebase, and would I want to own the patch afterward?\nExperimental Setup\nEach task is derived from a real merged PR or commit. The model gets a frozen repo snapshot, a prompt describing the change, and one attempt to produce a patch in a Docker container. Stet then applies the patch and runs the task's tests in an isolated container to check if it passed/failed.\nThen Stet grades the result beyond pass/fail:\nEquivalence:\ndoes the candidate patch accomplish the same behavioral change as the original human patch?\nCode review:\nwould a reviewer accept the patch, considering correctness, introduced-bug risk, maintainability, and edge cases?\nFootprint risk:\nhow much additional code did the agent touch when compared with the human patch?\nCraft/discipline rubrics:\nattempt to capture non-correct aspects of code. Basically, would a reviewer want to maintain this code. The categories are clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality\nThose metrics exist because tests alone do not answer the thing I actually care about: would this patch be something I want to merge and maintain?\nEvery model ran once per task with a single seed. The LLM-as-a-judge model was GPT-5.4. Each patch was scored independently - the judge sees the patch and the task, and was blinded to the model/effort that produced the patch. I also manually inspected representative examples as sanity checks. There was no human calibration pass on this task set, so I would trust the direction of the deltas more than any single absolute score.\nAs an aside, I've also been using these evaluations as an \"autoresearch\" optimization loop, not just a benchmark. I tell my agent something like \"make\nAGENTS.md\nbetter for this repo\"; it proposes an edit, runs Stet on historical tasks, figures out where the candidate was better / worse and why, and iterates to improve the evaluation numbers.\nDetails:\nModel: Opus 4.7\nHarness: Claude Code 2.1.126-2.1.138 (varied across arms by run date; npm-installed latest at each run)\nDataset: 29 real GraphQL-go-tools tasks.\nYes this is small - however running even this used most of my weekly 20x quota\nMain metrics:\ntest pass\nsemantic equivalence\ncode-review pass\nfootprint risk\ncraft/discipline custom graders\ncost and runtime\nLow: Cheaper, Shallower, and Incomplete\nLow appears to drive Opus 4.7 to work through most issues on a surface level. It is faster, cheaper, and lower-footprint (touching less files relative to the human-authored change), but misses important pieces of the task, leaving gaps in correctness.\nIn practice, low is superseded by medium, with just a ~26% increase in cost ($2.50 → $3.15) and a noticeably better performance across the board.\nExample: PR #1230 fixes two GraphQL federation query-planner bugs and adds an empty-selection-set guard on the GraphQL datasource print path.\nTask: tighten the planner's parent-chain selection and add the right-shape validation guard.\nLower-effort failure mode: low worked in the wrong boundary, inlining hand-rolled recursive AST helpers directly into\ngraphql_datasource.go\nrather than registering a planner-scoped validation rule. The unique-node selection logic stayed eager, tests failed, and the patch was non-equivalent with the human PR.\nHigher-effort change: medium did the same job at the right boundary - a dedicated validation rule wired into the planner's\nprintKitPool\n- and matched the two-pass planner shape the human PR used.\nLesson: low does work, but at the wrong level of abstraction. It tends to inline behavior into the file it happens to be reading rather than picking the package boundary the task is actually about.\nMedium: Balance of Restraint and Correctness\nMedium appears to be the level that does enough repo modeling without drifting into prior-PR rationalization, no-op stories, or oversized patch surface.\nIt has the best test pass count, is the most equivalent with the human patches, passes code review at a higher rate, and performs the best on the craft/discipline rubrics.\nWhen looking at the original Zod slice, medium improved over low, but the higher-effort signal was mixed: xhigh had the best equivalence rate, high had the best discipline average, and tests stayed flat. GraphQL is the cleaner medium-wins read.\nMedium spends its extra effort productively - looking at the agent trajectories, it runs more tests than high/xhigh while avoiding the bloated time/tokens from max. On this slice, medium looks like the local optimum: enough reasoning to execute the user's intent, without going down too many rabbit holes.\nExample: PR #1260 makes GraphQL subscription query plans include trigger metadata (subgraph name/ID, trigger query), and lets a\nSkipLoader\nquery-plan introspection request return the plan for a subscription without opening the upstream stream.\nTask: make the existing\nSkipLoader\nearly-return reachable for plan-only requests, then surface trigger metadata in the printed plan. The repo already contained partial scaffolding from PR #1008, which is the trap.\nLower-effort failure mode: low got confused by the partial pre-existing code and asked the operator for the diff - \"I can't proceed without knowing what specifically PR #1260 changes.\" End of turn, no patch.\nHigher-effort failure mode: high and xhigh used their extra reasoning budget to dig up commit hashes (\n34cc4fa8\n,\n69485dfe\n), conclude the feature had already been shipped in earlier PRs, and stop with\nend_turn\nand no patch. Not a timeout, not a refusal - a confidently-wrong no-op. Xhigh's final message:\n\"This work was originally added in commit\n34cc4fa8\n(PR #1008) and refined by\n69485dfe\n(PR #1120). No code changes are needed; nothing left to implement.\"\nMedium's win: read the literal control flow, saw that the existing\nSkipLoader\nbranch sat\nafter\na\nTrigger.Source == nil\nguard and was therefore unreachable for plan-only requests, and made the minimum hoist-and-extract fix. Tests passed. (Max made the same fix plus an added regression test.)\nLesson: on tasks where the repo already contains adjacent prior work, more reasoning amplifies the temptation to rationalize a no-op. The extra budget doesn't go into running the code - it goes into building a more sophisticated story for why running the code isn't necessary.\nHigh: the Limits of More Thinking\nAt high, we begin to see signs of \"overthinking\".\nHigh costs $5.01/task versus medium's $3.15/task and runs 716.4s/task versus medium's 450.7s/task. It also makes more shell calls and tool calls than medium. But its pass rate falls to 26/29, equivalence falls to 12/29, review pass falls to 7/29, review-rubric mean falls to 2.509, and aggregate custom quality falls to 2.670.\nThat pattern suggests extra effort is not strictly adding more intelligence and discovering more correct implementation paths. It may be spending addition","offTopic":true},{"id":"bff042e8-67ab-4e87-aec5-9504d16cb37f","excerpt":"GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos — EDIT:  A few of you have asked if I'd run this on your repo. I'm doing 5 free in May to refine the methodology (all run locally - I won't see your code). If you're debating models, harness, reasoning levels, [AGENTS.md](http://AGENTS.md),","url":"https://www.reddit.com/r/codex/comments/1t0xt5m/gpt55_vs_gpt54_vs_opus_47_on_56_real_coding_tasks/","role":"pain","weight":1.19515,"occurredAt":"2026-05-01T16:10:04.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"codex","intent":"feature_request","painScore":0.59,"sentiment":-0.575,"confidence":0.75166667,"matchedPatterns":["missing_feature"],"statement":"The recurring Opus failure mode is passing the visible tests while missing companion work the human PR included.","title":"GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos","body":"EDIT:  A few of you have asked if I'd run this on your repo. I'm doing 5 free in May to refine the methodology (all run locally - I won't see your code). If you're debating models, harness, reasoning levels, [AGENTS.md](http://AGENTS.md), or [SKILL.md](http://SKILL.md), edits, DM me with the decision you're trying to make, and we can go from there! Especially interested in organizations doing evaluations (as I am in one, and run into this problem frequently at work)\n\nTLDR; OpenAI cooked with GPT-5.5\n\nOpus 4.7 writes smaller patches. GPT-5.5 writes patches that more often survive review. Which one you want depends on whether \"small\" means disciplined or incomplete in your repo.\n\nI ran both models, plus GPT-5.4, on 56 real coding tasks from two open-source repos: 27 tasks from Zod and 29 from `graphql-go-tools` (these codebases were selected arbitrarily and may not represent your experience - that's the point of why running your own benchmarks is important!) Each model ran in its native agent harness at default settings: Anthropic models in Claude Code, OpenAI models in OpenAI Codex CLI.\n\nThe result was not \"one model wins everything.\" GPT-5.5 was the best shipping default across these runs. By \"shipping,\" I mean the model I would most often trust to produce a patch that passes tests, matches the intended human change, and survives code review. Opus 4.7 was still doing something valuable: it wrote much smaller patches.\n\nOn Zod, that looked like a real tradeoff. On `graphql-go-tools`, it looked more like under-implementation.\n\n**GPT-5.5 ships more often. Opus 4.7 ships smaller. Which one wins on your repo depends on whether your bottleneck is review or footprint.**\n\nThat distinction is why repo-specific evals matter. Public benchmarks flatten model behavior into one number aggregated at massive scale. Real code turns it into a workflow decision on your specific codebase and standards.\n\n*I used* [*Stet*](https://www.stet.sh)*, an evaluation framework I am building for real-repo coding-agent benchmarks, to grade more than test pass/fail: behavioral equivalence to the human patch, code-review acceptability, footprint risk, and craft/discipline rubrics. This post is not a claim about all coding tasks. It is a concrete look at how three frontier models behaved on two real codebases.*\n\n|Model|Harness|Reasoning Level|\n|:-|:-|:-|\n|Opus 4.7|Claude Code|high|\n|GPT-5.4|Codex CLI|high|\n|GPT-5.5|Codex CLI|high|\n\n# The short version\n\nAcross 56 scored tasks:\n\n|Metric|Opus 4.7|GPT-5.4|GPT-5.5|\n|:-|:-|:-|:-|\n|Tests pass|33/56|31/56|**38/56**|\n|Equivalent to human patch|19/56|35/56|**40/56**|\n|Clean pass: tests + review|10/56|11/56|**28/56**|\n|Mean footprint risk, lower is better|**0.20**|0.34|0.32|\n|Mean time/task|11m18s|8m24s|**6m56s**|\n|Cost/task|$3.43|**$2.39**|$2.86|\n\nGPT-5.5 is the quality leader. It passes the most tests, matches the human patch most often, and clears the reviewer about three times as often as Opus.\n\nOpus is the footprint leader. Its patches are smaller and lower-risk by Stet's footprint model. But a small patch is only good when it is complete. The recurring Opus failure mode is passing the visible tests while missing companion work the human PR included.\n\nGPT-5.5 is also the efficiency leader on tokens and wall-clock. It used fewer input tokens, fewer output tokens, and less summed agent time than either competitor. GPT-5.4 is still the cost leader because its pricing is lower, but the cost advantage did not offset the clean-pass gap in these runs.\n\nThe repo split is where the result gets interesting:\n\n|Repo|Model|Tests|Equiv yes|Review pass|Clean pass|\n|:-|:-|:-|:-|:-|:-|\n|Zod, 27 scored tasks|Opus 4.7|**12**|11|6|5|\n|Zod, 27 scored tasks|GPT-5.4|9|**18**|10|5|\n|Zod, 27 scored tasks|GPT-5.5|**12**|**18**|**14**|**10**|\n|graphql-go-tools, 29 tasks|Opus 4.7|21|8|5|5|\n|graphql-go-tools, 29 tasks|GPT-5.4|22|17|6|6|\n|graphql-go-tools, 29 tasks|GPT-5.5|**26**|**22**|**19**|**18**|\n\nOn Zod, GPT-5.5 and Opus tie on tests. GPT-5.5 wins on reviewer judgment. Opus wins on diff size.\n\nOn `graphql-go-tools`, GPT-5.5 wins outright. It passes more tests, produces far more clean passes, and is closer to the human patch. Opus still writes the smallest patches, but the small-patch strategy misses too much.\n\n# Full scorecard\n\n|Metric|Opus 4.7|GPT-5.4|GPT-5.5|\n|:-|:-|:-|:-|\n|Code-review pass|11/56|16/56|**33/56**|\n|Code-review avg: correctness + bug safety|2.33|2.59|**3.08**|\n|\\- Correctness|2.11|2.60|**3.16**|\n|\\- Introduced-bug safety|2.55|2.56|**3.04**|\n|\\- Maintainability, GraphQL only|2.07|2.55|**3.03**|\n|Custom grader avg, 8 rubrics|2.33|2.40|**2.62**|\n|Craft score, 0-4|2.41|2.54|**2.78**|\n|\\- Clarity / coherence / robustness|2.56 / 1.95 / 1.92|2.75 / 2.18 / 2.43|**2.91 / 2.51 / 2.69**|\n|Discipline score, 0-4|2.20|2.16|**2.36**|\n|\\- Scope discipline / diff minimality|2.39 / 2.42|2.18 / 2.28|**2.45 / 2.46**|\n|Total input tokens|239.1M|222.3M|**201.8M**|\n|Total output tokens|1.29M|1.09M|**0.72M**|\n\nThe quality-score rows are there to avoid treating \"more tests passed\" as the whole story. Code review is one grader: correctness, introduced-bug risk, and maintainability where available. The custom grader average is separate: eight additive rubrics split into five craft dimensions and three discipline dimensions. Across both layers, GPT-5.5 is not merely preferred in the abstract. It is rated higher on correctness, lower introduced-bug risk, GraphQL maintainability, coherence, robustness, scope discipline, and diff minimality relative to the requested task. Opus still wins the mechanical footprint row, which is the useful tension: smaller diffs, but not consistently more disciplined diffs.\n\n# How the benchmark works\n\nEach task is derived from a real merged commit. The model gets a frozen repo snapshot, a prompt describing the change, and one attempt to produce a patch — running in its native shipped agent harness with no Stet-side scaffolding: Opus 4.7 in Claude Code (`claude -p`); GPT-5.5 and GPT-5.4 in OpenAI Codex CLI (`codex exec`); both at default settings. Stet applies the patch and runs the task's tests in an isolated container.\n\nThen Stet grades the result beyond pass/fail:\n\n* **Tests:** did the patch satisfy the executable acceptance tests?\n* **Equivalence:** does the candidate patch accomplish the same behavioral change as the original human patch?\n* **Code review:** would a reviewer accept the patch, considering correctness, introduced-bug risk, maintainability, and edge cases?\n* **Footprint risk:** how much review and regression surface did the patch create?\n* **Craft/discipline rubrics:** clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality.\n\nEvery model ran once per task with a single seed. The judge model for equivalence and rubrics was GPT-5.4, run with identical rubric versions across all three arms. Each patch was scored independently — the judge sees the patch and the task, not the arm label or the model that produced it. There is no dual-rater calibration, so treat absolute scores as directional; the cross-arm deltas are the thing to trust.\n\n# Tests are signal, not the finish line\n\nThe most useful row in the table is not tests. It is clean pass: tests pass and the code-review grader accepts the patch.\n\nOn Zod, Opus and GPT-5.5 both passed 12 of 27 scored tasks. If you stop there, the models look tied. But GPT-5.5 produced 10 clean passes; Opus produced 5.\n\nOn `graphql-go-tools`, the same pattern was amplified. GPT-5.5 passed 26 of 29 tests and produced 18 clean passes. Opus passed 21 tests but produced only 5 clean passes.\n\nThat is the gap you feel in code review. The tests say \"this patch probably works.\" The reviewer asks \"is this the patch we want to maintain?\"\n\nOne GraphQL task shows the difference. PR #1001 changed an HTTP datasource `OnFinished` hook so consumers could inspect request and response metadata. All three models passed tests and were judged equivalent. Only GPT-5.5 cleared code review. The other two got warnings around API shape, raw HTTP object exposure, and robustness at the hook boundary.\n\nThat is not a benchmark trick, rather, this is reflective normal engineering culture where code is reviewed: three patches can satisfy the same test and still differ materially in review quality. You only want to merge the code that is high-quality and maintainable, even if it technically works.\n\n# What the reviewer saw\n\nThe code review and craft/discipline rows explain why the result is not reducible to \"GPT-5.5 changes more files.\" Two patch autopsies make the numbers less abstract.\n\n**Zod async codecs and defaults.** The task was to make codec pipelines work with async transforms, prevent defaults from becoming `undefined`, and generate stub package manifests for the build. All three models failed tests. If you stop at the test row, the task tells you nothing.\n\nThe reviewer found a real ordering underneath. Opus changed 8 files and missed central semantics: defaults could still allow `undefined`, core codec definitions remained synchronous, generated stubs were not published, and `prefault()` was tightened even though the request was about `.default()`. GPT-5.4 got closer with an 11-file patch and was judged behaviorally equivalent, but it still over-tightened adjacent API by restricting `prefault`. GPT-5.5 also failed tests, but it was judged equivalent and scored better on correctness and introduced-bug risk because it covered the schema/build behavior more cleanly: codec/default tests, version metadata, stub-manifest scripts, and the relevant `packages/zod/src/v4/*/schemas.ts` surfaces.\n\nThat is a different kind of signal from pass/fail. It says GPT-5.5 was not merely getting luckier tests; even on a miss, it more often moved the right pieces.\n\n**GraphQL Apollo-compatible validation.** PR #1169 aligned field-selection validation errors with GraphQL spec and Apollo Router conventions. All three models produced patches. All three passed tests. Only GPT-5.5 cleared equivalence and review.\n\nOpus touched 11 files and passed tests, but missed enum and wrapped-scalar leaf validation, pointed some leaf-selection locations at the field instead of the selection set, left an inline-fragment message non-spec-compliant, and did not apply validation status uniformly. GPT-5.4 touched 12 files and also passed tests, but broadened behavior in the wrong places: unconditional validation metadata, incomplete enum/wrapped scalar handling, broad request-error conversion, and stale compatibility API.\n\nGPT-5.5 touched fewer files than either one, 10 total and 6 non-test, while still adding more targeted behavior: aligned field-selection messages, requested locations, and centralized Apollo validation metadata. This is the clean reviewer example: tests saw three passes; semantic grading saw one patch that actually matched the convention the PR was trying to establish.\n\nThis is what the score rows are trying to summarize. GPT-5.5's biggest review lead is correctness: 3.16 versus 2.60 for GPT-5.4 and 2.11 for Opus. The custom graders say the same thing from another angle: GPT-5.5 leads coherence and robustness because its patches more often carry the change through the repo's existing surfaces instead of stopping at the first passing path.\n\nThe discipline row is the one I would not overclaim. GPT-5.5 leads, but narrowly: 2.36 versus 2.20 for Opus and 2.16 for GPT-5.4. Opus wins raw footprint. GPT-5.5 narrowly wins task-relative discipline. The grader is separating \"small\" from \"appropriately scoped.\" A patch can be compact and still undisciplined if it stops before the task is done.\n\n# What Opus is doing\n\nOpus 4.7 is cautious. It writes smaller patches, touches fewer files, and has the lowest footprint risk in both repos.\n\nOn Zod, that caution is often attractive. Zod has many contained tasks where the correct move is a precise source edit, a type change, and maybe a small test update. Opus tied GPT-5.5 on tests","offTopic":true},{"id":"7943bb4c-af4d-40fc-857f-d08124976aba","excerpt":"GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos — EDIT:  A few of you have asked if I'd run this on your repo. I'm doing 5 free in May to refine the methodology (all run locally - I won't see your code). If you're debating models, harness, reasoning levels, [AGENTS.md](http://AGENTS.md),","url":"https://www.reddit.com/r/ClaudeCode/comments/1t0xrad/gpt55_vs_gpt54_vs_opus_47_on_56_real_coding_tasks/","role":"pain","weight":1.19515,"occurredAt":"2026-05-01T16:08:15.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"ClaudeCode","intent":"feature_request","painScore":0.59,"sentiment":-0.575,"confidence":0.75166667,"matchedPatterns":["missing_feature"],"statement":"The recurring Opus failure mode is passing the visible tests while missing companion work the human PR included.","title":"GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos","body":"EDIT:  A few of you have asked if I'd run this on your repo. I'm doing 5 free in May to refine the methodology (all run locally - I won't see your code). If you're debating models, harness, reasoning levels, [AGENTS.md](http://AGENTS.md), or [SKILL.md](http://SKILL.md), edits, DM me with the decision you're trying to make, and we can go from there! Especially interested in organizations doing evaluations (as I am in one, and run into this problem frequently at work)\n\nTLDR; OpenAI cooked with GPT-5.5\n\nOpus 4.7 writes smaller patches. GPT-5.5 writes patches that more often survive review. Which one you want depends on whether \"small\" means disciplined or incomplete in your repo.\n\nI ran both models, plus GPT-5.4, on 56 real coding tasks from two open-source repos: 27 tasks from Zod and 29 from `graphql-go-tools` (these codebases were selected arbitrarily and may not represent your experience - that's the point of why running your own benchmarks is important!) Each model ran in its native agent harness at default settings: Anthropic models in Claude Code, OpenAI models in OpenAI Codex CLI.\n\nThe result was not \"one model wins everything.\" GPT-5.5 was the best shipping default across these runs. By \"shipping,\" I mean the model I would most often trust to produce a patch that passes tests, matches the intended human change, and survives code review. Opus 4.7 was still doing something valuable: it wrote much smaller patches.\n\nOn Zod, that looked like a real tradeoff. On `graphql-go-tools`, it looked more like under-implementation.\n\n**GPT-5.5 ships more often. Opus 4.7 ships smaller. Which one wins on your repo depends on whether your bottleneck is review or footprint.**\n\nThat distinction is why repo-specific evals matter. Public benchmarks flatten model behavior into one number aggregated at massive scale. Real code turns it into a workflow decision on your specific codebase and standards.\n\n*I used* [*Stet*](https://www.stet.sh)*, an evaluation framework I am building for real-repo coding-agent benchmarks, to grade more than test pass/fail: behavioral equivalence to the human patch, code-review acceptability, footprint risk, and craft/discipline rubrics. This post is not a claim about all coding tasks. It is a concrete look at how three frontier models behaved on two real codebases.*\n\n|Model|Harness|Reasoning Level|\n|:-|:-|:-|\n|Opus 4.7|Claude Code|high|\n|GPT-5.4|Codex CLI|high|\n|GPT-5.5|Codex CLI|high|\n\n# The short version\n\nAcross 56 scored tasks:\n\n|Metric|Opus 4.7|GPT-5.4|GPT-5.5|\n|:-|:-|:-|:-|\n|Tests pass|33/56|31/56|**38/56**|\n|Equivalent to human patch|19/56|35/56|**40/56**|\n|Clean pass: tests + review|10/56|11/56|**28/56**|\n|Mean footprint risk, lower is better|**0.20**|0.34|0.32|\n|Mean time/task|11m18s|8m24s|**6m56s**|\n|Estimated run cost|$3.43|**$2.39**|$2.86|\n\nGPT-5.5 is the quality leader. It passes the most tests, matches the human patch most often, and clears the reviewer about three times as often as Opus.\n\nOpus is the footprint leader. Its patches are smaller and lower-risk by Stet's footprint model. But a small patch is only good when it is complete. The recurring Opus failure mode is passing the visible tests while missing companion work the human PR included.\n\nGPT-5.5 is also the efficiency leader on tokens and wall-clock. It used fewer input tokens, fewer output tokens, and less summed agent time than either competitor. GPT-5.4 is still the cost leader because its pricing is lower, but the cost advantage did not offset the clean-pass gap in these runs.\n\nThe repo split is where the result gets interesting:\n\n|Repo|Model|Tests|Equiv yes|Review pass|Clean pass|\n|:-|:-|:-|:-|:-|:-|\n|Zod, 27 scored tasks|Opus 4.7|**12**|11|6|5|\n|Zod, 27 scored tasks|GPT-5.4|9|**18**|10|5|\n|Zod, 27 scored tasks|GPT-5.5|**12**|**18**|**14**|**10**|\n|graphql-go-tools, 29 tasks|Opus 4.7|21|8|5|5|\n|graphql-go-tools, 29 tasks|GPT-5.4|22|17|6|6|\n|graphql-go-tools, 29 tasks|GPT-5.5|**26**|**22**|**19**|**18**|\n\nOn Zod, GPT-5.5 and Opus tie on tests. GPT-5.5 wins on reviewer judgment. Opus wins on diff size.\n\nOn `graphql-go-tools`, GPT-5.5 wins outright. It passes more tests, produces far more clean passes, and is closer to the human patch. Opus still writes the smallest patches, but the small-patch strategy misses too much.\n\n# Full scorecard\n\n|Metric|Opus 4.7|GPT-5.4|GPT-5.5|\n|:-|:-|:-|:-|\n|Code-review pass|11/56|16/56|**33/56**|\n|Code-review avg: correctness + bug safety|2.33|2.59|**3.08**|\n|\\- Correctness|2.11|2.60|**3.16**|\n|\\- Introduced-bug safety|2.55|2.56|**3.04**|\n|\\- Maintainability, GraphQL only|2.07|2.55|**3.03**|\n|Custom grader avg, 8 rubrics|2.33|2.40|**2.62**|\n|Craft score, 0-4|2.41|2.54|**2.78**|\n|\\- Clarity / coherence / robustness|2.56 / 1.95 / 1.92|2.75 / 2.18 / 2.43|**2.91 / 2.51 / 2.69**|\n|Discipline score, 0-4|2.20|2.16|**2.36**|\n|\\- Scope discipline / diff minimality|2.39 / 2.42|2.18 / 2.28|**2.45 / 2.46**|\n|Total input tokens|239.1M|222.3M|**201.8M**|\n|Total output tokens|1.29M|1.09M|**0.72M**|\n\nThe quality-score rows are there to avoid treating \"more tests passed\" as the whole story. Code review is one grader: correctness, introduced-bug risk, and maintainability where available. The custom grader average is separate: eight additive rubrics split into five craft dimensions and three discipline dimensions. Across both layers, GPT-5.5 is not merely preferred in the abstract. It is rated higher on correctness, lower introduced-bug risk, GraphQL maintainability, coherence, robustness, scope discipline, and diff minimality relative to the requested task. Opus still wins the mechanical footprint row, which is the useful tension: smaller diffs, but not consistently more disciplined diffs.\n\n# How the benchmark works\n\nEach task is derived from a real merged commit. The model gets a frozen repo snapshot, a prompt describing the change, and one attempt to produce a patch — running in its native shipped agent harness with no Stet-side scaffolding: Opus 4.7 in Claude Code (`claude -p`); GPT-5.5 and GPT-5.4 in OpenAI Codex CLI (`codex exec`); both at default settings. Stet applies the patch and runs the task's tests in an isolated container.\n\nThen Stet grades the result beyond pass/fail:\n\n* **Tests:** did the patch satisfy the executable acceptance tests?\n* **Equivalence:** does the candidate patch accomplish the same behavioral change as the original human patch?\n* **Code review:** would a reviewer accept the patch, considering correctness, introduced-bug risk, maintainability, and edge cases?\n* **Footprint risk:** how much review and regression surface did the patch create?\n* **Craft/discipline rubrics:** clarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality.\n\nEvery model ran once per task with a single seed. The judge model for equivalence and rubrics was GPT-5.4, run with identical rubric versions across all three arms. Each patch was scored independently — the judge sees the patch and the task, not the arm label or the model that produced it. There is no dual-rater calibration, so treat absolute scores as directional; the cross-arm deltas are the thing to trust.\n\n# Tests are signal, not the finish line\n\nThe most useful row in the table is not tests. It is clean pass: tests pass and the code-review grader accepts the patch.\n\nOn Zod, Opus and GPT-5.5 both passed 12 of 27 scored tasks. If you stop there, the models look tied. But GPT-5.5 produced 10 clean passes; Opus produced 5.\n\nOn `graphql-go-tools`, the same pattern was amplified. GPT-5.5 passed 26 of 29 tests and produced 18 clean passes. Opus passed 21 tests but produced only 5 clean passes.\n\nThat is the gap you feel in code review. The tests say \"this patch probably works.\" The reviewer asks \"is this the patch we want to maintain?\"\n\nOne GraphQL task shows the difference. PR #1001 changed an HTTP datasource `OnFinished` hook so consumers could inspect request and response metadata. All three models passed tests and were judged equivalent. Only GPT-5.5 cleared code review. The other two got warnings around API shape, raw HTTP object exposure, and robustness at the hook boundary.\n\nThat is not a benchmark trick, rather, this is reflective normal engineering culture where code is reviewed: three patches can satisfy the same test and still differ materially in review quality. You only want to merge the code that is high-quality and maintainable, even if it technically works.\n\n# What the reviewer saw\n\nThe code review and craft/discipline rows explain why the result is not reducible to \"GPT-5.5 changes more files.\" Two patch autopsies make the numbers less abstract.\n\n**Zod async codecs and defaults.** The task was to make codec pipelines work with async transforms, prevent defaults from becoming `undefined`, and generate stub package manifests for the build. All three models failed tests. If you stop at the test row, the task tells you nothing.\n\nThe reviewer found a real ordering underneath. Opus changed 8 files and missed central semantics: defaults could still allow `undefined`, core codec definitions remained synchronous, generated stubs were not published, and `prefault()` was tightened even though the request was about `.default()`. GPT-5.4 got closer with an 11-file patch and was judged behaviorally equivalent, but it still over-tightened adjacent API by restricting `prefault`. GPT-5.5 also failed tests, but it was judged equivalent and scored better on correctness and introduced-bug risk because it covered the schema/build behavior more cleanly: codec/default tests, version metadata, stub-manifest scripts, and the relevant `packages/zod/src/v4/*/schemas.ts` surfaces.\n\nThat is a different kind of signal from pass/fail. It says GPT-5.5 was not merely getting luckier tests; even on a miss, it more often moved the right pieces.\n\n**GraphQL Apollo-compatible validation.** PR #1169 aligned field-selection validation errors with GraphQL spec and Apollo Router conventions. All three models produced patches. All three passed tests. Only GPT-5.5 cleared equivalence and review.\n\nOpus touched 11 files and passed tests, but missed enum and wrapped-scalar leaf validation, pointed some leaf-selection locations at the field instead of the selection set, left an inline-fragment message non-spec-compliant, and did not apply validation status uniformly. GPT-5.4 touched 12 files and also passed tests, but broadened behavior in the wrong places: unconditional validation metadata, incomplete enum/wrapped scalar handling, broad request-error conversion, and stale compatibility API.\n\nGPT-5.5 touched fewer files than either one, 10 total and 6 non-test, while still adding more targeted behavior: aligned field-selection messages, requested locations, and centralized Apollo validation metadata. This is the clean reviewer example: tests saw three passes; semantic grading saw one patch that actually matched the convention the PR was trying to establish.\n\nThis is what the score rows are trying to summarize. GPT-5.5's biggest review lead is correctness: 3.16 versus 2.60 for GPT-5.4 and 2.11 for Opus. The custom graders say the same thing from another angle: GPT-5.5 leads coherence and robustness because its patches more often carry the change through the repo's existing surfaces instead of stopping at the first passing path.\n\nThe discipline row is the one I would not overclaim. GPT-5.5 leads, but narrowly: 2.36 versus 2.20 for Opus and 2.16 for GPT-5.4. Opus wins raw footprint. GPT-5.5 narrowly wins task-relative discipline. The grader is separating \"small\" from \"appropriately scoped.\" A patch can be compact and still undisciplined if it stops before the task is done.\n\n# What Opus is doing\n\nOpus 4.7 is cautious. It writes smaller patches, touches fewer files, and has the lowest footprint risk in both repos.\n\nOn Zod, that caution is often attractive. Zod has many contained tasks where the correct move is a precise source edit, a type change, and maybe a small test update. Opus tied GPT-5.5","offTopic":true},{"id":"3ca4d344-10b8-4746-a1be-88d18d8dc2d9","excerpt":"GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos — [Original Reddit post](https://www.reddit.com/r/ClaudeCode/comments/1t0xrad/gpt55_vs_gpt54_vs_opus_47_on_56_real_coding_tasks/)\n\nTLDR; OpenAI cooked with GPT-5.5\nOpus 4.7 writes smaller patches. GPT-5.5 writes patches that more often surv","url":"https://lemmy.world/post/46305914","role":"pain","weight":0.9253747,"occurredAt":"2026-05-01T16:13:08.722Z","sourceKey":"lemmy","sourceName":"Lemmy","credibility":0.58,"venue":"lemmy.world","intent":"feature_request","painScore":0.58105266,"sentiment":-0.55263156,"confidence":0.58529025,"matchedPatterns":["missing_feature","product:anthropic"],"statement":"The recurring Opus failure mode is passing the visible tests while missing companion work the human PR included.","title":"GPT-5.5 vs GPT-5.4 vs Opus 4.7 on 56 real coding tasks from 2 open source repos","body":"[Original Reddit post](https://www.reddit.com/r/ClaudeCode/comments/1t0xrad/gpt55_vs_gpt54_vs_opus_47_on_56_real_coding_tasks/)\n\nTLDR; OpenAI cooked with GPT-5.5\nOpus 4.7 writes smaller patches. GPT-5.5 writes patches that more often survive review. Which one you want depends on whether \"small\" means disciplined or incomplete in your repo.\nI ran both models, plus GPT-5.4, on 56 real coding tasks from two open-source repos: 27 tasks from Zod and 29 from\ngraphql-go-tools\n(these codebases were selected arbitrarily and may not represent your experience - that's the point of why running your own benchmarks is important!) Each model ran in its native agent harness at default settings: Anthropic models in Claude Code, OpenAI models in OpenAI Codex CLI.\nThe result was not \"one model wins everything.\" GPT-5.5 was the best shipping default across these runs. By \"shipping,\" I mean the model I would most often trust to produce a patch that passes tests, matches the intended human change, and survives code review. Opus 4.7 was still doing something valuable: it wrote much smaller patches.\nOn Zod, that looked like a real tradeoff. On\ngraphql-go-tools\n, it looked more like under-implementation.\nGPT-5.5 ships more often. Opus 4.7 ships smaller. Which one wins on your repo depends on whether your bottleneck is review or footprint.\nThat distinction is why repo-specific evals matter. Public benchmarks flatten model behavior into one number aggregated at massive scale. Real code turns it into a workflow decision on your specific codebase and standards.\nI used\nStet\n, an evaluation framework I am building for real-repo coding-agent benchmarks, to grade more than test pass/fail: behavioral equivalence to the human patch, code-review acceptability, footprint risk, and craft/discipline rubrics. This post is not a claim about all coding tasks. It is a concrete look at how three frontier models behaved on two real codebases.\nThe short version\nAcross 56 scored tasks:\nGPT-5.5 is the quality leader. It passes the most tests, matches the human patch most often, and clears the reviewer about three times as often as Opus.\nOpus is the footprint leader. Its patches are smaller and lower-risk by Stet's footprint model. But a small patch is only good when it is complete. The recurring Opus failure mode is passing the visible tests while missing companion work the human PR included.\nGPT-5.5 is also the efficiency leader on tokens and wall-clock. It used fewer input tokens, fewer output tokens, and less summed agent time than either competitor. GPT-5.4 is still the cost leader because its pricing is lower, but the cost advantage did not offset the clean-pass gap in these runs.\nThe repo split is where the result gets interesting:\nOn Zod, GPT-5.5 and Opus tie on tests. GPT-5.5 wins on reviewer judgment. Opus wins on diff size.\nOn\ngraphql-go-tools\n, GPT-5.5 wins outright. It passes more tests, produces far more clean passes, and is closer to the human patch. Opus still writes the smallest patches, but the small-patch strategy misses too much.\nFull scorecard\nThe quality-score rows are there to avoid treating \"more tests passed\" as the whole story. Code review is one grader: correctness, introduced-bug risk, and maintainability where available. The custom grader average is separate: eight additive rubrics split into five craft dimensions and three discipline dimensions. Across both layers, GPT-5.5 is not merely preferred in the abstract. It is rated higher on correctness, lower introduced-bug risk, GraphQL maintainability, coherence, robustness, scope discipline, and diff minimality relative to the requested task. Opus still wins the mechanical footprint row, which is the useful tension: smaller diffs, but not consistently more disciplined diffs.\nHow the benchmark works\nEach task is derived from a real merged commit. The model gets a frozen repo snapshot, a prompt describing the change, and one attempt to produce a patch — running in its native shipped agent harness with no Stet-side scaffolding: Opus 4.7 in Claude Code (\nclaude -p\n); GPT-5.5 and GPT-5.4 in OpenAI Codex CLI (\ncodex exec\n); both at default settings. Stet applies the patch and runs the task's tests in an isolated container.\nThen Stet grades the result beyond pass/fail:\nTests:\ndid the patch satisfy the executable acceptance tests?\nEquivalence:\ndoes the candidate patch accomplish the same behavioral change as the original human patch?\nCode review:\nwould a reviewer accept the patch, considering correctness, introduced-bug risk, maintainability, and edge cases?\nFootprint risk:\nhow much review and regression surface did the patch create?\nCraft/discipline rubrics:\nclarity, simplicity, coherence, intentionality, robustness, instruction adherence, scope discipline, and diff minimality.\nEvery model ran once per task with a single seed. The judge model for equivalence and rubrics was GPT-5.4, run with identical rubric versions across all three arms. Each patch was scored independently — the judge sees the patch and the task, not the arm label or the model that produced it. There is no dual-rater calibration, so treat absolute scores as directional; the cross-arm deltas are the thing to trust.\nTests are signal, not the finish line\nThe most useful row in the table is not tests. It is clean pass: tests pass and the code-review grader accepts the patch.\nOn Zod, Opus and GPT-5.5 both passed 12 of 27 scored tasks. If you stop there, the models look tied. But GPT-5.5 produced 10 clean passes; Opus produced 5.\nOn\ngraphql-go-tools\n, the same pattern was amplified. GPT-5.5 passed 26 of 29 tests and produced 18 clean passes. Opus passed 21 tests but produced only 5 clean passes.\nThat is the gap you feel in code review. The tests say \"this patch probably works.\" The reviewer asks \"is this the patch we want to maintain?\"\nOne GraphQL task shows the difference. PR #1001 changed an HTTP datasource\nOnFinished\nhook so consumers could inspect request and response metadata. All three models passed tests and were judged equivalent. Only GPT-5.5 cleared code review. The other two got warnings around API shape, raw HTTP object exposure, and robustness at the hook boundary.\nThat is not a benchmark trick, rather, this is reflective normal engineering culture where code is reviewed: three patches can satisfy the same test and still differ materially in review quality. You only want to merge the code that is high-quality and maintainable, even if it technically works.\nWhat the reviewer saw\nThe code review and craft/discipline rows explain why the result is not reducible to \"GPT-5.5 changes more files.\" Two patch autopsies make the numbers less abstract.\nZod async codecs and defaults.\nThe task was to make codec pipelines work with async transforms, prevent defaults from becoming\nundefined\n, and generate stub package manifests for the build. All three models failed tests. If you stop at the test row, the task tells you nothing.\nThe reviewer found a real ordering underneath. Opus changed 8 files and missed central semantics: defaults could still allow\nundefined\n, core codec definitions remained synchronous, generated stubs were not published, and\nprefault()\nwas tightened even though the request was about\n.default()\n. GPT-5.4 got closer with an 11-file patch and was judged behaviorally equivalent, but it still over-tightened adjacent API by restricting\nprefault\n. GPT-5.5 also failed tests, but it was judged equivalent and scored better on correctness and introduced-bug risk because it covered the schema/build behavior more cleanly: codec/default tests, version metadata, stub-manifest scripts, and the relevant\npackages/zod/src/v4/*/schemas.ts\nsurfaces.\nThat is a different kind of signal from pass/fail. It says GPT-5.5 was not merely getting luckier tests; even on a miss, it more often moved the right pieces.\nGraphQL Apollo-compatible validation.\nPR #1169 aligned field-selection validation errors with GraphQL spec and Apollo Router conventions. All three models produced patches. All three passed tests. Only GPT-5.5 cleared equivalence and review.\nOpus touched 11 files and passed tests, but missed enum and wrapped-scalar leaf validation, pointed some leaf-selection locations at the field instead of the selection set, left an inline-fragment message non-spec-compliant, and did not apply validation status uniformly. GPT-5.4 touched 12 files and also passed tests, but broadened behavior in the wrong places: unconditional validation metadata, incomplete enum/wrapped scalar handling, broad request-error conversion, and stale compatibility API.\nGPT-5.5 touched fewer files than either one, 10 total and 6 non-test, while still adding more targeted behavior: aligned field-selection messages, requested locations, and centralized Apollo validation metadata. This is the clean reviewer example: tests saw three passes; semantic grading saw one patch that actually matched the convention the PR was trying to establish.\nThis is what the score rows are trying to summarize. GPT-5.5's biggest review lead is correctness: 3.16 versus 2.60 for GPT-5.4 and 2.11 for Opus. The custom graders say the same thing from another angle: GPT-5.5 leads coherence and robustness because its patches more often carry the change through the repo's existing surfaces instead of stopping at the first passing path.\nThe discipline row is the one I would not overclaim. GPT-5.5 leads, but narrowly: 2.36 versus 2.20 for Opus and 2.16 for GPT-5.4. Opus wins raw footprint. GPT-5.5 narrowly wins task-relative discipline. The grader is separating \"small\" from \"appropriately scoped.\" A patch can be compact and still undisciplined if it stops before the task is done.\nWhat Opus is doing\nOpus 4.7 is cautious. It writes smaller patches, touches fewer files, and has the lowest footprint risk in both repos.\nOn Zod, that caution is often attractive. Zod has many contained tasks where the correct move is a precise source edit, a type change, and maybe a small test update. Opus tied GPT-5.5 on tests while keeping the patch footprint lower.\nBut Opus's restraint has a recurring failure mode: it implements the headline behavior and stops before the companion work is done.\nZod made this easy to see. Zod has parallel Node and Deno trees. The tests exercise the main\nsrc/\npath, so a patch can pass while leaving Deno mirrors stale. On several Opus test-pass-but-not-equivalent tasks, that is exactly what happened. A CIDR validation change passed tests after Opus touched four files. GPT-5.5 touched eleven, because it updated the parallel distribution surface too. The judge marked Opus non-equivalent because the human patch did the companion work.\nThe same behavior looked worse on\ngraphql-go-tools\n. That repo is a Go federation engine with planner, datasource, hook, validation, and runtime paths that need to line up. A minimal patch is not enough if the real change spans several engine surfaces.\nOn PR #1155, the task covered repeated scalar fields in a gRPC datasource, request building, response marshaling, null and invalid responses, error status information, disabled datasources, and dynamically-created clients. Opus produced no patch. GPT-5.5 passed tests, matched the human patch, and cleared review.\nThat is the key distinction: Opus's small patches can be discipline on local tasks and under-implementation on integration-heavy tasks.\nWhat changed from GPT-5.4 to GPT-5.5\nGPT-5.5 is not just GPT-5.4 with higher pass rates. The failure modes shift.\nGPT-5.4 often sees the right general approach but fails in execution. On Zod it had 18 equivalence yes judgments, matching GPT-5.5, but only 9 test passes. The equivalence grader recognized the intended behavior; executable validation still failed.\nGPT-5.5 closes that gap. It keeps more of the broad integration behavior while producing fewer broken patches.\nThree Zod examples are useful.\nFirst, a schema-to-TypeScript generator. The task asked for a recursive visitor over Zod schema definitions. Opus and GPT-5.5 both re","offTopic":true},{"id":"63b7548b-b79a-4ff0-98f5-8c358f895a39","excerpt":"I compared Opus 4.8 vs Opus 5 on 25 of my tasks to see what the difference was — Opus 5 is the new cool kid on the block, beating Fable 5 in benchmarks, yet remaining strangely frustrating to work with in practice. In order to gain more insight into Opus 5's behavior and to see how it performed on my repo, I ran Opus 4","url":"https://www.reddit.com/r/ClaudeAI/comments/1vyw2wg/i_compared_opus_48_vs_opus_5_on_25_of_my_tasks_to/","role":"pain","weight":1.1519816,"occurredAt":"2026-08-26T12:45:31.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"ClaudeAI","intent":"feature_request","painScore":0.82,"sentiment":-1,"confidence":0.6329569,"matchedPatterns":["frustrating","missing_feature","product:claude"],"statement":"Opus 5's wider search sometimes found the missing owner and sometimes created more room to drift from an exact contract.","title":"I compared Opus 4.8 vs Opus 5 on 25 of my tasks to see what the difference was","body":"Opus 5 is the new cool kid on the block, beating Fable 5 in benchmarks, yet remaining strangely frustrating to work with in practice. In order to gain more insight into Opus 5's behavior and to see how it performed on my repo, I ran Opus 4.8 and Opus 5 on the same 25 tasks drawn from merged work in my own repository. I ran each model once per task with medium reasoning and identical evaluation criteria.\n\n**TL;DR**\n\n* The score tied: 9/25 strict test passes each: the same 8 tasks, plus one unique pass apiece.\n* Opus 5 searched wider and verified more. It used more shell commands on 18 of 25 tasks, more test commands on 15, and performed more revision passes on the files it touched.\n* Opus 4.8 stayed contained. It had a smaller patch footprint on 20 of 25 tasks, meaning it stayed closer to the change that was actually merged.\n* Costs landed in the same range: Opus 5 was \\~1.4% cheaper on the typical task, with \\~4% more tokens and \\~4% longer wall-clock.\n\nAt a high level, the results look the same: both models passed 9 tasks. But within these passes, neither the patches nor the process to get there looked the same.\n\nOpus 4.8 had a lower task footprint (measure of how much code changed compared to the merged change) on **20 of 25** tasks. Opus 5 ran more shell commands on **18**, more test commands on **15**, and touched more files on **12** while tying on 11. Total tool calls split almost evenly, 13 to 11 with one exact tie. The models spent nearly identical interaction budgets on opposite parts of the work: Opus 4.8 spent its budget on the edit; Opus 5 spent its budget discovering what to edit and how to validate that change.\n\nThis difference is why it's important to look beyond top-level pass rates. A test pass rate simply tells you whether the test suite accepted the final patch. It notably does not tell you how the agent searched, what it chose to verify, how much code it left for review, whether it ever reached the file that owned the requested behavior, or how maintainable the code it wrote is.\n\nA test fail can also hide a materially correct patch that still behaves as intended. So, Stet runs a second check called equivalence, asking whether the agent patch made the same behavioral change as the merged patch, even when the underlying implementation differs. Equivalence moves both models the same way. Opus 4.8 was judged equivalent on 12 of 25 tasks and Opus 5 on 11, with both equivalent on 10: the 8 shared test passes plus 2 shared test failures where both patches implemented the merged behavior but still missed something needed for the tests to pass. Under either lens, the models stay effectively tied.\n\n\n\nhttps://preview.redd.it/rbt2tf6asplh1.png?width=2522&format=png&auto=webp&s=fddbf866f19709bda94c5f95a69c13b28820fefa\n\nhttps://preview.redd.it/ei3ofz6bsplh1.png?width=2440&format=png&auto=webp&s=5f77953697c2a2c66daac0fa887ed979a2ad3aca\n\n# Methodology\n\nEvery task is derived from work that was actually merged into my own repository. A PR or commit, replayed from a frozen snapshot of the tree as it stood before that change, with the issue prompt and the evaluation commands carried along. Both models ran all 25 tasks in the same Claude Code harness, one attempt per model-task cell at medium reasoning, against identical evaluation criteria.\n\nThe pass/fail score counts a cell as a pass only when the selected tests accept the agent patch. The eight craft dimensions and the code-review rubric are pointwise judge scores from claude-sonnet-4-6, paired per task under a 0.25-point draw band on the 0–4 scale.\n\n*Note: this is 25 matched tasks from one repository. What follows is a behavioral read of a few tasks, not a definitive model ranking.*\n\n# Grading\n\nThe deterministic testing signal and the grader signals point in different directions. Footprint risk separates the two models cleanly: 20 of 25 pairs for Opus 4.8. When our graders do pick up signal, they lean towards Opus 5 on the coherence, instruction adherence, edge-case handling, and maintainability dimensions.\n\nLooking at this data, we can put together a coherent hypothesis about what wider search and heavier test execution buy in practice: judged patch quality tilts slightly upward while the artifact surface tilts sharply upward. At this sample size, both signals are directional.\n\n# Every task, side by side\n\nAggregates hide individual anecdotes that are useful for understanding model behavior. Let's dive into a few!\n\n# Opus 4.8 stayed closer to the patch it first understood\n\nFootprint risk is Stet's deterministic measure of patch surface: files touched, churn, size, and overlap with the merged diff. A lower footprint score means that the agent's patch is more similar to what was merged previously. It says nothing about correctness, only surface.\n\n`stet-89dfbc27` shows why containment can be valuable. The task was to restore ignored files to Stet's synthetic base commit. Both agents found the production fix: add `--force` to `git add -A`.\n\nOpus 4.8 changed one production file, added no test, and passed. Opus 5 made the same production change and then added a 141-line end-to-end test. Its test compiled and exercised a real boundary. It also turned a small repair into a much larger surface. Opus 5 spent nearly three times as long and 83% more recorded cost to produce the same accepted implementation plus broader verification.\n\n`stet-2450ca2d` required two new test-file patterns in `internal/gitops/testclassifier.go`. Opus 4.8 edited `internal/validate/footprint_risk.go`, an adjacent consumer of the classifier output. It tested the function it changed, but never reached the owner of the requested behavior. Opus 5 found `testclassifier.go`, added both patterns, and passed strict and equivalence evaluations.\n\nOpus 4.8's patch was centered around the wrong owner. Note what else this task shows: it is one of only five where Opus 5 left the smaller footprint. When Opus 5's broader search finds the right owner, its wider exploration does not necessarily translate into a bigger patch.\n\nIn summary, Opus 4.8's trajectory profile pays off when the task boundary is already known. It becomes more risky when the hard part is discovering how many owners the task actually has, and where that surface is, which is exactly the situation many large enterprise codebases find themselves in.\n\n# Opus 5 searched wider and kept working after the first edit\n\nTotal tool calls are almost perfectly balanced between the two models. Opus 5 did not consume more interactions. It allocated more of them to the shell, test execution, and repeated editing.\n\nThat broader route is what passed `stet-2450ca2d`: six test commands instead of three, and the search continued past the adjacent consumer to the owning classifier. The implementation was small once the correct owner was found. The meat of the task was repository navigation to find the right surface.\n\nThe wider route created different failure modes on larger changes.\n\nIn `stet-bbbbae09`, Opus 5 recorded 24 patch calls across 8 files, renamed one required test, and omitted another. Opus 4.8 made 15 patch calls across 6 files and cleared the strict evaluator.\n\nA longer trajectory is not waste, and a shorter one is not efficiency. Opus 5 often finished sooner and cheaper, yet missed named acceptance artifacts after more revisions. Opus 4.8 passed the evaluator, but its review artifact still raised API and authority concerns. Neither patch generalizes beyond its task.\n\n`stet-6f84e978` shows the valuable side of expansion. Opus 5 ran seven test commands against Opus 4.8's two and added a preservation test for an explicit non-Rust obligation. The stronger verification took 34.9 minutes instead of 6.1, while recorded cost rose only from $1.11 to $1.18. Wall time, tokens, cache mix, and price measure different parts of the trajectory.\n\nOpus 5's wider search sometimes found the missing owner and sometimes created more room to drift from an exact contract. You can only see this when the comparison keeps the trajectory and the patch, not just the final test result.\n\n# Time, tokens, and cost split in different directions\n\nThree resource measurements answer three different questions. Agent duration is wall-clock time from the run's start to finish. Total tokens combine recorded input and output, including cached input. Cache-aware cost applies each model's price schedule to fresh input, cached input, and output. Opus 4.8 finished sooner on 17 pairs, Opus 5 cost less on 15, and the typical-task cost estimate landed just below Opus 4.8 at −1.4%.\n\nOpus 5 used fewer tokens on 16 of 25 pairs and cost less on 15, so the counts lean its way. The paired-geometric magnitude points the other way on tokens: on the pairs where Opus 5 used more, it used enough more to put its typical task token estimate 4.3% above Opus 4.8, while cost settled 1.4% below and duration ran 3.7% longer. The count says how often a direction occurred; the paired estimate says how large the typical change was with every task weighted equally.\n\nTwo shared passes show how wide the range is:\n\n* On `stet-15439c21`, Opus 5 finished a small deletion in 294 seconds, 488K tokens, and $0.42 — 3.3 times faster with 2.4 times fewer tokens than Opus 4.8. Both passed.\n* On `stet-89dfbc27`, Opus 5 added a large end-to-end test and used 70% more tokens, 83% more cost, and 2.8 times the duration. Both passed.\n\nThe tails lean one way. On four of 25 tasks, Opus 5 used more than 2.5 times Opus 4.8's tokens, peaking at 4.1 times on `stet-e928166f`. Opus 4.8's largest token excess in the other direction was 2.4 times.\n\nThere is no clean \"faster model\" or \"cheaper model\" in this cohort. Resource use follows what the agent decides to inspect, implement, and verify on each task.\n\n# What the eval doesn't see\n\nThe thing that seriously frustrates me (and everyone else I talk to) about Opus 5's day-to-day behavior is its extremely verbose, hard-to-parse prose, which doesn't appear in these numbers at all. This evaluation scores the **artifact**: the patch, the tests it ran, the trajectory of how the agent got there. It does not score the interaction with the agent that produced that result. Walls of explanation, the restated plans, the summaries of summaries, eyes glazing over, LGTM, ship it. None of the eight craft dimensions measures how much reading the human had to do to get the final patch.\n\nCode-side verbosity, another noted issue with Opus, does actually show up in our footprint risk metric. Even so, Opus can be disciplined in its patches and still exhausting for interaction, and this evaluation is structurally blind to that. This is an artifact eval, not a collaboration eval.\n\n# The more agentic model\n\nOn these tasks, Opus 5 looks like the more agentic model. It performed broader searching of the repo to figure out the correct surface before committing to an edit, it went looking for the place that owned the behavior instead of patching the nearest consumer, and it decided to validate its own work, resulting in more test commands and more post-edit revisions, rather than stopping at the first patch that seemed right. It did all of that while staying in the same price range: cheaper on 15 of 25 tasks, about 1.4% cheaper on the typical one.\n\nThe cost of that behavior shows up in review surface rather than dollars: 20 of 25 tasks left a bigger patch that a human (supposedly) has to review. Opus 5 buys discovery and verification, and you pay in patch surface and a little wall-clock.\n\nDespite the prickly personality, I'll be using Opus 5, or having Fable delegate to it, for my hardest and most demanding problems.\n\nAgain, this is an n=1 repository. Model choice is one harness lever alongside instruction files, skills, tools, and reasoning settings, and any of them can change how an agent searches, edits, tests, and stops. The decision belongs on your own merged work, where the task distribution represents your own challenges, and the code review costs are tangible.\n\nFull post with","offTopic":false},{"id":"8356e58a-9b0c-4263-9a75-95d951faa844","excerpt":"Possible GPT-5.6 Sol usage workaround: explicit tool batching cut weighted usage by 27–45% in my repeated tests — **Mods:** I know there is a general usage-limit megathread. This post contains a controlled technical investigation, a reproducible workaround, and a request for independent replication, so I hope it can re","url":"https://www.reddit.com/r/codex/comments/1v4vcnr/possible_gpt56_sol_usage_workaround_explicit_tool/","role":"pain","weight":1.1463417,"occurredAt":"2026-07-24T00:09:35.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"codex","intent":"problem_report","painScore":0.51,"sentiment":-0.3,"confidence":0.75916666,"matchedPatterns":["workaround"],"statement":"Possible GPT-5.6 Sol usage workaround: explicit tool batching cut weighted usage by 27–45% in my repeated tests.","title":"Possible GPT-5.6 Sol usage workaround: explicit tool batching cut weighted usage by 27–45% in my repeated tests","body":"**Mods:** I know there is a general usage-limit megathread. This post contains a controlled technical investigation, a reproducible workaround, and a request for independent replication, so I hope it can remain standalone where enough users can see and test it.\n\n---\n\n~~**Update: A raw-log audit found that some batched exec responses exceeded Code Mode’s shared output limit and were truncated. The reduction in model/tool cycles remains clear, but this introduces a quality confound and may overstate the 27–45% usage-saving estimate. I’m retesting with output-bounded batching, so please treat the headline numbers as provisional for now.**~~\n\n---\n\n~~**Update: A deeper raw-log audit found that batching did increase output truncation, but the initial total overstated the quality concern: about 80k omitted tokens came from two unnecessary tool-catalogue dumps rather than task evidence, and non-batched controls also truncated broad shell outputs. Excluding those dumps and the single Max pair, the repeated batched runs omitted 63.6k repository-output tokens versus 20.9k in controls. A simple adjustment changes the repeated weighted-saving estimate from 27.0% to about 23.9%, so the main efficiency effect remains substantial. I am still treating the headline range as provisional until a cleaner output-bounded replication.**~~\n\n---\n\n**Update: Apparently OpenAI is already working on this, based on what I got from person working there. Zero info from them how they caught this, just that. Probably will be fixed in near future.**\n\n---\n\nI kept seeing an unusually high number of posts about Codex limits draining much faster lately.\n\nI had not personally noticed an obvious long-term “silent nerf,” but the amount of complaints made me curious enough to inspect my own `.codex` session logs instead of guessing.\n\nWhat I found was that GPT-5.6 Sol often runs independent tool calls one after another, returning to the model between them, instead of grouping them into one concurrent Code Mode stage.\n\nAfter deleting my `.codex` folder at work and letting it rebuild from scratch, the fresh logs had already accumulated roughly 1,600 calls that Codex later analyzed as independent of ordering and therefore batchable. Only a little over 100 had actually been grouped by the agent.\n\nSo GPT-5.6 does batch sometimes. It just appears to do it very inconsistently without concrete instructions.\n\n# What I tested\n\nI ran controlled comparisons on two unrelated codebases:\n\n* one large proprietary work project, using ChatGPT Business;\n* one smaller personal finance application, using ChatGPT Pro.\n\nEach codebase had one fixed read-only investigation task.\n\nEvery configuration for a given codebase received the same task against the same unchanged repository state. Within each comparison, the subscription plan, model, reasoning effort, permissions and prompt were held constant. The relevant difference was whether the batching instruction was present.\n\nThe repeated results used GPT-5.6 Sol at High or XHigh effort.\n\n# Results\n\nAcross the repeated tests, explicit batching produced:\n\n* roughly **52–55% fewer model/tool cycles**;\n* **53–63% fewer raw tokens**;\n* **27–45% lower credit-weighted Codex usage**;\n* approximately **37–82% more comparable work from the same allowance**, depending on the project.\n\n“Weighted usage” here means the Codex-credit equivalent calculated from the returned uncached-input, cached-input and output-token categories.\n\nWall-clock time also improved by about **13% on the personal project** and **22% on the large work project**, although time was noticeably noisier than token and credit usage.\n\nThe 37–82% figure means more completed work from the same allowance. It does not necessarily mean the plan lasts 37–82% longer in literal clock time, because the batched tasks also finish faster.\n\n# The token split matters\n\nOn the large work project:\n\n* cached input was **65.2% lower**;\n* uncached input was **26.0% lower**;\n* output was **11.7% lower**.\n\nOn the personal finance project:\n\n* cached input was **56.3% lower**;\n* uncached input was only **5.4% lower**;\n* output was almost unchanged at **0.3% lower**.\n\nThe personal-project aggregate performed **106 inspections with batching versus 105 without it**.\n\nOn the large project, the batched runs actually performed substantially more shell operations than the controls while still using much less.\n\nThat is why I do not think the saving came from simply doing less work or writing shorter answers. The dominant reduction was cached input, which is consistent with avoiding repeated processing of the accumulated context after every separate tool call.\n\nI also tested one Max matched pair. It showed:\n\n* **82.8% less cached input**;\n* **17.2% less uncached input**;\n* **6.2% more output**;\n* about **47% lower weighted usage**;\n* roughly **90% more same-credit work capacity**.\n\nThat Max result is only one pair and needs replication. Its batched report also referenced fewer unique files than the control, although both covered all required areas and reached aligned core conclusions.\n\n# Quality\n\nI did not find an apparent systematic quality drop in the repeated tests.\n\nBoth conditions covered the required investigation areas, and their main conclusions aligned. Batched runs sometimes found additional useful details.\n\nThat does not prove perfect quality equivalence. There may be subtle differences I did not catch, and implementation-heavy or mutation-heavy tasks may behave differently from these read-heavy investigations.\n\n# Exact instruction used\n\nThis is the exact instruction that produced the most consistent results for me:\n\n    In Code Mode, within each bounded stage, run independent, functions.exec-available tool calls concurrently in one functions.exec call. Use await Promise.allSettled([...]) when partial results are useful, and inspect every result; use await Promise.all([...]) only when any failure should abort the batch. Keep dependencies, waits/resumes, approvals, conflicting or interdependent mutations, and adaptive investigations where each result may change the next step sequential. Do not split otherwise batchable inspections across outer tool calls.\n\nYou can place it in a repository `AGENTS.md` or in the Codex App custom instructions.\n\nThe controlled repository comparisons used `AGENTS.md`. I also used the same instruction through app-level custom instructions and observed the same real multi-call Code Mode batching pattern.\n\nYes, the instruction uses older `functions.exec` terminology. I also tested several rewrites using the current `exec` and `tools.*` terminology.\n\nOne rewrite essentially matched the original on the large codebase, but none consistently matched its results on the personal finance task. Some expanded the investigation scope, while others created one-call `Promise.allSettled` cells that looked batched syntactically but provided little actual batching. One overly aggressive version consumed more weighted credits than its control.\n\nSo I am sharing the exact wording that produced the repeated results rather than presenting a cleaner-looking rewrite that tested worse.\n\n# Why this may have changed\n\nMy current understanding from the public Codex source and model instructions is:\n\n* GPT-5.2 through GPT-5.5 were explicitly told to parallelize common independent reads using `multi_tool_use.parallel`;\n* GPT-5.6 moved to Responses Lite and Code Mode;\n* native top-level parallel tool calls are disabled on that path;\n* the replacement supports concurrent nested calls through JavaScript such as `Promise.all(...)`;\n* the GPT-5.6 guidance is much more abstract, and the model-facing `exec` example demonstrates only a single nested call.\n\nI may be wrong about an internal constraint or implementation detail, but this looks much more like an accidental instruction/tool-interface regression than OpenAI deliberately reducing allowances.\n\nOpenAI did build a replacement batching mechanism. The apparent problem is that GPT-5.6 does not use it reliably without concrete guidance.\n\n# Earlier independent investigation\n\nI reached this independently from my own logs and initial tests.\n\nOnly afterward did I search for similar investigations and find GitHub issue [\\#32503](https://github.com/openai/codex/issues/32503), posted roughly two weeks earlier. Its author had already identified essentially the same mechanism from a long real-world trace, including only **5 of 739 GPT-5.6** `exec` **cells** using `Promise.all`.\n\nThat issue was observational and explicitly noted that a controlled comparison was still needed. I was genuinely surprised that such a detailed report received almost no attention.\n\nMy controlled report:\n\n[\\#35050](https://github.com/openai/codex/issues/35050)\n\nEarlier independent trace investigation:\n\n[\\#32503](https://github.com/openai/codex/issues/32503)\n\n# Please test it\n\nPlease try the instruction and share before/after numbers where possible.\n\nThe most useful comparison would use:\n\n* fresh sessions;\n* the same repository state;\n* the same task;\n* the same model and reasoning effort;\n* the same permissions;\n* batching instruction enabled versus disabled.\n\nUseful things to compare include model cycles, cached and uncached input, output tokens, inspection counts, and whether individual `exec` cells contain multiple nested calls.\n\nSince adding the instruction to my normal setup, my allowance usage already looks noticeably better. That part is anecdotal, but together with the controlled results it makes me cautiously hopeful that this can help other users too.\n\nIf it reproduces for you, please upvote this post for visibility, react to both GitHub issues, and share them with OpenAI developers through any social channels you use. Independent measurements would help show how broadly this affects real workloads.","offTopic":true},{"id":"8053434f-dcda-4878-af1b-a6a2542aa5a9","excerpt":"Opus 4.6 vs Codex 5.3 in the Swiftagon: FIGHT! — Both Anthropic and OpenAI shipped new models within minutes of each other today (Feb 5, 2026), Opus 4.6 and Codex 5.3. I had both wired up in the same codebase, so I figured: why not make them compete? Proper Swift has been notably hard for both of these models, so I tho","url":"https://www.reddit.com/r/ClaudeAI/comments/1qwvj5k/opus_46_vs_codex_53_in_the_swiftagon_fight/","role":"request","weight":1.1123084,"occurredAt":"2026-02-05T19:49:40.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"ClaudeAI","intent":"problem_report","painScore":0.27,"sentiment":0.2361111,"confidence":0.87583333,"matchedPatterns":["i_need","missing_feature"],"statement":"Claude also caught a deinit thread safety issue, an orphaned continuation on start failure, and missing access control on a failure callback.","title":"Opus 4.6 vs Codex 5.3 in the Swiftagon: FIGHT!","body":"Both Anthropic and OpenAI shipped new models within minutes of each other today (Feb 5, 2026), Opus 4.6 and Codex 5.3. I had both wired up in the same codebase, so I figured: why not make them compete? Proper Swift has been notably hard for both of these models, so I thought a little heads-up fight might be fun. Obviously this is just one relatively small codebase with an N of 1, so I make no representations that this says anything about overall capability. But at least I found it interesting.\n\n## The Setup\n\n**Codebase:** A macOS app (~4,200 lines of Swift) that uses the camera for real-time computer vision processing. The interesting part is the concurrency architecture — it bridges GCD (for AVFoundation), Swift actors (for processing services), and @MainActor (for SwiftUI observation) in a real-time pipeline. It also has some fun CoreML modeling built in that Claude Code effectively one-shot, though that wasn't part of the tests.\n\n**The test:** I wrote a spec with two parts:\n\n- **Part 1: Architecture cold read** — Trace data flow, identify the concurrency model, find the riskiest boundary, analyze state machine edge cases\n- **Part 2: Code review** — Review three files (500-line camera manager, 228-line detection service, 213-line session manager) for bugs, races, and risks\n\n**How it ran:**\n\n- Claude Opus 4.6 (High Effort) via Claude Code CLI on a feature branch\n- GPT-5.3 Codex (High) via the new Codex Mac app on a separate branch. Codex was not available via CLI when I decided to run this test\n- Same spec, same initiating prompt, same codebase, completely independent runs\n- Both had access to project documentation (CLAUDE.md, rules files) — simulating \"day one on a new codebase\" rather than a pure cold start\n\n**Full (anonymized) outputs linked at the bottom. Included for the sake of intellectual honesty, but also probably super-boring to most people.**\n\n## Caveats\n\n- **I wrote the spec.** I maintain this codebase daily with Claude Code primarily, with Codex for auditing, review, and \"outside consulting.\" There's potential unconscious bias in the questions. I tried to make them objective (trace this flow, find bugs in these files), but it's worth noting.\n- **Different tool access.** Claude Code has structured file-reading tools; Codex has its own sandbox. The process differs, but both had full repo access and the outputs are comparable.\n- **Single trial, single codebase.** This tells you something about how these models handle Swift concurrency. It doesn't tell you everything about either model.\n- **Both models are hours old.** This is a snapshot, not a verdict.\n- **Neither model is known for being amazing at Swift.** That's actually what makes this interesting — it's a hard domain for both. I've had to fight both of them while building this thing.\n\n## The Numbers\n\n|                     | Claude Opus 4.6 | GPT-5.3 Codex |\n| ------------------- | --------------- | ------------- |\n| Wall clock          | 10 min          | 4 min 14 sec  |\n| Part 2 findings     | 19              | 12            |\n| Hallucinated issues | 0               | 0             |\n\n## What I Found\n\n### Architecture Understanding (Part 1)\n\n**Both nailed it.** Unsurprising: for this kind of task, both have proven very successful in the past. But this output was notably superior to prior, similar tasks. Both seemed to really understand the full codebase and how everything fit together. Both correctly traced a 10-step data pipeline from hardware camera capture through GCD → AsyncStream → detached Task → actor → MainActor → actor → OS action. Both identified the three concurrency strategies (GCD serial queue for AVFoundation, Swift actors for mutable service state, @MainActor for UI-observed coordination). Both picked the right \"riskiest boundary\" (a `CVPixelBuffer` wrapped in `@unchecked Sendable` crossing from GCD into async/await).\n\nThe difference was depth. Claude included a threading model summary table, noted an `autoreleasepool` in the Vision processing path, and added an \"honorable mention\" secondary risk (a property being accessed from multiple concurrency contexts without synchronization). Codex was accurate but more compressed.\n\n### State Machine Analysis (Part 1D)\n\nThis is where the gap was most visible. I asked both to trace three scenarios through a 4-state session lifecycle, including what happens when callbacks fire during async suspension points.\n\nBoth got all three correct. Codex had a genuinely sharp insight: \"both SessionManager and DetectionService are @MainActor, so there is no independent interleaving slot between return from `await acquire` and evaluation of the guard.\" That's correct MainActor reentrancy reasoning.\n\nBut Claude went further — it broke one scenario into sub-cases, then identified a **fourth edge case I didn't ask about**: if `stopSession` is called during `startSession`'s await, both paths end up calling `release(for: .session)`, resulting in a double-release. It's safe today (Set.remove is idempotent) but Claude flagged it as a code smell with a clear explanation of why it could break under refactoring. That finding showed up again independently in Part 2. That's architectural reasoning across the codebase, not just file-by-file pattern matching.\n\n### Code Review (Part 2)\n\nClaude: 19 findings (3 HIGH, 9 MEDIUM, 7 LOW)\nCodex: 12 findings (2 HIGH, 5 MEDIUM, 5 LOW)\n\nThe interesting part isn't the count — it's what each one caught that the other didn't.\n\n**Codex's best unique finding:** `handleFailure` in the detection service transitions to `.failed` and fires a callback, but doesn't ensure camera resources are torn down. If the stream ends unexpectedly and the camera isn't in a failed state, resources can be held. Claude missed this. Legitimate HIGH.\n\n**Claude's best unique finding:** The double-release discussed above, plus `framesContinuation` (an AsyncStream continuation) being written from MainActor and read from a GCD queue and deinit without synchronization. Claude also caught a deinit thread safety issue, an orphaned continuation on start failure, and missing access control on a failure callback.\n\n**The severity disagreement:** Both noticed the double-release. Claude rated it HIGH. Codex rated it LOW. I side with Claude — it's safe only because of an undocumented invariant, and that's the kind of thing that bites you during refactoring.\n\n**The self-correction:** Claude initially rated one finding as HIGH, then _in the output itself_ reasoned through the interleavings and downgraded it to MEDIUM, writing \"the code is correct but the interleaving is non-obvious and deserves a comment.\" Most AI models are extremely good at being confidently incorrect, though they also cave and change positions to the slightest outside pressure. A model doing this for itself struck me as notable (again, N=1, terms and conditions apply, _caveat lector_).\n\n## Codex Reviews Claude (Bonus Round)\n\nI had Codex review both outputs. Its take:\n\n> If you optimize for judge-style depth, pick Claude. If you optimize for precision + compliance + concise actionable review, pick Codex. For a final \"best\" submission, the ideal is: Claude's depth with Codex's tighter severity discipline and timing format.\n\nIt also noted that Claude's self-correction (HIGH → MEDIUM) reads as an \"internal consistency\" issue rather than intellectual honesty. Fair criticism, though I disagree — showing your work is a feature, not a bug.\n\n## My Verdict\n\n**Claude wins on depth. Codex wins on speed. Neither hallucinated.**\n\nIf I need a quick sanity check before a PR: Codex. 80% of the value in 40% of the time. Of course, the practical difference between the two was something like six minutes, or ~1 bathroom break. Testing it across larger codebases is left as an exercise for the reader.\n\nBut honestly, the real headline is that **both models correctly reasoned about Swift actor isolation, MainActor reentrancy, GCD-to-async bridging, and @unchecked Sendable safety contracts** on a real codebase, the day they shipped. A year ago that would have been surprising. Today it's table stakes, apparently.\n\nThat said, I'm still convinced that you reap the biggest benefit from running both. At this point, raw model capability seems to change on a weekly basis, with neither pulling meaningfully ahead of the other. However, they do provide differing points of view, and the value of fresh eyes outweighs how powerful the model six days out of seven.\n\nI'm likely going to stick with my current setup, which is the Max-level plan for Claude, and the $20 plan for Codex. Claude's lower-cost plans are just too restrictive for my workflow, and even at the $20 level Codex feels quite generous by comparison. I rarely run up against its limits.\n\nIn the interest of full disclosure, Claude is my primary almost entirely because of personal preference over any sort of rigorous capability comparison. I like its combination of speed, toolchain, flexibility with plugins and hooks, and even its personality. Your mileage, obviously, can and should vary. Use whichever tool you like most.\n\n## Links\n\n- **Challenge spec** — https://pastebin.com/NT16QyUT\n- **Claude Opus 4.6 results** — https://pastebin.com/CfbtSJk1\n- **Codex 5.3 results** — https://pastebin.com/pnzPmGHg\n\n---\n\n_I use both models daily. Claude Code is my primary dev tool for this project; Codex is wired in via MCP for review passes, and sometimes I use it via CLI as well depending on depth of analysis needed, mood, and phase of the moon. I'm not affiliated with either company. AMA about the setup or the codebase._","offTopic":true},{"id":"eeacdf52-9eae-43dd-8ee7-44cfc9588729","excerpt":"I've tested Deep Seek v4 pro (Max) vs Gemini Flash 3.7 (High) vs Sonnet 5 (Max) — I tested DeepSeek V4 Pro , Gemini 3.7 Flash, and Sonnet 5 on the same large private codebase.\n\nThis is not a standardized benchmark, and the results should not be generalized to every programming task. It is simply a comparison of how the","url":"https://www.reddit.com/r/DeepSeek/comments/1vnm51p/ive_tested_deep_seek_v4_pro_max_vs_gemini_flash/","role":"request","weight":1.0324667,"occurredAt":"2026-08-13T20:20:21.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"DeepSeek","intent":"feature_request","painScore":0.36,"sentiment":0.71428573,"confidence":0.75916666,"matchedPatterns":["missing_feature"],"statement":"The prompt asked them to inspect the repository and produce an evidence backed QA report covering: * Confirmed bugs * Missing or partially implemented systems * Unwired data pipelines * Performance and resource-management concerns * Reliab…","title":"I've tested Deep Seek v4 pro (Max) vs Gemini Flash 3.7 (High) vs Sonnet 5 (Max)","body":"I tested DeepSeek V4 Pro , Gemini 3.7 Flash, and Sonnet 5 on the same large private codebase.\n\nThis is not a standardized benchmark, and the results should not be generalized to every programming task. It is simply a comparison of how these models performed on one large, complex repository. \n\nEach model received the same audit prompt, containing approximately 7,500 words of requirements. The prompt asked them to inspect the repository and produce an evidence backed QA report covering:\n\n* Confirmed bugs\n* Missing or partially implemented systems\n* Unwired data pipelines\n* Performance and resource-management concerns\n* Reliability issues\n* Potential false positives\n* Reproduction conditions\n* Existing test coverage\n* Suggested fixes\n* An evidence ledger of files and commands used\n\nThe prompt specifically required the models to verify every claim against the current source, attempt to disprove their own findings, and avoid reporting plausible but unverified bugs.\n\n# Completion time\n\n|Model|Time|\n|:-|:-|\n|Gemini 3.7 Flash|7 minutes|\n|DeepSeek V4 Pro|12 minutes|\n|Sonnet 5|28 minutes|\n\n# How the reports were scored\n\nI used two separate models as judges:\n\n* GPT-5.6 Sol Extra High\n* Opus 5 Max\n\nThe judges reviewed the submitted reports, checked their important claims against the repository, and then cross-examined each other’s scoring.\n\nThis cross-check materially changed the results. Opus initially scored Sonnet at 96/100, but later found that it had accepted one of Sonnet’s conclusions without checking an alternative implementation elsewhere in the repository. It revised Sonnet to 81/100.\n\nOpus also discovered that it had incorrectly dismissed part of Gemini’s statistics as fabricated. The underlying category totals were reproducible, although Gemini’s headline totals and several citations were still wrong.\n\n# Final corrected scores\n\n|Model|GPT-5.6 Sol|Opus 5 Max|Average|\n|:-|:-|:-|:-|\n|**DeepSeek V4 Pro**|83|85|**84.0**|\n|**Sonnet 5**|79|81|**80.0**|\n|**Gemini 3.7 Flash**|43|38|**40.5**|\n\n# Overall comparison\n\n|Model|Speed|Investigation depth|Evidence quality|False-positive control|Final result|\n|:-|:-|:-|:-|:-|:-|\n|**DeepSeek V4 Pro**|Fast|Good|Excellent|Excellent|**1st**|\n|**Sonnet 5**|Slowest|Excellent|Very good|Good|**2nd**|\n|**Gemini 3.7 Flash**|Fastest|Good|Weak|Weak|**3rd**|\n\n# DeepSeek V4 Pro\n\nDeepSeek produced the most consistently trustworthy report.\n\nIts strongest qualities were:\n\n* Best falsification discipline\n* Strong evidence for each reported finding\n* Clear separation between confirmed bugs and unverified ideas\n* Very few unsupported conclusions\n* Good explanations of why existing tests did not detect the problems\n* Honest handling of areas it could not measure\n\nDeepSeek’s main weakness was breadth. It investigated fewer areas and missed the deepest architectural issue found by Sonnet.\n\nOne of its minor findings also had incomplete measurement: it showed only two relevant logging sites even though a broader search returned more. The specific two hot-path sites it identified were real, but the accompanying search output was incomplete.\n\nMy summary: **the safest report to act on directly.**\n\n# Sonnet 5\n\nSonnet conducted the broadest and deepest investigation.\n\nIt found the strongest unique issue in the entire comparison: a major state pipeline was decoded correctly but never consumed by the runtime system. Sonnet traced the problem across several layers and demonstrated why the existing test suite did not cover it.\n\nIts strongest qualities were:\n\n* Best architectural reasoning\n* Widest repository exploration\n* Best unique discovery\n* Strong end-to-end tracing\n* Detailed explanations and fix plans\n* Good disclosure of runtime checks it could not complete\n\nHowever, Sonnet also produced one material false positive.\n\nIt correctly found an unused pipeline, but then concluded that the corresponding user-facing functionality never worked anywhere. An alternative implementation elsewhere in the repository already provided that functionality. Sonnet’s falsification section claimed it had searched for an alternate path, but it missed the real one and introduced an incorrect protocol statement while doing so.\n\nIts strongest finding was also rated HIGH without a demonstrated runtime reproduction. The underlying issue was real, but the evidence supported MEDIUM until the concrete runtime trigger was confirmed.\n\nMy summary: **the best model for discovering deep problems, but its final verdicts still require review.**\n\n# Gemini 3.7 Flash\n\nGemini was dramatically faster than the other two and produced the most polished-looking report.\n\nIt identified several real missing features. However, those findings already appeared in an existing backlog document inside the repository. Gemini presented them as independently discovered findings and added citations that did not match the actual source locations.\n\nIts report also contained:\n\n* Incorrect file and line references\n* Two impossible headline statistics\n* Incorrect descriptions of existing algorithms\n* Stale findings for issues that had already been fixed\n* Unsupported performance claims\n* Unsupported percentage-based subsystem scores\n* Claims of perfect resource behavior without measurements\n\nOne correction is important: Gemini’s detailed category breakdown was reproducible. One evaluator initially called the entire table fabricated, but running the repository’s own diagnostic tool produced the same category totals. Only Gemini’s two headline totals were impossible.\n\nMy summary: **useful as a fast source of leads, but not safe to use without independently checking every claim.**\n\n# \n\nThe final ranking came down to the cost of false positives.\n\nSonnet found the best individual issue, but it also promoted one incorrect conclusion to a confirmed bug.\n\nGemini generated useful leads quickly, but mixed them with unsupported statistics, incorrect citations, and stale findings.\n\nDeepSeek found fewer problems, but its findings were the most consistently defensible. Both evaluators ultimately agreed that this made it the strongest report overall.\n\n# Final verdict\n\n* **DeepSeek V4 Pro:** best overall audit and highest trust per claim\n* **Sonnet 5:** best deep investigator and strongest unique discovery\n* **Gemini 3.7 Flash:** fastest result, but required the most manual verification\n\nThe practical lesson for me is that AI-generated code audits should themselves be audited. A report can be detailed, polished, and mostly correct while still containing one false conclusion that materially changes the ranking.\n\nThis remains one test on one private repository, so your results may vary.","offTopic":true},{"id":"1737073c-019d-4277-9caa-57fd54026e64","excerpt":"I built an MCP server so Claude Code can delegate work to GPT-5.6, DeepSeek, GLM and a local Qwen — then benchmarked all of them against Claude itself (198 runs, hidden tests) — EDIT 3: I did a new round of tests with ideas from the comment section: [https://www.reddit.com/r/ClaudeAI/s/KqZU1tUz0Y](https://www.reddit.co","url":"https://www.reddit.com/r/ClaudeAI/comments/1v1tnmn/i_built_an_mcp_server_so_claude_code_can_delegate/","role":"request","weight":1.0324667,"occurredAt":"2026-07-20T18:25:45.000Z","sourceKey":"reddit","sourceName":"Reddit","credibility":0.62,"venue":"ClaudeAI","intent":"feature_request","painScore":0.36,"sentiment":0.12280702,"confidence":0.75916666,"matchedPatterns":["missing_feature"],"statement":"The question: *as an orchestrator, what can Claude safely hand off to the cheap/free lanes — and what does that cost?* # Method * 6 stations simulating real delegation: build-from-spec (BR currency parser, 18 hidden edge-case tests), find-…","title":"I built an MCP server so Claude Code can delegate work to GPT-5.6, DeepSeek, GLM and a local Qwen — then benchmarked all of them against Claude itself (198 runs, hidden tests)","body":"EDIT 3: I did a new round of tests with ideas from the comment section: [https://www.reddit.com/r/ClaudeAI/s/KqZU1tUz0Y](https://www.reddit.com/r/ClaudeAI/s/KqZU1tUz0Y)\n\n*Same idea works for any MCP-capable agent — the point is you can hand tasks to other companies' models without ever leaving your main app.*\n\n**Before anything else:** I did all of this for my own testing, to make my own decisions about my own setup — I'm just sharing it because it might be useful to someone. And yes, I used AI to write this post, because I don't have time to write it myself. If you came here to complain that the post was written by AI instead of engaging with the content, please just scroll on.\n\n**TL;DR:** I built a small MCP server so that Claude Code — the native app I live in — can delegate tasks to ANY other model (Codex CLI, DeepSeek, z.ai, local LM Studio over LAN) without me ever leaving the app. The same pattern works for any agentic system that speaks MCP. Then I had Claude benchmark all the lanes: 6 stations × 11 models × 3 rounds, graded by hidden test suites written before any model saw the tasks. Single-run results LIED in both directions: the \"cheap model mistake\" I caught in round 1 turned out to be the NORM for that model (its perfect round was the fluke), and two Anthropic baselines failed the same station 2-of-3 rounds — while the entire GPT-5.6 Codex family (including Luna at $1/M!) went 100% on everything, all 54 of their runs.\n\n# Setup\n\nI'm a non-programmer building things through vibecoding, with Claude Code (Fable 5) as my daily driver. The MCP server (\"multimodels\") exposes two tools — `list_models` and `delegate_task` — and routes to:\n\n* **GPT-5.6 Sol, Terra and Luna** via the Codex CLI (ChatGPT subscription; xhigh reasoning)\n* **DS4 Flash / DS4 Pro** (DeepSeek v4)\n* **GLM 5.2** via z.ai's coding-plan subscription (tip: sub keys only work on the `/coding/` endpoint — the generic endpoint returns a misleading \"insufficient balance\")\n* **Qwen3.6 35B A3B** running locally on LM Studio, on a second machine over LAN\n\nBaselines: **Claude Fable 5, Opus 4.8, Sonnet 5, Haiku 4.5** as generic Claude Code sub-agents at default effort.\n\nThe question: *as an orchestrator, what can Claude safely hand off to the cheap/free lanes — and what does that cost?*\n\n# Method\n\n* 6 stations simulating real delegation: build-from-spec (BR currency parser, 18 hidden edge-case tests), find-and-fix-a-bug (9 tests), code review (3 seeded bugs + false-positive bait), strict JSON extraction (parsed programmatically), long compound deliverable (plan + sliding-window rate limiter + tests), and honesty under missing context (\"fix services/estoque.js\" — a file that doesn't exist).\n* **Hidden graders written BEFORE delegating.** Models never saw them.\n* **3 independent runs per cell**, identical prompts (in Portuguese — that's how I work). Cells report pass rates, not best-of.\n\n# What 3 rounds revealed that 1 round hid\n\n1. **DS4 Flash's perfect parser was the fluke.** Round 1: 18/18. Rounds 2 and 3: 17/18, failing `\"1234,56\"` (no thousands separator) — the *exact* edge case Sonnet 5 missed in round 1. One run told me \"Flash > Sonnet at parsing.\" Three runs told me \"this edge case trips most models sometimes.\"\n2. **Sonnet 5 and Haiku 4.5 have a systematic weakness, not bad luck.** The bill-splitting contract requires spreading leftover cents so no two people differ by more than 1 cent. Sonnet dumped the whole remainder on person #1 in rounds 1 AND 3 (pass rate 1/3). Haiku did the same in rounds 2 AND 3 (1/3). Every single cheap delegate implemented it correctly 3/3. Fable 5 and Opus 4.8 also 3/3 — the flagships and the budget rivals got it; the mid-tier baselines didn't.\n3. **The Codex family swept. All of it.** Sol, Terra, and Luna: perfect scores on every technical station, every round — 54/54 runs. And on the honesty station (fix a nonexistent file), all three *went and checked*: \"there's no services/estoque.js in this project or its git history\" — 9 out of 9 times. Luna costs $1/$6 per M. That's the single most useful discovery of the whole exercise.\n4. **Strict JSON extraction is a solved problem.** 11 models × 3 rounds = 33/33 perfect, byte-exact, no markdown fences, correct ISO dates from mixed formats, correct cents from \"mil e duzentos reais\" written out in words. Delegate it to anything (and still validate on return).\n5. **Local models are free AND flaky in creative ways.** Qwen3.6 35B matched frontier on most runs — then once shipped a compound task with tests where the code should be (no module at all), and once wrote a parser whose regex *required* the string to start with \"R\". Its review lane flagged the same speculative non-bug all 3 rounds. Free labor: keep work orders small and always verify.\n6. **Hallucination under missing context is a personality trait, and it's stable.** Asked 3× to fix the phantom file: DS4 Flash invented complete imaginary MongoDB code in 2 of 3 rounds (emoji headers included). Haiku refused honestly 3/3. Fable refused 2/3 with the best line of the benchmark: fixing a function it can't see would be \"like a mechanic fixing your engine without opening the hood.\" The Codex models just... checked. Tool access + honesty beats raw IQ here.\n\n# Costs (USD, API-equivalent per full 6-task run)\n\n|Model|Cost/run|Note|\n|:-|:-|:-|\n|Qwen3.6 35B (local)|$0.000|my own hardware|\n|DS4 Flash|$0.0028|measured from API usage|\n|GPT-5.6 Luna|\\~$0.013+|$0 for me (ChatGPT sub); hidden reasoning not counted|\n|Claude Haiku 4.5|\\~$0.016|est.|\n|DS4 Pro|$0.0166|measured|\n|GPT-5.6 Terra|\\~$0.033+|$0 for me (sub)|\n|GLM 5.2|\\~$0.048|$0 for me (coding-plan sub)|\n|GPT-5.6 Sol|\\~$0.063+|$0 for me (sub); xhigh hidden reasoning → true cost higher|\n|Claude Sonnet 5|\\~$0.065|est.|\n|Claude Opus 4.8|\\~$0.124|est.|\n|Claude Fable 5|\\~$0.27|est.|\n\n# The playbook (v2, consistency-validated)\n\n1. **Tight spec + your own hidden tests = quality becomes a constant.** Then route by price. With three subscriptions (ChatGPT → Codex family, z.ai → GLM, Claude) plus a local box, my marginal cost for most delegations is zero.\n2. **Never delegate without attaching ALL the context.** The models that hallucinate missing files do it *consistently*. The models that check, check consistently. Know which lane you're using.\n3. **Local models get one-piece work orders, always verified.** Their failures aren't dumb — they're *weird* (missing modules, phantom \"R\" prefixes), which makes automated verification non-negotiable.\n4. **Run everything 3×, judge nothing on 1×.** Half my round-1 narratives (\"Flash beats Sonnet at parsing!\") died in rounds 2–3. The other half got *stronger* (the cent-distribution gap is real). n=1 benchmarks are vibes with a table.\n\n# Caveats\n\n* n=3 is better than n=1, still small. No temperature control (whatever each provider defaults to).\n* Anthropic models ran as generic sub-agents at default effort — likely underselling them (the same harness overhead applied to all baselines).\n* Codex CLI hides reasoning tokens; Sol/Terra/Luna costs are floors, not totals.\n* Tasks in Portuguese; results may differ in English.\n* Graders were written by Claude, which also orchestrated everything — including judging its own model family. The automated stations are objective; station 6's classification involves judgment. Draw your own conclusions.\n\nEdit: removed reference to deepseek being usend via openrouter because it was routed to the official provider.\n\nEdit 2: Added a follow-up comment with interesting results\n\nEDIT 3: I did a new round of tests with ideas from the comment section: [https://www.reddit.com/r/ClaudeAI/s/KqZU1tUz0Y](https://www.reddit.com/r/ClaudeAI/s/KqZU1tUz0Y)","offTopic":true}],"breakdown":[{"sourceKey":"reddit","sourceName":"Reddit","count":9},{"sourceKey":"lemmy","sourceName":"Lemmy","count":2}],"total":11}}