What Senior Devs Catch That AI Misses

Here is a change we caught before it merged. It is small, it is plausible, and it is wrong in a way you cannot see without knowing Rails.

- Propshaft replaces Sprockets as the default asset pipeline in Rails 8, and the
- difference is dramatic: in our experience, build times drop from 45-60 seconds
- to under 5 seconds for medium-sized apps.
+ Propshaft replaces Sprockets as the default asset pipeline in Rails 8. It drops
+ the transpilation and concatenation stages entirely, so asset precompilation
+ stops being a build step that scales with your asset count.

The deletion is correct. That timing figure had no measurement behind it and deserved to go.

Read the addition again.

It is wrong.

Propshaft still walks every asset, fingerprints it, and copies it into place. Its own README says so:

All assets in the load path will be copied (or compiled) in a precompilation step for production that also stamps all of them with a digest hash

The work scales with how many assets you have. What drops is the cost of each one, because transpiling and bundling are gone - which is a real and useful thing to say, and not what the sentence said.

So a made-up number was swapped for a made-up mechanism. That is the worse trade, because a mechanism reads as reasoning rather than as a claim someone should go and check - and reasoning gets waved through where a number gets questioned.

Nobody skimming that paragraph would have stopped #

The sentence had a subject, a cause and an effect, and it used the right vocabulary throughout. It also agreed with the general direction of the truth - Propshaft is faster - while getting the reason for it wrong.

You cannot catch that by reading carefully.

You catch it by already knowing what assets:precompile does. Sean Goedecke put the general version well in “LLMs reward expertise”: “The most important skill in prompting is expertise in the domain you’re prompting for.”

He calls the thing experts do “steering” - you recognise a suboptimal suggestion and redirect it. This diff is that mechanism running backwards. Without someone who knows the asset pipeline, there is nothing to steer against and the confident answer wins by default.

Senko Rašić pushed back on that framing a fortnight later, insisting that “creating good code is a craft that requires skill, patience, attention to detail, experience and wisdom”. The diff above argues for his side better than it does for Goedecke’s. Writing that sentence took no craft at all. Knowing it was wrong took every item on his list.

Note what the change was for. The task was removing an unsourced number, and the same edit introduced a new defect while completing it. Cleanup is where this happens most, because a correction feels like tidying rather than authorship.

What actually caught it #

A second model, told to attack the diff.

The fix was not a better model or a longer prompt. It was a different one, with a brief that made disagreement its job rather than a risk.

It came back with four findings. One was this: on applications with many assets, Propshaft still enumerates, fingerprints and copies every asset during assets:precompile, so the work still scales with asset count. Dropping transpilation and concatenation lowers the cost per asset without making the build independent of how many there are, and the new wording promised the wrong thing.

Then a person had to decide whether the objection was correct, and that step took either already knowing the answer or being willing to go and read the Propshaft source until you did.

Three links, and the last one is the only one you cannot automate. A model wrote, another model challenged, and someone with domain knowledge decided which was right.

Drop the third link and you have two confident systems agreeing with each other.

It is not only the model that is confident #

There is a measurement for this, and it is worse than the anecdote. METR ran a randomised trial with sixteen experienced open-source developers across 246 real issues, in repositories they had worked in for years.

They were 19% slower with AI.

Going in, those developers expected a 24% speedup. Coming out, having just lived through the slowdown, they still estimated AI had made them 20% faster.

Forty points between what happened and what they believed happened, in experts, on code they knew.

METR is careful about what that does not show: early-2025 models, Claude 3.5 and 3.7 Sonnet through Cursor Pro, on mature codebases with demanding quality standards, and they explicitly decline to claim it generalises to most developers or to later tools.

What travels is the gap itself. Experience did not make those developers better at estimating their own speed. It made them more certain about an estimate that was forty points off.

Partial checking feels exactly like checking #

Here is the same failure again, from later the same day, after the rule above had been written down.

Take a Laravel migration guide that recommends upgrading to 11. Checking it against Laravel’s support table shows 11 left security support in March, so the obvious correction is: do not land on 11, go to 12, same PHP floor.

Two of those three facts are right.

Laravel 12 stopped getting bug fixes on 13 August 2026. Laravel 13 has been current since March and requires PHP 8.3, not the 8.2 that correction promises. The table has four rows, and stopping after two produces a fix that is still wrong.

Nobody skipped the check there. The check ran, felt complete, and stopped one row short. That is harder to design against than not checking at all, because it produces the identical feeling of having been careful.

What this means if you are the one paying for it #

You are not buying keystrokes any more. That part got cheap, and pretending otherwise is how founders end up overpaying for output they could have generated themselves.

What stayed expensive is the ability to look at a fluent, well-structured, technically-worded paragraph and say that specific clause is false. I have not found a shortcut to it. It comes from having been wrong about the same thing before, which is a slow way to acquire anything.

So when you are deciding who to hire, or whether the shop you are already paying is worth it, the question changed. It is no longer “can they build this.” It is: when the AI hands them something plausible, do they check it, and can they?

Three things worth asking, none of which require you to read code:

• Ask them to show you something the AI got wrong recently. Not a crash - a crash announces itself. Something that ran, looked right, and was wrong anyway. If nothing comes to mind, either they are not reviewing or they are not catching.
• Ask who reviews the AI’s work, and whether it is the same person who prompted it. Self-review by the author fails the same way here as it always has, except faster and with better grammar.
• Ask what happens when a check fails. A team that treats “stop and don’t ship this” as a normal outcome has a working process. A team where every review ends in approval has a ritual.

We wrote about the team structure that makes this hold up if you want the operating detail. The short version fits in a sentence: the reviewer has to be someone other than the writer, and they have to be allowed to say no.

Sources #

• Sean Goedecke, “LLMs reward expertise”
• Senko Rašić, "‘Code was never the hard part’ is an insult to all programmers"
• METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” - the 19% slowdown, the 24% predicted speedup, the 20% believed speedup, and METR’s own limits on what it shows
• Propshaft - the asset pipeline whose behaviour the claim got wrong. Its README settles both halves. It is faster: “a dramatically simpler and faster asset pipeline compared to previous options, like Sprockets.” And it still does per-asset work: “All assets in the load path will be copied (or compiled) in a precompilation step for production that also stamps all of them with a digest hash.” The original sentence took the first half as the reason for the second.