Your Link Checker Is Probably Checking Nothing

A link checker reporting zero broken links is telling you one of two things, and the output looks identical either way: it inspected everything and found nothing, or it inspected almost nothing. Ours had been doing the second since the day it shipped.

Here is the run that caught it:

🔍 149516 Total  🔗 31937 Unique  ✅ 15642 OK  🚫 0 Errors  👻 133874 Excluded

133,874 excluded of 149,516. Only 15,642 links were ever looked at, and the gate had been reporting that as a pass on every pull request since the day it shipped.

The bug is one flag, and you probably have it too #

We use lychee with --offline, which is correct: internal link checking should not hit the network. The problem is what “internal” means after Hugo renders.

Production emits absolute URLs. Your /blog/foo/ becomes https://jetthoughts.com/blog/foo/ in the built HTML. And --offline drops every http(s) URI as external, by design.

So the crawler saw a page full of absolute links, classified all of them as “not my problem”, and reported success on what remained - which on our homepage was a single skip-link anchor.

Adding --remap points the public origin back at the built tree:

# Rakefile
task :links do
  dir  = ENV.fetch("OUTPUT_DIR", "_dest/public-linkcheck")
  root = File.expand_path(dir)

  sh("lychee", "--offline", "--no-progress",
    "--remap", "https://jetthoughts.com/(.*) file://#{root}/$1",
    "--root-dir", root, "#{dir}/**/*.html")
end

Same command, same flags, one addition.

Checked links went from 15,642 to 114,050 on the run that landed the fix - roughly a sevenfold increase in what the gate could actually see, with the excluded count dropping from 133,874 to the genuinely external links it should have been skipping all along.

Run this on your own repo before you keep reading #

The diagnostic is cheap.

Whatever checker you run, it works the same way.

Compare what your checker says it inspected against how many links your built site actually contains.

# how many internal links does the built site actually contain?
grep -rhoE 'href="[^"]+"' public/ | wc -l

# now compare that to the "checked" number your CI prints

If your checker reports a few hundred links on a site that renders tens of thousands of them, it is not passing your build so much as abstaining from it.

What it found the moment it could see #

Five real defects, invisible until that flag changed:

• two wrong blog slugs, five links between them
• a post whose own canonical_url pointed at a 404
• a dead /contact/ on a conversion page
• a closing section offering a downloadable ROI calculator - itemising five things inside it, promising “no email required, instant download” - for a spreadsheet that had never existed

That last one had been live long enough that nobody remembered writing it. It was a template ending nobody ever filled in, and every green run since had quietly confirmed that the page was in good shape.

Then we went looking on purpose #

The link checker was found by accident, which was the uncomfortable part. So we ran a deliberate exercise: eight realistic defects, planted one at a time, with a prediction written down before each one about which check should catch it.

Three of eight were caught.

Those written-down predictions mattered more than the score. Writing “the banned-phrase ratchet will catch this” before planting it turns a vague sense of coverage into a falsifiable claim - and two of those claims were wrong in a specific way.

One ratchet was carrying slack. It was set to fail above 14 hits when the tree actually had 11, so a planted phrase landed in the gap and the suite stayed green. A ratchet with three spare slots does not guard the last three defects.

# The fix is boring: set the baseline to the MEASURED count,
# then prove it is exact by dropping it one lower and watching it fail.
RENDERED_BASELINE = 11   # was 14, against an actual 11

def test_rendered_pages_do_not_regress_on_banned_phrases
  violations = rendered_files.flat_map { |path| rendered_hits(path) }.sort

  assert_operator violations.size, :<=, RENDERED_BASELINE,
    "Banned phrases in BUILT HTML went up (baseline #{RENDERED_BASELINE}, " \
    "now #{violations.size}):\n  " + violations.join("\n  ")
end

Setting it to 10 and watching it fail takes fifteen seconds. It is the only thing separating a ratchet from a decoration.

The one that should genuinely worry you #

Our visual regression suite was passing because it compared screenshots against nothing.

Run from a git worktree, it lost its reference images and wrote fresh captures over them instead. Every run green. It had been green for a while.

Someone finally tested the tester: injected body { background: red !important }, confirmed the rule reached the built CSS bundle, confirmed the page referenced that fingerprinted file, then measured the captured PNG against the baseline.

Candidate [255,0,0] against a baseline of [255,255,255], for a difference level of 0.68.

The run reported 0 failures.

Three earlier injections had failed to go red, and the person doing it had blamed their own injections each time. That is the honest shape of this problem: when a check is broken, the evidence that it is broken is indistinguishable from everything being fine.

Say the number, not the name #

Every one of these failures shared a tell, and it is cheap to look for: the check reported a verdict without ever reporting the size of the thing it had just examined.

A gate that reports 0 failures is telling you about its exit code and nothing else. A gate that reports 114,050 links checked, 0 errors is telling you what it actually inspected before it decided everything was fine. Only the second kind can be caught lying.

So: make every check print its denominator, and read it.

[snap_diff] 55 screenshots compared      # a real number you can watch move
lychee: 114,050 links checked            # not "link check passed"

If your CI output cannot distinguish “inspected everything and found nothing” from “inspected nothing”, it is not a check. It is a green icon with a job title.

What we do now, and what it costs #

SQLite is the case that should settle this. Its test suite is famously larger than the database itself, and it still carried a data-race bug for sixteen years. Tailscale hit it in production and wrote up the hunt; the detail worth stealing is what the maintainers had to do to see it at all - the bug was “so rare, the SQLite developers had to add code to deliberately trigger it in their testing environments.”

They broke it on purpose. Until they did, every run was green, and green meant nothing about that bug.

So: a new test is not finished until someone has broken the thing it guards and watched it fail. Flaky failures do not count; this has to be deliberate, and someone has to be watching when it goes red.

That adds maybe two minutes to writing a test.

It is the highest-return two minutes in the suite, and worth more than a coverage percentage that cannot tell a working check apart from a decorative one.

If you want the exercise: pick your three most important checks, plant one realistic defect against each, and write down beforehand which one should catch it. You will learn more in an afternoon than a coverage report has told you all year.

Sources #

• lychee - the link checker, and its --offline and --remap behaviour
• Tailscale, “Tracking down the 16-year-old WAL-reset SQLite bug” - SQLite carried a data-race bug for sixteen years, and its developers “had to add code to deliberately trigger it in their testing environments” before any test could see it