The Green Test That Defended the Bug

A passing test suite tells you that your code does what your tests say. It tells you nothing about whether your tests say the right thing.

In one week we found two tests that were green, well-written, clearly named — and defending bugs. Not neglected tests. Not commented-out tests. Tests that ran on every commit and asserted, precisely and confidently, the wrong behaviour.

Both were caught the same way, and it was not by reading them.

The first: a test that asserted the bug

Our product offers a fallback when a user's cloud workspace fails to start: we provision one on our own infrastructure so they are not dead-ended. The logic gating it looked reasonable.

export const RESCUE_CODES: ReadonlySet<string> = new Set([
  'CODESPACE_BILLING_BLOCKED',
  'CODESPACES_UNAVAILABLE',
  'CODESPACE_LIMIT_REACHED',
])

An allowlist. Three specific failures get the fallback; everything else does not.

Over sixty days, the most common terminal failure in production was a code that was not on that list. It hit eleven people. Every one of them got a dead end with no fallback, including a user who had run thousands of tasks, failed nine deploys across two days, and then churned. We learned about her by email.

Here is the test that had been passing the whole time:

it('leaves non-rescue codes untouched', async () => {
  const out = await decorateFailureWithRescue(
    { code: 'PROVISION_FAILED', message: 'm' }, fleet, 'u1',
  )
  expect(out).toEqual({ code: 'PROVISION_FAILED', message: 'm' })
  expect(fleet.markEligible).not.toHaveBeenCalled()
})

Read it as a specification and it says: when the most common failure occurs, offer the user nothing. That is the bug, written down, asserted, and defended on every CI run. Anyone who fixed the behaviour would have seen a red test and, quite reasonably, assumed they had broken something.

The test was not wrong about the code. It was wrong about the product.

The second: a fixture that modelled the defect

The other one is subtler and, I think, more common.

We record an event when a subscription renews. Our webhook handler marked every incoming "active" notification as a renewal, with a comment explaining why that was safe: these always arrive after the row already exists, so they must be renewals.

They are not. Payment providers echo the initial purchase through the same webhook. One live subscriber generated a "renewal" event eleven milliseconds before his actual first activation, and another two hours later. On a monthly plan.

The test covering this was called "fires isRenewal=true on a RENEWED notification", and it passed. Here is the fixture:

const existingSub = {
  googleExpiresDate: new Date(Date.now() + 30 * 24 * 3600 * 1000),
  currentPeriodEnd:  new Date(Date.now() + 30 * 24 * 3600 * 1000),
}

The stored period end is thirty days out. The mocked provider response returns... thirty days out. The period does not advance.

A real renewal moves the billing period forward. A fixture where it does not is not modelling a renewal — it is modelling the echo. The test was asserting on the exact shape of the bug and calling it the correct case. When we fixed the code to require the period to actually advance, that test went red, and for a moment it looked like the fix was wrong.

Neither was found by reading code

Both were found by mutation: deliberately breaking the implementation and checking that a test noticed.

The mechanic is trivial. Delete a guard, invert a condition, remove a line. Run the tests. If they still pass, the test does not cover what its name claims.

We do this on anything load-bearing, and it takes seconds:

cp src/thing.ts /tmp/thing.bak
# remove the guard
sed -i '' '/if (input.alreadyTried) return false;/d' src/thing.ts
npx jest src/thing.spec.ts     # must go RED
cp /tmp/thing.bak src/thing.ts

The same week, this caught a third one — a test of ours, freshly written, that did not test anything at all. It asserted that headings inside fenced code blocks are excluded from a table of contents. The fixture used # install deps inside a shell block. But the extractor only matches ## and ###, so an # heading was never a candidate. We removed the guard entirely and the test still passed. Rewritten with an ## inside the fence, it now fails when it should.

That one had been green for about twenty minutes before we broke it on purpose. The other two had been green for months.

What actually distinguishes the two failure modes

It is worth separating them, because they need different habits.

The test that asserts the bug happens when you write tests from the implementation instead of from the requirement. You read what the code does, describe it accurately, and ship. The test is a faithful record of current behaviour — which is exactly what makes it dangerous, because it converts an oversight into a specification. The defence is to write the assertion from the user-facing rule first: "a user whose deploy fails should be offered a fallback" is a sentence you can check against the world.

The fixture that models the defect happens when you build test data from what the code already produces rather than from what reality produces. Nobody asked "what does a real renewal look like on the wire?" — the fixture was assembled to make the function run. The defence is to state, in a comment next to the fixture, what real-world event it represents. Writing "a real renewal moves the period forward" above the data makes the mismatch obvious immediately.

The uncomfortable conclusion

Coverage numbers cannot see either of these. Both files were covered. Both suites were green. Both tests had accurate names and clear intent.

A test is a claim about correct behaviour, and claims can be wrong. The only cheap way we know to check the claim is to break the code on purpose and see whether the test objects. If it does not, the test is decoration — and worse than decoration, because it is decoration that will actively stop the next person from fixing the bug.