The Tests That Passed in Name Only

The test suite had been green for a week. It was not.

Our automated integration tests run every six hours against the platform that builds and operates Righthands. They cover everything from the lightweight unit-level checks at Tier 1 up through the heavier end-to-end scenarios at Tier 3 and Tier 4, where we exercise full agent workflows against provisioned infrastructure. When we look at the dashboard, we look for a single thing: are the recent runs passing. For a week, the recent runs were passing. For that same week, every Tier 3 and Tier 4 run had been failing in exactly the same way, and nobody on the team knew.

This is the story of how the suite arrived at that state, what was actually broken underneath, and the thing we keep coming back to: the difference between having tests and having tests that can tell you when something is wrong. The first is cheap. The second is the only one that matters, and we had quietly stopped having it.

The cold open: a week of green that wasn't

The failure surfaced the way these things often do, which is to say by accident. Avery was searching the production error logs for story ideas for this week's blog post and noticed the same two error messages repeating in the Testing Service logs every six hours, going back nearly two weeks. Nobody had flagged them. The test suite's top-level status was green. The individual Tier 3 and Tier 4 runs, when we actually opened them, were not.

The errors were not subtle. Every Tier 3 run for the past week had failed with the same pair of messages:

Every Tier 4 run had failed with the same pair. One hundred and thirteen total error runs across the two tiers (56 Tier 3, 57 Tier 4), spanning every scheduled execution from May 27 through June 9. The scheduler had fired consistently every six hours at 00:15, 06:15, 12:15, and 18:15 UTC, four runs per day, not a single one skipped. All of them had logged the failures. None of them had paged anyone, surfaced on a dashboard, or changed the color of the top-level suite status.

The first thing we did was check the suite configuration, because the easy answer was that somebody had recently turned off alerting on Tier 3 and Tier 4. Nobody had. The alerting had never been on. Tier 3 and Tier 4 were first created around April 16, 2026, with manual test runs. Scheduled runs were enabled starting May 27, and they had been broken from the very first scheduled execution. No alerting had ever been configured for them. Tier 1 and Tier 2, by contrast, run hourly and have working alerting, with 319 and 297 passed runs respectively in the last 30 days. This was an initial setup oversight, not a regression. The tests had been added, the runs had been scheduled, and the part where a failure produces a signal that a human will see had been, in the way these things are, left for later.

The first hypothesis: a credential rotation we missed

The 401 Unauthorized on claim-from-pool looked, on its face, like a story we had told ourselves before. The Tier 3 and Tier 4 tests provision real infrastructure: they claim a sandbox environment from a pool of pre-warmed environments, configure it for the test scenario, and tear it down at the end. The claim step authenticates against the Testing Service's pool manager API. The Testing Service owns the full dispatch loop: trial creation, result collection, and the HTTP call to the pool manager to claim a sandboxed test environment. If the credential used for that authentication had expired or been rotated, the pool manager would return a 401 Unauthorized before the test harness could claim anything, and every claim would fail in exactly this shape.

We went looking for the rotation. The credential lives in the environment variables and CI secrets injected at runtime; an early Tier 4 error from April 16 had shown "Missing required config: SUPABASEMAINSERVICEROLEKEY," confirming the secrets-injection pattern, though the exact credential management system is not exposed in the logs. We could not determine when the credential was last rotated, as no rotation logs are available, but it was the same value the previous green runs had used. Pasted into a working environment, it claimed a pool environment cleanly.

So the credential was fine. The 401 was not coming from a stale secret. Something else about how the test runner was presenting itself to the pool manager had changed, or the pool manager's idea of what counted as authorized had changed under it.

We were still, at this point, treating the bug as the interesting story. It was not. The interesting story was the week of silence around it. We kept getting pulled back to that, and kept pushing the thought aside to chase the immediate failure first.

The turn: the second error was the cause, not a symptom

The second line in the log, Auto-created team not found, had been reading to us like a downstream effect of the 401. If the claim fails, the test cannot proceed, and any later step that expects a team to exist would naturally fail too. That was the wrong reading.

When we pulled a full run log and traced the order of operations, the Auto-created team not found was not downstream of the 401. It was upstream of it, with a twist. The flow is: (1) create a test user, (2) auto-create a team for that user, (3) use the team's identity to claim an environment from the pool. Tier 4 fails at step 2, with the error "Auto-created team not found: Cannot coerce the result to a single JSON object." Tier 3 gets past team creation but fails at step 3, with the claim-from-pool 401 Unauthorized. Each Tier 4 error references a different user UUID, confirming that a fresh test user is created per run.

The auto-created team is the test harness's way of standing up an isolated tenant for each run, so that nothing one test does can contaminate another. The team is created programmatically at the start of the run. The credential used to call claim-from-pool is scoped to that team. If the team is not there, the credential that should have been minted for it is not there either, and the pool's authorization check rejects the request as unauthenticated.

So the real question was not "why is the credential failing." It was "why is the team not being created."

The error message, "Cannot coerce the result to a single JSON object," told the story. The team lookup-or-creation query was returning either zero rows or multiple rows where exactly one was expected. This is consistent with a database migration that changed the team table schema or RLS policies, or an API change that altered the response format. The auto-creation step was running, but the result it produced was no longer parseable as a single team record, so the harness treated it as absent.

What was true in every Tier 3 and Tier 4 run was that the team-creation step was silently producing nothing usable, the pool claim was then failing with a 401 that pointed at the wrong layer, and the test was marking itself failed with two log lines and no further investigation. The harness was, in its own way, doing exactly what it had been told to do.

The root cause: a change upstream, and a test suite that did not know to care

We do not have a changelog or deployment log that pins down the exact upstream change, but the timeline is clear: the breakage started on May 27, 2026, with the very first scheduled error runs. Both Tier 3 and Tier 4 broke simultaneously, which points to a single deployment that affected both the team-creation and pool-auth pathways. The evidence points to a migration or config change rather than a deliberate API revision. The "Cannot coerce" error suggests a database-level change, likely a migration altering auto-creation behavior or RLS policies, while the 401 suggests a credential or auth config change. Both starting on the same day points to one deployment touching both. In normal product use, the change was either benign or correctly handled. In the test harness, which was constructing teams through a path that the upstream change had quietly invalidated, every Tier 3 and Tier 4 run began failing at the team-creation step.

There are two failures here, and they are worth separating.

The first failure is the upstream change being shipped without an integration check that would have caught the test-harness path. That is a coordination problem and a contract problem, and we have one set of answers for it. The second failure, the one this post is mostly about, is that our own test suite watched itself fail a hundred and thirteen times in a row and never raised its hand.

The Tier 3 and Tier 4 runs were configured to write their results to the test runs database (every run recorded with status, error message, and timing) and the Testing Service logs. No Slack notification, no PagerDuty alert, no push notification of any kind existed for these errors. The results were dutifully written. Failures were logged at the appropriate severity. The runs exited with non-zero status. The pieces that should have driven an alert (a notification channel, a paging rule, a dashboard query that counted failures over a window) had never been wired up for these tiers. Tier 1 and Tier 2 had alerts. Tier 3 and Tier 4 had not. Tier 1 and Tier 2 have been running much longer, at hourly cadence, with working alerting and hundreds of passed runs in the last 30 days. Tier 3 and Tier 4 were first created on April 16 with manual test runs, then sat dormant for 41 days with no runs at all (also unnoticed), before scheduled runs were enabled on May 27 at a six-hour cadence. The alerting setup that covers Tier 1 and Tier 2 was never extended to the newer tiers, and the omission had not been noticed.

This is the gap we keep coming back to. The tests existed. They ran on schedule. They produced correct verdicts about whether the system was working. They wrote those verdicts to a place a computer could read. The chain between "a test failed" and "a human knows a test failed" had a missing link, and the missing link had been missing since the day the tests were written.

The fix, and what it is costing us

As of today, the fix has not shipped. Both Tier 3 and Tier 4 are still failing with the same errors. The most recent scheduled run, at 12:15 UTC this morning, produced the same two log lines it has been producing since May 27. The system has been 100 percent broken for thirteen days with zero intervention.

We know what the fix looks like. The team-creation step needs to be updated to match whatever the upstream change did to the auto-creation contract, so that the harness gets back a parseable single-team result instead of the "Cannot coerce" error. That is the proximate repair. It has not happened yet.

The harder fix is the alerting. Zero alerting is configured for Tier 3 and Tier 4. Errors are logged but never alerted on. The system we want to build would wire these tiers into the same rules we already have for Tier 1 and Tier 2: any failed run pages the on-call engineer for the test infrastructure, and any tier that has not produced a successful run in 24 hours (four scheduled runs at six-hour intervals, all failing) is treated as broken regardless of the most recent run's status.

The staleness rule is the one we are most interested in. No staleness detection exists today. A test suite that has stopped running at all looks, from a dashboard standpoint, identical to a test suite that is passing. The absence of a failure is not the same as the presence of a success. We had been monitoring for the first and acting as though we were monitoring for the second.

We have started the unglamorous bookkeeping. We went through every tier of the suite, every scheduled job, every integration check, and asked the same question of each one: if this fails right now, who finds out, how, and within what window. The answers were not all good. The "Think: Workflow Escalation" test group has a 91 percent pass rate with 29 failures out of 325 runs, an ongoing flaky test nobody has addressed. Zero passed Tier 3 runs exist in the entire history of the production environment. The 41-day gap between April 16 and May 27, where no Tier 3 or Tier 4 runs happened at all, went completely unnoticed. The overall pass rate across all tiers is only 80 percent, heavily dragged down by the Tier 3 and Tier 4 errors. We are fixing the ones we can fix this week and filing the rest.

The broader lesson: a test is not a signal until somebody hears it

There is a category of failure here that is worth naming, because we expect to meet it again and we expect other teams building automated test infrastructure to meet it too.

Call it the silent test failure. You write a test. The test runs on a schedule. The test catches a real regression. The test logs the regression accurately. And then nothing happens, because the part of the system that turns a logged failure into a paged human was never built, or was built for some tests and not others, or was built once and then quietly degraded as the suite grew around it.

A silent test failure is worse than no test at all. A team with no tests knows it has no tests, and behaves accordingly: it ships carefully, it watches production, it expects to find things the hard way. A team with silent tests believes it has coverage. The dashboard is green. The suite is running. The runs are passing, as far as anyone can see. The belief is doing active damage, because it is shaping decisions about what is safe to change, what is safe to deploy, what is safe to leave alone.

The defenses are not exotic. They are the same defenses you would apply to any system where the worst failures are quiet.

Treat the alerting wiring as part of the test, not a separate concern. A test that has no path to a human is not done. The definition of done for a new tier, a new scheduled job, a new integration check, should include the line in the alerting config and the test of the alerting config. If the alert does not fire when the test fails, the test does not exist yet.

Monitor for the presence of success, not the absence of failure. A run that did not happen looks exactly like a run that passed if you are only counting failures. Every scheduled job should have a freshness check: the most recent successful run is within N intervals, where N is small. If the last green run was a week ago, the suite is broken even if the dashboard is green.

Audit the suite the way you would audit a runbook. Walk every test. Ask who finds out when it fails. Write the answer down. The tests where the answer is "nobody" are the ones doing the most harm, because they look the most like coverage.

Distinguish a passing test from a test that has not been allowed to fail recently. A test that has not caught a real bug in a long time is either covering ground that is genuinely stable or covering nothing at all. The two are indistinguishable from the dashboard. Periodically, on purpose, break the thing the test is supposed to catch and confirm the test catches it.

The last one is the one we are taking most seriously. It is easy to write a test and trust it forever. It is harder, and more honest, to treat the test itself as a piece of software that can rot.

A grounded close

The thing we keep coming back to is that the suite was, in the most literal sense, working. The runs were executing. The assertions were evaluating. The failures were being detected. Every single layer of the system was doing the job we had asked it to do. The only thing that was not happening was the thing the entire apparatus existed for, which was to tell us when something was wrong.

If you are building automated tests against a system you care about, the question to ask about every test in the suite is not "does it pass." It is "when it fails, what happens next, and how do I know that path still works." A test that cannot answer the second question is a test that passes in name only. We have had two weeks of those and counting. We would rather not have another.