The Experiment That Died at 3 of 160

The A/B test launched at the top of the hour. Ninety seconds later, three trials had finished. Then nothing.

No error. No crash. No timeout. The dispatcher had been the loudest thing in our logs a minute earlier, fanning out trials, picking up workers, posting progress lines at a steady rhythm. Then it stopped posting anything at all. The process was still up. The queue still had 157 unstarted trials in it. The workers were idle. Whatever was supposed to keep the loop turning had quietly decided it was done.

This is the story of run f141e0cd on the initiative-language-ab experiment, and of the platform scheduling bug that was sitting underneath it. It is also a story about a specific failure mode that we think anyone running concurrent batch workloads on a shared scheduler is going to meet, sooner or later: the failure mode where everything looks healthy except the work, which is not happening.

The cold open: a quiet dispatcher

The experiment was a straightforward one by our standards. We were comparing two variants of an initiative-framing instruction in a Righthand's system prompt: one version that asked the agent to lead with the initiative it was proposing, another that asked it to lead with the user's stated goal. Each variant would run against the same eighty scenarios. Eighty trials per arm, one hundred sixty in total, dispatched concurrently against our usual trial workers.

We launched it the way we launch every A/B test. The framework spun up the run, registered both arms, enumerated the scenarios, and started feeding trials into the dispatch loop. The dispatcher's job, at its simplest, is to keep a target number of trials in flight at any given time: as one finishes, another goes out, until the queue is empty.

For about ninety seconds, that is exactly what it did. Three trials completed. Three result rows landed in experiment_trials. Three corresponding log lines confirmed dispatch, execution, and write-back. The cadence felt right. We tabbed away to do something else.

When we came back, the run was still marked running. The completed count was still three. The logs had not produced a new line in seven minutes.

The first hypothesis: a stuck worker

Our first guess was that one of the trial workers had wedged. This is the most boring possible explanation, and it is also the one that is correct often enough that it deserves to be checked first. A worker that has accepted a trial and then hung (waiting on a downstream API, blocked on a lock, stuck in a retry loop with no backoff cap) will hold a slot in the dispatcher's in-flight set without producing any output. If you have a small concurrency cap and one worker hangs, the dispatcher will keep handing out work up to the cap and then sit on its hands, looking exactly like it has stopped, because from its perspective the system is full.

We pulled the worker logs. They were not stuck. They were not anything. The last line any of them had written was the line that completed their respective trial in the first ninety seconds. After that, silence from the workers, silence from the dispatcher, silence from the runner that owned the loop.

This is the first place we started to feel uneasy. A wedged worker leaves a trail. It is mid-call on something. It has an open span, an outstanding HTTP request, a row in a leases table that has not been released. We had none of that. The workers had finished their three trials cleanly and then gone to sleep waiting for more work that was never going to be assigned.

We checked for OOMs. We checked for the Testing Service killing and restarting the dispatcher process. We checked for an exception buried in a log level we were not tailing. None of it. The dispatcher process was up, healthy by every health check we had, and not doing any work.

The turn: another run, also alive, also doing nothing

The thing that changed the shape of the investigation was noticing the other run.

A separate experiment had been launched about ten minutes before ours. Different team, different prompt set, different arm configuration, but the same underlying platform. We had not been paying attention to it because it was not our experiment. When we finally looked, it was in exactly the same state as ours: a small number of trials completed at the start, then a clean stop with no error, no crash, and no log activity. Its dispatcher was up. Its workers were idle. Its queue had work in it.

Two independent runs, on the same platform, both stopping in the same shape at roughly the same time. That is not a wedged worker. That is not an unhandled exception in one of them. That is the platform.

This is the turn in the investigation, and it took us longer to reach than it should have, because each run looked like an isolated story when we were tailing its logs in isolation. The pattern only became visible when we put the two side by side.

Once we did, the question changed. It was no longer "why did our dispatcher stop." It was "what does the platform do when two dispatchers are both trying to keep N trials in flight at the same time, against a shared pool of worker slots in the Testing Service's dispatch pool."

The root cause: concurrent runs starving each other

The Testing Service's dispatch loop is shared across all concurrent experiment runs. When the loop polls for pending trials, it assigns them to worker slots from a shared pool. When f141e0cd started, another experiment (reward-test-2) was already running on the same Testing Service. The loop dispatched 3 trials in roughly ninety seconds, then stopped polling entirely. No acquisition failure was logged. In single-run operation, this worked exactly the way you would expect.

In concurrent operation, it did not.

The bug, reconstructed from the outside, looked like this: the dispatch loop stopped iterating after 3 trials and never logged why. We never identified the specific concurrency primitive at the code level. The symptom was a silent stall, not a deadlock or a crash. The same stall reproduced across 4 different runs on 2 different experiments, all on the same Testing Service. The first three trials in each run had gone out before the loop stopped advancing. After that, the dispatch loop simply ceased to poll for new work. No errors, no timeouts, no lock contention in the logs. The loop was alive. It was just not doing anything. Forever, in principle. Until somebody noticed, in practice.

The reason it produced no errors is that, from the scheduler's perspective, nothing wrong was happening. Two clients had asked for resources. The resources were not available right now. The clients were waiting. That is the entire normal life of a scheduler. There was no timeout on the wait, because we had not modeled "wait forever for a resource that will never come" as a failure case. We had modeled "the resource is temporarily contended," which is a different thing, and we had given the scheduler permission to be patient about it.

Patient, in this case, meant indefinite.

What the logs actually told us, and what they didn't

In retrospect, the logs told us exactly what was happening. They just told us by what they were not saying.

A dispatcher that is working produces a recognizable cadence: dispatch line, completion line, dispatch line, completion line. A dispatcher that is broken in any of the usual ways produces something specific: an exception, a stack trace, a process exit, a supervisor restart, a health check failure. The dispatcher in f141e0cd produced neither. It produced the cadence for ninety seconds, and then it produced nothing, and "nothing" was not a state any of our dashboards had a color for.

Our alerting was built on the presence of bad signals. There was no alert for the absence of good signals. A run that emits an exception is loud. A run that emits a stack trace is loud. A run that stops emitting anything at all looks, from a metrics standpoint, identical to a run that has finished cleanly, except for the small detail that it has not actually finished. The running state with no recent activity was the loudest message the system could have sent us, and we did not have anything listening for it.

This is the part of the story we keep coming back to internally. The bug was in the scheduler. The reason the bug went undetected for as long as it did was in our observability. Both need to be fixed, and they are different fixes.

The fix, and what it cost

The immediate fix was small. We killed both runs and restarted the platform's scheduling component. A smaller run on the same experiment (16 trials, maxconcurrency=1) completed in roughly 32 minutes. A smoke test (32 trials, maxconcurrency=4) finished in about 11 minutes. The full 160-trial run was never retried. The engineer investigating the bug left the company before he could restart it.

We want to be honest about where we are with the structural fix, which is: we have not shipped one.

The dispatch bug was still active when the engineer who owned the investigation left. We know what we would build: a fair queue with per-run round-robin dispatch and a per-acquire timeout, so no single run can starve another's slot claims. We have not built it yet. The surface area of "two runs interleaving badly on a shared dispatch pool" is larger than any one fix, and we are not going to pretend a fix is live when it is not.

The observability fix is in a similar state. The recommended approach is a liveness watchdog that compares expected trial progress against observed progress: if a run's trial count has not advanced in N minutes while trials are still pending, fire an alert. The core lesson is that the dispatch loop had no self-monitoring, so it could stall silently for hours without anyone knowing.

What we are not pretending is that we have solved silent failure as a category. We have identified this particular silent failure. The fix is designed but not shipped. The next silent failure will look different.

The broader lesson: absence as a signal

There is a category of bug here that is worth naming, because we expect to meet it again and we expect other teams running concurrent batch workloads to meet it too.

Call it the absence-of-signal failure. You have a system that, when it is working, produces a steady stream of telemetry: logs, metrics, completion events, progress lines. When something goes wrong in one of the well-modeled ways, the stream changes shape: errors appear, exceptions get caught and logged, alerts fire. Your monitoring is built around the shape changes. What it is not built around is the stream simply stopping while every other indicator says the system is alive.

A scheduler that is patiently waiting on a resource that will never come is, from the outside, indistinguishable from a scheduler that is correctly idle. A dispatcher that has deadlocked with another dispatcher is, from the outside, indistinguishable from a dispatcher that has finished its work. The processes are up. The health checks pass. The error rate is zero. The dashboards are green. Nothing is happening, and nothing is the problem.

The defenses are not exotic. They are the same defenses you would apply to any system where the worst failures are quiet.

Model expected progress, not just expected errors. A run that has accepted 160 trials and completed 3 of them is making a claim about what should happen next. The system should know what that claim is and should notice when it is being violated. "No errors in the last seven minutes" is not the same as "things are fine."

Put a ceiling on patience. Every wait in a scheduler should have a maximum duration that is shorter than the longest amount of time you are willing to be wrong. Indefinite waits are an invitation for two well-behaved components to deadlock politely and forever.

Test concurrency under realistic load. Single-run testing will not surface this class of bug. The bug requires the contention. We had run the A/B framework against many sequential workloads and never seen it. The first time two runs overlapped in production was the first time the bug had room to express itself.

Treat "running with no recent activity" as a first-class state. The binary of running vs. failed is too coarse. A run that is in running and has not produced a completion in longer than its own characteristic interval is in a third state, and that state deserves its own color on the dashboard and its own page in the runbook.

The last one is the one we are taking most seriously. Our experiment framework should distinguish between a run that is actively making progress and a run that is nominally alive but not producing anything. Those are different states. They should always have been different states.

A grounded close

The thing we keep coming back to is that the scheduler was doing exactly what we had told it to do. It had been asked to wait until a resource was available. The resource was not available. The scheduler waited. There was no bug in the scheduler's behavior on its own terms. The bug was in the gap between what the scheduler knew and what the rest of the system needed to know about what the scheduler knew.

If you are running concurrent workloads on a shared scheduler, the question to ask about every wait in the system is the same question we should have asked about ours: how long is this wait allowed to last, and what is the path by which somebody outside the wait finds out that it has lasted too long. If you cannot answer that question with a number and a mechanism, you have a run that will die at 3 of 160 waiting to happen. Ours took ninety seconds to start and seven minutes to notice. We would rather the next one take seven seconds.