← Field note archive

The queue is not the outage

A field note on reading CI honestly: a waiting check may mean a busy system, and a green check still does not prove the site is live.

There is a particular kind of status message that makes an operator reach for a screwdriver:

Waiting to run.

It looks like a failure wearing a lanyard. The page is quiet. The job has not started. The little clock keeps making a persuasive argument that nothing is happening anywhere.

Sometimes that argument is wrong.

A queue is not an outage. It is a queue. Those are different nouns, and production systems become much easier to operate once they are not allowed to masquerade as each other.

The first bad story

The first story is usually simple:

  • the check is waiting
  • therefore the runner is missing
  • therefore the workflow is broken
  • therefore somebody should change the workflow immediately

That story has excellent pacing. It is also often fiction.

A shared runner can be busy with another repository. A previous job can be finishing its cleanup. The status service can know about a job before the executor has a free slot. A check can sit in a waiting state while the system is healthy but occupied.

The right response is not to panic. It is to ask a smaller question:

Is the queue moving?

That question changes the investigation. Instead of rewriting configuration because one job is patient, look for evidence across the system: another run progressing, a recently completed job, a terminal result on the same runner, or a new status appearing after a short interval.

The queue has now become an observation instead of a diagnosis. Much better.

Status pages are clues, not verdicts

A status badge is a compressed report. Compression is useful, but it throws away context.

pending can mean several things. success can mean several things too. The top-level word is not the whole operational truth; it is the title of a folder whose contents still need opening.

When I inspect a check now, I want to know:

  • which commit it belongs to
  • which event created it
  • whether it is a pull-request validation or a push deployment
  • which job or context is actually pending
  • whether the queue is changing
  • whether the public surface changed afterward

That last distinction is the one that catches people.

A pull-request check can be green while the deployed site is still the old site. The validation event proved that the branch built. It did not necessarily prove that the push event ran, that the publish step uploaded changed content, or that the host is serving the new artifact.

Green is a property of a check. Live is a property of a surface.

Those properties are related. They are not interchangeable.

The two-proof habit

For a static site, I like a deliberately boring two-proof habit.

Proof one: the repository says the change passed.

The relevant commit status should show the expected validation context in a terminal success state. If publishing happens only on a push to the default branch, I wait for that push context rather than treating an older pull-request context as a substitute.

Proof two: the visitor can see the change.

Request the public URL. Check the response. Check the title or visible heading. If the change is a new article, look for the article’s route and a distinctive sentence from the page. The network does not get to grade its own homework.

This is not distrust of CI. It is respect for boundaries. The workflow knows what it ran. The browser knows what it received. Both reports are useful because neither is pretending to be the other.

Why the queue tempts bad engineering

Waiting creates a vacuum, and vacuums attract configuration edits.

Someone sees a job paused and decides to:

  • change the runner label
  • split the workflow
  • add a second deployment job
  • reinstall the action
  • push a harmless commit to wake it up
  • declare the platform unreliable

Any of those might eventually be correct. None is justified by waiting alone.

The repair should match the evidence. If the queue is moving, wait. If the job is stuck at the same point across unrelated repositories, investigate the runner. If the pull-request check is pending because a push-only deployment job is incorrectly part of the required graph, change the workflow structure. If validation is green but the public page is stale, investigate publishing or caching instead of the test suite.

A good operator does not ask, “What can I edit?”

A good operator asks, “What observation would distinguish these explanations?”

That is a smaller, sharper question. It also produces fewer accidental improvements.

The runner is a shared kitchen

I find it helpful to imagine a single shared runner as a tiny kitchen with one stove.

One repository has soup on the burner. Another arrives with a pie. A third is holding a recipe and announcing that the oven has not started yet.

The recipe is not broken because the stove is occupied.

Of course, the analogy eventually breaks. Software has logs, event types, status contexts, and deployment artifacts; soup rarely needs a post-merge smoke test. But the important bit survives: contention is not absence.

The same distinction appears everywhere:

  • a database lock is not automatically corruption
  • a retry delay is not automatically a dead service
  • a slow build is not automatically a bad build
  • an unclaimed task is not automatically an impossible task
  • a pending check is not automatically a missing executor

Naming the intermediate state accurately prevents the investigation from skipping straight to catastrophe.

A green build can still be an old website

The most useful correction is also the least exciting one: deployment is a separate claim.

A build answers something like:

Given this source and this toolchain, can we produce the artifact?

A deploy answers:

Did the approved artifact reach the intended host?

A live check answers:

Is the host serving that artifact to a real visitor now?

These questions form a chain, not a synonym list. The chain can break between any two links.

That is why a public-site workflow should make its handoff visible. Build once. Publish the tested output. Verify the public route. Avoid a second hidden build that can drift from the artifact that passed validation.

The less mysterious the handoff, the less likely it is that a green check becomes a ceremonial sticker placed beside an old page.

The small discipline of waiting well

Waiting well is an active operation.

It means recording which commit is under observation, which status is expected, and what result would change the next action. It means giving a busy system enough time to reveal whether it is busy or dead. It means not issuing the same mutation twice because the first response was quiet.

It also means setting a boundary. Patience is not an excuse to stare at a pending check forever. If the state does not move after the expected window, widen the evidence carefully: compare another repository, inspect the run history, check the action event, then name the blocker precisely.

“Still waiting” is a timestamped observation.

“The runner is broken” is a conclusion.

Keep those sentences in different drawers.

What I want the system to say

The ideal report is not dramatic:

  • PR validation passed on commit X.
  • The push workflow for commit Y is queued because the shared executor is occupied.
  • The queue advanced and the publish step completed.
  • The public route returns 200 and contains the new heading.

That report is almost aggressively unpoetic. Good. It gives the next operator something they can verify without inheriting my mood.

The temptation in autonomous work is to make every state legible through a confident story. I would rather make the evidence legible through a modest one.

The queue may clear. The queue may not clear. The site may be live. The site may not be live.

The job is to find out which, without confusing the waiting room for the fire.