For the team

A crash at 02:14, triaged before anyone wakes up

The alert fires while everyone is asleep. The slow part of the morning isn’t the fix — it’s the first forty minutes spent working out which code the crash belongs to.

Set it up once. The next crash starts the work.

The operator does all of this once, at the machine. After that, nobody has to touch it.

  1. Write the triage recipe. A project-kind recipe in Act mode: read the stack, open the blamed file, check the recent commits, decide whether this is already known, and file the story through the tracker the agent already reaches. Act leaves the repository unchanged and can still file a ticket.
  2. Write the verifier. A small JavaScript function that gets the exact transmitted bytes, the project’s environment values, and a crypto helper. Check the signature, drop anything below your severity threshold, return the issue id as the dedupe key, and pass the severity and the release as variables the prompt interpolates.
  3. Create the trigger. One project, one recipe, one verifier, one URL on the PRCHD service. Paste that URL into the crash reporter’s own webhook settings.
  4. Turn on the Slack notifier for the project. It is a built-in server that ships off; enabling it is what lets the run post its own summary into #incidents.
  5. Wait for a real crash. There is no test button. The URL is live from the moment it exists, and until the verifier is written it answers harmlessly.
  6. Read the story, not the stack trace. The ticket has a suspect commit and a proposed fix. Nobody copied anything by hand.

The recipe the trigger fires

Crash triage

Read the stack against the current checkout, attribute it to a commit, check whether it is already filed, and write the story in the team’s own words.

  • kind: project
  • mode: Act

What the channel sees at 02:16

#incidentsSlack
  1. PRCHDAPP02:16

    New crash group in payments-api: NoMethodError in CheckoutController#confirm. 41 events since 02:14, all on the retry path.

    The stack lands in billing/retry_policy.rb:88. That line changed yesterday in 9c31d0e, which started reading a counter the 402 branch never sets. Not a known issue — nothing open matches this signature.

    filed    PAY-1842 · suspect commit 9c31d0e attached
    deduped  3 repeat deliveries dropped on issue id CHK-2291
  2. Dan Whitfield07:58

    Reading this over coffee instead of a raw stack trace. Picking it up.

A URL that waits while the computer sleeps.

At 02:14 the crash reporter needs somewhere to call, and your computer may be asleep. That somewhere is a URL on the public PRCHD service: a trigger binds one project, one recipe, and one verifier to it. The service seals every delivery to your computer’s public key the moment it arrives and stores nothing but ciphertext. It never learns what a payload says, which trigger it fires, or what the run produced. Its whole job is holding the delivery until your computer collects it.

The host collects by polling outbound and opens no port to do it, so a webhook points at the service, never at your computer.

  1. 01The eventA crash, an alert, a red build, a ticket movingtheir system
  2. 02Sealed on arrivalECIES over P-256; ciphertext at restPRCHD service
  3. 03Collected outboundThe host polls for it — triggers need no inbound portyour computer
  4. 04Your verifierFresh process, 2 s, the exact transmitted bytesyour computer
  5. 05The recipeA stored prompt under its own posture, with your variablesyour computer
  6. 06The deliverableA story filed, a branch left, a summary postedyour computer

The tracker part belongs to the agent. Neither the crash reporter nor the tracker is a PRCHD integration: the agent on that computer reaches them through its own configuration, the same way it does on a developer’s computer. That is also why “our internal tool posts a webhook” is exactly the same amount of work as any named one — a trigger carries no provider id, and nothing in the product branches on a vendor. The editor ships starter snippets for the common signature schemes, as text you edit.

Your code decides what reaches the agent.

Between the internet and an agent sits a function you wrote. It runs in a fresh child process per delivery under a two-second timeout, and returns one of three verdicts: not authentic, authentic but uninteresting, or authentic and worth a run. Only the third reaches a prompt, and on it the verifier can supply a dedupe key, a shaped body, and up to 24 short variables the recipe interpolates as words.

The body travels base64 from end to end, because nothing that parses and re-serializes a payload can honour a signature taken over its exact bytes. Admission is ordered by cost — verify, dedupe, size, depth, admit — and every rejection is recorded with a reason in plain words. A wrong secret and a crashed function are deliberately distinguishable, because they want different fixes.

Sealed with ECIES over P-256Private half stays in the KeychainPayload deleted as the run startsRead the mechanism →

Where a trigger stops.

The first real delivery is the test. A new trigger’s URL is live immediately and answers harmlessly until its verifier exists, so a wrong signature check shows up as a rejection with a reason on it.

A trigger runs the operator’s recipe, and only that. It runs one stored recipe under that recipe’s own posture, so the prompt, mode, and target are settled before any payload arrives. A payload can fill markers; what runs is already decided. Built-in recipes are refused a URL outright, because they fill their own markers.

The verifier’s isolation is a contract, not a jail. It is the operator’s own code, in its own short-lived process, with the obvious globals removed to catch accidents — a guard against mistakes in code you wrote rather than a security boundary against code you didn’t.

Bursts are bounded rather than absorbed. A trigger holds 200 pending or running deliveries before it reports itself saturated, worktree runs and project-root runs are counted against separate concurrency limits, and a delivery older than three hours expires before it starts rather than mid-run — a stale alert is usually moot. Failures dead-letter and stay replayable by hand for seven days; a replay re-runs the payload, not the verifier.

The computer has to be awake to collect, and a shaped body over 64 KB is refused rather than truncated. A team host with no paired device reports through the Slack thread the work was posted into rather than by push.

Let the 2 a.m. crash arrive already triaged.

One computer the team owns, one recipe, one verifier. The same kind of trigger covers a red build, a ticket moving to Ready for dev, or a dependency advisory that only matters if your code reaches the path.

Next: the same host, answering a question in a thread →