How a Dagu server behind a firewall takes webhooks

TL;DR The Dagu scheduler holds one outbound WebSocket to Dagu Cloud. Dagu Cloud stores each request until the server answers for it, and the server uses the delivery ID as the run ID, so a request delivered twice starts one run. Signing secrets stay on the server, which checks signatures against the time Dagu Cloud received the request, so a request that waited still passes. Dagu Cloud stores only a digest of each URL's token.

Public webhook URLs let GitHub, Stripe, or Slack start a DAG on a server that nothing on the internet can reach. On the surface the feature is small: a switch, a URL, and a list of deliveries. This post is about what had to hold underneath, mostly from the first-release section of the triggers RFC.

The constraints

Four came first:

• Nothing connects in. Whatever reaches the server rides a connection the server opened.
• Nothing new to run. No agent, sidecar, or tunnel next to Dagu.
• The server is sometimes away. Restarts, deploys, and laptops that sleep. A request that arrives then must still start its run, and only one.
• Dagu Cloud learns as little as possible. No workflows, no signing secrets, and the request body only for as long as it has to.

The shape

GitHub, Stripe, Slack, ...
    │  POST https://hooks.dagu.sh/<server>/<token>
    ▼
Dagu Cloud relay            stores the request and answers 202,
one mailbox per server      keeps it for up to 7 days
    ▲
    │  WebSocket over HTTPS, opened by the server
    │  hello · hooks/hooked · delivery/delivered · missed · ping
    ▼
Dagu scheduler              verifies, filters, and queues the run,
holding the lock            then answers with an outcome

The sender gets 202 Accepted as soon as Dagu Cloud has stored the request. Whether a run started is the server's business, and it shows up in the server's own delivery list, not in the sender's response. That split is what lets the server be offline: the sender's half of the exchange never waits for it.

One process holds the connection

A Dagu deployment can run several dagu server processes and several schedulers. Only one scheduler at a time holds the scheduler lock, and that one holds the relay connection. It is already the process that queues runs, so a delivery never crosses a process boundary on its way to the queue. If a second connection from the same server ever appears, the relay keeps the newer one and closes the older with the reason replaced.

The connection starts with a ticket. The scheduler asks Dagu Console for one with the same license credentials it uses to check in. The ticket lasts five minutes, which is plenty to dial, and names how many URLs the plan allows: 1 on free monitoring, 1,000 on a trial or paid plan. The relay ends every connection after 12 hours and the scheduler asks for a fresh ticket, so the limit the relay enforces is never more than half a day old. When Dagu Console disconnects the server, the relay closes the socket with the reason dropped, the next ticket request is refused, and the scheduler asks again only every 10 minutes.

Keeping the socket honest takes both directions. The scheduler pings every 15 seconds, and the relay takes a socket silent for 45 seconds as offline. A socket can also stay open on the server's side alone after a laptop sleeps or the network changes, so when a ping has gone unanswered for 30 seconds and nothing at all has arrived for 45, the scheduler closes the socket and dials again. Reconnects back off from 1 second to 1 minute with jitter, and start from 1 second again after any connection that lasted a minute.

The server owns the set of URLs

The URL ends in a token the server makes: 32 random bytes, stored encrypted beside the webhook's HMAC secret so the page can show the URL again. The relay never receives the token. The server sends it the SHA-256 of each token, and the relay hashes the token in each incoming path to find the URL.

The server sends its URLs as one whole set, which replaces whatever set the relay held. A set is easier to get right than a stream of additions and removals: there is no ordering to lose across reconnects, and the first set on a new connection repairs anything the last one missed.

A set has two sharp edges. The first is that a set leaving a URL out drops the requests waiting for it, so a set built from a partial read would destroy data. If a single webhook record cannot be read or decrypted, the scheduler keeps the last set it sent rather than sending a smaller one, and the card says the webhooks could not be read.

The second is the empty set. Turning off the last URL while the scheduler is stopped leaves the relay holding it, and a scheduler with no public URLs has no reason to connect. So the scheduler records in its status that the relay may still hold URLs, and when it next starts, it connects once more to send the empty set. In production, a URL turned off while no scheduler ran was gone two seconds after the scheduler came back.

Each URL is keyed by the webhook's ID, not the DAG's name. Renaming a DAG keeps its webhook, so it keeps the URL and the requests waiting for it. That made one more invariant matter: one webhook per DAG, across processes. Two dagu server processes creating a webhook for the same DAG at the same moment could otherwise leave two webhooks, and two URLs, behind one name. Each write stores the webhook's record first and then claims the DAG name with an atomic create. A claim whose webhook is gone, or names another DAG, can only be left over from a crash between the two steps, and the next writer takes it over.

One request, one run

The relay sends a delivery again, on the next connection, until it hears an outcome for it. That is the only way to survive a server that dies mid-delivery, and it means the same delivery can arrive twice: the scheduler can queue the run and crash before its answer leaves.

So the delivery ID is the run ID. Queueing a run whose ID already exists counts as queued, and the second copy changes nothing. A sender can still send the same event twice, as GitHub does when you press Redeliver. That is a new request to Dagu Cloud with a new delivery ID, and it starts a new run, which is what pressing Redeliver means.

Every delivery ends in one of four answers to the relay:

The server logs finer outcomes than it sends. A wrong signature is Rejected with its reason, and a filtered event or GitHub's ping is Skipped, so the card says why nothing ran.

A retry also holds back that URL's later deliveries until it comes due, so a full queue pauses the URL instead of letting newer requests skip ahead. Waiting at the relay, rather than piling runs onto a full queue, keeps a burst of pushes from turning into hundreds of queued runs. If the burst keeps coming, the relay's limit of 1,000 waiting requests per server pushes back on the sender with 503.

Signatures on the server, time from Dagu Cloud

The obvious place to check a GitHub or Stripe signature is where the request lands. That would mean handing Dagu Cloud every signing secret, so the check lives on the server instead, with the secret encrypted at rest beside the webhook. Dagu Cloud forwards the body byte for byte, and the headers without credentials and proxy headers such as Authorization, Cookie, and X-Forwarded-For.

Timestamps were the catch. Stripe, Slack, and Standard Webhooks sign a timestamp to stop replays, and the usual rule rejects anything more than 5 minutes off. A request that waited three hours for a rebooting server would fail that rule, though nothing was wrong with it. So the relay records when it received each request and sends that time with the delivery, and the server checks the signed timestamp against it, not against its own clock.

That trades one thing for another, and it is worth saying which. The signature still proves the sender wrote that body. The freshness check now trusts Dagu Cloud about when the request arrived. Dagu Cloud already holds the body while it waits, so the check never could stop Dagu Cloud itself from replaying it. What it stops is a stranger replaying a captured request later, and for that, the receipt time Dagu Cloud records is the time of the replay.

Slack adds one more wrinkle. When you save a Request URL, Slack sends a challenge and wants it echoed back at once, which a server that may be offline cannot promise. The server marks Slack URLs in its set, and the relay answers the challenge itself. It starts no run.

The body, as sent

Local webhook requests wrap their payload in {"payload": ...}. Public URLs cannot, since GitHub and Stripe decide what they send. The run gets the body exactly as it arrived, in WEBHOOK_PAYLOAD and in a file named by WEBHOOK_PAYLOAD_FILE.

The file exists because of Linux. A single environment string there is capped at 128 KiB, and a GitHub push can be larger, up to the relay's 512 KiB. A step whose environment carried a larger body failed before it started, with argument list too long, and the body rode twice: once in WEBHOOK_PAYLOAD, and again inside DAG_PARAMS_JSON. Now every webhook run gets the body as a file in a private temporary directory, readable only by the run's user and removed when the run ends. A body too large for one variable is left out of both variables, and the steps read the file. Windows gets the same treatment at its own limit of 32,767 characters.

The relay accepts only UTF-8 bodies. Deliveries travel as JSON text and end up in environment variables, and neither carries arbitrary bytes well. Every sender this release targets sends JSON or form-encoded text, so a binary body gets 415 rather than a lossy conversion.

What the relay keeps and refuses

Nothing is dropped silently. Requests that expired after 7 days, and requests refused while 1,000 waited, are counted per URL and reported to the server when it connects. The card shows them until an admin dismisses the notice. Whether 512 KiB is enough is still an open question in the RFC, since large push payloads can exceed it.

Checking it against production

Each behavior above was checked against production, with a build connected to Dagu Cloud the way any server is, before the release and again on v2.20.1:

• A GitHub push signed with the real secret started a run, and its payload file matched the sent body byte for byte.
• GitHub's ping and an issues.opened event outside the event list were skipped, and a push with a bad signature was rejected.
• A wrong token got 404 from Dagu Cloud without reaching the server.
• A request sent while the server was stopped ran when it came back.
• Renaming the DAG kept the URL.
• New URL stopped the old URL in about 5 seconds, and turning the URL off returned 404 within about 10.

What comes next

This release gives existing per-DAG webhooks a public address. The RFC goes further: triggers declared in the DAG, each with a source, an event type, a condition, and params mapped from the event, all going through one event store that starts at most one run per event and trigger. The RFC also says the trigger model follows only if public URLs get used, so how you use them decides what gets built.

The public webhook URL docs cover every sender, outcome, and status code, and the RFC has the full design. If something here looks wrong, or you need a sender that isn't listed, tell me on Discord.