Distributed Execution
One workflow graph. Workers on every platform.
Dagu's distributed mode is the same single binary in two roles: a coordinator that dispatches work and workers that poll for it. Start dagu worker on Linux, macOS, or Windows, describe the machine with labels, and route whole DAGs or single steps with worker_selector. Workers dial out over gRPC secured with mutual TLS, so there is no broker, no shared database, and no inbound port on the worker.
# nightly-close.yaml
schedule: "0 2 * * *"
steps:
# Windows box exports from the line-of-business system
- id: export_sales
action: dag.run
with: { dag: export-sales }
worker_selector:
os: windows
# Linux worker near the data transforms it
- id: transform
action: dag.run
with: { dag: transform-sales }
worker_selector:
os: linux
region: us-east-1
depends: [export_sales]
# GPU worker scores the result
- id: score
action: dag.run
with: { dag: score-models }
worker_selector:
gpu: "true"
depends: [transform]
Execution order
One binary is the server, the coordinator, and every worker
Workers run on Linux, macOS, and Windows, on amd64 and arm64
worker_selector routes a whole DAG or a single step by labels
Workers dial out over mTLS-secured gRPC, so NAT and private networks are fine
At a glance
Distributed Dagu vs a broker-based worker stack
Coordinator and workers from one binary; no broker or result backend.
Scheduler, broker, result store, and workers deployed and upgraded separately.
Native workers on Linux, macOS, and Windows.
Workers are commonly Linux-only; Windows goes through WSL or containers.
Declarative worker_selector labels per DAG or step.
Named queues wired into both worker config and task code.
Workers dial out on one mTLS-secured port.
Workers need reachable broker endpoints and shared credentials.
In depth
Where each tool fits
Scale out without adopting a platform
Distributed execution is a deployment choice, not a rewrite. The YAML that runs on one machine runs on a fleet, the queue still gates concurrency before dispatch, and a DAG that must stay on the main instance pins itself there with worker_selector: local.
- A worker is one command: dagu worker --worker.coordinators=<host>:50055 --worker.labels gpu=true
- DAG definitions travel to workers over gRPC at dispatch time, so workers keep no copy of the repository
- default_execution_mode: distributed sends every run to the fleet; without it, only DAGs with a worker_selector are dispatched
Labels route work, machines stay interchangeable
A worker advertises what it is with key-value labels. A DAG declares what it needs with worker_selector. The coordinator matches the two, so adding capacity means starting another worker with the same labels, not editing workflows.
- An empty selector matches any worker; a selector with labels must match every key exactly
- Workers with extra labels still match, so one machine can serve several pools at once
- worker_selector on a dag.run step sends that sub-DAG to a different worker than its parent
A fleet that mixes operating systems and architectures
Workers are the same Go binary, released for Linux, macOS, and Windows on amd64 and arm64. A Windows worker runs PowerShell steps next to a Linux worker running bash in the same graph, with one place to see status, logs, and history.
- Label by convention, for example os=windows or arch=arm64, and route with the same selectors
- Shared-nothing mode streams logs and status to the coordinator over gRPC, so no NFS or shared volume is needed
- Workers only dial out, so machines behind NAT, on a VPN, or in another cloud can join the fleet
Cloud and Kubernetes ready
The official Helm chart deploys the UI, scheduler, coordinator, and optional worker pools on Kubernetes. Workers can also join from far outside the cluster: VMs, bare metal, or an office Windows host, with mutual TLS authenticating both ends when traffic crosses a boundary.
- helm repo add dagu https://dagucloud.github.io/dagu, then helm install with your values
- The coordinator needs one reachable host:port; a Kubernetes Service or internal load balancer is enough
- The coordinator verifies worker certificates and workers verify the coordinator over mutual TLS
FAQ
Practical questions before adopting Dagu
Do I need a message broker or an external database?
No. The coordinator dispatches tasks over gRPC and workers poll it, sending heartbeats, status, and logs back on the same connection. In shared-nothing mode there is no shared storage at all; in shared-filesystem mode workers write to the same volume the server reads.
Can workers sit behind NAT or in a private network?
Yes. The only required path is worker to coordinator on one TCP port, and the coordinator never opens a connection back to the worker. Machines behind NAT, on VPNs, or in another cloud can join by dialing the coordinator address.
How do Windows workers fit in?
Install the same binary, run dagu worker with labels such as os=windows, and give Windows DAGs a matching worker_selector. Steps on that machine run under the shell you configure, for example shell: powershell -NoProfile, while the rest of the graph runs elsewhere.
Can I run all of Dagu on Kubernetes?
Yes. The official Helm chart renders deployments for the UI, scheduler, coordinator, and any worker pools you define, with a ClusterIP Service in front of the coordinator. Workers outside the cluster point at that Service through whatever ingress or load balancer you expose it with.
What happens when a worker goes offline mid-run?
Workers send heartbeats every second. When a worker's heartbeat has been stale for over 30 seconds, the coordinator marks that worker's running tasks as failed, so failure handlers and notifications fire instead of a run hanging forever.
Keep reading
Next step
Start with one workflow.
Install Dagu, move one fragile script or agent task into YAML, and decide from a real run history.