dbt Core scheduling

dbt Core has no scheduler. This is the smallest thing that is a real one.

dbt Core ships a CLI and stops there, so scheduling is yours to solve. The usual choices are cron, which gives you no retries, no history, and no alert when the nightly build dies, or a full orchestration platform with a metadata database to operate. Dagu is one binary that stores state in files.

Install Dagu Browse recipes

One binary, no metadata database to run alongside dbt

dbt stays the CLI you already know, invoked as a command

run_results.json captured on failure, so dbt retry works

Free to self-host, with no per-seat or per-job pricing

The gap is real, and it is the reason dbt Cloud costs money

dbt Core does not include a scheduler. Every team running it in production has picked something to invoke it, and the choice is usually made under time pressure rather than on merit.

cron runs dbt and tells you nothing: no retry, no history, no notification when last night's build failed.
Airflow, Dagster, and Prefect all solve it, and all bring a metadata database, a scheduler process, and an upgrade path to own.
dbt Cloud solves it by charging for the part that Core left out.

dbt is a CLI, so keep the CLI as the boundary

Dagu does not wrap dbt in an operator or parse its DAG. A step runs dbt exactly as you would at a terminal, which means the version, the flags, and the profiles stay yours and nothing needs updating when dbt changes.

Freshness gates the build: if sources are stale the build never starts, instead of quietly modelling yesterday's data.
There is deliberately no blind retry on the build. Re-running dbt build after a partial failure re-runs the models that already succeeded, which costs warehouse credits to redo work that was fine.
handler_on.exit runs whether the build passed or failed, which is what makes the artifacts usable.

Nightly dbt build with a freshness gate

# dbt-nightly.yaml
schedule: "0 3 * * *"
max_active_runs: 1
working_dir: /opt/analytics/dbt

artifacts:
  enabled: true

env:
  - DBT_PROFILES_DIR: /opt/analytics/dbt/.dbt

steps:
  - id: deps
    run: dbt deps

  - id: freshness
    run: dbt source freshness --target prod
    depends: deps

  # No blind retry here: re-running a partial build re-runs models that
  # already succeeded. Resuming is dbt retry's job, in dbt-retry.yaml.
  - id: build
    run: dbt build --target prod
    depends: freshness

# run_results.json matters most when the build failed, so keep it either way
handler_on:
  exit:
    run: |
      cp -f target/manifest.json target/run_results.json \
        "${context.paths.artifacts_dir}/" || true
  failure:
    run: ./scripts/notify-dbt-failure.sh

mail_on:
  failure: true

Keep run_results.json, because that is what dbt retry reads

dbt retry resumes the last invocation from its point of failure by reading run_results.json. That only helps if the file survived the failure, which is exactly the moment a naive pipeline throws it away.

The exit handler copies manifest.json and run_results.json into the run's artifact directory on every run, failed or not.
Artifacts are attached to the run in history, so the state that produced a failure stays with the record of that failure.
A resume is then a small workflow of its own rather than an engineer reconstructing flags from a log.

dbt retry does nothing when the previous invocation failed before executing any node, for example on a warehouse connection or permission error. In that case the run needs to start over rather than resume.

Put an approval in front of a production rerun

Resuming a build against a production warehouse spends real credits, so it is not a button everyone should have. A human task pauses the workflow until somebody decides. Because dbt retry reuses the previous invocation's configuration, the only question worth asking is whether to resume at all.

The workflow waits without holding a process, so the decision can sit until somebody is actually awake.
Completion happens from the web UI, the REST API, or the CLI, and is recorded against the person who did it.
Nothing about the previous run has to be reconstructed by hand, which is the step where people usually get the target or the selector wrong.

Approval-gated resume from the last failure

# dbt-retry.yaml
working_dir: /opt/analytics/dbt

env:
  - DBT_PROFILES_DIR: /opt/analytics/dbt/.dbt

steps:
  # dbt retry reuses the previous invocation's configuration, so the only
  # decision left is whether to spend the credits resuming it.
  - id: confirm
    action: human.task
    with:
      prompt: Resume last night's failed dbt build from its point of failure?

  - id: retry
    run: dbt retry
    depends: confirm

mail_on:
  failure: true

Where the warehouse is not the only system involved

Most dbt schedules are one step in a longer chain. The extract that feeds dbt, the reverse ETL that follows it, and the on-premises system at one end are usually not dbt's problem, and often not the warehouse's either.

Run extraction, dbt, and downstream publication as ordered steps in one workflow instead of three unrelated cron entries hoping about timing.
Reach on-premises databases and file systems over SSH from the same workflow.
Split models across schedules with dbt's own selectors, so hourly models do not wait on the nightly full build.

When a different tool is the better answer

Dagu is a scheduler, not a data platform, and that boundary is deliberate.

If you want asset-level lineage, backfills driven by a data catalogue, and a UI built around tables rather than tasks, Dagster is built for that and Dagu is not.
If your team already runs Airflow well, adding a second orchestrator to save some memory is not a good trade.
The scheduler is a single instance with no built-in leader election, so if the nightly build is on a critical path, the availability design is yours to build.

FAQ

Practical questions before adopting

Does Dagu parse the dbt DAG into separate tasks?

No, and that is intentional. dbt already resolves its own model graph, so re-deriving it in the orchestrator means keeping a parser in sync with dbt's releases and getting a second, subtly different graph. A step runs dbt build and dbt owns the ordering inside it. Where you want coarser splits, use dbt's own selectors across separate steps or workflows.

How is this different from just using cron?

cron starts a process. What it does not give you is a retry on a transient warehouse error, a notification when the build fails, a history you can look at three weeks later, a guarantee that a slow run does not overlap the next one, or a gate that stops the build when sources are stale. Those are the things people build badly around cron, and they are fields on a workflow here.

Do we need a database for this?

No. Dagu is a single binary that keeps run state in files, so there is no metadata database to provision, back up, or upgrade next to your warehouse. That is the main practical difference from Airflow, Dagster, and Prefect for a team whose whole orchestration need is scheduling dbt properly.

Can it run dbt inside a container?

Yes. A step can specify a container image, so the dbt version and its adapter are pinned to the image rather than to whatever is installed on the host. The workflow definition stays the same either way.

What about dbt docs?

dbt docs generate is another command, so it runs as a step like the rest. The generated site is a directory of static files, which you publish wherever you already serve internal documentation, or attach to the run as artifacts if you only need the history.

Next step

Start with one workflow.

Install Dagu, move one script that runs on cron today into YAML, and decide from a real run history.

Install Dagu Browse recipes