DuckDB

Scheduled DuckDB pipelines, without putting a database in front of your database.

DuckDB runs in-process with no server, which is exactly why it has no scheduler, no retry, and no run history. Dagu adds that operational layer from a single binary, so the stack stays two executables and a data directory.

Install DaguBrowse recipes
One binary orchestrating one binary, with no database to operate
The default local queue enforces DuckDB's single-writer rule
Cursors survive process restarts for incremental loads
Runs next to the data, including inside closed networks
01

Embedded means no scheduler, by design

DuckDB runs inside your process. There is no daemon, no port, and nothing listening to run a query at 3am. That is the point of the design, not an oversight, and it means the schedule has to come from outside.

  • A DuckDB pipeline is usually a CLI invocation, which makes cron the default answer and the only one most teams reach for.
  • cron gives you execution and nothing else: no retry when S3 times out, no history, and no signal when last night silently did nothing.
  • The community cronjob extension schedules inside the process, so the schedule dies with the process and leaves no run history behind.
02

Do not undo the reason you chose DuckDB

The appeal of DuckDB is that there is no cluster, no server, and no warehouse bill. Putting a heavyweight orchestrator in front of it gives all of that back. Airflow wants a scheduler, a metadata database, and a Python DAG framework. Kestra wants a JDBC database, object storage, and four components. Dagu is one binary with file-backed state, and it stays that way here: DuckDB arrives through a versioned action rather than compiled into the orchestrator, which is what keeps the core binary portable and cgo-free.

  • Adding Postgres to schedule a serverless analytics engine is a strange trade to make.
  • Workflow definitions stay in git next to the SQL they run, rather than in a separate platform's metadata.
  • The whole stack fits on one host, which is usually the same host the data already lives on.
  • The duckdb@v1 action carries a pinned DuckDB with it, so a worker does not have to be prepared by hand and every run uses the same version.
03

Single-writer is a scheduling problem

The standard production advice for DuckDB is to guard against overlapping writes with an OS-level lock, because the database takes a single writer. When the guard is cron plus a lockfile, it is one script away from being wrong. Declaring it on the workflow removes the class of bug.

  • The default local queue allows one active run, so a slow run delays the next one rather than corrupting the database.
  • resources.limits.memory caps a run before a large join takes the host down with it.
  • retry_policy covers the failures that are genuinely transient, such as an object store timing out mid-scan.

This constrains concurrent runs of the workflow. It does not make DuckDB multi-writer: another process writing to the same file is still your problem to prevent.

Nightly rollup over Parquet on object storage
# duckdb-nightly-rollup.yaml
schedule: "0 3 * * *"

resources:
  limits:
    memory: "8Gi"

steps:
  - id: rollup
    action: duckdb@v1
    with:
      database: /data/analytics.duckdb
      query: |
        INSTALL httpfs; LOAD httpfs;
        CREATE OR REPLACE TABLE daily_sales AS
          SELECT order_date, region, sum(amount) AS amount
          FROM read_parquet('s3://warehouse/raw/orders/*.parquet')
          GROUP BY 1, 2;
    retry_policy:
      limit: 2
      interval_sec: 300

  - id: export
    action: duckdb@v1
    with:
      database: /data/analytics.duckdb
      query: |
        COPY daily_sales TO '/data/export/daily_sales.parquet' (FORMAT parquet);
    depends: rollup

  - id: count_rows
    action: duckdb@v1
    with:
      database: /data/analytics.duckdb
      readonly: true
      query: SELECT count(*) AS row_count FROM daily_sales;
    depends: export

  - id: verify
    env:
      - COUNT_JSON: ${steps.count_rows.outputs.result}
    run: test "$(printf '%s\n' "$COUNT_JSON" | jq -r '.[0].row_count')" -gt 0
    depends: count_rows

handler_on:
  failure:
    run: /opt/analytics/notify-failure.sh

mail_on:
  failure: true
04

Incremental loads need a cursor that outlives the process

Reloading everything on every run is the thing that turns a fast local query into a slow expensive one. An incremental load needs to remember where the last successful run finished, and that memory has to survive a restart without a database to keep it in.

  • Dagu stores a small JSON cursor across runs, so each run reads only the window it has not loaded yet.
  • The cursor is saved after the load step succeeds, so a failed run leaves it untouched and the next run retries the same window.
  • Bounding the window at both ends keeps a run from racing rows that arrive while it is still going.
Incremental append with a persisted watermark
# duckdb-incremental-load.yaml
schedule: "*/15 * * * *"

steps:
  - id: load_cursor
    action: state.get
    output: CURSOR
    with:
      key: cursors/events-loaded-through
      default:
        loaded_through: "2026-01-01T00:00:00Z"

  - id: window
    run: |
      printf 'since=%s\n' "$(printf '%s\n' "$CURSOR" | jq -r .value.loaded_through)" >> "$DAGU_OUTPUT_FILE"
      printf 'until=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$DAGU_OUTPUT_FILE"
    outputs:
      - name: since
      - name: until
    depends: load_cursor

  - id: append_new_events
    action: duckdb@v1
    with:
      database: /data/analytics.duckdb
      query: |
        INSTALL httpfs; LOAD httpfs;
        INSERT INTO events
          SELECT * FROM read_parquet('s3://warehouse/events/*.parquet')
          WHERE ingested_at >  TIMESTAMP '${steps.window.outputs.since}'
            AND ingested_at <= TIMESTAMP '${steps.window.outputs.until}';
    depends: window
    retry_policy:
      limit: 3
      interval_sec: 60

  - id: save_cursor
    action: state.set
    with:
      key: cursors/events-loaded-through
      value:
        loaded_through: "${steps.window.outputs.until}"
    depends: append_new_events

mail_on:
  failure: true
05

Where this is the weaker choice

Dagu schedules DuckDB. It does not change what DuckDB is, and there are workloads where the pairing is the wrong answer.

  • DuckDB is single-node and single-writer. If several services need to write concurrently, no orchestrator fixes that.
  • It is not built for high-frequency small writes; that remains PostgreSQL's job.
  • If you already operate Airflow or a warehouse with its own scheduler, adding a second orchestrator for one pipeline is rarely worth it.

FAQ

Practical questions before adopting

Is there a DuckDB executor or plugin?

There is no built-in SQL executor for it, and that is deliberate. DuckDB ships as the official duckdb@v1 action, which carries its own pinned copy of the DuckDB CLI. Compiling DuckDB into the orchestrator would cost the core binary its portability and leave a layer to keep in sync with DuckDB's releases, while giving you less than the CLI already offers.

How do I stop two runs from corrupting the database file?

No extra setting is needed for one workflow: Dagu's default local queue allows one active run. A run that is still going blocks the next scheduled run instead of opening a second writer. This only governs Dagu runs of that workflow, so keep other processes off the same file.

What happens when a large query runs out of memory?

Set resources.limits.memory on the workflow so the run is capped before it takes the host with it, and give the step a retry policy for the failures that are transient rather than structural. A query that needs more memory than the host has is a query to rewrite, not to retry.

DuckDB has a community cronjob extension. Why not use that?

It schedules inside the DuckDB process, so the schedule exists only while that process does. There is no run history, no retry, no alert when a job fails, and nothing to look at the next morning. It suits a long-lived embedded application rather than batch work on a server.

Can DuckDB read from S3 in a scheduled run?

Yes, through the httpfs extension, which is how most scheduled DuckDB work reads Parquet without a separate extract step. Because the object store is a network dependency, that is exactly the step worth giving a retry policy.

Next step

Start with one workflow.

Install Dagu, move one script that runs on cron today into YAML, and decide from a real run history.

Install DaguBrowse recipes