File transfer and EDI

The file did not arrive, and nobody noticed until Monday.

File transfer is the oldest job in the building and usually the least supervised: a shell script on cron, no retry, no alert, no record of what moved. Dagu keeps the transfer commands and adds the arrival waits, verification, retries, and history around them.

Install Dagu Browse recipes

Waiting for a file is a step, not a polling loop you maintain

Retries and failure alerts on every transfer

Runs inside the network, so files never take a detour through a cloud service

Free to self-host, with no per-connection licence

The integration nobody modernises

Order files, invoices, payroll extracts, bank data, and inventory feeds still move as files on a schedule. The transfer itself is a solved problem. What is usually missing is everything around it.

The script knows how to send the file. It does not know what to do when the far side is down.
Failure is discovered by a person downstream, often days later, because a missing file looks the same as a quiet day.
Nobody can answer which file moved, when, and whether it was complete, because nothing recorded it.

Waiting for a file is a step

Most schedulers make you write a polling loop with a sleep and a counter, and then maintain it. Dagu has a wait action, so an arrival window is declared rather than coded, and a file that never shows up fails the run instead of hanging it.

wait.file polls for a path to appear, or to disappear, at an interval you choose.
A timeout turns a missed delivery into a failed run with an alert, rather than a job stuck until somebody looks.
Verification runs as its own step, so a truncated or corrupt file stops the import instead of feeding it.

Inbound: wait, verify, import, archive

# edi-inbound.yaml
schedule: "0 * * * *"

s3:
  bucket: corp-edi-archive
  region: ap-northeast-1

steps:
  - id: wait_for_delivery
    action: wait.file
    with:
      path: /var/spool/edi/orders.csv
      poll_interval: 30s
    timeout_sec: 1800

  - id: verify
    run: |
      cd /var/spool/edi
      sha256sum -c orders.csv.sha256
    depends: wait_for_delivery

  - id: import
    run: /opt/core/import-orders.sh /var/spool/edi/orders.csv
    depends: verify
    retry_policy:
      limit: 2
      interval_sec: 300

  - id: archive
    action: s3.upload
    with:
      key: edi/inbound/orders.csv
      source: /var/spool/edi/orders.csv
    depends: import

  - id: mark_done
    action: file.move
    with:
      source: /var/spool/edi/orders.csv
      destination: /var/spool/edi/done/orders.csv
      create_dirs: true
    depends: archive

handler_on:
  failure:
    run: /opt/edi/notify-failure.sh

mail_on:
  failure: true

Silence is the failure mode

A transfer that fails loudly is a small problem. A transfer that fails quietly becomes a reconciliation project. The operational layer is the part worth adding.

retry_policy covers the far side being briefly unreachable, which is most transfer failures.
mail_on.failure and handler_on.failure make a missed delivery page someone the same hour.
Dagu's default local queue stops a slow transfer overlapping the next scheduled one and sending the same file twice.

Both ends stay inside your boundary

Files that cannot leave the network are the normal case in finance, healthcare, and the public sector. A single binary running on your own host reaches the partner over SFTP and the core system locally, with no third-party service in the path.

sftp.upload and sftp.download move files with key-based authentication and an optional bastion host.
The same workflow can reach an on-premises system over SSH and an object store for the archive copy.
Nothing is relayed through a vendor's cloud, so data residency and audit questions stay simple.

Outbound: extract, checksum, send

# edi-outbound.yaml
schedule: "30 18 * * 1-5"

ssh:
  user: edi
  host: sftp.partner.example.com
  key: ~/.ssh/edi_key

steps:
  - id: extract
    run: /opt/core/export-invoices.sh ./outgoing/invoices.csv
    retry_policy:
      limit: 2
      interval_sec: 120

  - id: checksum
    run: |
      cd ./outgoing
      sha256sum invoices.csv > invoices.csv.sha256
    depends: extract

  - id: send_file
    action: sftp.upload
    with:
      source: ./outgoing/invoices.csv
      destination: /inbound/invoices.csv
    depends: checksum
    retry_policy:
      limit: 3
      interval_sec: 60

  - id: send_checksum
    action: sftp.upload
    with:
      source: ./outgoing/invoices.csv.sha256
      destination: /inbound/invoices.csv.sha256
    depends: send_file

handler_on:
  failure:
    run: /opt/edi/notify-failure.sh

mail_on:
  failure: true

Keep the evidence, not just the file

The question after an incident is rarely what the file contained. It is when it arrived, whether it was complete, and who reran it.

Every run keeps per-step logs, status, timing, and retry counts.
Archiving to object storage as a workflow step makes retention a schedule rather than a habit.
Reruns go through the same recorded path, so a manual recovery is as visible as the automated run.

Priced per server, not per connection

Managed file transfer products are usually licensed per connection, per partner, or per transferring server, which is why the bill grows every time the business adds a trading partner.

Community self-host is free, with unlimited servers and workers.
The licensed tier is priced per Dagu server and adds SSO, role separation, and audit logging.
Adding a partner means adding a workflow file, which changes no licence count.

When a managed file transfer product is the better answer

Dagu schedules and supervises transfers. It is not a full MFT platform, and some requirements genuinely call for one.

Protocol breadth: if you need AS2, OFTP2, or a certified EDI VAN connection, use a product built for it.
Partner self-service portals, per-partner credential rotation, and non-repudiation receipts are MFT features, not scheduler features.
Compliance regimes that require a certified transfer product will not accept a general-purpose orchestrator, whatever it does technically.

FAQ

Practical questions before adopting

Does Dagu replace a managed file transfer product?

For scheduled SFTP and file-based integration, usually yes: it runs the transfer, waits for arrivals, verifies, retries, alerts, and keeps history. For AS2, OFTP2, certified EDI VAN connectivity, partner self-service portals, or non-repudiation receipts, no. Those are MFT platform features and a scheduler should not pretend otherwise.

How do I trigger on a file arriving rather than on a schedule?

Run the workflow on a short schedule and open it with a wait.file step, which polls for the path and proceeds as soon as it appears. Give the step a timeout so a delivery that never comes fails and alerts instead of waiting forever. A run can also be started externally through a webhook when the sending side can call one.

What protocols are supported?

SFTP over SSH is built in as sftp.upload and sftp.download, and object storage transfers are built in for S3-compatible endpoints. Anything else runs as an ordinary command step, so existing tools for FTPS, rsync, or a vendor client keep working with scheduling and supervision wrapped around them.

How are credentials handled?

Key-based SSH authentication is the default, with a bastion host supported when the partner is behind one. Secrets are declared at workflow level and resolved from a provider such as environment variables, a file, Vault, or a cloud secret manager, and their values are redacted in run logs.

Can it run where there is no internet access?

Yes. Dagu is a self-contained binary with no required external database or broker, so it runs on-premises and in closed networks. Distributed execution uses a coordinator and workers over gRPC inside your own network.

Next step

Start with one workflow.

Install Dagu, move one script that runs on cron today into YAML, and decide from a real run history.

Install Dagu Browse recipes