Upgrade

How to move a running Ductus to a new version: what the version numbers promise, how to find out what the new version will do before you install it, the steps for each way of installing, what happens to the database, and how to go back. For deploying a change to your own project (new interface code, a new interface), the same tools apply: see Deploy a change.

The short version, for a server installed from the Linux package:

1. Unpack the new version's portable archive anywhere on the server.
2. ./ductus upgrade --check --replay 1d --env-file /etc/ductus/ductus.env --root /var/lib/ductus/project: it changes nothing and exits 1 when something would go wrong.
3. Install the new package. It hands the running engine over to the new version: receiving never stops.
4. Look: systemctl status ductus, then the console's Overview ("took over from …").

Versions and what's supported

What: a version is MAJOR.MINOR.PATCH (semver): 0.6.0, 0.6.1, 1.2.0. ductus --version prints it; the console's Overview, GET /api/v1/engine and ductus upgrade --check show the version of every engine on the database.

What each kind of version promises:

Why: a hospital upgrades rarely and needs to know an upgrade won't strand a project or a database. Deprecating in minors and removing only at majors means a project that validates without warnings on 4.9 starts on 5.0. A database that only grows within a major is what makes going back a matter of installing the older program.

What's supported today: Ductus is pre-release. There are no released versions yet, only development builds of the latest commit, and only that is supported (SECURITY.md). Long-term-support versions, for sites that stay on one version for a year or more, are planned once clustering (v1.2) is stable; until then, follow the latest version.

Several engines on one database may run different versions while you upgrade them one at a time: each runs its own interfaces, and the older ones keep working on what the newer one added. Don't leave them on different versions for longer than the upgrade: a fix you upgraded for isn't on the ones you haven't.

Before every upgrade: check, then replay

ductus upgrade --check

What: the new version's program reads the database and the project and says what its first start would do. It changes nothing: not the database, not the project folder.

Why: otherwise the first place you learn that a database change works through a 50 GB table, or that your tests fail under the new SDK, is the restart itself, in the change window.

When: before every upgrade, from the new version's portable archive (no install needed), on the server or any machine that reaches the database, with the service's settings. A read-only login is enough (GRANT pg_read_all_data TO ductus_check). Run it again on the day: the database and the project may have changed since.

Default: it checks everything below; --replay 1d adds a replay of a day's traffic; --json gives one document (for a change ticket). Exit 0: nothing in the way. Exit 1: something would refuse the start or fail the project.

From a lab engine (DESKTOP-GCT6E6V is the machine it ran on):

$ ./ductus upgrade --check
ductus upgrade --check: what version 0.6.0-dev would do here (nothing is changed)

The database
✓ migrations: nothing to apply

Engines on this database
✓ 18e39e5085ef2289-3ee on DESKTOP-GCT6E6V (pid 1006): version 0.6.0-dev, environment dev, since 2026-10-31 12:52 UTC, runs 1 interface(s)

The project (., environment dev)
✓ validate: valid
✓ test: 1 passed, 0 failed

Between versions
✓ required stops: none (versions before 1.0 upgrade to any later one)
✓ deprecated settings: none used

ready to upgrade to 0.6.0-dev (checked in 0.6 s)

What each section means, and what to do when it isn't ✓:

A failing check, here a test the new version fails, ends with the count and exits 1:

$ ./ductus upgrade --check --replay 1h
…
The project (., environment dev)
✓ validate: valid
✗ test: 0 passed, 1 failed
    adt-in
      ✗ sets the receiving application and keeps the patient
          expected "RECEIVER", got "LAB"
…
The last 1h replayed through 0.6.0-dev (nothing is sent)
✓ replay against the record and kept outputs: 300 message(s), 600 comparison(s), 0 identical, 600 same decision (outputs not compared), 0 different
    note: no installed version found (/usr/bin/ductus): outputs compared with kept outputs only; name a binary with --against
⚠ replay: outputs not compared: 600 comparison(s) matched by decision only (both sent), not by what was sent, because no version ran beside this one and no destination keeps its outputs; to compare outputs, run it on the server, where the installed version runs beside it, or name the installed binary with --against PATH

1 problem(s): fix them before upgrading (checked in 0.7 s)

The replay there passed although the code now sets MSH-5 to "LAB" in every message: this lab has no installed version and keeps no outputs, so all it could compare was that each message was still sent. That's what the ⚠ is for. "Identical" is only ever claimed for an output that was compared.

On a database no version has run on yet, the first section says so instead: "a new database: the first start creates every table (25 migrations)".

--json gives the same as one document: ok and problems, every check with its section, level and text, and the details (migrations.pending with each one's tables, engines, the project's validate and test output):

$ ./ductus upgrade --check --json
{
    "version": "0.6.0-dev",
    "ok": true,
    "problems": 0,
    "seconds": 0.61,
    "checks": [
        {
            "section": "database",
            "level": "ok",
            "text": "migrations: nothing to apply",
            "more": []
        },
…
    "migrations": {
        "new_database": false,
        "pending": [],
        "newer": [],
        "edited": [],
        "failed": []
    },
…

ductus replay: what it does to real traffic

What: ductus replay --since 24h runs a period's stored messages through the new version's engine and your project's code, sends nothing and writes nothing, and compares what each destination's code decides (sent and exactly what, filtered and why, quarantined and why) with the installed version run beside it on the same messages, with what was recorded, and with kept outputs. ductus upgrade --check --replay 1d runs it as part of the check.

Why: "what will change?" is the question a change board asks, and the usual evidence is weeks of testing by hand on a test system. A replay of production's own traffic answers it in minutes, message by message, field by field.

When not: it isn't a load test (one thread, one message at a time: measured), and it can't compare code that writes (db().execute, a POST): those messages are reported as not comparable (why).

What it compares with, by default: the version installed on the server (/usr/bin/ductus, C:\Program Files\Ductus\ductus.exe), run beside it; that needs the installed one to be 0.6.6.4 or later (why). Without an installed version, outputs are compared only where they were kept (save_output: true). Every other destination is compared by its decision alone, and the report counts what it could compare separately:

When no output was compared at all, the report ends with ⚠ outputs not compared and the fix, in the terminal, the --json document (outputs_not_compared) and the HTML page alike; upgrade --check --replay shows it as a warning (exit 0: nothing differed, but little was checked). On a machine without the installed version, name it: --against /path/to/old/ductus.

A difference the installed version shares with the new one isn't the upgrade's: the data it looks up changed since the message was sent. The report calls it drift and doesn't fail for it (nuance).

For a change board, --report replay.html writes a page with the versions, the period, the counts and a SHA-256 checksum. Examples in it quote field values from real messages: keep it as you keep patient data, or make it with --hide-values (nuance). The same tool compares a code change with --baseline <git revision>:

$ ductus replay --since 1h --baseline HEAD --report replay.html
adt-in: 300 message(s) replayed in 0.0 s
ductus replay: version 0.6.0-dev against the project at HEAD, 2026-10-31 15:25 to 2026-10-31 16:25 UTC, environment dev (nothing was sent)

adt-in: 300 message(s) in 0.0 s
  ✗ archive: 300 compared, 300 different
      vs the project at HEAD: MSH-5 differs in 300 message(s) (e.g. message 406: "RECEIVER" → "LAB")
  ✗ lab: 300 compared, 300 different
      vs the project at HEAD: MSH-5 differs in 300 message(s) (e.g. message 406: "RECEIVER" → "LAB")

300 message(s), 600 comparison(s): 0 identical, 600 different, 0 drift, 0 not comparable, 0 waiting (0.1 s)
report: …/replay.html (sha256:794fea6a30fe050d12f3253ba1240f39f40f5219f780783cb192b06c3d20489f)

Here the earlier project ran beside this one, so every output was compared: the change shows field by field, on the same messages that passed by decision above.

More on replay, with what the clock and Math.random do in it: Replay a period before a change.

Upgrading, by how you installed

Linux packages: the handover

sudo apt install ./ductus_0.6.1_amd64.deb          # Ubuntu, Debian
sudo dnf install ./ductus-0.6.1-1.x86_64.rpm       # RHEL, Rocky, Alma

What happens: the package replaces the program and runs systemctl reload ductus. The new version starts beside the running engine, on the same listening sockets: it reads the project, applies the database changes and loads every interface's code while the old one carries on. Once it's ready, the old engine stops cleanly, lets go of its interfaces, and the new one takes them and sends. Receiving never stops and sending pauses for a moment (measured under real systemd, 4 senders at 200 messages/s: 0 refused, longest wait for an ACK 0.17–0.22 s, nothing lost: nuance).

The handover's own lines, from an engine handing over to the program now on disk (ductus handover is what systemctl reload runs, with the service's control socket; these are the lines journalctl -u ductus shows):

$ ductus handover --control …/control
started …/ductus (pid 1312): it reads the project, migrates the database and loads every interface's code while this one runs
ductus 0.6.0-dev (pid 1312) is ready after 104 ms and receives beside this one: stopping this one
this engine stopped in 1 ms and let go of its interfaces: the new one takes them
handed over to ductus 0.6.0-dev (pid 1312): receiving never stopped, sending paused 41 ms

• A new version that can't start (a broken project, a failed database change, another engine owning an interface) leaves the old one serving: the install says so, journalctl -u ductus says why, and senders notice nothing. Fix what it names and sudo systemctl reload ductus (nuance).
• A start longer than 5 minutes (a large database change) is given up by the handover: sudo ductus handover --timeout 30m instead of the reload.
• For a few seconds two engines run, so the server needs memory for both.
• From a development build before 0.6.6.3 (no released version is), the package restarts instead of handing over; from one before 0.6.6.2 that restart refuses connections for a moment (nuance).
• A drain refuses a handover: resume first, or stop and start.

Windows installer

msiexec /i ductus-0.6.1-x64.msi /qn          # as an administrator; no need to uninstall first

What happens: the installer stops the service cleanly, replaces the program and starts it again; C:\ProgramData\Ductus (settings, project, logs) is never touched. The engine is down for a few seconds (an MSI upgrade measured 3.1–3.5 s: a few connections refused or reset, nothing lost), so choose a quiet moment (nuance). A handover like Linux's is planned for v1.0. Details (a service stopped on purpose, a password account, Restart Manager): Windows installer.

The portable archive

Replace the program and hand over, as the package would:

sudo install -m 0755 ductus-0.6.1/ductus /usr/bin/ductus
sudo systemctl reload ductus

Without the package's service (ductus run under your own supervisor), stop and start it with the new program; the senders see a short gap. To hand over there too, start the engine with DUCTUS_CONTROL=/some/dir/control and run ductus handover --control /some/dir/control after replacing the program (nuance).

Docker

Build the new version's image (docker build -t ductus:dev . in the new version's source, as in Docker), then docker compose -f packaging/docker/compose.yaml up -d: Compose sees the new image and replaces the container. Its ports close while the new one starts, so senders are refused for a moment. Docker is for trying and developing until v1.0.

ductus dev on a laptop

Replace the program and start ductus dev again. Its Postgres (in .ductus/postgres) is kept and updated by the new version like any other database; ductus dev --reset starts from an empty one.

Database changes (migrations)

What: a new version may change the database: a new table, a new column, a new index. Each change is a numbered migration, applied once, in order, by the first start of the version that carries it, and recorded in the database (_sqlx_migrations). ductus upgrade --check lists the ones still to apply, each with a sentence saying what it does, for example 0025: "keeps the names accounts had before, and the directory's id for each directory account".

They only add. Within a major version, a migration never renames or removes anything the version before uses: the old program keeps working on the new tables. That's what makes going back safe, and what lets several engines on one database be upgraded one at a time. A store test, on every push, starts the engine on a database that has a migration it doesn't know, as an older program meets a newer one's database.

Heavy ones. Most take milliseconds. One that works through an existing table row by row (an index built, a column filled) is marked ⚠ with the table and its size now: while it runs, the start waits, and storing or sending waits as it says (nuance). Under the Linux package's service senders wait in the queue rather than being refused, for as long as it takes. Plan the window by the size shown, and time it on a restored backup first. There have been none since v0.5 (0011, the queue index, was the last).

When one fails, the start refuses with the database's reason, and so does every later start of that version until the cause is fixed. Each migration runs in one transaction, so a failure leaves nothing half done: the database is as it was before it, and the version before still runs on it (going back keeps you running meanwhile). A migration recorded as failed (upgrade --check shows it ✗ with its number) is one that couldn't run in a transaction; none so far does. Then every version refuses to start until it's repaired: don't edit _sqlx_migrations by hand, ask for help.

Backups are the DBA's. The engine needs no backup of its own before an upgrade, but your usual point-in-time recovery (pgBackRest, WAL archiving) is what undoes anything at all (PostgreSQL).

Going back (rollback)

What: installing the previous version again. The database stays as the new version left it; the previous version runs on it, ignoring what it doesn't know.

When: the new version misbehaves in a way upgrade --check and replay didn't show. Decide by what you see: a project problem is fixed in the project and deployed (systemctl reload); a problem in Ductus itself is a rollback, and an issue.

What going back doesn't undo: whatever the new version wrote stays in the database: tables and columns the older one doesn't read. Messages received meanwhile are kept and sent as usual; features only the new version had stop until you upgrade again. Going back across a major version isn't possible (from 1.0 on: the major may have removed what the older one needs).

What going back does to a database the new version migrated: nothing. The older program doesn't undo the new version's migrations, can't, and needn't: they only added. At its start it sees migrations it doesn't know in _sqlx_migrations, leaves them as they are, and runs on the tables it knows; what the new version added (a table, a column) stays, unused, until you upgrade again, when the new version finds its migrations already applied and applies nothing twice. ductus upgrade --check lists them as "applied by a newer version than this one" (✓, not a problem). Two things to know:

• Data written only into what the new version added isn't read by the older one while it runs. Messages themselves are in tables every version reads, so none is lost or left waiting.
• A migration that failed left nothing behind: each runs in one transaction, so a failure undoes all of it. The new version refuses to start (each time, with the reason) and the older one runs on the database as it was (Database changes).

Both are shown on a server, under traffic and tested on every push: a store test starts the engine on a database with a migration it doesn't know, and the v0.6 demo goes back with a package while a sender sends.

On a server: the v0.6 demo

What: an upgrade and a downgrade of the Linux package, each while a sender sends, on the server Your first interface set up (Ubuntu 24.04, MLLP over TLS with client certificates, Postgres on another machine). The commands and outputs are from scripts/demo-v06.sh, which runs them under real systemd on every push.

Why: "receiving never stops" is a claim; this is what a sender saw while it happened, and what the database looks like after going back.

The demo's two packages (0.6.0~dev.1 and ~dev.2) hold the same program, so they show the mechanics of an upgrade, not a version's changes; a release's files are named 0.6.0, 0.6.1.

Check first, from the new version's archive

The new version's portable archive, unpacked anywhere, checks the running server with the service's settings, and replays the last day's traffic through the new version beside the installed one:

$ cd /tmp
$ tar -xzf ductus-0.6.0-dev.2-linux-x64.tar.gz
$ sudo -u ductus ./ductus-0.6.0-dev.2/ductus upgrade --check --replay 1d --env-file /etc/ductus/ductus.env --root /var/lib/ductus/project
ductus upgrade --check: what version 0.6.0-dev would do here (nothing is changed)

The database
✓ migrations: nothing to apply

Engines on this database
✓ 18e3c2043934cad1-14e on ductus1 (pid 334): version 0.6.0-dev, environment prod, since 2026-10-31 23:46 UTC, runs 1 interface(s)

The project (/var/lib/ductus/project, environment prod)
✓ validate: valid
    ⚠ database: plain, no TLS: the server doesn't offer it (sslmode=prefer); whoever answers at its address is believed: add sslmode=verify-full to DUCTUS_DATABASE_URL (sslrootcert=/path/ca.pem for a private CA)
✓ test: 1 passed, 0 failed

Between versions
✓ required stops: none (versions before 1.0 upgrade to any later one)
✓ deprecated settings: none used

The last 1d replayed through 0.6.0-dev (nothing is sent)
✓ replay against version 0.6.0-dev (installed): 100 message(s), 100 comparison(s), 100 identical, 0 different

ready to upgrade to 0.6.0-dev (checked in 0.9 s)

On the server, the replay found the installed version (/usr/bin/ductus) and ran it beside the new one, so every output was compared whole: "100 identical", where the lab example above could only compare decisions. (The database warning is the demo's database server offering no TLS; a real one should.)

The upgrade, under traffic

A sender keeps 4 connections open and sends 1,000 messages at 200 a second, over TLS with its client certificate; 1.5 s in, the package is installed:

$ sudo apt install ./ductus_0.6.0~dev.2_amd64.deb
…
Unpacking ductus (0.6.0~dev.2) over (0.6.0~dev.1) ...
Setting up ductus (0.6.0~dev.2) ...
$ journalctl -u ductus -o cat --no-pager | grep "^handed over"
handed over to ductus 0.6.0-dev (pid 637): receiving never stopped, sending paused 180 ms

What the sender saw (ductus gen's last line), and what the receiver got:

retries 4, rejected 0, refused 0, dropped 4, longest wait 504 ms
unique 1000 | duplicates 0 | out of order 0 | patients 100

• Refused 0: no connection was turned away at any moment.
• Dropped 4, retries 4: each of the 4 open connections was closed once, by the old engine as it let go, with no message lost on it, and opened again at once on the new one. A sender's MLLP client does this by itself; to its operators it looks like nothing.
• Longest wait 504 ms: the slowest ACK, from first try to ACK, for a message that waited out the close, the reconnect and the TLS handshake.
• Every message once and in order at the receiver; sending paused 180 ms while the new engine took the interfaces.

Over four runs of the demo on a laptop (an upgrade and a downgrade each), the longest wait was 0.34–0.52 s and sending paused 26–180 ms; on CI's runner 143 ms (upgrade) and 157 ms (downgrade), paused 14 and 18 ms. Refused was 0 and the receiver complete, with no duplicate and nothing out of order, every time. Without TLS, in the same kind of container, demo-packages.sh measured a reload at 75–81 ms longest wait and a package upgrade at 0.30–0.45 s.

Going back, under traffic

To see what an older version does on a database a newer one migrated, the demo first records one migration the older program doesn't know (number 9999, with a table of its own), as if the newer version had added it; both packages hold one program, so this stands in for a real one. Then the same sender sends while the older package goes in:

$ sudo apt install -y --allow-downgrades ./ductus_0.6.0~dev.1_amd64.deb
…
The following packages will be DOWNGRADED:
  ductus
…
dpkg: warning: downgrading ductus from 0.6.0~dev.2 to 0.6.0~dev.1
…
Unpacking ductus (0.6.0~dev.1) over (0.6.0~dev.2) ...
Setting up ductus (0.6.0~dev.1) ...
$ journalctl -u ductus -o cat --no-pager | grep "^handed over" | tail -1
handed over to ductus 0.6.0-dev (pid 832): receiving never stopped, sending paused 26 ms
$ cd /var/lib/ductus/project
$ sudo -u ductus ductus upgrade --check --env-file /etc/ductus/ductus.env
ductus upgrade --check: what version 0.6.0-dev would do here (nothing is changed)

The database
✓ migrations: nothing to apply
✓ applied by a newer version than this one: 9999 (fine: migrations only add, so this version runs on them)
…
ready to upgrade to 0.6.0-dev (checked in 0.5 s)

retries 4, rejected 0, refused 0, dropped 4, longest wait 349 ms
unique 1000 | duplicates 0 | out of order 0 | patients 100

A downgrade is a handover like an upgrade: refused 0, every message once and in order. The newer version's migration and its table are still there, and the older engine runs beside them. (upgrade --check with the installed program describes this version's view: nothing to apply, and one migration it doesn't know.)

On Windows: the same, with a gap

demo-windows.ps1 (every push, on CI's Windows runner) upgrades the MSI while a sender sends over TLS with its client certificate: 4 connections, 200 messages a second for a minute, the installer started 3 s in and done 3.4 s later. What the sender and the receiver saw:

sent 12000 messages over 4 connection(s) in 60.0s (200/s), retries 12, rejected 0, refused 8, dropped 4, longest wait 3291 ms
received 12200 | unique 12200 | duplicates 0 | out of order 0 | patients 300

(the receiver's count includes the 200 sent before). While the installer stopped, replaced and started the engine, the sender's connections were closed (dropped 4) and its new ones refused for a moment (refused 8); its slowest message waited 3.3 s for its ACK. Nothing was lost, sent twice or out of order: the sender resent on its own, as MLLP senders do. That's the Windows gap until the handover comes to Windows (v1.0): pick a quiet moment.

Going back on Windows is uninstall then install: the demo upgrades to a build numbered as the next minor version, tries the older MSI over it (refused, exit 1603: "A newer version of Ductus is already installed. Uninstall it first to go back."), uninstalls, installs the older one, and starts it on the same settings, project and database (Windows installer).

After the upgrade: confirm

• systemctl status ductus shows "running N interfaces" (Windows: Get-Service Ductus is Running, and the Event Log has the start, ID 1000).
• The console's Overview says the version, and "took over from v…" after a handover; the audit log has an engine.handover entry (audit actions).
• ductus status --token … shows every interface ok within a minute; ductus doctor checks every connection from the server.
• If you kept the replay's report, file it with the change: it's the record of what the upgrade was expected to change.