Nuances: upgrades

What could surprise you here, why it is so, and what to do. Each entry has a level, 1 (day one) to 5 (expert): the index lists them all by level.

Run ductus upgrade --check with the new version before every upgrade (level 1)

• What: the new version's program (the portable archive: ductus-<version>-linux-x64.tar.gz or -windows-x64.zip, no install needed) reads the database and the project and says what its first start would do: the migrations it applies, with the tables they work through and their size now; the engines on the database, their hosts and versions; the project validated against the database and its tests run with the new SDK and type checker; settings it deprecates. Exit 1 means something would refuse the start or fail the project. It changes nothing, in the database or in the project folder.
• Why: otherwise the first place you learn that a migration builds an index on a 50 GB table, or that a test fails under the new SDK, is the restart itself, in the change window.
• What to do: run it from the unpacked archive on the server (or a workstation that reaches the database) before the change window, with the service's settings: ./ductus upgrade --check --env-file /etc/ductus/ductus.env (Windows: C:\ProgramData\Ductus\ductus.env). A read-only login is enough: GRANT pg_read_all_data TO ductus_check. --json gives one document to attach to the change ticket.

A migration marked heavy holds up a table while it runs (level 2)

• What: most migrations add a table or a column and take milliseconds. One that works through a big table row by row (an index built, rows copied) is shown with ⚠, the table and its size: while it runs, the start waits, and storing or sending waits as it says. The version running before keeps working on the new tables afterwards (migrations only add), so rolling back is installing the previous version.
• Why: an index is built in one pass over the table; Postgres holds the table's writes until it's done.
• What to do: plan the window by the size shown; for a large table, time the upgrade on a copy (a restored backup) first. There have been none since v0.5 (0011, the queue index, was the last).

On Linux a restart refuses no connection: senders wait for it (level 2)

• What: under the package's systemd service, systemctl restart ductus and a crash don't refuse senders (an upgrade doesn't either: it hands over, below). systemd keeps the engine's listening sockets open while it restarts and hands them to the new start; a sender connecting meanwhile waits in the kernel's queue until the engine accepts, then gets its ACK. Each open MLLP connection is closed once by the stop (after its last ACK) and the sender reconnects. Measured with 4 senders at 200 messages/s (Ubuntu 24.04 and Rocky Linux 9): a restart 0 refused, longest wait for an ACK 0.2–0.3 s (the same restart without systemd's help: 4 refused, 0.6 s), a package upgrade 0 refused, 0.2–0.35 s; a kill -9 0 refused, 5.4 s (systemd waits RestartSec, 5 s, before starting it again). Nothing lost, nothing twice, in order.
• Why: a refused connection is what senders alarm on, and some wait a minute or more before they try again; a wait of a second inside one ACK isn't noticed. A start also has to connect and migrate before it can take messages: the queue covers that.
• What to do: nothing; systemctl status ductus shows "running N interfaces" once it's up (systemctl start and restart wait for it now). The first upgrade to this version still refuses for a moment (the version before never gave systemd its sockets). A long migration makes senders wait as long: plan it (upgrade --check). Outside the package's service (ductus run by hand, ductus dev) a restart refuses as before.

systemctl reload hands over: receiving never stops, sending pauses briefly (level 2)

• What: on Linux, systemctl reload ductus (what the package's upgrade runs) starts the program now on disk beside the running engine, on the same listening sockets. The new engine reads the project, migrates the database and loads every interface's code while the running one carries on; from then on both store what arrives. Once it's ready, the running engine stops as a clean stop does (each MLLP connection closed once, after its last ACK: the sender reconnects and lands on the new engine), lets go of its interfaces and exits; the new one takes them and sends. Measured under real systemd (Ubuntu 24.04, Rocky Linux 9), 4 senders at 200 messages/s: a reload 0 refused, longest wait for an ACK 60–67 ms (a restart: 0.2–0.3 s), sending paused ~35 ms; a package upgrade 0 refused, 0.17–0.22 s; nothing lost, twice or out of order (one interface under 300 messages/s in WSL: sending paused 80–170 ms); at 1,001 interfaces sending paused 0.6 s (most of it statements waiting on a database-driver delay that a coming sqlx release removes). The reload returns once the new engine sends, and says so in the log: "handed over to ductus 0.6.1 (pid 5233): receiving never stopped, sending paused 180 ms". The audit log has an engine.handover entry; the Overview says "took over from v0.6.0".
• Why: a restart, even with systemd holding the sockets, makes senders wait for the whole start (0.5–1 s, longer with a migration) and sends nothing meanwhile; a handover overlaps the start with the running engine.
• What to do: use systemctl reload ductus for upgrades (the package does) and deploys (below). Run ductus upgrade --check first as before. For a few seconds two engines run, so the server needs room for both (memory roughly doubles while they overlap). The first upgrade to 0.6.6.3 restarts instead: the version before has no handover.

A new version or project that can't start leaves the running one serving (level 2)

• What: anything that keeps the new engine from getting ready (a broken interface.yaml or code, a failed migration, another engine owning one of its interfaces, a time limit of 5 minutes) stops the new one, not the old: systemctl reload fails, journalctl -u ductus says why ("the new engine can't take over: … This engine (pid 4120, ductus 0.6.0) goes on."), and senders notice nothing. A package upgrade whose new version can't take over leaves the package installed and the old version running, and says so.
• Why: an upgrade or deploy should never be what takes the engine down.
• What to do: fix what it names and systemctl reload ductus again (or systemctl restart ductus to start the new version the plain way). A migration longer than 5 minutes: ductus handover --timeout 30m (as the engine's user or root). A drain refuses a handover too: resume, or stop and start instead.

Deploying project changes: systemctl reload (level 1)

• What: the reload takes the project as it is on disk now, so changed interface code, a new interface or a changed destination is deployed with no refused connection: pull the project (git pull in /var/lib/ductus/project), then systemctl reload ductus. ductus.env is read again too (the database URL, the environment). A change the running engine can't hand over is refused and named: lowering an interface's lanes (ordering) needs a restart, which checks for messages waiting in the lanes that go.
• Why: the engine never reloads code in place; a new process starting beside the old one gives the same result with nothing to get half-way wrong.
• What to do: reload after every project change; ductus validate and ductus test first. Changes to the systemd unit itself (systemctl edit ductus) still need systemctl daemon-reload and a restart. Rolling back is the same: put the previous project (or install the previous package) and reload. Outside the package's service (ductus run by hand) set DUCTUS_CONTROL=/some/dir/control and run ductus handover --control /some/dir/control; ductus dev has no handover.

ductus replay shows what a change would do to real traffic, before you make it (level 1)

• What: ductus replay --since 24h runs the last day's messages (as received: originals are never changed) through this version's code, every destination, and sends nothing. For each message and destination it compares what the code decides (sent and exactly what, filtered and why, quarantined and why) with three things, whichever exist: the installed version run beside it on the same messages (from the new version's archive, before installing it: the default), what the destination recorded (its status and reason, kept for every message), and what it kept (save_output: true). The report groups differences where they are: HL7 by field ("PID-8 differs in 1,204 messages", with examples), JSON by path, other formats by line; --report replay.html writes a page with the versions, the period, the counts and a checksum on top (for a change board to sign), --json the same as one document. Each message and destination counts as identical (compared whole), same decision (outputs not compared: both sent, nothing to compare the output with), different, drift, not comparable or waiting. Exit 1 when anything differs. For a code change, --baseline main (a Git revision, or a folder) compares the project as it is now with that one, same report. ductus upgrade --check --replay 1d adds it to the upgrade check.
• Why: the usual evidence for an upgrade or a mapping change is weeks of test-system validation by hand; a replay of production's own traffic answers "what changes?" in minutes, message by message, and says where.
• What to do: run it from the unpacked archive before an upgrade (./ductus replay --since 24h --env-file /etc/ductus/ductus.env --report replay.html), and with --baseline before deploying a code change. A read-only login or a replica is enough. --limit 1000 samples the first thousand per interface; --interface picks some.

A replay compares outputs with the installed version from 0.6.6.4 on (level 2)

• What: the installed version takes part by running ductus replay --emit itself, so it must be 0.6.6.4 or later. Before that (and on a server with no installed version: /usr/bin/ductus, C:\Program Files\Ductus\ductus.exe), outputs are compared with kept outputs only, decisions with the record, and the report says so first. A message both sides sent with no output to compare is counted same decision (outputs not compared), never identical: a changed field can't show there. When no output was compared at all, the report ends with "⚠ outputs not compared" and the fix (also outputs_not_compared in --json, and on the HTML page), and upgrade --check --replay shows it as a warning. --against PATH names another binary.
• Why: comparing with a version run now, on the same messages at the same moment, isolates exactly what the upgrade changes, and needs nothing stored. Without it, a pass by decision alone would read as proof the outputs were checked.
• What to do: run the replay on the server, where the installed version is, or copy that binary and name it with --against. For the upgrade to 0.6.6.4 itself, keep outputs where you need them compared (save_output: true on the destinations that matter, a while before; it costs ~1.15 GB per million deliveries).

A replay doesn't write: those messages can't be compared (level 3)

• What: nothing leaves a replay. No destination is connected and routes store nothing. db().query runs (Postgres sessions are read-only, so a query that writes is refused by the server), db().execute and fetch() other than GET and HEAD aren't made: the call fails, and the message is reported "not comparable: writes to database billing: not made in a replay" for that destination, even if the code caught the error and went on. SQL Server has no read-only session setting: a db().query that writes there isn't stopped.
• Why: a replay of a day must never charge a patient twice or insert a day of rows again.
• What to do: keep writes in their own destination function where you can, so the other destinations of the interface stay comparable. Use db().query only for reads.

A difference from the record can be the data, not the code: drift (level 3)

• What: lookups run against the data as it is now, not as it was when the message was sent. When the installed version, run beside, gives the same result as the new one but the record or a kept output differs, the report calls it drift: shown, but it doesn't fail the replay (the upgrade didn't cause it). Without an installed version to compare with, every difference counts, and destinations whose code looked something up are marked "a difference may be the data".
• Why: both versions see the same lookups at the same moment, so only what they share can come from the data.
• What to do: read drift as "this interface's results depend on data that changes": expected for lookups, worth a look otherwise.

In a replay the clock is the message's arrival, and Math.random repeats (level 3)

• What: during a replay new Date() and Date.now() give the time the message was received (and the period's start in a module's top-level code), and Math.random() gives the same sequence for the same message and destination, in every version. new Date(x), Date.UTC and instanceof Date work as always.
• Why: otherwise every output that carries a timestamp or a random id would differ between two runs, and the report would drown in noise.
• What to do: nothing for comparisons between versions. A kept output that carries the time it was sent (a few milliseconds after it arrived, or hours later after an outage) differs from the replay in that field: use meta.receivedAt when you mean the message's time.

What a replay costs, and where (level 3)

• What: one thread runs the code, one message and destination at a time (the other version, beside it, its own thread). Measured in WSL (scripts/bench-replay.sh): 100,000 ADT messages through three destinations (an HL7 mapping, a JSON mapping, the message as is) in 7.5 s alone (13,400 messages/s, ~40,000 function calls/s) and 8.6 s with a second version beside it: a day of 1 million messages takes about 1.5 minutes on such an interface. The database reads the period once, in arrival order, 500 messages a statement. Lookups dominate when an interface makes them: at 1 ms a query, a million messages wait 17 minutes on lookups alone (they run one at a time).
• Why: both versions must produce their results in the same order to be compared as they stream, without holding a day in memory.
• What to do: for a quick answer use --limit 1000 (the first thousand per interface) or --interface; run the full day where lookups are few, or against a replica (DUCTUS_DATABASE_URL), outside peak hours when lookups hit production systems.

A replay's report holds patient data unless you hide it (level 2)

• What: examples show field values from real messages ("PID-5: "DOE^JOHN" → "DOE^JON""). --hide-values leaves values out (field names, counts and message ids stay). The replay reads every payload of the period with a database login, outside the API's permissions and its audit log, like ductus audit export.
• Why: the values are what make a difference understandable; whoever runs the replay can read the database already.
• What to do: keep reports with values as you keep patient data; for a change board outside the integration team, make it with --hide-values. Give the replay's login only pg_read_all_data.

A stopped service, a failed start and a drain refuse connections, on purpose (level 2)

• What: systemctl stop ductus closes the ports (systemd lets go of the sockets when the service stops); a start that fails (the database unreachable, a broken project, another engine owning the interfaces) gives them up too, so while it fails and systemd retries every 5 s senders are refused; a drain closes its MLLP listeners for real.
• Why: a port that takes connections with no engine behind it would leave senders waiting on ACKs that never come, looking connected. Refused is the honest answer when the engine isn't coming back by itself.
• What to do: watch for refusals at the senders as before; journalctl -u ductus says why a start failed.

Windows and containers: a restart is a short gap (level 2)

• What: on Windows the service manager runs one process, so a restart has a gap: Restart-Service Ductus measured 0 refused, a few connections reset and 1.1–1.2 s longest wait for an ACK (Windows itself retries a refused connection for ~1–2 s, so a short gap shows as a wait); an MSI upgrade had the engine down 3.1–3.5 s (5 refused, 4 reset, 3.2 s longest wait, nothing lost). msiexec itself took 100–165 s on a machine where Group Policy turns Restart Manager off, but the engine kept running until its last few seconds (see the Event Log nuance). A container's restart closes its published ports, so senders are refused until the new one listens.
• Why: passing the listening sockets on Windows needs a resident launcher (planned for v1.0); a new container can't share the old one's ports (Kubernetes rolling upgrades are planned for v1.2).
• What to do: senders resend on their own; for a planned restart, a quiet moment helps. Upgrading Windows: stop the service first where Restart Manager is off, to save the wait.