Introduction

Ductus is an integration engine: it receives messages, runs your code on them, and delivers them reliably to the systems that need them. Messages come in over HL7 v2's MLLP, HTTP, files in a folder or on an SFTP server, or are made on a schedule; they go out the same ways, or into another interface's queue. Interfaces are written as code (TypeScript, with a typed SDK) and configured in plain YAML files that live in Git, next to their tests.

Every message is stored in PostgreSQL before the sender is told it arrived, delivered at least once, in order where you ask for it, and kept for a window you choose. What happened to each one can be looked up, sent again or corrected, by people with the right to, and every such action is in the audit log.

Pre-release. Ductus is under active development. Downloads before v1.0 are development builds: there is no production support, there has been no external security audit, and the Windows installer isn't signed yet. Try it, test it, tell us what's missing; don't send real patient data through it yet.

What it is, and what it isn't

It is one program (ductus, with its web UI inside) and a PostgreSQL database. There is no message broker, no Java runtime and no Node.js to install: the engine runs your TypeScript in an embedded V8, and uses Postgres as its store and its queue. It runs as a service on Linux (systemd) or Windows, in a container, or on a laptop with a database of its own (ductus dev).

It is for the work an interface team does: feeds from an EHR to a lab or billing system, an API that other systems call and wait on, a nightly file from a partner, a lookup in a database on the way. It's written with healthcare in mind (HL7 v2 is a first-class format, retention and the audit log follow HIPAA's habits), and it works the same for any feed, API or batch job.

It isn't a low-code tool: there are no drag-and-drop mappings. Simple interfaces are a few lines of YAML (a feed relayed as it is needs no code at all), and anything more is TypeScript with types, tests and an editor that knows HL7's field names. It isn't a database or an archive either: messages are kept for one, three or seven weeks, and longer history belongs in the systems that produced it. And before v1.0 it doesn't cluster: one engine runs each interface, and a standby is started by a person (why).

Where to start

How to read these docs

• Guides (Install, Configure and the pages around them) explain how to do something, start to finish, and why each step is there. They link to the reference for every setting they name.
• The reference lists everything: every command, API operation and setting, with its default and an example. It's made from the engine itself, so it always matches the version it comes with.
• Nuances say what could surprise you, why it is so, and what to do about it, each with a level from 1 (everyone needs it on day one) to 5 (expert). Guides link to them rather than repeat them.
• Architecture is how the engine works inside: for the curious, and for anyone who must explain its guarantees to someone else.

Every setting, command and page aims to say what it does, why you'd use it, when not to, its default, and an example. If one doesn't, that's a bug: please report it.

Choosing Linux or Windows

Ductus runs on both, from one code base and one project format: the same interfaces, the same YAML, the same TypeScript, the same database. A project moves from one to the other with only its file paths changed. Where they differ is in how each operating system starts, stops and upgrades a service, how it logs in to other Windows systems, and how much the build has been exercised. This page lists every difference we know of, each with the measurement or nuance behind it.

Our recommendation: Linux, unless the engine must log in to SQL Server with Windows accounts, or your team runs only Windows servers and would rather not keep one Linux server. Both are supported, and both behave the same in everything not in this table.

The differences

Plan for these on either

• Reboots (patch nights): a reboot is a stop and a start. The engine stops cleanly (Windows gives the service its preshutdown time, 90 s; systemd sends SIGTERM and waits 45 s), and it starts with the server. While the server is down, senders are refused and retry, as with any engine; systemd's socket store doesn't survive a reboot. Reboot in a quiet moment, or move the interfaces to another engine first (several engines).
• Antivirus and endpoint protection that scans every file write slows the database far more than the engine: the engine writes only its logs; Postgres writes every message. Ask for the usual exclusions for Postgres's data and WAL folders on its server (your DBA will know them), and for the engine's project folder if a scan on read slows its start.
• Postgres is the same on both: the DBA's server, on Linux or Windows, whatever the engine runs on (PostgreSQL).
• Time zone: cron schedules, quiet hours and monitor: times are in the server's local time on both. A server (or container) that runs on UTC reads "06:00-22:00" as UTC.

How the two are kept the same

One code base and one project format; the whole Rust test suite runs on both on every push; Windows-only behaviour is written down as a nuance the day it's found (the Windows section, and the Windows rows in other sections). When a difference is fixed (a resident launcher for zero-downtime Windows upgrades is planned for v1.0, a Kerberos keytab on Linux too), its row here changes in the same commit.

Install

Ductus is one program, ductus (ductus.exe on Windows): the engine, its command line and the web UI, all in one file of about 140 MB. It needs a PostgreSQL database, which is never bundled: in production that is your DBA's Postgres, on the same server or another. Everything the engine maintains in it (tables, daily partitions, retention, cleanup) it does by itself, while it runs: there are no cron jobs or scheduled tasks to set up.

Which way to install

Not sure between Linux and Windows? Choosing Linux or Windows lists every difference.

Each release on GitHub holds all of these, plus the docs for that version as a zip, SBOMs and SHA256SUMS. Check a file before you install it: Downloading and checking a release.

What you need first

• A server: 64-bit x86 (x86_64/amd64; no ARM builds yet), Linux with glibc 2.34 or later (RHEL 9 and its rebuilds, Ubuntu 22.04, Debian 12 and later; RHEL 8 can't run it), or 64-bit Windows (tested on every push on GitHub's current Windows Server runner, and on Windows 11).
• Memory: the engine itself is small (measured: 1,001 interfaces loaded and idle in 111 MB); what grows is what interface code holds, up to isolates.heap_mb (512 MB) per worker thread at most, which most interfaces never come near. Watch the process once real traffic runs (systemctl status ductus shows its memory; Task Manager on Windows), and see memory for interface code for what makes it grow. During a Linux upgrade two engines run for a few seconds, so leave room for a second copy.
• CPU: one worker thread per core by default. A feed of a million HL7 messages a day averages 12 a second; one core runs a realistic HL7 transform at about 14,800 a second (measured), so CPU is rarely what limits it: the database's disk is.
• PostgreSQL 14 or later (18 recommended), with a login the engine owns its tables with: PostgreSQL.
• The ports your senders connect to, and the ones the engine connects out on, opened in the firewall: Ports and firewalls.
• A project: the folder of interfaces the engine runs, usually a Git repository. A new one starts with ductus init; Your first interface walks through it.

What every install has in common

The engine is configured by a few environment variables, the same on every platform (the packages read them from a settings file: /etc/ductus/ductus.env on Linux, C:\ProgramData\Ductus\ductus.env on Windows):

Every ductus command takes --env-file FILE to read a file of these (NAME=value lines) before anything else: ductus doctor --env-file /etc/ductus/ductus.env runs with exactly the service's settings. Variables already set win over the file.

After installing

1. Give it a database: PostgreSQL.
2. Put the project in place and set DUCTUS_PROJECT and DUCTUS_ENV.
3. Run ductus doctor --env-file … on the server: it checks every connection the project makes, from where the engine will make it, and says in plain words what a failure means.
4. Start the service, and make the first admin account: ductus user create you --role admin --env-file … (people).
5. Open the web UI: http://127.0.0.1:7700 on the server itself, or set up api.tls to use it from your own computer.

Downloading and checking a release

Every release is on GitHub: github.com/thorst/ductus/releases. Each one holds the packages for every way to install, the docs for that version as a download, lists of every library inside (SBOMs), and what you need to check that a file is the one the project built: its SHA-256 checksum and GitHub's record of the build (an attestation).

Check before you install, every time: a file downloaded through a proxy, a mirror or a colleague's share could have been changed on the way, and on Windows the installer isn't code-signed before v1.0, so nothing else tells you it's genuine. You can skip it for the ductus dev laptop setup built from source, where nothing was downloaded.

Pre-releases until v1.0. Every release before 1.0 is marked pre-release: for trying, testing and pilots, not production. Settings can change between releases; the upgrade page and ductus upgrade --check say what each one changes.

Which file

Every file comes from the same CI run, on the tagged commit, and only after every test and demo passed on it: the Rust tests on Linux and Windows, the end-to-end demos, the packages installed, upgraded and removed under real systemd, the MSI installed, upgraded and removed as a service. Nothing is built by hand.

Check the checksum

SHA256SUMS lists every file of the release with its SHA-256. Download it beside the files you took, and check them.

On Linux, in the folder with the downloads (--ignore-missing skips the files you didn't download):

sha256sum --check --ignore-missing SHA256SUMS

Each file says OK. Anything else (FAILED, No such file) means the file isn't the one the release holds: download it again from the release page, and don't install it. On the release job's dry run (version 0.0.0-dryrun.118; a release's files are named the same way with its version), with only the .deb downloaded beside SHA256SUMS, then the same file with one byte added to it:

$ sha256sum --check --ignore-missing SHA256SUMS
ductus_0.0.0~dryrun.118_amd64.deb: OK
$ sha256sum --check --ignore-missing SHA256SUMS
sha256sum: WARNING: 1 computed checksum did NOT match
sha256sum: SHA256SUMS: no file was verified
ductus_0.0.0~dryrun.118_amd64.deb: FAILED

The second exits 1, so a script can stop on it. (A pre-release's .deb and .rpm are named with ~ where the version has -: that is how Debian and RPM spell "sorts before the release".)

On Windows, in PowerShell, the hash of the file, then the line SHA256SUMS has for it:

(Get-FileHash .\ductus-0.6.0-x64.msi -Algorithm SHA256).Hash
Select-String ductus-0.6.0-x64.msi .\SHA256SUMS

The two must be the same 64 characters (Get-FileHash writes them in capitals, SHA256SUMS in small letters: the case doesn't matter). On the dry run's MSI:

PS> (Get-FileHash .\ductus-0.0.0-dryrun.118-x64.msi -Algorithm SHA256).Hash
3F8DB975E899350CCC54FC0DAF3F659A7BC5A1BA69DE074ADE95AF47BE43A47D
PS> Select-String ductus-0.0.0-dryrun.118-x64.msi .\SHA256SUMS

SHA256SUMS:6:3f8db975e899350ccc54fc0daf3f659a7bc5a1ba69de074ade95af47be43a47d  ductus-0.0.0-dryrun.118-x64.msi

Check who built it: the attestation

A checksum proves the file matches SHA256SUMS; it doesn't prove who wrote SHA256SUMS. Every file of a release, SHA256SUMS included, is also attested by GitHub: a signed record, made by the release job as it built the file, that names the repository, the workflow and the exact commit. GitHub signs it through Sigstore's public log, so the record can't be changed afterwards, even by the project.

Check one with the GitHub CLI, signed in with any GitHub account (gh auth login):

gh attestation verify ductus_0.6.0_amd64.deb -R thorst/ductus

It says whether it found an attestation for exactly this file, made by a workflow of thorst/ductus, and ends with the verification succeeding. If it finds none, or one from another repository, don't install the file. On Windows it's the same command (gh installs with winget install GitHub.cli); it's the check that replaces a code signature until v1.0 (nuance).

When to use which: the checksum always (it needs nothing but the system's own tools); the attestation once per release on the machine you download on, before the file is copied to servers, or wherever your security policy asks for proof of origin.

The SBOMs: every library inside

A Software Bill of Materials lists every library a program is made of, with its version and license, in a format tools read: these are CycloneDX 1.5 JSON, the format Dependency-Track, Microsoft Defender and most scanners import.

On the dry run: 530 crates in the Linux binary's, 534 in the Windows one's, 95 npm packages in the web UI's, about 700 KB, 700 KB and 100 KB. They leave out the tools used only while building (code generators, the build scripts' own libraries): those aren't in the program you run. The same lists, for reading, with the notable libraries first (V8, TypeScript, rustls, sqlx, tokio), are on Libraries and versions; that page also counts build-time crates, so its numbers are a little higher. Use the SBOMs when your security team asks "what's in it", or to be told when a library in it has a published vulnerability. Scanning them for you is planned for v0.9.

The docs as a download

ductus-docs-0.6.0.zip is this site as it is for that version: unzip it anywhere, open ductus-docs-0.6.0/index.html in a browser. Every page says the version in its title. It's about 3.2 MB (3,212,719 bytes on the dry run). It's for:

• A server room or network without internet access: the docs are with the download, nothing is fetched from anywhere.
• The version you run, not the newest: docs.getductus.com describes the latest release; the zip describes the one you installed, and stays that way.

Search works from the unzipped folder (opened as file://): it's mdBook's own search, which loads its index from the folder. The site's search (Pagefind) needs a web server, so the zip doesn't have it. Don't put the zip's folder on a web server as your copy of the site: it works, but people then read an old version's docs as current; link to docs.getductus.com instead.

Before a release: development builds

Until a release exists, every push to main builds the packages, the archives and the installer in CI as development builds (versions like 0.6.0-dev.2 and 0.6.0-ci, kept 90 days as the run's artifacts linux-packages and windows-installer; no checksums, attestations or SBOMs), and the packages and the installer can be built from a clone of the repository. They are for testing the project itself: use a release for anything else.

Linux packages

The .deb (Ubuntu, Debian) and .rpm (RHEL, Rocky, Alma) install the engine as a systemd service with its own user, a settings file and a folder for the project. Upgrades hand the running engine over to the new version with no connection refused.

Use them on any Linux server that runs Ductus for real. Don't where you can't install packages (then the portable archive has the same unit and settings file as samples), or in a container (the image).

Both packages hold the same binary, built once on Rocky Linux 9: it runs wherever glibc is 2.34 or later (RHEL 9 and its rebuilds, Ubuntu 22.04, Debian 12 and later). RHEL 8 (glibc 2.28) can't run it.

Install

# Ubuntu, Debian
sudo apt install ./ductus_0.6.0_amd64.deb
# RHEL, Rocky, Alma
sudo dnf install ./ductus-0.6.0-1.x86_64.rpm

Both files are on the release page. Check the one you downloaded first, in its folder with the release's SHA256SUMS (each file you have says OK):

sha256sum --check --ignore-missing SHA256SUMS
gh attestation verify ductus_0.6.0_amd64.deb -R thorst/ductus    # GitHub built it, from the tagged source

Before v0.6.0 is released, packages are development builds: docker build --target packages --build-arg VERSION=0.6.0-dev.1 --output dist . in a clone of the repository makes both.

The install creates the ductus user and group, and doesn't start anything: nothing is configured yet. It prints what to do next, which is the rest of this page.

What the package installs

The sandbox. The unit lets the engine use the network and its own folder, and nothing else: the rest of the file system is read-only to it (ProtectSystem=strict), home folders are hidden (ProtectHome=yes), and it has no capabilities. That's on purpose: one bad interface can't write anywhere on the server. It means a file destination or a file source that moves files must work in a folder you give it:

sudo systemctl edit ductus
# in the editor that opens:
#   [Service]
#   ReadWritePaths=/data/outbound /data/inbound
sudo systemctl restart ductus

Without that the interface's messages fail with "Read-only file system", and wait to be tried again (nuance). A folder for a source needs the same when the source moves or deletes what it read (the default, after: move). A port below 1024 (an MLLP port of 104, say) needs a capability too; the unit says which two lines to add with systemctl edit.

Configure

1. Settings. Edit /etc/ductus/ductus.env (as root). At least the database:

   # /etc/ductus/ductus.env
   DUCTUS_DATABASE_URL=postgres://ductus:PASSWORD@db1.hospital.org/ductus?sslmode=verify-full
   DUCTUS_PROJECT=/var/lib/ductus/project
   DUCTUS_ENV=prod
   DUCTUS_SECRET_KEY=…
   

   The file's comments explain each line; every variable is in the install overview. For the database login and TLS, see PostgreSQL. DUCTUS_SECRET_KEY is the key your project's secrets/prod.json was encrypted with (secrets).

2. The project. Clone it as the service's user, so the files are its own:

   sudo -u ductus git clone git@git.hospital.org:interfaces/ductus.git /var/lib/ductus/project
   

   The ductus user has no shell (nologin), but sudo -u ductus runs a command as it. For a repository that needs a login, give the user a deploy key that can only read (/var/lib/ductus/.ssh/), or copy the project in from your deployment tool. The engine only reads the project: it never writes there.

3. Check before starting. As root or the ductus user, with the service's own settings:

   sudo -u ductus ductus validate --env-file /etc/ductus/ductus.env --root /var/lib/ductus/project
   sudo -u ductus ductus doctor --env-file /etc/ductus/ductus.env --root /var/lib/ductus/project
   

   validate checks the project as the engine will read it, in the environment it will run in, and against the database (reading only): lane changes that would strand waiting messages, retention changes still running. --root is needed: unlike run, these commands don't read DUCTUS_PROJECT. doctor connects to everything the project connects to, from this server, and says in words what a failure means: a firewall, a certificate, a login.

Start, stop, look

sudo systemctl enable --now ductus     # start now and at every boot
systemctl status ductus                # "running 12 interfaces" once every one is up
journalctl -u ductus -f                # the log; "started interfaces=12 …" when ready

• systemctl start waits until every interface is up (the engine tells systemd when it's ready). A start migrates the database first, so after an upgrade it can take longer; there's no time limit.
• A stop is clean: no new connections, the messages in hand finished and saved, exit 0 (what finishes). systemd waits 45 s, longer than the engine's own limit (DUCTUS_STOP_TIMEOUT, 30 s).
• A restart or a crash refuses no connection: systemd keeps the listening sockets and hands them to the next start; senders wait in the queue (0.2–0.3 s for a restart, ~5 s after a crash: measured). A stop, a failed start and a drain do close the ports, on purpose (why).
• A refused start isn't retried. Exit code 3 means another engine runs some of these interfaces on the same database, or a change to an interface's lanes would strand waiting messages. Trying again changes neither, so systemd doesn't (RestartPreventExitStatus=3): the log says what to do, then systemctl start ductus. Any other failure is retried every 5 s.
• Logs go to the journal, plain text with fields (interface=adt-in). RUST_LOG in ductus.env sets how much: info by default.

Deploy a change, upgrade

Project changes (new code, a new interface): pull the project, then reload. The new project starts beside the running engine and takes over once it's ready; nothing is refused, and a project that can't start leaves the running one serving (deploying):

sudo -u ductus git -C /var/lib/ductus/project pull
sudo -u ductus ductus validate --env-file /etc/ductus/ductus.env --root /var/lib/ductus/project
sudo systemctl reload ductus

Upgrades: run the new version's ductus upgrade --check from its portable archive first, then install the new package the same way as the first. The package hands the running engine over to the new version (systemctl reload again: 0 refused, sending paused for a moment: measured). A new version that can't start leaves the old one running, and the install says so. Rolling back is installing the previous package: the database changes a version makes only ever add, so the version before runs on them.

Remove

sudo apt remove ductus (or dnf remove) stops and disables the service and removes the program. The ductus user, /etc/ductus/ductus.env and /var/lib/ductus (the project) stay, and so does everything in the database: removing Ductus never deletes messages.

Windows installer

The MSI installs the engine as a Windows service, Ductus, with a settings file, a folder for the project and its logs, and an Event Log source. It installs silently for Group Policy, SCCM or Intune, and upgrades in place.

Use it on a Windows server that runs Ductus for real. Don't where you can't run an installer (then the .zip has the program alone), or to try Ductus on your own machine (ductus dev needs no installer and no administrator).

Unsigned before v1.0. The installer isn't signed with a code-signing certificate yet, so Windows SmartScreen says "Windows protected your PC" and "unknown publisher" when it's run from a download. Check the file before you run it: its SHA-256 against the release's SHA256SUMS (Get-FileHash ductus-0.6.0-x64.msi), and that GitHub built it from the tagged source (gh attestation verify ductus-0.6.0-x64.msi -R thorst/ductus): both are on Downloading and checking a release. Then "More info" → "Run anyway", or unblock it first (Unblock-File). Signing comes with v1.0 (nuance).

Install

As an administrator:

msiexec /i ductus-0.6.0-x64.msi /qn          # silent; without /qn, the usual wizard

Before v0.6.0 is released, MSIs are development builds: packaging/windows/build.ps1 in a clone of the repository makes one (it needs Rust, the Visual Studio Build Tools and WiX 5: see packaging/README.md).

The install registers the service (it starts automatically at boot) but doesn't start it: nothing is configured yet, and without a database it would stop at once.

What the installer puts where

C:\ProgramData\Ductus can be read by administrators, SYSTEM and the service only: ductus.env holds the database password. Keep it that way.

The service's account

The account is what SQL Server, file shares and anything else on the network see when the engine logs in with Windows. Three choices (nuance):

The installer (and ductus service account) grants the account "Log on as a service". Where Group Policy manages that right, ask for the account to be added to the policy, or the next policy refresh takes it away (nuance). ductus service account with no name says who the service runs as now.

Folders outside C:\ProgramData\Ductus. The service can touch only what you give it: a file interface's inbox, archive or output folder needs a grant (nuance):

icacls "D:\Interfaces" /grant "NT SERVICE\Ductus:(OI)(CI)M"

That works whatever account runs the service (its own SID is in its token). For a network share, grant the computer account (HOSPITAL\SERVER01$), or the gMSA.

Configure

1. Settings. Edit C:\ProgramData\Ductus\ductus.env as an administrator (Notepad run as administrator works). At least the database:

   DUCTUS_DATABASE_URL=postgres://ductus:PASSWORD@db1.hospital.org/ductus?sslmode=verify-full
   DUCTUS_PROJECT=C:\ProgramData\Ductus\project
   DUCTUS_ENV=prod
   DUCTUS_SECRET_KEY=…
   

   No quotes are needed, and backslashes are plain backslashes here. Every variable is in the install overview; the database's login and TLS are on the PostgreSQL page. Secrets are never installer properties (they'd be written to the install log): the password lives in this file only.

2. The project. Clone it into C:\ProgramData\Ductus\project (or point DUCTUS_PROJECT elsewhere and grant the service read access there). In YAML, write Windows paths with their drive and in single quotes ('D:\Interfaces\in'): in double quotes a backslash starts an escape (nuance).

3. Check before starting, with the service's own settings:

   ductus validate --env-file C:\ProgramData\Ductus\ductus.env --root C:\ProgramData\Ductus\project
   ductus doctor --env-file C:\ProgramData\Ductus\ductus.env --root C:\ProgramData\Ductus\project
   

   These run as you, not as the service's account: a folder or a SQL Server Windows login can pass here and fail for the service. The service's log says "access denied" then.

Start, stop, look

Start-Service Ductus           # after a change to ductus.env: Restart-Service Ductus
Get-Service Ductus
Get-ChildItem C:\ProgramData\Ductus\logs\ductus.*.log | Sort-Object LastWriteTime |
  Select-Object -Last 1 | Get-Content -Tail 50 -Wait      # the newest log, as it grows

The log files are named by the date in UTC (ductus.2026-10-30.log), so near midnight the newest one may carry tomorrow's or yesterday's date.

• Starting and stopping report progress to the service manager, so a long migration or a stop finishing the messages in hand isn't taken for a hang.
• A stop is clean, whether from the service manager, a Windows shutdown (the service takes its preshutdown time, 90 s) or ductus drain --stop: the messages in hand are finished and saved (what finishes).
• A restart is a short gap: measured 0 refused, a few connections reset, the longest wait for an ACK ~1.1 s (measured). Senders resend by themselves.
• If it fails it's started again after a minute, twice, then waits for a person. A start refused because another engine runs these interfaces isn't retried.
• The Event Log gets what operations watch for: started, stopped, couldn't start (with the reason), an interface stopped, a critical alert, and other errors at most once a minute per kind (IDs, nuance). The detail is in the log files.

Upgrade

Run the new version's ductus upgrade --check from its .zip first, then the new MSI the same way as the first (msiexec /i … /qn; no need to uninstall). The installer stops the service cleanly, replaces the program and starts it again; settings, project and logs are never touched. The engine is down for 3.1–3.5 s (measured: a few connections refused, nothing lost), so pick a quiet moment (measured). A sender sending through an upgrade over TLS, on CI's runner: 8 connection attempts refused, the slowest message 3.3 s to its ACK, 12,000 of 12,000 received once each and in order (the demo's output).

• A service someone stopped on purpose stays stopped after the upgrade (nuance).
• A password account leaves the service stopped, waiting for ductus service account NAME --start (Event Log 1005 says so).
• Where Group Policy turns Restart Manager off, msiexec may wait 100–160 s before it stops the service (the engine keeps running meanwhile). Close Event Viewer first, or stop the service yourself first to save the wait (why).
• Rolling back is uninstalling, then installing the previous MSI: the installer refuses to put an older version over a newer one ("A newer version of Ductus is already installed. Uninstall it first to go back."). C:\ProgramData\Ductus and the database stay; give SERVICEACCOUNT= again if the service ran as another account. The database changes a version makes only ever add, so the version before runs on them (Going back). demo-windows.ps1 goes back this way on every push.

Uninstall

"Apps" in Settings, or msiexec /x ductus-0.6.0-x64.msi /qn: removes the service, the program and the Event Log source. C:\ProgramData\Ductus (settings, project, logs) stays, and so does everything in the database.

The portable archive

The engine as one program, without an installer: ductus-0.6.0-linux-x64.tar.gz and ductus-0.6.0-windows-x64.zip, each unpacking into a folder ductus-0.6.0. Unpack it anywhere and run it; nothing is registered, and nothing is installed. Both are on each release, with SHA256SUMS to check them against.

Use it to check an upgrade before installing it (the main reason it exists), to try a new version beside the installed one, to run the command line on a workstation (doctor, replay, user), and on servers that can't take packages. Don't use it as the way to run production when a package will do: the packages set up the service, its user, its sandbox and its folders, and upgrade in place with no connection refused.

Before an upgrade

Run the new version's program against the database and project your engines use now, before the change window:

tar xzf ductus-0.6.1-linux-x64.tar.gz && cd ductus-0.6.1
./ductus upgrade --check --env-file /etc/ductus/ductus.env --root /var/lib/ductus/project

Expand-Archive ductus-0.6.1-windows-x64.zip -DestinationPath C:\Temp     # makes C:\Temp\ductus-0.6.1
C:\Temp\ductus-0.6.1\ductus.exe upgrade --check --env-file C:\ProgramData\Ductus\ductus.env --root C:\ProgramData\Ductus\project

It reads only, changing nothing in the database or the project, and says what the new version's first start would do: the database changes it makes (and how big the tables they work through are now), which engines run on the database and on which versions, whether the project validates and passes its tests with the new version, and which settings it deprecates. It exits 1 on anything that would refuse the start or fail the project; --json gives one document to attach to a change ticket (nuance).

A read-only database login is enough, and safer for a program you're trying out:

CREATE ROLE ductus_check LOGIN PASSWORD '…';
GRANT pg_read_all_data TO ductus_check;

Then DUCTUS_DATABASE_URL=postgres://ductus_check:…@db1.hospital.org/ductus?sslmode=verify-full in place of --env-file (keep --env prod and the secrets key, which the check needs to read the project as the engine does). The same login suits ductus replay, which runs a day's traffic through the new version and compares (nuance).

Installing from the archive

Prefer the packages. Without them, on Linux, the archive's samples give the same setup by hand:

sudo install -m 0755 ductus /usr/bin/ductus
sudo useradd --system --home-dir /var/lib/ductus --create-home --shell /usr/sbin/nologin ductus
sudo install -d -m 0755 /etc/ductus
sudo install -m 0640 -g ductus ductus.env /etc/ductus/ductus.env   # then edit it
sudo install -m 0644 ductus.service /usr/lib/systemd/system/ductus.service
sudo systemctl daemon-reload && sudo systemctl enable --now ductus

Everything on Linux packages from "What the package installs" on then applies, except that upgrades are yours: replace /usr/bin/ductus and run sudo systemctl reload ductus (the handover, as the package does). On Windows, the .zip has no service setup: use the MSI for a service; ductus.exe from the zip is for the command line and for trying a version.

The program needs glibc 2.34 or later on Linux (RHEL 9 and its rebuilds, Ubuntu 22.04, Debian 12 and later), and 64-bit Windows.

Docker

The image runs the same engine as the packages, in a container: Debian 13 slim, the binary at /usr/local/bin/ductus, the system's CA certificates, running as user ductus (uid 10001), never root. The project is mounted read-only at /project.

Use it to try Ductus with its database in one command, and to develop with containers. Don't use it for production before v1.0: the image is a development build, it isn't published to a registry yet (you build it), and a container's restart refuses senders for a moment, where the Linux package's service refuses none (nuance). From v1.0, production images will be built FROM the engine's image with the project inside: one image per release of your interfaces. Kubernetes, with rolling upgrades, is planned for v1.2.

Try it with Compose

From a clone of the repository:

docker build -t ductus:dev .
PROJECT=$PWD/examples/adt-relay docker compose -f packaging/docker/compose.yaml up

That starts Postgres 18 and the engine on the example project. Then, in a second terminal, make yourself an account (the database is the container's, so the command runs there):

docker compose -f packaging/docker/compose.yaml exec engine ductus user create me --role admin

and sign in to the web UI at http://127.0.0.1:7700.

The Compose file is short and worth reading before you change it:

# compose.yaml (packaging/docker/compose.yaml, in part)
services:
  engine:
    image: ${IMAGE:-ductus:dev}
    environment:
      DUCTUS_DATABASE_URL: postgres://ductus:${POSTGRES_PASSWORD:-ductus}@postgres/ductus
      DUCTUS_ENV: ${DUCTUS_ENV:-dev}
      DUCTUS_SECRET_KEY: ${DUCTUS_SECRET_KEY:-}
    volumes:
      - ${PROJECT:?set PROJECT to your project folder}:/project:ro
    ports:
      - 127.0.0.1:7700:7700   # the API and web UI: this machine only
      - 6661:6661             # each interface's listener
    stop_grace_period: 45s

What to know

• Settings are environment variables, the same as everywhere (the list). Keep DUCTUS_SECRET_KEY and the database password out of the Compose file: Compose reads ${...} from a .env file beside it (one you don't commit), and an orchestrator can pass its secrets in as environment variables. The engine reads them only as variables, not as files.
• The API listens on 0.0.0.0:7700 inside the container, so Docker can publish it: the image sets DUCTUS_API_LISTEN=0.0.0.0:7700, which wins over api.listen in ductus.yaml (set the variable to move it). Publish it to 127.0.0.1 (as above) unless the API has tls: or a TLS proxy is in front: the web UI's sign-in needs HTTPS anywhere but on the machine itself.
• Publish each interface's listener (6661:6661 for an MLLP port). ductus ports lists every port a project listens on.
• Give a stop time. Docker waits 10 s by default before it kills a container; a clean stop can need up to DUCTUS_STOP_TIMEOUT (30 s). stop_grace_period: 45s (or docker stop --time 45) lets the engine finish the messages in hand (what finishes).
• The project is read-only: interfaces change through Git and a new container, never inside a running one. A file destination needs a volume of its own, writable by uid 10001.
• The database in the Compose file is for trying it out. For anything that matters, point DUCTUS_DATABASE_URL at the DBA's Postgres (PostgreSQL).
• Time zone: the image runs on UTC unless you set TZ. Cron schedules, quiet hours and monitor: times are read in the container's local time.

On your laptop: ductus dev

ductus dev runs a project with a PostgreSQL of its own: no database to install, nobody to ask, nothing to configure. It's how you write and try interfaces on your own machine, on Linux or Windows.

Use it for development, demos and learning. Don't use it to run anything that matters: it uses Postgres's default settings, nobody backs its data up, and it has no service, no handover and no restart on failure (nuance). Everywhere else, use ductus run (or a package) with your DBA's Postgres.

Get the program

Download the portable archive for your system and put ductus (ductus.exe) on your PATH, or build it from source (cargo build --release in a clone; see the repository's README). No administrator rights are needed on either system.

Run it

ductus init hospital      # a new project, with one interface to start from
cd hospital
ductus dev

The first run downloads Postgres 18.6 once for your user (about 30 MB, from Maven Central), then prints where everything is:

Postgres 18.6.0 is running for this project:
  postgresql://postgres:…@127.0.0.1:42731/ductus
  (data in ./.ductus/postgres/data; stops with the engine)
Web UI login: user `dev`, password `…` (kept in ./.ductus/dev-login)
… INFO ductus_runtime::api: REST API listening (/api/v1) host=127.0.0.1 port=7700 https=false
… INFO ductus_runtime: started interfaces=1 mllp=1 http=0 files=0 schedules=0 …

• The web UI is at http://127.0.0.1:7700, with the dev login it printed.
• The database URL works with any SQL tool (psql, DBeaver, DataGrip) while the engine runs, and with every command that needs a database: export DUCTUS_DATABASE_URL=… (PowerShell: $env:DUCTUS_DATABASE_URL = "…"), then ductus status, ductus user create, ductus token create and the rest talk to it.
• Ctrl-C stops the engine cleanly, then its Postgres.
• The data stays in .ductus/postgres between runs, and so does its port. ductus init's .gitignore leaves .ductus/ out of Git.

All of them are in the reference.

Without internet access

The download is the only thing ductus dev fetches, and it keeps it in a per-user folder: ~/.cache/ductus/postgres on Linux ($XDG_CACHE_HOME when set), %LOCALAPPDATA%\ductus\postgres on Windows. Where the download is blocked, copy that folder to the same place from a machine of the same system that has run ductus dev once; or unpack Postgres's binaries yourself into a folder holding bin/, and point DUCTUS_POSTGRES_DIR at it. Or skip dev and run ductus run against any Postgres you have, with DUCTUS_DATABASE_URL.

What's different from a server

• One engine per project folder at a time: a second ductus dev in the same folder says it's already running.
• No handover: systemctl reload is the packages'. After a change to the project, stop and start ductus dev again (it starts in well under a second).
• DUCTUS_DATABASE_URL is ignored: dev always uses its own Postgres.

PostgreSQL

Ductus keeps everything in PostgreSQL: every message as received, each destination's delivery and its status, the queue, the audit log, accounts, alerts. There is no other store and no message broker. This page is what to ask your DBA for, and why; it's written so you can hand it to them.

What the engine does in its database, by itself: creates its tables at its first start, and applies a new version's changes at that version's first start (they only ever add: the version before keeps working on them); makes each day's partitions ahead of time, hourly; drops whole partitions once their messages are past retention (no row-by-row deletes, so no bloat from them); holds one session for the whole time it runs, with an advisory lock that says which engine owns which interfaces. It needs no superuser, no extension, and no cron job or scheduled task.

What it doesn't do: back the database up, replicate it, or tune it. Those are the DBA's, as for any database that matters (backups).

Version

PostgreSQL 18 is what Ductus is tested on, on every push, and what ductus dev runs. Any version from 14 on should work: the oldest feature the engine relies on is DETACH PARTITION … CONCURRENTLY (new in 14), which lets it drop old messages without blocking anyone. 14–17 aren't tested yet. For a new install, use 18.

Managed services work the same: Azure Database for PostgreSQL (flexible server), Amazon RDS and Aurora PostgreSQL, Google Cloud SQL. They require TLS: see TLS.

The login and its database

Give Ductus a database of its own, owned by its login:

CREATE ROLE ductus LOGIN PASSWORD '…';
CREATE DATABASE ductus OWNER ductus;

(Or, on the database server, sudo -u postgres createuser --pwprompt ductus and sudo -u postgres createdb --owner ductus ductus.)

• Why the owner: the engine creates and changes its own tables, functions and partitions: at each new version's first start, and every hour for partitions. Owning the database (and so its public schema) is the one grant that covers all of it, now and in later versions.

• Why its own database: its tables are named for it (messages, deliveries, users…) and live in public; sharing a database with an application invites a clash. Several engines share one database without trouble: each runs its own interfaces, and an engine started with interfaces another runs is refused (nuance).

• No superuser, no CREATEDB, no CREATEROLE, no extensions.

• A read-only login for checks and audits: ductus upgrade --check and ductus replay only read, so give them one that can't write (Postgres 14+):

  CREATE ROLE ductus_check LOGIN PASSWORD '…';
  GRANT pg_read_all_data TO ductus_check;
  

Then the engine's URL, in DUCTUS_DATABASE_URL:

postgres://ductus:PASSWORD@db1.hospital.org:5432/ductus?sslmode=verify-full

A password with @, :, /, ? or # in it must be percent-encoded in the URL (@ is %40). The URL means what it means to psql and every libpq tool, so a URL that works there works here.

Postgres on the engine's own server can skip the password and TLS: connect over its Unix socket, postgres:///ductus?host=/var/run/postgresql, and Postgres's peer login maps the ductus service user to the ductus role (local ductus ductus peer in pg_hba.conf, often there already as local all all peer). It's also the fastest connection.

Connections

The engine opens up to 32 connections to its database (DUCTUS_DB_CONNECTIONS), as it needs them, plus one it holds for its lock. Budget for:

• each engine on the database: 33;
• twice that for a few seconds during a Linux upgrade or deploy: the new engine starts beside the running one, with its own pool, before the old one leaves (handover);
• a few more for the command line (ductus status, doctor, user) and the DBA's own tools.

The pool is for the engine's own database only. The databases interfaces query (db(), in resources.yaml) have pools of their own, set per entry (max_connections).

PgBouncer and other poolers: session mode only. In transaction mode the engine's lock is lost at once, since each transaction lands on another server connection (nuance). The engine pools its own connections, so a pooler in front of it saves little.

TLS to the database

Without sslmode in the URL, the connection is libpq's prefer: TLS when the server offers it, and the server's certificate is never checked. That stops someone listening, not someone answering in the database's place. For a database on another machine, ask for its certificate to be checked (nuance):

The CA is trusted when the engine's server trusts it (Linux: update-ca-certificates; Windows: the machine's store), or when the URL names it: &sslrootcert=/etc/ductus/tls/db-ca.pem (nuance). The engine's start log, ductus validate, ductus doctor and the web UI's Connections page all say how the connection is protected ("TLS 1.3, verified (sslmode=verify-full)"), and warn when a database on another machine isn't verified. A verified database's certificate is also watched: its expiry alerts at 30, 14 and 7 days like every other certificate the engine uses.

Managed Postgres (nuance):

• Azure: sslmode=verify-full. Its certificates chain to public roots servers already trust.
• Amazon RDS / Aurora: sslmode=verify-full&sslrootcert=/etc/ductus/tls/rds-global-bundle.pem, with Amazon's RDS CA bundle downloaded from AWS: its CAs aren't in the system's store.
• Google Cloud SQL: sslmode=verify-full&sslrootcert=… with the instance's server CA, downloaded from the console. Connect by a name its certificate has.

Always connect by the endpoint's DNS name, the one on its certificate, not its address.

A client certificate instead of (or beside) a password, where the DBA logs the engine in by certificate (cert in pg_hba.conf): &sslcert=/etc/ductus/tls/db-client.pem&sslkey=/etc/ductus/tls/db-client.key. A renewed pair is used by the next new connection (nuance).

A verified connection's certificate (verify-full, verify-ca) is watched like any other: alerts at 30, 14 and 7 days (nuance). Opening a connection over TLS costs about 60 ms for now, which pools hide (measured).

Sizing

The disk decides the speed. Every message acknowledged with durability: strict (the default) waits for one durable commit, so a sender that sends one message at a time can't go faster than the database's disk can flush: about 400 a second at 2.4 ms a flush. Commits from many senders at once are grouped, so the engine as a whole goes much faster. ductus doctor measures the commit time from the engine's server and says what it means (nuance). Put Postgres's write-ahead log on the fastest disk you have (NVMe, or a cloud disk with low write latency). For scale: a million HL7 messages a day averages 12 a second.

Space is roughly: messages a day × their average size × the days kept (retention: 7, 21 or 49 days; retention), plus a small row per destination per message (its delivery and status), the indexes, what destinations keep with save_output (about 1.15 GB per million deliveries, measured), and anything waiting or quarantined past its window (kept until it's resolved: the held area). Measure it on your own traffic after a week: SELECT pg_size_pretty(pg_database_size('ductus')). Space comes back as whole partitions are dropped, a day or a week at a time: there's nothing to vacuum away.

Long queries against the message tables (a report run on the store) can make the engine's hourly partition work wait; keep them short, or run them on a replica (nuance). Autovacuum's defaults are fine.

Backups are the DBA's

Back the database up as you back up any database that holds records you must not lose: with continuous WAL archiving and base backups (pgBackRest, Barman, or the managed service's own point-in-time restore). The engine doesn't take backups and needs nothing from them; nothing in it is scheduled outside the database.

What a backup means for messages:

• A restore to an earlier point loses what was acknowledged after it. Senders were told those messages arrived, so they won't send them again: get them resent from the source systems for the gap. The same holds for a failover to a replica that was behind (asynchronous replication): if no message may be lost, the DBA's answer is synchronous replication to the standby.
• The engine reconnects by itself after a restart or failover of the database. While it's cut off, nothing can be stored, so nothing is acknowledged and senders try again; when it can't hold its lock, it pauses sending until it can (nuance).
• Messages are kept 1, 3 or 7 weeks by design, not forever: longer history belongs in the source systems or an archive, not in backups of this database.

Ports and firewalls

What to open for Ductus, in each direction, and how to get the exact list for a firewall request. The short answer: in, one port for the web UI and API, and each interface's own listener; out, the database and whatever your interfaces connect to. Nothing else: no thick client, no extra ports for the web UI, no cluster port.

What connects to the engine (in)

The API and web UI listen on this machine only by default (api.listen: 127.0.0.1:7700), so opening them to the network is a choice you make. When you do, give them a certificate and an allow list, in ductus.yaml:

# ductus.yaml, in part
api:
  listen: 0.0.0.0:7700
  tls: { certificate: ductus-api }          # a certificates: entry in resources.yaml
  allow: [127.0.0.1, 10.1.50.0/24]          # this machine, and the operators' subnet

Without tls, passwords and tokens cross the network readable and the web UI's sign-in doesn't work at all (its cookie is only sent over HTTPS: nuance). Keep 127.0.0.1 in the list, or commands run on the server itself are refused (nuance). Developers need this port and whatever they deploy with (SSH, Git), nothing more.

Interfaces' listeners: each MLLP source has a port of its own; HTTP sources can share one, by path. ductus new interface picks a free one from the ranges in ductus.yaml (ports: MLLP 6661–6699, HTTP 8080–8099 by default). Ask the firewall team for a range once, set ports: to it, and new interfaces land inside it with no new request:

# ductus.yaml, in part
ports:
  mllp: 7000-7099     # the range the firewall team opened
  http: 9080-9099

The engine checks who may connect itself too: allow: on a source lists the senders' addresses, subnets or host names, and anyone else is disconnected before a byte is read (who may connect). Keep the firewall rules as well: each catches the other's mistakes (nuance).

A port below 1024 (MLLP on 104, say) needs a capability under the Linux package's service; the unit file says which two lines to add with systemctl edit ductus. Above 1024 needs nothing.

What the engine connects to (out)

The engine never connects anywhere to "phone home": no update checks, no telemetry.

The exact list for a firewall request

ductus ports prints every port a project listens on and every host it connects to, for the environment you name, with conflicts (two interfaces on one port) flagged:

For production, give it --env prod. Here, the project from Your first interface, with TLS:

$ ductus ports
environment: dev

Listening
  PORT  PROTOCOL       HOST       INTERFACE          ROUTE    TLS                                                            ALLOWED
  6661  MLLP over TLS  0.0.0.0    adt-in                      presents ductus-mllp, clients need a certificate from test-ca  anyone
  7700  HTTP           127.0.0.1  (engine REST API)  /api/v1  none                                                           anyone

Connects to
  (nothing)

✓ no port conflicts

ductus doctor, run on the engine's server, then checks each outbound connection from there: DNS, TCP (refused, or no answer at all, which is a firewall), TLS, and each system's login. For each one it prints the source address the connection left from: that is the address a firewall request must name, which on a server with several networks isn't always the one you'd guess. --report doctor.txt saves it as text for the network team or a vendor (doctor.json as JSON) (what a pass means).

Opening a range on the server itself

# RHEL, Rocky, Alma (firewalld)
sudo firewall-cmd --permanent --add-port=6661-6699/tcp && sudo firewall-cmd --reload
# Ubuntu (ufw)
sudo ufw allow from 10.20.30.0/24 to any port 6661:6699 proto tcp

# Windows
New-NetFirewallRule -DisplayName "Ductus MLLP" -Direction Inbound -Protocol TCP -LocalPort 6661-6699 -RemoteAddress 10.20.30.0/24 -Action Allow

Your first interface

From nothing to HL7 messages flowing over MLLP, then the same over TLS with client certificates, on your own machine, in about fifteen minutes. Every command here is real: run them as you read. You need the ductus program on your PATH (how) and, for the TLS part, openssl (it plays the hospital's certificate authority).

1. A new project

ductus init hospital
cd hospital
git init        # a project is a Git repository: what runs is what's committed

ductus init makes a project that passes its tests and validates as it is:

hospital/
├── ductus.yaml              the project's settings (all optional; the defaults written out)
├── resources.yaml           databases, servers and APIs interfaces share (none yet)
├── environments/
│   ├── dev.yaml             values for `--env dev` (the default)
│   └── prod.yaml            the same names, production's values
├── interfaces/
│   └── adt-in/              one interface: HL7 over MLLP on port 6661, written to files
│       ├── interface.yaml   where messages come from and go
│       ├── transform.ts     what happens to each one
│       ├── transform.test.ts
│       └── fixtures/sample.hl7
├── .ductus/, tsconfig.json, .vscode/settings.json    for your editor (below)
├── .gitignore, README.md

Open the folder in VS Code. With Red Hat's YAML extension, every setting in the YAML files completes as you type and shows its description on hover; in transform.ts, hovering msg.get("PID-5") shows PID-5 Patient Name. The schemas and typings in .ductus/ come from the ductus you have installed, so they match it (editor support).

2. The interface

interfaces/adt-in/interface.yaml, as generated (its comments show what else you could write):

# interfaces/adt-in/interface.yaml
name: adt-in
about:
  description: ""
  owner: ""
  from: { application: "", vendor: "", hosts: [] }   # hosts: where senders connect from (IPs or subnets)
  to: { application: "", vendor: "" }
source: { mllp: { port: 6661 } }
script: transform.ts
destination: { file: { dir: out/adt-in } }

• source: listen for HL7 over MLLP on port 6661. Each message is stored before its ACK goes back to the sender.
• script: run transform.ts on each message.
• destination: write each result as a file. The folder is relative to the interface's own folder, so the files land in interfaces/adt-in/out/adt-in/.
• about: what the interface is for, who owns it, and the systems on each side. Nothing here changes how messages flow; it's what ductus impact and the web UI read.

The code, transform.ts, gets each message and returns what to send:

import type { Message } from "@ductus/sdk";

export function transform(msg: Message) {
  // Example: set the receiving application.
  msg.set("MSH-5", "RECEIVER");
  return msg;
}

Returning null would drop the message (filtered, with the reason kept); throwing would quarantine it for a person to look at (nuance).

3. Test and validate

$ ductus test
adt-in
  ✓ sets the receiving application and keeps the patient (0ms)

1 passed, 0 failed

The tests run in the same embedded JavaScript engine the interfaces run in, with no database or network. Then check the whole project as the engine would read it:

$ ductus validate
environment: dev
✓ adt-in: config and code load
⚠ adt-in: about.from: add the sending system's `application` or `vendor`
⚠ adt-in: about.to: add the receiving system's `application` or `vendor`
  (warnings: `ductus impact` can't find what isn't named; `require_about: true` in ductus.yaml makes them errors)
note: not checked against a database (set DUCTUS_DATABASE_URL): lanes that would strand waiting messages, retention changes
✓ types: no errors

valid

The warnings are about about:, which is still empty. Fill it in: the day a vendor has an outage or a breach, ductus impact --vendor "Epic Systems" lists every interface it touches, but only the ones that say so.

# interfaces/adt-in/interface.yaml
name: adt-in
about:
  description: ADT from the EHR, kept as files for the lab's archive
  owner: interfaces@hospital.org
  from: { application: Epic, vendor: Epic Systems, hosts: [10.20.30.0/24] }
  to: { application: Lab archive, vendor: In-house }
source: { mllp: { port: 6661 } }
script: transform.ts
destination: { file: { dir: out/adt-in } }

ductus validate now has no warnings.

4. Run it

$ ductus dev
Postgres 18.6.0 is running for this project:
  postgresql://postgres:…@127.0.0.1:42731/ductus
  (data in ./.ductus/postgres/data; stops with the engine)
Web UI login: user `dev`, password `…` (kept in ./.ductus/dev-login)
… INFO ductus_runtime::api: REST API listening (/api/v1) host=127.0.0.1 port=7700 https=false
… INFO ductus_runtime: started interfaces=1 mllp=1 http=0 files=0 schedules=0 internal=0 …

ductus dev is the engine with a Postgres of its own (downloaded once, the first time: more). In a second terminal, send it ten synthetic ADT messages, as an EHR would:

$ ductus gen --count 10
sent 10 messages over 1 connection(s) in 0.0s (297/s), retries 0, rejected 0, refused 0, dropped 0, longest wait 23 ms

Each one was stored before its ACK went back ("longest wait" is the slowest ACK). Look at what arrived:

ls interfaces/adt-in/out/adt-in/       # 1.hl7 … 10.hl7, named by message id

and open the web UI at http://127.0.0.1:7700, signing in with the dev login ductus dev printed. The interface's Messages tab lists the ten, and each message shows its HL7 fields by name, what was received, and what happened at each destination.

5. Change it

Make the code do something of its own: say the lab wants patient names in capitals. In transform.ts:

import type { Message } from "@ductus/sdk";

export function transform(msg: Message) {
  msg.set("MSH-5", "LAB");
  msg.set("PID-5.1", msg.get("PID-5.1").toUpperCase());   // PID-5.1: family name
  return msg;
}

and say so in its test, transform.test.ts:

import { expect, input, type Message, test } from "@ductus/sdk";
import { transform } from "./transform.ts";

test("addresses the lab and capitalizes the family name", () => {
  const out = transform(input<Message>("sample.hl7"));
  expect(out.get("MSH-5")).toBe("LAB");
  expect(out.get("PID-5.1")).toBe("SMITH");
  expect(out.get("PID-3.1")).toBe("123456");
});

ductus test, then stop ductus dev (Ctrl-C) and start it again: it starts in well under a second, and new messages get the new code. HL7 is numbered as the standard numbers it, from 1, and a field that isn't there reads as "", never an error (nuance).

6. The same over TLS, with client certificates

MLLP crossing a network you don't own should be encrypted, and the listener should know which systems may send. That's two settings: a certificate the listener presents, and the CA whose certificates it takes from senders.

A certificate authority for the lab. In real life the hospital's PKI team signs your certificate. Here, openssl plays that part (keep its folder out of Git: ca.key signs anything, and the server will trust the hospital's CA, not this one):

mkdir -p lab-ca tls
echo "lab-ca/" >>.gitignore
openssl req -x509 -newkey rsa:2048 -nodes -keyout lab-ca/ca.key -out lab-ca/ca.pem -days 365 -subj "/CN=Test Hospital CA"

The listener's certificate. ductus cert request makes a private key that never leaves this machine, and a request (CSR) to send to the CA. It says what to send and what comes back:

$ ductus cert request tls/mllp --dns localhost --ip 127.0.0.1
wrote tls/mllp.key (the new RSA 2048 private key: it stays on this server; only you can read it) and tls/mllp.csr (the request)

Send tls/mllp.csr to the hospital's CA (its PKI or security team). It asks for a certificate for localhost, 127.0.0.1, for a TLS server and client (one certificate can serve listeners and be the engine's client certificate). Never send tls/mllp.key.
…

Sign it as the CA would, and do the same for a sender (in real life, the sending system's team asks their CA, or yours, for theirs):

openssl x509 -req -in tls/mllp.csr -CA lab-ca/ca.pem -CAkey lab-ca/ca.key -CAcreateserial -out tls/mllp.pem -days 90 -copy_extensions copy
ductus cert request tls/sender --dns sender.test
openssl x509 -req -in tls/sender.csr -CA lab-ca/ca.pem -CAkey lab-ca/ca.key -CAcreateserial -out tls/sender.pem -days 90 -copy_extensions copy

Name them in resources.yaml, so interfaces refer to them by name:

# resources.yaml
certificates:
  ductus-mllp: { cert_file: tls/mllp.pem, key_file: tls/mllp.key }
  test-ca: { ca_file: lab-ca/ca.pem }

certificates holds both kinds: a pair the engine presents (ductus-mllp), and a CA it trusts or checks clients against (test-ca). Ask ductus cert show what the engine will make of one, before a sender finds out:

$ ductus cert show ductus-mllp
✓ certificate `ductus-mllp` (resources.yaml): ./tls/mllp.pem
  subject  CN=localhost
  names    localhost, 127.0.0.1
  issuer   issued by Test Hospital CA
  valid    2026-10-31 → 2027-01-29: 89 days left
  chain    complete: localhost, issued by Test Hospital CA (the CA `test-ca` in resources.yaml)
  key      matches (./tls/mllp.key)
→ Senders must trust Test Hospital CA (the hospital's CA): ask them to confirm.

(Shown the file alone, ductus cert show tls/mllp.pem warns that the chain is incomplete: it can't know Test Hospital CA is a root until resources.yaml names it. With a real CA, append its intermediate certificates to mllp.pem after yours: why.)

Turn TLS on for the listener, and ask senders for a certificate from test-ca naming sender.test:

# interfaces/adt-in/interface.yaml
name: adt-in
about:
  description: ADT from the EHR, kept as files for the lab's archive
  owner: interfaces@hospital.org
  from: { application: Epic, vendor: Epic Systems, hosts: [10.20.30.0/24] }
  to: { application: Lab archive, vendor: In-house }
source:
  mllp:
    port: 6661
    tls:
      certificate: ductus-mllp
      clients: { ca: test-ca, subjects: ["CN=sender.test"] }
script: transform.ts
destination: { file: { dir: out/adt-in } }

ductus validate checks the certificates as the engine's start will (a certificate it can't use stops the start, with the reason), and ductus ports shows the port is TLS now:

$ ductus ports
…
  PORT  PROTOCOL       HOST       INTERFACE          ROUTE    TLS                                                            ALLOWED
  6661  MLLP over TLS  0.0.0.0    adt-in                      presents ductus-mllp, clients need a certificate from test-ca  anyone

Send. Start ductus dev again, then send as a sender with its certificate, trusting the lab CA for the engine's:

$ ductus gen --count 10 --to localhost:6661 --ca lab-ca/ca.pem --cert tls/sender.pem --key tls/sender.key
sent 10 messages over 1 connection(s) in 0.1s (79/s), retries 0, rejected 0, refused 0, dropped 0, longest wait 110 ms

And see what other senders get. One without TLS (plain ductus gen --count 1) sees its connection closed with no ACK, and keeps retrying as a real sender would (Ctrl-C to stop it):

sending failed, retrying: connection closed before ACK

while the engine's log says why, once a minute per sender address (nuance):

WARN ductus_runtime::tls: TLS handshake refused (logged once a minute per sender) listener="interface `adt-in` (MLLP)" peer=127.0.0.1 reason=the sender isn't speaking TLS (a plain MLLP or HTTP sender on a TLS port?)

One with TLS but no certificate of its own (--ca lab-ca/ca.pem without --cert) is told TLS: the receiver wants a client certificate (name one in --cert).

The certificate is watched from now on: the engine alerts 30, 14 and 7 days before it expires, and a renewal is two files replaced, with no restart (renewing).

7. To a server

What you built is a folder of files: commit it, and the server runs what's committed.

git add -A && git commit -m "adt-in: ADT from Epic to the lab archive"

.gitignore already leaves out what mustn't be in Git (.ductus/, *.key). For the server:

1. Install the engine there, and give it a database.
2. Clone the project as the service's user, and set DUCTUS_PROJECT and DUCTUS_ENV=prod.
3. Certificates on the server: run ductus cert request there, so the key never leaves it, with the name senders connect to (--dns mllp.hospital.org), and send the CSR to the hospital's CA. Paths that differ per server belong in the environment (environments), and the CA is the hospital's, not lab-ca.
4. ductus doctor on the server, then start the service.

The Configure pages go through every file in turn.

On a new server, start to finish

What: the steps above on a fresh Ubuntu 24.04 server, with the Linux package, a database on another machine (db1.hospital.org), MLLP over TLS with client certificates, and a sending system that has its own certificate. Every command and output below is from the v0.6 demo (scripts/demo-v06.sh), which runs them under real systemd on every push to the repository; only what's marked … is left out.

Why: a laptop hides what a server adds: a service account that owns the files, a database elsewhere, a CA that isn't yours, and senders that must be told what to trust. Seeing it once end to end shows where each of those comes in.

When not: this builds the project on the server, to show every step in one place. In real life the project comes from Git (step 2 above), and only the certificate's key is made on the server.

Install, then tell the engine where its database is. The package doesn't start the engine:

$ cd /tmp
$ sudo apt install ./ductus_0.6.0~dev.1_amd64.deb
…
Unpacking ductus (0.6.0~dev.1) ...
Setting up ductus (0.6.0~dev.1) ...

Ductus is installed. It isn't running yet: first tell it where its database and project are.
  1. Edit /etc/ductus/ductus.env (the database URL, the project folder, the environment).
  2. Put the project in /var/lib/ductus/project (git clone), readable by the ductus user.
  3. sudo systemctl enable --now ductus
  4. journalctl -u ductus -f      (it says "started" once every interface is up)

(0.6.0~dev.1 is the demo's development package; a release is ductus_0.6.0_amd64.deb.) In /etc/ductus/ductus.env the demo sets one line, the database; DUCTUS_ENV=prod and DUCTUS_PROJECT=/var/lib/ductus/project are already the package's defaults (the file):

# /etc/ductus/ductus.env, in part
DUCTUS_DATABASE_URL=postgres://ductus:…@db1.hospital.org/ductus

The project, made as the ductus user so the service owns it (on a real server: sudo -u ductus git clone … instead), and the listener's certificate, its key made on this server:

$ cd /var/lib/ductus
$ sudo -u ductus ductus init project
created interfaces/adt-in/ (MLLP on port 6661)
…
$ cd /var/lib/ductus/project
$ sudo -u ductus ductus cert request tls/mllp --dns mllp.hospital.org
wrote tls/mllp.key (the new RSA 2048 private key: it stays on this server; only you can read it) and tls/mllp.csr (the request)

Send tls/mllp.csr to the hospital's CA (its PKI or security team). It asks for a certificate for mllp.hospital.org, for a TLS server and client (one certificate can serve listeners and be the engine's client certificate). Never send tls/mllp.key.
…

The hospital's PKI team signs the request and sends back the certificate and their CA's own. In the demo openssl plays the CA, on the same machine:

$ openssl x509 -req -in tls/mllp.csr -CA /root/hospital-ca/ca.pem -CAkey /root/hospital-ca/ca.key -CAcreateserial -out tls/mllp.pem -days 90 -copy_extensions copy
Certificate request self-signature ok
subject=CN = mllp.hospital.org
$ cp /root/hospital-ca/ca.pem tls/hospital-ca.pem

The certificates by name, and the interface sending to the lab system's receiver instead of files (a service writes only where its unit lets it):

# resources.yaml
certificates:
  ductus-mllp: { cert_file: tls/mllp.pem, key_file: tls/mllp.key }
  hospital-ca: { ca_file: tls/hospital-ca.pem }

# interfaces/adt-in/interface.yaml
name: adt-in
about:
  description: ADT from the EHR to the lab system
  owner: interfaces@hospital.org
  from: { application: Epic, vendor: Epic Systems }
  to: { application: Lab system, vendor: In-house }
source:
  mllp:
    port: 6661
    tls:
      certificate: ductus-mllp
      clients: { ca: hospital-ca, subjects: ["CN=epic.hospital.org"] }
script: transform.ts
destination: { mllp: { host: 127.0.0.1, port: 6662 } }

Check before starting, with the service's own settings (--env-file reads ductus.env as the service does), then start it and let ductus doctor check every connection from the server:

$ sudo -u ductus ductus cert show ductus-mllp
✓ certificate `ductus-mllp` (resources.yaml): ./tls/mllp.pem
  subject  CN=mllp.hospital.org
  names    mllp.hospital.org
  issuer   issued by Test Hospital CA
  valid    2026-10-31 → 2027-01-29: 89 days left
  chain    complete: mllp.hospital.org, issued by Test Hospital CA (the CA `hospital-ca` in resources.yaml)
  key      matches (./tls/mllp.key)
→ Senders must trust Test Hospital CA (the hospital's CA): ask them to confirm.
$ sudo -u ductus ductus validate --env-file /etc/ductus/ductus.env
environment: prod
✓ resources: 0 database(s), 0 SFTP server(s), 0 API(s), 2 certificate(s)
✓ adt-in: config and code load
✓ certificate `ductus-mllp`: CN=mllp.hospital.org (mllp.hospital.org), expires 2027-01-29 (89 days)
✓ certificate `hospital-ca`: a CA
⚠ database: plain, no TLS: the server doesn't offer it (sslmode=prefer); whoever answers at its address is believed: add sslmode=verify-full to DUCTUS_DATABASE_URL (sslrootcert=/path/ca.pem for a private CA)
✓ database: new (no engine has run on it yet): no messages wait, no retention is applied
✓ types: no errors

valid
$ sudo systemctl enable --now ductus
Created symlink /etc/systemd/system/multi-user.target.wants/ductus.service → /usr/lib/systemd/system/ductus.service.
$ systemctl show -p StatusText ductus
StatusText=running 1 interface
$ sudo -u ductus ductus doctor --env-file /etc/ductus/ductus.env
ductus doctor · environment prod · 2026-10-31 23:46:41 +00:00
on ductus1 · ductus 0.6.0-dev
checked 3 connection(s) in 0.1 s: 1 ok, 2 warning(s), 0 failed

⚠ engine database  db1.hospital.org:5432 Postgres
    used by  the engine (DUCTUS_DATABASE_URL)
    DNS      db1.hospital.org → 172.18.0.2 (0.9 ms)
    connect  172.18.0.2:5432 from 172.18.0.3 in 0.2 ms
  ⚠ TLS      none: the server doesn't offer TLS (sslmode=prefer), the connection is plain
    login    as `ductus`, database `ductus`: PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2); plain, no TLS: the server doesn't offer it (sslmode=prefer)
    commits  2.2 ms median of 20 (each waits for the disk): strict ingest ≈ 450 msg/s per sender connection, one commit per ACK; more connections share commits. A 1M/day ADT feed averages ~12 msg/s.
  → Queries, their results and the login cross the network readable: ask the DBA to turn on ssl, then set sslmode=verify-full.

✓ MLLP destination  127.0.0.1:6662 MLLP
    used by  adt-in
    address  127.0.0.1 (an address, nothing to look up)
    connect  127.0.0.1:6662 from 127.0.0.1 in 0.3 ms

⚠ listening port  0.0.0.0:6661 MLLP over TLS
    used by  adt-in
  ⚠ port     in use: by a running engine, or another program
    address  127.0.0.1 (an address, nothing to look up)
    connect  127.0.0.1:6661 from 127.0.0.1 in 0.2 ms
    TLS      TLS 1.3, post-quantum key exchange (X25519MLKEM768), serves `ductus-mllp`: certificate for mllp.hospital.org, issued by Test Hospital CA, valid until 2027-01-29 (89 days); covers mllp.hospital.org
    chain    complete: mllp.hospital.org, issued by Test Hospital CA (the CA `hospital-ca` in resources.yaml)
  → If the engine isn't running on this server, another program holds port 6661: find it with `ss -ltnp` (Linux) or `netstat -ano` (Windows).
  → Senders must trust Test Hospital CA (the hospital's CA): ask them to confirm.
  → Doctor can't know the name senders connect with: it must be one of mllp.hospital.org (ask senders which name or address they use).

Doctor passes (nothing failed: it exits 0) with two warnings to read, not ignore. The database connection isn't encrypted: the demo's database server offers no TLS, and a real one should (PostgreSQL); the warning goes once DUCTUS_DATABASE_URL says sslmode=verify-full. Port 6661 is in use: here by the engine itself, which doctor can't tell from another program; run before the start, the same check says whether the port is free.

The sending system gets its own certificate from the same CA (in real life its team asks for it), and sends:

$ ductus cert request /root/epic/sender --dns epic.hospital.org
…
$ openssl x509 -req -in /root/epic/sender.csr -CA /root/hospital-ca/ca.pem -CAkey /root/hospital-ca/ca.key -CAcreateserial -out /root/epic/sender.pem -days 90 -copy_extensions copy
Certificate request self-signature ok
subject=CN = epic.hospital.org
$ ductus gen --to mllp.hospital.org:6661 --ca /root/hospital-ca/ca.pem --cert /root/epic/sender.pem --key /root/epic/sender.key --count 100
sent 100 messages over 1 connection(s) in 0.6s (156/s), retries 0, rejected 0, refused 0, dropped 0, longest wait 139 ms

The lab system's receiver (ductus sink in the demo) got all 100, once each, in order. A sender without a certificate of its own is refused before anything is read, and the engine's log says why, once a minute per sender:

$ timeout 5 ductus gen --to mllp.hospital.org:6661 --ca /root/hospital-ca/ca.pem --count 1
sending failed, retrying: TLS: the receiver wants a client certificate (name one in --cert)
$ journalctl -u ductus -o cat --no-pager | grep "TLS handshake refused" | tail -1
2026-10-31T23:47:03.775177Z  WARN ductus_runtime::tls: TLS handshake refused (logged once a minute per sender) listener="interface `adt-in` (MLLP)" peer=127.0.0.1 reason=the sender presented no client certificate

The demo then upgrades this server and goes back, both under traffic: Upgrade shows it.

The project

Everything an engine runs is in one folder of plain files, the project: its interfaces, their code and tests, and a few files of settings. It lives in Git, so every change is reviewed, recorded and can be undone; the engine runs what's in the folder, and never writes there.

Why files and Git rather than settings in a database behind a GUI: a change is a diff someone reviews before it goes live, the history says who changed what and why, the same project runs in test and production with only its environment's values differing, and rolling back is a git revert. Nothing about an interface exists only inside a running engine.

The layout

hospital/
├── ductus.yaml                    the project's settings: retention, the API, sign-in, alerts
├── resources.yaml                 what interfaces share, by name: databases, servers, APIs, certificates
├── docs-schema.yaml               your company's own documentation fields (optional)
├── environments/
│   ├── dev.yaml                   each environment's values: hosts, ports, paths
│   ├── test.yaml
│   └── prod.yaml
├── secrets/
│   ├── dev.json                   each environment's secrets, encrypted (committed)
│   └── prod.json
├── interfaces/
│   ├── adt-in/
│   │   ├── interface.yaml         one interface: where messages come from and go
│   │   ├── transform.ts           its code (optional)
│   │   ├── transform.test.ts      its tests
│   │   └── fixtures/              sample messages the tests read
│   └── orders-to-lab/
│       └── …
└── lib/                           code several interfaces import (optional; any name)

ductus init makes a new project with all of these but docs-schema.yaml and secrets/, and one interface; ductus new interface <name> --source mllp|http|file|sftp|schedule adds another, with its code, a test and a sample message, on a free port.

How the engine reads it

At start, the engine reads ductus.yaml, then the environment it was started with (DUCTUS_ENV, or --env), fills every ${NAME} and ${secret:NAME} in from it, reads resources.yaml, then every interfaces/*/interface.yaml, and loads each interface's code. Any mistake stops the start with the file and what's wrong: a typo in a setting's name too, since every file refuses keys it doesn't know. The engine doesn't read its configuration again while it runs (only certificate files, every minute, so a renewal needs no restart); a change takes effect at the next start, or with systemctl reload ductus on Linux (deploying).

ductus validate reads it the same way without starting anything, for an environment (--env prod), and also type-checks every interface's code with the TypeScript compiler built into the engine. Run it before every commit, and in CI; with DUCTUS_DATABASE_URL set it also checks the project against the database, reading only. ductus test runs every interface's tests.

In your editor

ductus types (which ductus test and ductus validate also run) writes what VS Code needs into the project:

• .ductus/types/: the SDK's typings, so transform.ts knows Message, db(), api() and every HL7 field's name, and tsconfig.json pointing at them;
• .ductus/schema/: a JSON Schema for each configuration file, and .vscode/settings.json mapping them to the files (unless it already maps them elsewhere).

With Red Hat's YAML extension, every setting then completes as you type, shows its description (the same text as the reference) on hover, and is underlined when it's wrong. They come from the ductus you have installed, so they match the engine that will run the project: run ductus types again after an upgrade. .ductus/ isn't committed (the .gitignore from ductus init leaves it out).

Values written once

Two rules keep a project readable as it grows:

• Anything that differs between environments is a variable (${LAB_HOST}), defined in each environments/<env>.yaml. The interface itself is the same file in test and production.
• Anything an interface connects to that others might too is a resource, by name: code says db("patient_db"), a destination says api: billing, a certificate is ductus-mllp. When the database moves, one line changes, and ductus impact can say which interfaces it touches.

Environments, variables and secrets

One project runs in every environment (a developer's laptop, test, production) with only its environment's values differing: the hosts and ports it connects to, the folders it uses, and its passwords and tokens. Interfaces never say which environment they're in.

The engine uses one environment, the one it's started with: DUCTUS_ENV=prod (the packages' settings file sets it), or --env prod on the command line. Without either it's dev.

Variables

Each environment's file holds its variables, by name:

# environments/prod.yaml
variables:
  EPIC_PORT: 6661
  LAB_HOST: lab.hospital.org
  LAB_PORT: 7001
  PATIENT_DB_HOST: patients-db.hospital.org
  BILLING_DB_HOST: billing-sql.hospital.org
  EMPI_URL: https://empi.hospital.org/api/v2
  QUEST_SFTP_HOST: sftp.quest.example.com
  OUTBOUND_DIR: /data/outbound
  TLS_DIR: /etc/ductus/tls

# environments/dev.yaml
variables:
  EPIC_PORT: 6661
  LAB_HOST: 127.0.0.1
  LAB_PORT: 7001
  PATIENT_DB_HOST: localhost
  BILLING_DB_HOST: localhost
  EMPI_URL: http://127.0.0.1:8090/api/v2
  QUEST_SFTP_HOST: 127.0.0.1
  OUTBOUND_DIR: out
  TLS_DIR: tls

Then any value in interface.yaml, resources.yaml, or the alerts: section of ductus.yaml can be a variable:

# interfaces/orders-to-lab/interface.yaml
name: orders-to-lab
source: { mllp: { port: ${EPIC_PORT} } }
destination: { mllp: { host: ${LAB_HOST}, port: ${LAB_PORT} } }

• A value that is only a reference keeps the variable's type: port: ${LAB_PORT} is the number 7001. Inside text it's text: url: "${EMPI_URL}/patients".
• Give every environment the same names. ductus validate checks the environment you name, and a variable missing from it stops the start, naming it: run ductus validate --env prod before deploying to production.
• $${ writes a literal ${, for the rare value that needs one.
• Not in the rest of ductus.yaml: only its alerts: section is read per environment (mail servers and webhooks differ; retention and sign-in rules shouldn't).
• Not in code: an interface's TypeScript reads the environment's values with env("NAME") and its secrets with secret("NAME"), from the same files.

When not to use a variable: for what is the same everywhere (an interface's port, when every environment uses the same one, is clearer written out), and never for secrets: environment files are plain text in Git.

A number in quotes is text. LAB_PORT: "6670" makes port: ${LAB_PORT} the text "6670", which the interface refuses ("invalid type: string"); write LAB_PORT: 6670 (nuance).

Secrets

Secrets are encrypted, one value at a time, in secrets/<env>.json, and committed to Git encrypted: the repository has them, readable only with the environment's key, which never is. That way a secret changes with a commit and a review like everything else, and a cloned project has all it needs except the key.

Make a key for each environment, once:

ductus secret init          # prints a new random key

Keep it where your other production secrets live (a vault, a password manager), and give it to the engine as DUCTUS_SECRET_KEY: in /etc/ductus/ductus.env on Linux, C:\ProgramData\Ductus\ductus.env on Windows. Use a different key for each environment, so a developer with the test key can't read production's secrets.

Set a secret (the value is read from the terminal, so it stays out of your shell's history):

export DUCTUS_SECRET_KEY=…              # the prod key
ductus secret set PATIENT_DB_PASSWORD --env prod
ductus secret list --env prod           # names only, never values

and use it anywhere a variable can go:

# resources.yaml
databases:
  patient_db:
    postgres:
      host: ${PATIENT_DB_HOST}
      database: patients
      user: ductus
      password: ${secret:PATIENT_DB_PASSWORD}
      tls: verify-full

• What belongs in a secret: database and SFTP passwords, SFTP private keys, API tokens (Authorization: "Bearer ${secret:EMPI_TOKEN}"), the directory's service password, the mail server's password, chat webhook URLs (a webhook's URL is its own password), the heartbeat's URL, certificates and keys kept in a vault rather than in files.
• Where values show: ductus secret list gives names only, and the engine shows a webhook's URL by its scheme and host only. Anyone who can run ductus on the server with the key can read the values, so the key is what to protect.
• Without the key, a project that uses a secret can't be read for that environment: the engine's start and ductus validate stop and say which secret is missing.
• Rotating a secret: ductus secret set the new value, commit, deploy (systemctl reload ductus on Linux). Rotating the key means setting every secret again with the new key.

Choosing environment names

dev (the default), test and prod cover most teams; any name works (staging, dr-site). Each name needs its environments/<name>.yaml (the engine refuses an environment without one, once the folder exists) and, if it uses secrets, its secrets/<name>.json and key. A disaster-recovery site that connects to different hosts is its own environment.

ductus.yaml: the project's settings

ductus.yaml sits at the top of the project and holds what applies to every interface in it and to the engine that runs them: how long messages are kept, where the web UI listens, how people sign in, the audit log, alerts, and the engine's own checks. It's optional: every setting has a default, and ductus init writes the main ones out so you can see them.

# ductus.yaml
ports:
  mllp: 6661-6699
  http: 8080-8099
retention: 7w
quarantined: keep
api:
  listen: 127.0.0.1:7700
isolates:
  heap_mb: 512
require_about: false

That is ductus init's file, and also exactly the defaults. Every setting is in the reference; this page says what each section is for and when to change it.

Only the alerts: section may use ${NAME} and ${secret:NAME} (environments): mail servers and webhooks differ per environment, but how long messages are kept and who may sign in shouldn't.

Ports for new interfaces

ports is where ductus new interface picks a new interface's port from: the first free one in the range. It doesn't limit anything: each interface's port is written in its own interface.yaml, and the engine listens on whatever that says. Set it to the ranges your firewall team opened, so new interfaces land inside them with no new request (ports and firewalls). Default: MLLP 6661-6699, HTTP 8080-8099.

How long messages are kept

retention is how long every interface keeps its messages, unless its interface.yaml says otherwise: 1w, 3w or 7w (the default), or the same in days (7d, 21d, 49d).

# ductus.yaml, in part
retention: 3w
quarantined: keep

• Why only three: each window is a set of partitions of the message tables, and old messages go by dropping a whole partition at once, which costs nothing however busy the feed. Deleting rows one at a time, as engines with any number of days must, slows the busiest tables and bloats them (nuance).
• Kept at least the window, and gone within a day (1w) or a week (3w, 7w) after it.
• Nothing waiting is ever deleted. A message still waiting for a destination past the window moves to the held area and is sent when it can be (nuance).
• Choose the window that covers how far back your team looks into problems: 1w for busy feeds nobody looks back on, 7w where problems surface weeks later (billing, a monthly reconciliation). Longer history belongs in the source system or an archive, not here.
• Shortening it later deletes messages, so the engine waits for ductus retention confirm <interface> before it applies; a longer window applies at once (nuance).

quarantined says what happens to quarantined messages (ones a destination refused for good, or the code rejected) once the window has passed: keep (the default) holds them in the held area until a person resends, corrects or dismisses them; purge lets them go with the rest. Keep is the safe choice: a quarantined message is a question nobody has answered yet. purge suits a feed whose sender resends anyway. Watch what's held with ductus retention and the ductus_held_messages metric.

The web UI and the API

api is where the REST API and the web UI listen (one port for both), whether over HTTPS, and who may connect. By default only this machine can reach them: 127.0.0.1:7700, plain HTTP.

# ductus.yaml, in part
api:
  listen: 0.0.0.0:7700
  tls: { certificate: ductus-api }
  allow: [127.0.0.1, 10.1.50.0/24]

• listen: host:port, or off to run without the API and the web UI (then the command line's status --token, drain and pause can't reach this engine either). Listen on the server's address, or 0.0.0.0, for people to use the web UI from their own computers. DUCTUS_API_LISTEN (or --api-listen) overrides it for one engine.
• tls: the certificate it presents, a certificates: entry in resources.yaml. Needed as soon as listen isn't this machine only: the web UI's sign-in cookie is only sent over HTTPS, and tokens would otherwise cross the network readable. The start and ductus validate warn without it (TLS for the web UI).
• allow: the addresses, subnets and host names that may connect (operators', developers', the monitoring server). List 127.0.0.1 too, or commands on the server itself are refused.

Every call needs a token or a signed-in person either way: the network settings decide who can reach the port, not who may do what (people).

By default the API and the console listen on this machine only. Anyone else reaches them once you set api.listen to an address on the network, and then they need tls (nuance).

Signing in

auth sets how people sign in to the web UI and the API: how long a session lasts (12 hours idle, 7 days at most), the shortest password (12 characters), how many wrong passwords lock a username (5 in 15 minutes, for 15 minutes), whether people may make their own API tokens, and a directory (Active Directory, OpenLDAP) to sign in with. The defaults suit most sites; People: users, tokens, the directory explains each, and when to change it.

The audit log

audit is how long the record of who did what is kept (sign-ins, resends, edits, pauses, account changes, payload reads) and whether reading a message's payload is recorded.

# ductus.yaml, in part
audit:
  keep: 6y              # the default: HIPAA's usual documentation period
  payload_reads: true   # the default: who looked at patient data

• keep: at least a year, in years (6y) or any duration. Six years is the usual HIPAA documentation period; ask your compliance office for yours. Older rows are deleted hourly.
• payload_reads: with patient data, who looked matters as much as who changed. Turn it off only where messages hold no personal data. Reading needs the payloads permission either way (nuance).

The audit log and the messages it names are kept for different times: the log for audit.keep (six years), a message for its interface's retention, so an audit row can outlive the payload it mentions (nuance). An engine that crashed shows in the log as a start whose previous run never stopped (nuance).

Alerts and owners

alerts says where notifications go (a mail server, channels: email lists and chat webhooks), which health states alert and how badly, and when to remind and to stay quiet. owners gives the address of owners that aren't accounts (a team's mailing list, by the name interfaces give). Nothing alerts until you write rules: Alerts and owners is the whole story.

The engine's own connection checks

doctor is what the engine checks by itself, as ductus doctor would: every six hours, only as far as each server's certificate (DNS, TCP, the TLS handshake; nothing logs in). That's what makes certificates alert 30, 14 and 7 days before they expire, including the ones of servers the engine connects to.

# ductus.yaml, in part
doctor:
  schedule: 6h          # the default; `off` to turn it off
  mode: certificates    # the default; `full` also logs in

• mode: full also logs in to each database, SFTP server and the directory, so a changed password is noticed before a message needs it. Use it where the servers' owners agree to a login every schedule (a service account's logins show in their logs).
• Turn it off only if something else watches every certificate the engine connects to.
• The first check runs a minute or two after the engine starts, at a random moment, so engines started together don't check together (nuance).

Memory for interface code

isolates.heap_mb is the memory limit, in MB, of each copy of the JavaScript engine that runs interface code (an isolate). Default: 512.

There is one isolate per thread of the shared pool (one thread per CPU core, or DUCTUS_THREADS), and one per worker of each interface with isolation: dedicated. So interface code can use up to about heap_mb × (threads + dedicated workers) in all; most interfaces use a few MB, so that's a ceiling, not what you'll see.

• Code that fills its isolate is stopped, its message quarantined and its interface stopped until a restart. The engine and every other interface keep running.
• In the shared pool the limit is shared by every interface on the thread, so the one that runs out isn't always the one that filled it: a cache in another interface that only grows (nuance, what you keep is what costs).
• Raise it for code that holds a lot on purpose (a large lookup table loaded once), or give that interface isolation: dedicated with its own isolates. Lower it on a small server with many cores, so the ceiling fits in its memory.

Insisting on documentation

require_about makes ductus validate fail, not only warn, when an interface doesn't name the application or vendor on each side in its about:. That's what lets ductus impact --vendor "Epic Systems" answer "what does this vendor's outage touch?" completely. Turn it on once the existing interfaces are documented, so every new one must be. Default: false.

resources.yaml: what interfaces share

resources.yaml defines, once and by a name, everything interfaces connect to that isn't a source or a destination of their own: the databases their code queries, SFTP servers, HTTP APIs, directories people sign in with, and certificates. Interfaces and their code use the name:

const row = await db("patient_db").one("SELECT mrn FROM patients WHERE id = $1", [id]);
const patient = await api("empi").get(`/patients/${mrn}`);

Why names: no address, login or password is ever written in an interface or its code. The code is the same in every environment; resources.yaml fills in each environment's host with ${NAME} and its password with ${secret:NAME} (environments). The engine logs in on the code's behalf, so the code never sees a credential. And because every connection is named, ductus doctor can check each one from the server, ductus impact can say which interfaces use it, and its certificate is watched for expiry.

When not: an interface's own source and destinations (an MLLP receiver, a folder) are in its interface.yaml, not here, unless several share them: an SFTP server and an HTTP API are always resources, so their login is in one place.

Every entry of every kind also takes application, vendor and owner (who to ask, best an email address: its certificate's expiry alerts reach them with to: owner), for ductus impact and alerts, and custom:, your own documentation fields.

Databases

databases are the databases interfaces' code queries with db("<name>"): lookups, inserts, a nightly extract. Each is postgres or sqlserver.

# resources.yaml
databases:
  patient_db:
    postgres:
      host: ${PATIENT_DB_HOST}
      database: patients
      user: ductus_lookup
      password: ${secret:PATIENT_DB_PASSWORD}
      tls: verify-full
      max_connections: 5
      application: Patient index
      owner: dba@hospital.org
  billing_db:
    sqlserver:
      host: ${BILLING_DB_HOST}
      database: Billing
      user: ductus
      password: ${secret:BILLING_DB_PASSWORD}
      application: Billing
      vendor: Cerner

• The login: give it only what the queries need, often read-only on a few tables or views. A wrong password makes messages wait and retry rather than quarantine: fixing it fixes them all (nuance).
• Connections: each entry has a pool, shared by every interface's calls, of up to max_connections (default 5), opened as needed. A call that finds them all busy waits up to 10 s, then fails as a temporary error (the message waits and is tried again). Raise it when calls queue while the database itself is quick, within what its DBA allows for every engine; don't raise it to cure slow queries. A busy interface with isolation: dedicated can have connections of its own (dedicated_pools).
• Postgres TLS works as for the engine's own database: tls: verify-full checks the server's certificate against this server's trusted CAs (ca: names a private one); without tls, it's prefer, which checks nothing (PostgreSQL's TLS).
• SQL Server is always TLS, verified: encrypt: mandatory by default, strict (TDS 8) for servers set to force it; there's no unencrypted mode. ca: for the domain's CA where this server doesn't trust it; name_in_certificate when connecting by an alias or a listener's name (nuance). A named instance is instance: PROD: its port comes from the SQL Server Browser (UDP 1434, which the firewall must let through).
• SQL Server Windows logins (auth: windows), where the DBA allows only those: without user, the engine logs in as the Windows service's own account (a gMSA is best: no password anywhere); with user and password, as that domain account. From Linux that is always NTLM, which some servers refuse (nuance, from Linux).

# resources.yaml
databases:
  billing_db:
    sqlserver:
      host: ${BILLING_DB_HOST}
      instance: PROD
      database: Billing
      auth: windows          # as the service's account: on Windows, best a gMSA

What code can do with one, and what to watch for: one result set per call on SQL Server, a query that writes is a write a crash can repeat, write names out (db("patient_db"), not db(name), so ductus impact can see it) (databases and APIs, running twice).

SQL Server, worth knowing before the first connection:

• encrypt: strict is TDS 8: TLS first, before anything else is said (nuance). SQL Server 2022 on Linux refuses it whatever it's given; 2025 accepts it (nuance).
• Connecting by an alias, a listener or an address the certificate doesn't list: name_in_certificate checks it for the right name (nuance).
• Its certificate is read through PRELOGIN, as the connection meets it, and watched when verified (nuance); verifying it costs nothing measurable (measured).
• Windows logins (auth: windows) from the service's own account are the computer's account on the network: give that a login (nuance). Doctor and the start log say whether Kerberos or NTLM was used (nuance). A refusal is 18452 (Windows said no) or 18456 (no login on SQL Server's side) (nuance), and Extended Protection set to Required refuses NTLM (nuance).

SFTP servers

sftp entries are the servers sftp sources read from and sftp destinations write to (server: <name>).

# resources.yaml
sftp:
  quest:
    host: ${QUEST_SFTP_HOST}
    user: hospital
    private_key: ${secret:QUEST_SFTP_KEY}
    host_key: "SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8"
    application: Quest Lab Results
    vendor: Quest

• Log in with a key where the server takes one: ssh-keygen -t ed25519 -f quest makes a pair; the private half goes in a secret (ductus secret set QUEST_SFTP_KEY --env prod <quest), the public half (quest.pub) to the server's owner. A password works too.
• host_key is the server's key fingerprint, so a server pretending to be this one is refused before any password or key reaches it. Leave it out the first time: the connection fails, and the error gives the fingerprint the server presented. Check it with the server's administrator, then paste it in. When their key changes (a rebuilt server), connections fail until you update it: on purpose, since that's what an impostor looks like too.

HTTP APIs

apis are the HTTP APIs the code calls with api("<name>"), and that http destinations send through (api: <name>): a base URL, the headers every request carries (the token), a timeout, and TLS.

# resources.yaml
apis:
  empi:
    url: ${EMPI_URL}
    headers: { Authorization: "Bearer ${secret:EMPI_TOKEN}" }
    timeout: 10s
    application: EMPI
    owner: empi-team@hospital.org

• Paths in code and destinations go under the URL: api("empi").get("/patients/1") calls ${EMPI_URL}/patients/1.
• A header the code or a destination sets itself replaces the API's of the same name (nuance).
• 5xx, 408 and 429 answers are retried by default (nuance).
• ca: for an API with a private CA's certificate, certificate: for one that asks for a client certificate (mutual TLS): both certificates: entries (TLS).

Directories

ldap entries are the directories people sign in to the web UI with (Active Directory, OpenLDAP): the server, its TLS and the service account that looks people up. auth.ldap in ductus.yaml names the one to use and maps its groups to roles: People: users, tokens, the directory.

Certificates

certificates are the certificates the engine presents (on its listeners, the API, and as a client certificate) and the CAs it trusts or checks clients against, each by a name: TLS and certificates.

interface.yaml: an interface

An interface is a folder under interfaces/: its interface.yaml, its code, its tests and sample messages. The YAML says where messages come from, what they are, where they go, and how they're delivered, kept and watched; the code says what happens to each one on the way.

Only three things are required: a name, a source, and a destination (or several, destinations). Everything else has a default that's safe to start with:

# interfaces/adt-relay/interface.yaml
name: adt-relay
source: { mllp: { port: 6662 } }
destination: { mllp: { host: ${LAB_HOST}, port: ${LAB_PORT} } }

That relays HL7 from whoever connects on port 6662 to the lab, as received: no code, every message stored before its ACK, delivered in arrival order, retried for as long as the lab is down or answers AE (an AR, a refusal for good, is quarantined for a person), kept seven weeks.

A fuller one, with each part explained on the pages below:

# interfaces/adt-to-lab/interface.yaml
name: adt-to-lab
about:
  description: ADT from Epic to the Sunquest lab
  owner: lab-it@hospital.org
  from: { application: Epic, vendor: Epic Systems, hosts: [10.20.30.0/24] }
  to: { application: Sunquest Lab, vendor: Sunquest }
source:
  mllp:
    port: ${EPIC_PORT}
    allow: [10.20.30.0/24]
script: transform.ts
timeout: 500ms
ordering: { key: PID-3.1 }
destinations:
  lab:
    mllp: { host: ${LAB_HOST}, port: ${LAB_PORT}, tls: true }
  billing:
    http: { api: billing, path: /charges }
    transform: toBilling
    output: json
retention: 3w
monitor:
  active_within: 30m
  between: "06:00-22:00"
  behind: { messages: 500, age: 10m }
alert: { to: [lab-team] }

The name

name is how the database, the web UI, logs, alerts and routes from other interfaces know it. Use the folder's name (as ductus new interface does), short and stable: lower-case words joined by -, saying what flows and between whom (adt-epic-to-lab). Don't rename an interface with messages waiting: messages are stored under the name, so a renamed interface is a new one; the old name's history stays under the old name, and anything still waiting there is sent by nobody.

The code

script is the TypeScript file, relative to the interface's folder, that exports transform(msg, meta) (and any function a destination names with transform:). ductus validate type-checks it, and the engine loads it at start.

• Without a script, messages go to their destinations as they came. That's right for a relay; most interfaces have one.
• A schedule source, and an HTTP source in respond mode, need one: there, the code makes the message or the answer.
• What the code returns decides: a message to send, null to filter it (kept, with the reason), or a thrown error to quarantine it for a person (nuance).

timeout is the longest the code may run for one message without pausing (time waiting on db(), fetch() or api() doesn't count), from 10ms to 10m; default 5s. It's the guard against code that never ends, such as a loop an unusual message never lets finish: past it, the code is stopped, its message quarantined, and the interface carries on; three in a row stop the interface until the engine restarts (nuance). Set it close to what the code needs: 500ms is plenty for most HL7 transforms. Interfaces in the shared pool wait while code on their thread runs, so a generous limit lets one runaway hold them all up that long; code that computes for seconds on purpose belongs in isolation: dedicated (delivery).

Your own fields

custom holds your company's documentation fields for the interface (a BAA reference, a change ticket, a runbook link), declared in docs-schema.yaml: Documentation fields. The engine ignores them; ductus validate checks them, and the web UI shows them.

Sources and formats

An interface's source is where its messages come from: exactly one of these.

Every source stores a message before it says it has it: an MLLP sender's ACK, an HTTP 202, a file's move or delete come only after the message is in the database. A sender that has its acknowledgement never needs to send again; a crash before it loses nothing the sender thinks was taken.

MLLP

# interfaces/adt-in/interface.yaml, in part
source:
  mllp:
    port: 6661
    host: 0.0.0.0            # the default: every IPv4 address of the server
    allow: [10.20.30.0/24]   # who may connect (below)

The engine listens on port; senders connect, send HL7, and get an ACK once each message is stored (AA), or an AE when it can't be (the database is unreachable: they send it again). Each MLLP source has a port of its own; ductus new interface picks a free one from ports.mllp in ductus.yaml, and ductus validate reports two on one port. A sender may open any number of connections. An MLLP source always carries HL7.

• host: which of the server's addresses to listen on: 0.0.0.0 (default, every IPv4 address), :: (IPv6 too), one address, or 127.0.0.1 (this machine only, for tests).
• tls: accept only TLS, presenting a certificate, and optionally take only senders with a client certificate from a CA you name: TLS on listeners. Senders on plain MLLP can't connect then, so agree on a date with them.

HTTP

# interfaces/orders-in/interface.yaml, in part
source:
  http:
    port: 8080
    path: /orders
    method: POST             # the default
    mode: queue              # the default
format: json

• mode: queue (the default): the request body is stored as a message, answered 202 Accepted once stored, and delivered like any other. For systems that push data.

• mode: respond: the code runs on the request and its result is the response, at once. Nothing is stored or queued, there's no destination, and if something fails the caller gets an error status and decides whether to try again (nuance). For REST APIs and lookups that other systems call and wait on:

  # interfaces/patient-lookup/interface.yaml
  name: patient-lookup
  source:
    http: { port: 8081, path: "/patients/{mrn}", method: GET, mode: respond }
  script: lookup.ts
  format: json
  

  {mrn} matches one path segment and is given to the code. In respond mode, calls the code makes one after another add up; await them together where you can (nuance).

• Several HTTP interfaces can share a port, each with its own path (two can't have the same method and path). They're one server then, so they must have the same tls and allow (nuance).

Files and SFTP

# interfaces/census-in/interface.yaml, in part
source:
  file:
    dir: '/data/inbound/census'   # Windows: 'D:\inbound\census', single-quoted
    pattern: "CENSUS_*.csv"
    poll: 30s
    after: move                   # the default: into dir/archive
    split: records                # each row is a message
format: csv

# interfaces/quest-results/interface.yaml, in part
source:
  sftp: { server: quest, dir: outbound, pattern: "*.csv", after: delete, split: records }
format: csv

An sftp source takes the same settings as a file one, plus server: an sftp: entry in resources.yaml with the address and login (SFTP servers).

• A file is read once it stops changing: its size and time the same at two looks in a row, poll apart (default 10s), so a file still being written is never read half done. Expect it one to two polls after its writer finishes (nuance). A writer that says when it's done (a .done file beside it) can use marker: .done instead.
• Files keep their order, oldest first, one at a time: if one fails, later ones wait (nuance).
• split: records makes each record a message (an HL7 batch at each MSH, CSV at each row with the header kept, fixed-length and text at each line), read as a stream, so a file of any size works; after a stop it carries on from the last record stored. Without it, a file is one message, up to 256 MB.
• after: move (the default, to archive_dir, by default archive inside dir) or delete, once everything in the file is stored. The engine never deletes what it moved: clean the archive yourself.
• The service must be allowed to move them: on Linux, add the folders to the unit (nuance); on Windows, grant the service's account (nuance).
• A relative dir is relative to the interface's folder.

Schedules

# interfaces/nightly-census/interface.yaml
name: nightly-census
source: { schedule: { cron: "0 2 * * *" } }   # 02:00 every day, the server's local time
script: census.ts
format: csv
destination: { sftp: { server: quest, dir: census } }

No input: the code runs on the timer and what it returns becomes the message. every: 15m runs at start, then at that interval (for polling); cron runs at the times a five-field expression gives (for a time of day). A run is skipped while the previous one's message is still waiting, and runs missed while the engine was down are made up by one run, not one per missed time (nuance).

Internal: from another interface

# interfaces/adt-to-billing/interface.yaml
name: adt-to-billing
source: internal
script: billing.ts
destination: { http: { api: billing, path: /charges } }

An interface with source: internal receives what other interfaces route to it with destination: { interface: adt-to-billing }: stored in the same transaction that marks it sent there, so nothing is lost or sent twice between the two (routes).

Who may connect: allow

# interfaces/adt-in/interface.yaml, in part
source:
  mllp:
    port: 6661
    allow: [10.20.30.0/24, 10.20.31.15, epic-bridge01.hospital.org]

allow on an MLLP or HTTP source lists who may connect: addresses, subnets and host names. Anyone else is disconnected before a byte is read, before any TLS handshake, and logged once a minute per address. Without it, anyone who can reach the port may connect.

• Why: it names the sending system, which "allow one connection only" (a habit from other engines) never did; a sender may open several connections, and its reconnect after a blip isn't refused (nuance).
• Keep the firewall rules too: each catches the other's mistakes (nuance).
• List the sender's addresses or subnet, its failover and DR hosts, and anything that relays for it. Host names are looked up at start and every 5 minutes, never per connection (nuance). Behind a proxy or load balancer, the address seen is the proxy's.
• ductus validate warns when about.from.hosts names a sender the list would refuse.
• HTTP interfaces on one port share one list: it's checked before the request is read, so before its path is known; validate refuses a port whose interfaces disagree (nuance).
• A sender that vanished (a pulled cable, a paused virtual machine) is noticed in about 30 seconds: every accepted connection has TCP keepalive on, so its lane isn't held by a dead connection for the operating system's two hours (nuance).

Formats

format is what the messages are, so the code gets them parsed and keys (an ordering key, dedupe keys) can be read from them: hl7 (the default), json, xml, text, csv or fixed (fixed-length records).

# interfaces/claims-in/interface.yaml, in part
format:
  fixed:
    fields:
      - { name: mrn, width: 10 }
      - { name: last_name, width: 20 }
      - { name: amount, width: 10, align: right, pad: "0" }

format: csv alone means commas and a header line; { csv: { delimiter: "|", header: false } } for others.

output is the format the code's results are written in when it isn't format: HL7 in, JSON out to an API. A destination can set its own output (destinations). Default: the same as format.

Destinations

Where an interface's messages go: one place with destination, or several, each by a name you choose, with destinations.

One or several

# interfaces/adt-to-lab/interface.yaml, in part
destinations:
  lab:
    mllp: { host: ${LAB_HOST}, port: ${LAB_PORT} }
  billing:
    http: { api: billing, path: /charges }
    transform: toBilling
    output: json
  archive:
    file: { dir: ${OUTBOUND_DIR} }

• Each destination gets every message and is delivered independently: its own queue, order, retries and pause. One that's down holds up only itself; the others carry on (nuance).
• Each runs its own function on the original message, the one named by transform: (default transform), not on another destination's result: toLab and toBilling each shape it for their receiver (nuance). output: sets the format that destination's result is written in.
• A message is finished once every destination is done with it: sent, filtered or quarantined. Filtering and quarantine are per destination too (nuance).
• destination: (one) is the same as a single destination named default. Use one or the other, not both.
• Adding a destination later gives it only messages that arrive from then on (and those still in flight), not older ones; removing one filters what was waiting for it (nuance).
• Many destinations on one interface: past a handful, consider splitting (when).

Each destination can also set its own retry, rate_limit and monitor (in place of the interface's: delivery), its own application, vendor and owner when they differ from the interface's about.to (about), and custom: fields.

Keeping what was sent. By default only the original message is stored, and what a destination was sent is reproduced by running its code again when you look (nuance). Where you must be able to prove exactly what was sent, save_output: true on that destination keeps it with its delivery (about 1.15 GB per million deliveries, measured).

Each destination runs its own function, but code that runs away after an await can be blamed only on its interface: every call of that interface in flight on the thread is quarantined, whichever destination it was for (nuance).

MLLP

# interfaces/orders-to-lab/interface.yaml, in part
destination:
  mllp:
    host: ${LAB_HOST}
    port: ${LAB_PORT}
    timeout_ms: 10000       # the default: connect, send and wait for the ACK
    on_reject: quarantine   # the default: an AR is for a person to look at
    tls: true               # TLS, trusting this server's CAs

The engine sends a message and waits for the ACK before the next one, on one connection it keeps open. AA is delivered; AE is tried again; AR follows on_reject: quarantine (the default) or retry, for receivers that answer AR when they're busy. A send that takes longer than timeout_ms is dropped and tried again; the receiver may have it already, so receivers that dedupe on MSH-10 take care of the repeat (twice). On Windows keep timeout_ms above 2000 (why).

tls: true sends over TLS, trusting the certificates this server trusts; tls: { ca, certificate } also trusts a private CA and presents a client certificate: TLS to receivers. A refused handshake is a temporary error: the message waits, with the reason on the destination (nuance).

ack: none sends without reading an answer, for the rare receiver that never sends ACKs: nothing then tells the engine a message was refused or lost.

HTTP

# interfaces/results-to-portal/interface.yaml, in part
destination:
  http:
    api: portal             # an apis: entry: its URL, token, timeout and TLS
    path: /results
    method: POST            # the default
output: json

Each message is a request: to url, or to path under an apis: entry in resources.yaml (api:), which brings the base URL, headers (the token), timeout and TLS, set once for every interface and environment (APIs). Prefer api: for anything with a token. A 2xx answer is delivered; 408, 429, 5xx and network errors are tried again; any other 4xx is quarantined: the receiver refused this message, and sending it again won't change that.

With a url of its own instead, a token belongs in a secret: headers: { Authorization: "Bearer ${secret:PORTAL_TOKEN}" }.

Files and SFTP

# interfaces/adt-to-archive/interface.yaml, in part
destinations:
  local:
    file: { dir: ${OUTBOUND_DIR} }
  partner:
    sftp: { server: quest, dir: inbound/adt }

Each message is a file named by the message's id and the output format (1234.hl7), written under a hidden temporary name and renamed when complete, so a reader never sees half a file. A file destination's dir is relative to the interface's folder unless written whole, and is made if it's missing; the service must be allowed to write there (Linux: the sandbox; Windows: the service's folders). An sftp destination's server is an sftp: entry in resources.yaml.

To another interface

# interfaces/adt-in/interface.yaml, in part
destinations:
  lab:
    mllp: { host: ${LAB_HOST}, port: ${LAB_PORT} }
  billing:
    interface: adt-to-billing

interface: puts each message into the queue of an interface with source: internal, in the same database transaction that marks it sent here: no network hop, and nothing lost or sent twice between the two (nuance). Use it to split one feed into teams' interfaces, or to chain steps each with its own owner, code, retries and alerts. The receiving interface's format must be what this destination outputs, its own keys, dedupe and destinations apply (nuance), and routes can't form a loop: ductus validate checks both, and ductus impact follows routes (nuance).

A drain stops what comes in from outside; routes between interfaces keep moving, so what's already inside reaches its last interface before the engine stops (nuance).

Pausing one

A destination can be paused (from the web UI, the API or ductus pause adt-in lab), at once, with a reason now or later (--reason): its messages wait, in order, until it's resumed, and a pause never alerts. For a receiver's planned downtime (nuance).

Delivery: order, retries, limits

How an interface's messages are delivered: in what order, how failures are tried again, what counts as a repeat, how fast a receiver may be sent to, when a sender gets its ACK, and where the code runs. The defaults are safe: strict order, retried until delivered, acknowledged only once on disk, in the shared pool.

Order: ordering

ordering says which messages must be delivered in the order they arrived: those with the same key.

# interfaces/adt-to-lab/interface.yaml, in part
ordering:
  key: PID-3.1        # one patient's messages in order; different patients may overtake

• Without a key (the default), the whole interface is one lane: delivered strictly in arrival order, one message at a time per destination. Correct for anything, and the right start for a feed whose receiver cares about order across patients.
• With a key, messages with the same value (one patient's MRN) stay in order with each other, and different values move independently, in 64 lanes by default (partitions). One patient's failing message then holds up only its own lane, not everyone (nuance).
• Choose the narrowest key correctness allows: by patient for ADT (PID-3.1), by order for results (ORC-2). A message whose field is empty goes to the empty value's lane.
• More workers help only with a key: with one lane, one worker does everything (nuance).
• Set partitions before the interface carries traffic. Lowering it with messages waiting is refused at start (they'd be in lanes no worker reads); raising it can reorder a patient's waiting and new messages until the backlog drains. ductus drain --wait first (nuance).

Failures: retry

retry is how a delivery that failed for a reason that may pass (a receiver down, a timeout, an AE, a database the code needs unreachable) is tried again: waits doubling from initial_backoff_ms (1 s) to max_backoff_ms (60 s), forever by default.

# interfaces/orders-to-lab/interface.yaml, in part
retry:
  max_attempts: 10              # then quarantine: "gave up after 10 attempts: …"
  max_backoff_ms: 300000        # at most 5 minutes between tries

• Forever is the default because a receiver that comes back should get every message it missed, in order, with nobody resending by hand (nuance).
• Set max_attempts where a person looking beats waiting: a receiver that refuses some messages for good with errors that look temporary. A message being retried holds up its lane.
• max_backoff_ms is how soon after a receiver comes back its messages start moving, at worst.
• A failure that can't pass (an AR, a 4xx, the code throwing) isn't retried: it's quarantined at once. The code's own temporary errors (a database down) are always retried.
• A destination can set its own retry.

Repeats: dedupe

dedupe recognizes a message sent again within a window (a sender resending after a lost ACK, a file dropped twice) by the values of keys, and doesn't deliver it twice. Off by default.

# interfaces/adt-in/interface.yaml, in part
dedupe:
  keys: [MSH-10]      # the default: HL7's message control ID
  window: 24h         # the default

A repeat is stored with the status duplicate, so it can be found, but never queued. Use it when the receiver can't tolerate repeats and the sender is known to resend. Most HL7 receivers already ignore a repeated MSH-10, so it's often not needed. Choose keys a resend repeats and a new message never does; cover the longest a sender may take to resend (after a weekend, 7d). Dedupe stops repeats from the sender; delivery itself is at least once, so the receiver may still see one message twice after a crash (nuance).

dedupe protects the receiver of this interface. When your code writes to another system (db().execute, an API POST), a "done" marker in a table of yours can't make that write happen exactly once: the two writes can't share a transaction. Make the write itself idempotent (an upsert keyed on the message's control id, an idempotency key header) (nuance).

Speed: rate_limit

rate_limit is the most messages a minute each destination may be sent, for receivers that can't take bursts (a device, a vendor API with a quota). No limit by default.

# interfaces/results-to-portal/interface.yaml, in part
rate_limit:
  per_minute: 600       # 10 a second, on average
  burst: 20             # at most 20 at once after a quiet spell
  over_limit: wait      # the default: held, in order

mode: hard counts per calendar minute instead, for quotas counted that way. over_limit: drop filters what's over the limit, with the reason: only for what is worthless late (frequent status updates where the next replaces the last). Each destination has its own allowance, and can set its own limit.

When senders get their ACK: durability

durability decides whether a sender's ACK (or HTTP 202) waits for the message to be on the database's disk.

• strict (the default): the ACK waits for the flush. Once a sender has it, the message survives a crash or power loss of any machine.
• relaxed: the ACK goes out as soon as the database has the message, before the flush. One sender connection can then go much faster than one message per disk flush (about 400 a second at 2.4 ms a flush). The cost: if the database server crashes or loses power, the last fraction of a second of acknowledged messages is lost, and senders won't resend them. An engine crash loses nothing in either mode (nuance).

Use relaxed only where one connection's peak is more than the disk allows and a lost message can be recovered another way (a resend from the source, a nightly reconciliation). Try a faster disk for Postgres's log, or more sender connections (they share flushes), first.

Dedicated threads and connections

By default an interface's code runs in the shared pool: one thread per CPU core, each running many interfaces, with one JavaScript engine (an isolate) per thread. Cheap, and right for most. isolation dedicated gives an interface threads of its own instead.

# interfaces/results-bulk/interface.yaml, in part
ordering: { key: PID-3.1 }
isolation: dedicated
workers: 8
dedicated_pools: { patient_db: 4 }

• workers (default 4, dedicated only): its threads, each with its own isolate, sharing the lanes. More help only with an ordering key, up to about the number of cores.
• dedicated_pools (dedicated only): connections of its own to resources.yaml databases, so a busy interface can't take every connection of the shared pool from the others, nor theirs from it (nuance).
• When: a busy interface that needs more than one worker (with a key, so there are lanes to share), code that computes for seconds on purpose, or code you don't trust yet: its load and its runaways then touch no other interface (shared pool).
• The cost: each worker is a thread and up to isolates.heap_mb of memory (memory).

Most interfaces never need it: one destination of an interface in the shared pool drained about 1,500 messages a second in the measurements, against an average of 12 a second for a million messages a day (measured). When one does, in this order: an ordering key, then dedicated, then more workers, then dedicated_pools.

Keeping and watching

How long an interface's messages are kept, what operations expects of it, and who hears when it doesn't happen.

How long: retention and quarantined

retention on an interface overrides the project's (ductus.yaml, 7w by default) for that interface: 1w, 3w or 7w.

# interfaces/adt-in/interface.yaml, in part
retention: 1w          # a busy feed nobody looks back on
quarantined: keep      # the default: until a person resolves them

The rules are the project's: whole partitions dropped, kept at least the window, nothing waiting ever deleted, a shorter window waiting for ductus retention confirm adt-in (how long messages are kept). Change it per interface when its feed is far busier or far more looked-back-on than the rest. One change at a time per interface (nuance).

quarantined is the same choice as the project's, for this interface's quarantined messages past the window: keep them in the held area until someone resends, corrects or dismisses them, or purge them with the rest (nuance).

Neither applies to an HTTP source in respond mode: it keeps nothing.

What's expected: monitor

The engine can't know that a quiet feed is a problem: ADT from an EHR that stops for half an hour in the afternoon is an outage, a lab's nightly file that doesn't come by 07:00 is one too, and a test feed that's quiet all week is fine. So nothing is expected until you say so (nuance), in the terms the receiving team uses, with monitor:

# interfaces/adt-in/interface.yaml, in part
monitor:
  active_within: 30m              # something received every 30 minutes...
  between: "06:00-22:00"          # ...during these hours (the server's local time)
  behind: { messages: 500, age: 10m }

# interfaces/quest-results/interface.yaml, in part
monitor:
  active_by: "Mon-Fri 07:00"      # the nightly file, by 07:00 on weekdays
  from: "18:00"                   # counting what arrived since 18:00 the evening before

Only receiving counts for active_within and active_by. A destination can set its own behind. Besides these, some states need no setting: a destination with messages waiting that has sent nothing for a minute is stalled, one with quarantined messages is in error, an interface whose code was stopped is stopped, and a paused destination is paused (never an alert). The web UI, ductus status and Prometheus show them all (what each state means).

Who hears: alert

States become notifications only through the alert rules in ductus.yaml (Alerts and owners): a rule says which states alert, how badly, and to which channels. alert on an interface adds recipients on top of the rule's, or silences the interface:

# interfaces/adt-in/interface.yaml, in part
alert:
  to: [lab-team, owner]     # channels from ductus.yaml, and/or whoever owns it

# interfaces/new-feed/interface.yaml, in part
alert:
  mute: true                # half-built: alerts open and show, but nobody is told

to: [owner] reaches whoever the interface's (or the destination's) owner names (who "owner" is). mute sends nothing about this interface; its alerts still open and show in the web UI, so nothing is hidden from people who look.

About and owners

about says what an interface is for, who owns it, and the systems on each side. Nothing in it changes how messages flow: it's what people and tools read when something goes wrong.

# interfaces/adt-to-lab/interface.yaml, in part
about:
  description: ADT from Epic to the Sunquest lab
  owner: lab-it@hospital.org
  from:
    application: Epic
    vendor: Epic Systems
    hosts: [10.20.30.0/24, epic-bridge01.hospital.org]
  to:
    application: Sunquest Lab
    vendor: Sunquest

Why fill it in

• ductus impact answers "what does this touch?": ductus impact --vendor "Epic Systems" for a vendor's outage or breach, ductus impact --host 10.20.30.0/24 for a network change, with how each interface is affected (it sends to them, receives from them, queries their database) and who owns it. It knows only what's named (nuance): the application and vendor on each side, the hosts senders connect from, and the same on resources.yaml entries.
• On call, the web UI shows the description and owner on the interface's page, and alert rules can send to the owner: the person woken at 3 a.m. knows what broke and who to call.
• hosts is the one thing the engine can't learn from the configuration (who connects to it): with it, a firewall change can be traced to the interfaces it affects, and ductus validate warns when the source's allow: list would refuse a host named here.

ductus validate warns about an interface that doesn't name the application or vendor on each side; require_about: true in ductus.yaml makes it an error, once your existing interfaces are documented.

Owners

about.owner is who to ask about the interface: a person or a team. Write it as an email address (lab-it@hospital.org): an address matches the person's account (by its email) and survives a change of name, where a name might match two people or nobody.

• A person with an account shows as that person in the web UI, and their Interfaces page opens on what they own ("Mine"). With a directory, people who own something get their account before they first sign in, when they're in a Ductus group (nuance).

• A team without an account is reached through owners: in ductus.yaml, by the name the interfaces give:

  # ductus.yaml, in part
  owners:
    Lab Systems:
      email: lab-systems@hospital.org
  

• A destination with another owner (its code or its receiver is another team's) says so on the destination: owner: billing-it@hospital.org. Otherwise a destination's owner is its interface's (nuance).

• resources.yaml entries have an owner too: who renews a certificate, who looks after a database.

Alert rules with to: owner mail whoever owns the interface or destination (Alerts and owners). ductus owners lists who owns what; with the database it marks which owners are people with an account, and --check fails on an owner that reaches nobody (run it in CI). When someone leaves, ductus owners reassign --from old@hospital.org --to new@hospital.org moves everything they owned, in interface.yaml and resources.yaml alike: a change to commit, like any other (nuance).

• An owner is a person when the text names exactly one account (by username, email or display name); two accounts at one level name nobody, and the Owners page says so (nuance).
• Operators who own something start on their own interfaces in the console ("Mine") (nuance).
• Moving ownership in the console makes a preview of the change to commit to the project, not a change: the project in Git stays the truth (nuance).

Your own fields

custom, beside about, holds the documentation your company keeps that Ductus doesn't ask for (a BAA reference, the change ticket, the analyst, a runbook link): Documentation fields.

TLS and certificates

TLS encrypts a connection and proves who is at the other end. The engine uses it in three places, all set up the same way, with certificates named once in resources.yaml:

Use TLS wherever messages cross a network you don't own, and for the API and web UI whenever they're reachable from another machine. It costs nothing you'll see on an MLLP feed: ingest measured the same with and without, about 1 ms per new connection, ~23 kB of memory per open one (measured). Listeners speak TLS 1.2 and 1.3 only, and every connection prefers a post-quantum key exchange where the other side offers one (nuance).

Certificates in resources.yaml

Every certificate and CA the engine uses is a certificates entry, by a name:

# resources.yaml
certificates:
  ductus-mllp:                              # a pair the engine presents
    cert_file: ${TLS_DIR}/mllp.pem          # the certificate, then its intermediates
    key_file: ${TLS_DIR}/mllp.key
    owner: infrastructure@hospital.org      # who renews it: expiry alerts reach them
  ductus-api:
    cert_file: ${TLS_DIR}/api.pem
    key_file: ${TLS_DIR}/api.key
  hospital-ca:                              # a CA alone: trusted, or checked against
    ca_file: ${TLS_DIR}/hospital-ca.pem
  partner-ca:
    ca: ${secret:PARTNER_CA}                # PEM text from a secret instead of a file

• A pair (cert_file + key_file) is what the engine presents: on a listener, or as its client certificate. cert_file holds the certificate first, then the intermediate certificates that issued it (not the root): the "full chain" a CA sends. Java and .NET senders don't fetch a missing intermediate, and refuse the connection (nuance).
• The key is unencrypted PEM, readable by the engine's account and nobody else (chmod 600, owned by ductus; on Windows, the service's account only).
• A CA (ca_file, or ca as text) is trusted for a connection that names it, or checks senders' client certificates.
• From a vault instead of files: cert and key as PEM text, usually ${secret:…}. Read at start only, so a renewal needs a restart, unlike files.
• Paths: relative to the project folder unless written whole; on Windows, write the drive (nuance). Keep the files outside the project on a server (/etc/ductus/tls, readable by the ductus group), with the path in the environment (${TLS_DIR}), so keys never go near Git.
• Checked before anything runs: every entry is loaded at start, before the database. A missing file, a key that isn't the certificate's, or an expired certificate stops the start with the entry and the reason (nuance). ductus validate checks the same, with each certificate's names and days left.

Getting a certificate from your CA

ductus cert request /etc/ductus/tls/mllp --dns mllp.hospital.org --subject "O=General Hospital, OU=Interfaces"

writes a new private key (mllp.key, readable only by you, never overwritten) and a request (mllp.csr), on the server that will use it, and says what to do next: send the .csr (never the .key) to the hospital's CA, put what comes back in mllp.pem (yours first, then the intermediates), and check it. The key is RSA 2048, which every CA takes (--key ecdsa for P-256); one certificate serves as both a server's and a client's (nuance).

ductus cert show /etc/ductus/tls/mllp.pem      # or the entry's name: ductus cert show ductus-mllp

says what the file holds and judges it the way the engine and senders will: the names, who issued it, days left (the chain's earliest), whether the chain is complete and in order, and whether the key matches. It exits 1 when the engine would refuse it, and says when a file is a CSR, a key or DER instead of PEM. Run it before naming a certificate in resources.yaml.

For a lab with no CA, ductus cert self-signed tls/lab --name localhost makes a pair to try TLS with; senders must be given its .pem to trust. Your first interface plays a CA with openssl for the whole path.

From a PFX (Windows, Active Directory Certificate Services)

Windows tools export a certificate with its key as a .pfx (PKCS#12) file under a password. The engine reads PEM: convert it with openssl, which is on every Linux server and comes with Git for Windows (C:\Program Files\Git\usr\bin\openssl.exe):

# the certificate first, then the CA certificates in the file
openssl pkcs12 -in mllp.pfx -clcerts -nokeys | openssl x509 -out mllp.pem
openssl pkcs12 -in mllp.pfx -cacerts -nokeys | sed -n '/BEGIN/,/END/p' >>mllp.pem
# the key, unencrypted (protect the file: chmod 600, or the service's account only)
openssl pkcs12 -in mllp.pfx -nocerts -nodes | openssl pkey -out mllp.key
ductus cert show mllp.pem

Each asks for the PFX's password. A PFX from an older Windows (RC2-encrypted) makes OpenSSL 3 say "unsupported … RC2-40-CBC": add -legacy to each openssl pkcs12. The PFX's CA certificates may include the root, which cert show notes as harmless to send; it checks the order too. In PowerShell without Git Bash, run the same with the full path to openssl.exe, and use Select-String or an editor in place of sed to keep only the BEGIN…END blocks.

On what the engine listens on

tls: on an MLLP or HTTP source makes its port TLS only, presenting the certificate named:

# interfaces/adt-in/interface.yaml, in part
source:
  mllp:
    port: 6661
    tls:
      certificate: ductus-mllp
      clients:
        ca: hospital-ca
        subjects: ["CN=epic-bridge*.hospital.org"]

• certificate: the pair it presents, for the name senders connect to.
• clients (optional): senders must present a certificate of their own, issued by ca and, with subjects, naming an allowed system: CN, O, OU… parts with * and ? wildcards, or DNS= for a certificate's DNS names. Keep subjects narrow: without it, every certificate that CA ever issued (often every machine in the hospital) is let in (nuance).
• Senders on plain MLLP can't connect once it's on, so agree on a date with them. One that tries gets its connection closed, no ACK; the engine's log says why, once a minute per sender (nuance).
• Senders on TLS 1.0 or 1.1 can't connect at all (nuance).
• HTTP interfaces sharing a port share its tls.
• allow: beside it says where senders may connect from; client certificates say which system they are. Use both where it matters (who may connect).

To see the chain a listener really serves, run the full checks (Connections → Run the checks): the engine connects to each of its TLS listeners and judges what it gets, a missing intermediate included. It can't know which name senders connect by, so a certificate without that name shows only at a sender (nuance).

The web UI and the API

# ductus.yaml, in part
api:
  listen: 0.0.0.0:7700
  tls: { certificate: ductus-api }
  allow: [127.0.0.1, 10.1.50.0/24]

Needed as soon as the API is reachable from another machine: the web UI's sign-in cookie is only sent over HTTPS (without tls, signing in from another computer doesn't work at all), and tokens and passwords would cross the network readable. The start and ductus validate warn about an API on the network without it (nuance). Browsers that have seen it over HTTPS won't try plain HTTP to it again for a year (HSTS). clients: works here too, for an API only certain systems may call.

On what the engine connects to

Every TLS connection the engine makes checks that the other side's certificate comes from a CA this server trusts and is for the host connected to. Add a private CA, or a client certificate, per connection:

# interfaces/orders-to-lab/interface.yaml, in part
destination:
  mllp:
    host: ${LAB_HOST}
    port: ${LAB_PORT}
    tls: { ca: hospital-ca, certificate: ductus-mllp }

• tls: true on an MLLP destination: TLS, trusting this server's CAs. ca: also trusts a certificates: CA, for that connection only, so a partner's CA isn't trusted by everything on the server (nuance). Ask the receiver's team for the CA that issued their certificate, not their certificate itself (which changes at every renewal).
• certificate: presents a pair as the engine's client certificate (mutual TLS). Tell the receiver's team its subject and the CA that issued it (nuance).
• Connect by the name on their certificate: one for lab.hospital.org is refused at 10.1.2.3 unless it lists that address.
• The same ca: and certificate: go on an apis: entry (which api() and http destinations through it share), an HTTP destination with its own https:// URL, a Postgres entry, the mail server, webhooks and the heartbeat; ca: alone on a SQL Server entry and the directory.
• A refused handshake is a temporary error: the message waits and is tried again, with the reason on the destination ("TLS: the receiver wants a client certificate") (nuance).
• ductus doctor --interface orders-to-lab makes the engine's own handshake, with the same settings, before you deploy (nuance).

An HTTP destination that goes through an apis: entry (http: { api: billing }) uses that entry's ca and certificate; on the destination itself they're only for one with its own https url, and validate refuses them elsewhere (nuance).

Renewing

Replace both files; no restart. The engine reads every cert_file and key_file again every minute (one task for the whole engine, comparing their contents). A new pair that loads is used for new connections from then on; connections already open keep theirs until they close. A new pair that doesn't load (half copied, a new certificate with the old key, expired) is logged once as an error, and the engine keeps presenting the pair it had: a bad renewal never takes a listener down (nuance). After writing them, look for certificate renewed in the log within a minute, or the "Ours" group on the web UI's Connections page.

You'll be told in time. Every certificate the engine presents or checks is watched: an alert at 30, 14 and 7 days before it expires, to its owner with to: owner, and critical once expired (alerts; nuance).

What's watched, and how the days are counted. Every TLS connection the project makes: HTTPS APIs and destinations, MLLP with tls:, the directory, Postgres and SQL Server when verified, mail and webhooks (the list). The countdown runs to the earliest expiry in the chain, intermediates included, in whole days (nuance). A check that can't connect shows the error and keeps the dates it last read (nuance); behind a load balancer that ends TLS, it's the balancer's certificate you see (nuance). The engine's own pairs count down from their files, with no connection made (nuance), and renewed files that can't be used show on the pair's row, not as an alert: the pair in use still counts down (nuance). A refused certificate is said once, in the check's result and the delivery's error, not logged as an error at every check (nuance).

By hand (a CA without automation, such as AD CS's web enrollment): a new request under another name, signed, checked, then copied over the files in use:

ductus cert request /etc/ductus/tls/mllp-2027 --dns mllp.hospital.org
# … the CA signs mllp-2027.csr; put the result in mllp-2027.pem …
ductus cert show /etc/ductus/tls/mllp-2027.pem
cp /etc/ductus/tls/mllp-2027.pem /etc/ductus/tls/mllp.pem
cp /etc/ductus/tls/mllp-2027.key /etc/ductus/tls/mllp.key

With a renewal tool: anything that writes new PEM files works, with no engine feature needed. Point it at the files the entry names (or copy into them from its own folder after each renewal), make sure the engine's account can read the result, and run ductus cert show on the first one it writes. Three common ones:

• certbot (Linux, ACME: Let's Encrypt for public names, or your own ACME CA with --server): a deploy hook copies each renewal into place:

  certbot certonly --standalone -d mllp.hospital.org \
    --deploy-hook 'install -o ductus -g ductus -m 600 "$RENEWED_LINEAGE/privkey.pem" /etc/ductus/tls/mllp.key &&
                   install -o ductus -g ductus -m 644 "$RENEWED_LINEAGE/fullchain.pem" /etc/ductus/tls/mllp.pem'
  

  fullchain.pem is the certificate with its intermediates, as cert_file wants. Let's Encrypt only issues for names that resolve publicly (or through DNS validation): most interface engines' names are internal, so certbot is mostly for an internal ACME CA.

• win-acme (Windows, ACME): its PEM files store writes the certificate and key into a folder (--store pemfiles --pemfilespath C:\ProgramData\Ductus\tls); name the certificate with its chain and the key in the entry (cert show says which file is which), and grant the service's account read access to the folder.

• step (Smallstep's CLI, with step-ca or another ACME CA): step ca certificate mllp.hospital.org mllp.pem mllp.key once, then step ca renew --daemon mllp.pem mllp.key keeps renewing the same two files in place.

Certificates from secrets (cert/key) and CAs change with a deploy and a restart.

When it doesn't work

To try a TLS listener or receiver before the real systems are ready, ductus gen --tls plays the sender and ductus sink --tls the receiver, checking certificates as the engine does and saying why a handshake fails (nuance).

ductus doctor checks every TLS connection the project makes, from the server, and names the step that fails.

People: users, tokens, the directory

Everyone who uses the web UI or the REST API is a person with an account (who signs in with a password, theirs or the directory's), or a token (for scripts and monitoring). Both have a role, and maybe extra permissions; everything either does that changes something is in the audit log, by name.

Roles and permissions

Two permissions go on top of a role, given only to the people who need them:

• payloads: read message contents, which is usually patient data (and edit them, as an operator). Without it the message page shows status, destinations and errors, and says the payload is hidden. Every read is in the audit log, written before the payload is shown (nuance).
• audit: read and export the audit log. For compliance and security staff, not operators (nuance).

Accounts

Local accounts are made on the server, with the command line (it works on the database directly, so it's also how the first admin is made, and how you get back in if locked out):

ductus user create ana --role admin --email ana@hospital.org --display-name "Ana Silva"
ductus user create ben --role operator --permission payloads --email ben@hospital.org --must-change
ductus user list

The password is asked for twice (--password-stdin for scripts); --must-change makes the person choose their own at the first sign-in. Admins also manage accounts on the web UI's Users page. Give everyone an --email: that's how owner texts (about.owner: ana@hospital.org) and alerts reach them.

• Passwords are at least 12 characters (auth.password.min_length, never below 8), can't be the username, one character repeated or a common one; there's no rule for mixing kinds of characters (length matters more). Stored as argon2id hashes.
• Lockout: five wrong passwords for one username within 15 minutes lock it for 15 minutes (auth.lockout); ductus user unlock ben or the Users page unlocks it at once (nuance).
• Disable (ductus user disable ben, or the Users page) ends their sessions at once and stops every token of theirs. Accounts are never deleted: their history in the audit log stays theirs. The last admin can't be disabled or demoted.
• Sessions in the browser last 12 hours unused and 7 days at most (auth.session).

# ductus.yaml, in part
auth:
  session: { idle: 8h, max: 1d }
  password: { min_length: 14 }
  lockout: { attempts: 5, window: 15m, for: 15m }
  token_sign_in: false
  user_tokens: true

All of it is under auth in ductus.yaml; the defaults suit most sites. Shorten session.idle where screens are shared or left unlocked.

Tokens

A token is for a script, a CI job or monitoring: Authorization: Bearer dux_… on every call.

ductus token create prometheus --role viewer            # printed once; stored only as a hash
ductus token create nightly-replay --role operator --permission payloads
ductus token list
ductus token revoke prometheus

• People can make their own under Account in the web UI (auth.user_tokens, on by default): a person's token can do no more than they can, and stops when their account is disabled.
• A token has no permissions unless it's given them: a monitoring token needs none.
• auth.token_sign_in (on by default) lets the sign-in page take a token instead of a password: how the first people get in before accounts exist. Turn it off once everyone has an account, so a leaked script token can't also open the web UI.

Signing in with the directory

With a directory (Active Directory, OpenLDAP, FreeIPA), people sign in with their usual username and password, and the directory's groups decide their role. Two pieces: the directory itself in resources.yaml, and how it's used in ductus.yaml.

# resources.yaml, in part
ldap:
  corp-ad:
    url: "ldaps://hospital.org"          # the domain's name reaches any controller
    bind_dn: "CN=svc-ductus,OU=Service Accounts,DC=hospital,DC=org"
    bind_password: ${secret:LDAP_BIND_PASSWORD}
    ca: hospital-ca                      # the domain's CA, where this server doesn't trust it

# ductus.yaml, in part
auth:
  ldap:
    resource: corp-ad
    user_base: "OU=Staff,DC=hospital,DC=org"
    groups:
      admin: "CN=Ductus Admins,OU=Groups,DC=hospital,DC=org"
      operator:
        - "CN=Interface Team,OU=Groups,DC=hospital,DC=org"
        - "CN=Night Operators,OU=Groups,DC=hospital,DC=org"
      viewer: "CN=Help Desk,OU=Groups,DC=hospital,DC=org"
      permissions:
        payloads: "CN=Ductus PHI,OU=Groups,DC=hospital,DC=org"
        audit: "CN=Ductus Auditors,OU=Groups,DC=hospital,DC=org"
    refresh: 15m

• The directory entry (ldap): ldaps:// (port 636), or ldap:// with starttls: true. Plain ldap:// is refused unless insecure: true, which is only for a test directory on the same machine: passwords would cross the network readable (nuance). The service account (bind_dn) needs read access to people and groups, nothing else; give it a password that doesn't expire, or note when it does: an expired one stops directory sign-ins.
• Who may sign in: people under user_base, found by user_filter ((sAMAccountName={username}) by default; (uid={username}) for OpenLDAP and FreeIPA), and in at least one role group. The highest role their groups give wins; permissions groups add to it. Someone in no role group is told they have no access (nuance).
• Write several groups one per line, as above, never [a, b]: DNs have commas (nuance).
• Their account is made at their first sign-in, marked as a directory account; its role, permissions, name and email are the directory's, read-only here. An admin can still disable it.
• Removal reaches Ductus within refresh (15 minutes by default): someone taken out of the groups, disabled or deleted in the directory loses their sessions and tokens at the next refresh. For someone who must lose access now, disable them on the Users page first (nuance).
• A renamed person keeps their account (found by their directory id, objectGUID or entryUUID), and a sign-in name given to someone new gets a new account (nuance).
• Nested groups: nested: true counts members of groups inside the configured ones. Prefer direct membership: one search per group, easy to audit (nuance).

Keep one local admin (ductus user create … --role admin), with a long password the on-call can reach: when the directory is down, directory sign-ins are refused ("the directory can't be reached right now") but local accounts and open sessions keep working (nuance). A name a local account holds is always checked locally, so the directory can't take it over.

Active Directory has not been tested against yet (the tests use OpenLDAP); its transitive-membership filter for nested groups is unit-tested, and the first pilot with a domain verifies it.

More on directory accounts:

• A person deleted and made again in the directory is a new id there, so a new account here: it can't be linked to the old one, which is refused at the next refresh (nuance).
• OpenLDAP follows a renamed person in their groups only with its refint overlay (nuance).
• Wrong passwords count twice: towards Ductus's lockout and towards the directory's own (nuance).

Alerts and owners

The engine judges every interface and destination every 15 seconds. When one is in a state your rules name (stalled, behind, silent…), it tells people once, reminds them while nobody has acknowledged it, tells them again if it gets worse, and says when it's fine again. Nothing alerts until you write rules: until then, states show in the web UI only.

All of it is the alerts section of ductus.yaml, the one section read per environment (so test and production can mail different people, and webhook URLs can be secrets):

# ductus.yaml, in part
alerts:
  smtp:
    host: smtp.hospital.org
    from: ductus@hospital.org
    username: ductus@hospital.org
    password: ${secret:SMTP_PASSWORD}
  channels:
    oncall: { email: interfaces-oncall@hospital.org }
    chat: { webhook: { url: "${secret:TEAMS_WEBHOOK}", format: teams } }
  rules:
    - { when: [error, stalled, stopped], severity: critical, to: [oncall, chat] }
    - { when: [behind, silent, missed, certificate], severity: warning, to: [chat] }
  repeat: 4h
  quiet: { between: "22:00-06:00", severities: [warning] }
  heartbeat: { url: "${secret:HEARTBEAT_URL}", every: 5m }

Channels: where alerts go

A channel is a name rules send to: an email list, a chat webhook, or both.

• Send to teams, not people: a team's mailing list or chat channel. People change jobs, go on leave and sleep; a team's list doesn't.
• Email needs smtp: your organization's relay or submission server, verified by its certificate (tls: starttls on 587 by default, tls on 465, none only for a relay on the same network segment). Ask the mail team to let from through their filters.
• Webhooks: format: teams (a Microsoft Teams Workflows webhook), slack (also Mattermost, Rocket.Chat), or json (every field, for your own incident tool). A webhook's URL is its own password, so keep it in a secret (nuance).
• Try one without an engine: ductus alerts test oncall sends a test notification now.

Rules: what alerts, and how badly

Each rule names states (when), a severity, and channels (to). A state is matched by the first rule that names it; a state no rule names doesn't alert, and ductus validate says which.

behind, silent and missed need an interface's monitor settings: nothing is expected until you say so. A paused destination never alerts (it's paused on purpose), and an alert that's open when someone pauses stays open, quietly, until it's resumed (nuance).

• warning: worth knowing today; held during quiet hours. critical: act now; never held. A warning that becomes critical is told again. On Windows a critical alert is also in the Event Log (1020).
• repeat (default 4h): while an alert is open and unacknowledged, its recipients are reminded this often; 0 never. Acknowledging it on the Alerts page stops the reminders until it gets worse or recovers. Acknowledging isn't fixing, and silencing isn't pausing (nuance).
• settle (default 2m): how long a state must be fine before its alert is over, so a receiver flapping up and down is one alert, not dozens. recovery (default on) tells the recipients when it is.
• quiet: hours when warnings wait until morning; one that comes and goes in the night is never sent, only kept in the history. Keep critical out of severities (nuance). The hours are the server's local time.
• certificates.warn_days (default [30, 14, 7]): the steps before a certificate expires. Make the first long enough for your slowest renewal (a vendor's CA, a change board).

Owners as recipients

to: owner in a rule, or an interface's alert.to, also tells whoever owns the interface or destination: its owner, else about.owner (nuance). An owner who is an account is mailed at the account's email; one written as an address is mailed at it; a team without an account gets its address from owners:

# ductus.yaml, in part
owners:
  Lab Systems: { email: lab-systems@hospital.org }

Lead with team channels, and add owner where a person should hear directly (an interface one analyst built and looks after, a certificate one person renews). Owners change; a rule that sends only to owners goes quiet when one leaves, which is why ductus owners --check fails on an owner that reaches nobody (owners).

The heartbeat: when the engine itself is down

An engine that is down, or can't reach its database, can't send alerts. heartbeat calls a URL every every (default 5 minutes) while the engine runs, for a service elsewhere that raises its own alarm when the calls stop: healthchecks.io, Better Stack, Uptime Kuma, or a monitoring tool's push check. Set its grace period to about twice every. Set it in production: it's the only way anyone hears that the engine is gone (nuance).

What to know

• Notifications are sent at least once: through an outbox in the database, retried until the channel takes them. A failing channel shows on the Alerts page, not as another alert (nuance).
• After a start, states the engine hasn't watched for a minute yet don't alert, so a restart doesn't page everyone (nuance).
• It costs little: 35 ms a round at 1,001 interfaces, measured (nuance).
• History (what fired, who was told, who acknowledged) is kept a year (alerts.keep).
• An interface can add recipients (alert: { to: [...] }) or be muted (alert: { mute: true }): keeping and watching.

Documentation fields

Every company keeps things about its interfaces that Ductus doesn't ask for: the business associate agreement, the change ticket that approved it, the analyst, a runbook, the go-live date. Documentation fields keep them beside the interface, in Git, checked like the rest, instead of in a spreadsheet nobody updates.

What you get: ductus validate reports a missing or wrong value; the web UI shows the fields on each interface, destination and resource's page and filters the interface list by them; ductus docs lists them (--missing for what's not filled in, --json for a report). What you don't: the engine ignores them entirely; a wrong value never stops anything running.

Declare them: docs-schema.yaml

At the top of the project, fields, by the key values will use:

# docs-schema.yaml
fields:
  baa_signed:
    type: enum
    values: ["yes", "no", n/a]
    required: true
    label: BAA signed
    help: Business associate agreement, from the contracts office
  change_ticket:
    type: text
    label: Change ticket
  analyst:
    type: person
    applies_to: [interface, destination]
  runbook:
    type: url
  go_live:
    type: date
  contract:
    type: text
    applies_to: [resource]

• required: true: every interface (or destination, or resource) the field applies to must have a value; ductus validate reports each missing one. The engine starts anyway.
• applies_to: interface (the default), destination, resource (resources.yaml entries), or several.
• label and help: what the web UI shows instead of the key, and a hint under it.

Fill them in: custom:

Values go in a flat custom: beside each one:

# interfaces/adt-to-lab/interface.yaml, in part
custom:
  baa_signed: "yes"
  change_ticket: CHG-1234
  analyst: ana@hospital.org
  runbook: https://wiki.hospital.org/interfaces/adt-to-lab
  go_live: 2026-11-15
destinations:
  lab:
    mllp: { host: ${LAB_HOST}, port: ${LAB_PORT} }
    custom: { analyst: ben@hospital.org }

# resources.yaml, in part
sftp:
  quest:
    host: ${QUEST_SFTP_HOST}
    user: hospital
    private_key: ${secret:QUEST_SFTP_KEY}
    custom: { contract: QUEST-2025 }

(custom on interfaces; the same key on destinations and on every resources.yaml entry.)

What to know

• Checked by ductus validate, not by the engine: a wrong value is a validate error and nothing else (nuance). Run validate in CI and the fields stay filled.
• The web UI shows them as of the engine's last start: it keeps a snapshot from the project it started with (nuance).
• Read-only in the web UI: a change is a commit, reviewed like any other.
• Keys are yours to choose, except names Ductus already uses beside custom (name, owner, vendor, …).

How interface code works

An interface's code is a TypeScript file in its folder, named by script in its interface.yaml. The engine calls one of its functions for each message and each destination, and sends on what the function returns. This page is how that works; the pages after it are the tasks: routing and filtering, mapping one message to another, formats, lookups and code tables and testing.

Say it in YAML first. Much of what interface code does in other engines is a setting here, checked by ductus validate and visible to ductus impact without reading any code. Write code for what's left:

What needs code: deciding which messages a destination gets, changing fields, building a message in another shape, and looking things up. There is no field-mapping screen or mapping file: the mapping is code, and a mapping can still be written as a table in that code (transforms), reviewed in Git and tested like the rest.

What the engine does with each message

1. A sender's message arrives and is stored in the database; only then does the sender get its acknowledgement. Nothing about your code delays or decides the ACK (except an HTTP source in respond mode, below).
2. For each destination, in its own queue and order, a worker calls that destination's function with the message as it was received, parsed by the interface's format.
3. What the function returns decides what happens, for that destination only:

4. A message is finished when every destination is done with it: sent, filtered or quarantined. Its overall status is quarantined if any destination quarantined it, else sent if any sent it, else filtered.

So a function's job is to say what this destination should get, or that it gets nothing, or that a person must look. Never swallow a problem with return null: a message nobody can see again is the failure people find months later. Quarantine it, with a reason a person can act on (nuance).

The functions

// interfaces/adt-in/route.ts, in part
export function toLab(msg: Message) {
  if (!LAB_EVENTS.has(msg.MSH[9][2])) return null; // not for the lab: filtered, no reason needed
  if (msg.PID[3][1].startsWith("TEST")) return kill("test patient");

• transform(msg, meta) is the function a destination calls unless it names another with transform:. With several destinations, export one function per destination (toLab, toBilling) and name each; one file can hold them all.
• The first argument is the message parsed by format: a Message for HL7, an object or array for JSON, an XmlElement for XML, a record (or a list of them) for CSV and fixed-width, a string for text (formats).
• The second, meta, is about this message: id (the stored message's id, the same on every retry and for every destination: give it to receivers that can ignore a repeat), receivedAt, destination (which destination this call is for) and origin (for a message routed from another interface, its id there).
• A function may be async and await lookups (lookups). While one waits, the thread runs other messages.
• What it returns is written in the destination's output format: a Message or an XmlElement as itself, an object as JSON, records as CSV or fixed-width lines, and a string as it is, whatever the format.
• Without a script, every destination gets the message exactly as received.

What code may and may not do

• It may only import project files and @ductus/sdk. No npm packages, no network or files of its own: databases, APIs, variables and secrets come through the SDK (db(), api(), fetch(), env(), secret()), so the engine can time them, retry them, mock them in tests and list them in ductus impact.

• It can run more than once for one message: after a temporary failure, a restart in the middle, a resend or a replay (nuance). So it must give the same answer for the same message, and anything it writes elsewhere (db().execute, an API POST) must be safe to repeat: use meta.id as the key.

• It shouldn't run long. A function that runs past timeout (5 s by default) without pausing is stopped, its message quarantined, and three in a row stop the interface (runaway scripts). Waiting on a lookup doesn't count.

• Missing fields read as "" and never throw, so a typo in a field number gives "" quietly: test with real messages and check the fields you map (nuance).

• timeout counts time the code runs, not time it waits on an await: a function waiting on a slow lookup isn't stopped for it (nuance). Code that runs away after an await interrupts every async call on its thread (nuance).

• An import used only for what it does when it loads is removed by TypeScript: import something from it you use, or call a function it exports (nuance).

Code shared between interfaces

A file outside any interface folder, such as lib/codes.ts, can be imported by several interfaces (import { translate } from "../../lib/codes.ts"). Use it for code tables, mappings and helpers that more than one interface needs. Two rules (shared pool):

• Keep it read-only: tables as const, functions that only use their arguments. A top-level let is one copy shared by every interface on that thread, and ductus validate warns about it (DX3).
• A heavy loop belongs in the interface's own file: a runaway in a shared file after an await can't be blamed on one interface (nuance).

All interfaces in an engine share one global scope per thread, so a global set by one is seen by others (nuance), and a shared library file is one copy per thread, not one per project (nuance).

A new interface, and the editor

ductus new interface orders-in (with --source mllp|http|file|sftp|schedule and --format) writes a working interface: its interface.yaml, a transform.ts, a test and a sample message, passing ductus test and ductus validate as written. Start from there.

ductus test and ductus validate write the SDK's typings into .ductus/types and a tsconfig.json, so VS Code (or any editor that speaks TypeScript) completes msg.PID[ with the fields' names and underlines mistakes as you type. After an upgrade, run ductus types so the editor knows the new version (nuance).

The examples on these pages

The code on these pages is from a small project in the repository, examples/guides: ADT routed to a lab and to billing, results to a FHIR server, an X12 835 to CSV, a census file to HL7. A code block that starts with a file's path, such as // interfaces/adt-in/route.ts, is a piece of that file, and the docs' tests check that the page and the file agree and that the project's own tests pass. Copy it, run ductus test in it, and change it.

Routes and filters

Which messages go where. In Ductus a route isn't a separate rule table: every destination gets every message, and its own function decides whether to send it, change it, or drop it. One interface with three destinations is three independent decisions on the same message.

Filtering: null and kill

// interfaces/adt-in/route.ts, in part
/** The visit events the lab system takes. */
const LAB_EVENTS = new Set(["A01", "A03", "A04", "A08"]);

export function toLab(msg: Message) {
  if (!LAB_EVENTS.has(msg.MSH[9][2])) return null; // not for the lab: filtered, no reason needed
  if (msg.PID[3][1].startsWith("TEST")) return kill("test patient");

• return null filters: this destination never gets the message, and nothing is said about why. Use it for the ordinary "not for this receiver" cases, the ones nobody will ask about: event types a receiver doesn't take, another facility's patients.
• return kill("reason") filters with a reason, shown on the message in the web UI and kept in the database: "filtered: test patient". Use it for anything someone might later ask about ("why didn't the lab get this patient?"): test patients, a duplicate you recognized, a message outside a date range.
• Filtering isn't an error. A filtered message is finished, raises no alert and is never retried. It's kept for the interface's retention, so the answer to "did we get it, and what did we do with it?" is always there.
• When not to filter: when the message is wrong. A missing MRN, an unknown code, a timestamp you can't read: those are for a person, so throw quarantine("…") (quarantine). Filtering a broken message hides it.

A filter on a field's value is written in code, not configuration, so it can say exactly what it means (startsWith, a set of codes, a date compared with today) and be tested with real messages (testing).

Routing by content: one destination per receiver

# interfaces/adt-in/interface.yaml, in part
destinations:
  lab:
    mllp: { host: ${LAB_HOST}, port: ${LAB_PORT} }
    transform: toLab
  billing:
    interface: adt-to-billing       # a route: into that interface's queue
    transform: toBilling

// interfaces/adt-in/route.ts, in part
/** Billing wants admissions and discharges, as received. */
export function toBilling(msg: Message) {
  return ["A01", "A03"].includes(msg.MSH[9][2]) ? msg : null;
}

Each destination names its function with transform:. Both get the message as it was received: toLab changing the patient class and removing Z segments changes nothing for toBilling (nuance). So "route ADT to the lab, and admissions and discharges to billing" is two functions of a few lines, each tested on its own, and a message can go to both, either or neither.

What each destination has of its own, and why that's the point of routing this way:

• Its own queue and order. The lab down for an upgrade holds up only the lab; billing keeps receiving (nuance).
• Its own outcome. The lab can quarantine a message billing sent; the message shows both (nuance).
• Its own pause, retries, rate limit and alerts (delivery).

When to split instead. Past a handful of destinations, or when destinations belong to different teams who change them on different days, give each receiver an interface of its own and route to it (below): each then has its own code, owner, tests and deploys (when to split).

To another interface

# interfaces/adt-to-billing/interface.yaml
name: adt-to-billing
about:
  description: Admissions and discharges from adt-in, as JSON encounters for the billing system
  owner: revenue-cycle-team
  to: { application: Billing, vendor: Experian }
source: internal                    # messages come only from another interface's route
script: billing.ts
ordering: { key: PID-3.1 }
output: json
destination:
  http: { api: billing, path: /encounters }

A destination with interface: adt-to-billing puts the message into that interface's queue, in the same database transaction that marks it sent from here. Nothing crosses a network, and nothing is lost or sent twice between the two (nuance). The receiving interface has source: internal and is an interface like any other: its own code, ordering key, dedupe, destinations, retries, owner and alerts (nuance). Its code sees meta.origin: the message's id in the interface it came from.

Use a route to:

• Hand a feed to another team. The integration team owns adt-in (the connection to the EHR, the ACKs, the dedupe); revenue cycle owns adt-to-billing, its mapping and its alerts.
• Chain steps that change at different speeds: a normalizing interface every downstream interface routes from, so a change of the EHR's quirks is made once.
• Fan one feed out to many receivers without one interface carrying a dozen destinations.

What to know:

• The route is fixed in interface.yaml. Code can't pick an interface at run time; it decides only whether this destination sends (null or the message). That's what lets ductus validate refuse a loop and check that the receiving interface reads the format this one writes, and ductus impact follow a vendor's messages through every hop (nuance).
• Order holds across a route when the sending interface keeps it (an ordering key), and the receiving one has a key too (nuance).
• A routed message waits up to 20 ms before the next interface sees it: nothing a feed notices (nuance).

Quarantine: for a person to look at

// lib/codes.ts, in part
export function translate(table: Readonly<Record<string, string>>, code: string, what: string): string {
  const value = table[code];
  if (value === undefined) throw quarantine(`unknown ${what} "${code}"`);
  return value;
}

throw quarantine("reason") holds this destination's delivery of the message for a person: it shows on the Quarantine page and in the interface's error count, alerts when your rules name error, and waits until someone resends it (after fixing the code or the table), corrects and resends it, or dismisses it (resend and correct). Write the reason for the person who will read it at 3 a.m.: what's wrong and with which value ("unknown patient class "X""), not "error".

Any other error your code throws (a TypeError from a field that isn't there, a failed JSON.parse) quarantines the message too, with the error's text. Prefer quarantine() for every case you can foresee: its reason is yours, not a stack trace.

Quarantine doesn't stop the queue. The next message for that destination goes on, so one bad message doesn't hold up the feed. Messages that wait instead, in order, are the ones that hit a temporary failure: a receiver or database that's down. Those are tried again until they go (nuance).

Routing by sender or event without code

Two cases often handled by routing rules elsewhere need none here:

• Different senders on different ports: give each sender its own interface (its own source), so each has its own about.from, allow: list and alerts. Two senders on one port can't be told apart reliably by content.
• A repeat of a message already received: dedupe on MSH-10 recognizes it before any code runs.

Transforms: mappings first, then code

A transform turns the message a sender sent into the one a receiver wants: a field moved, a code translated, a segment removed, or a whole new message in another format. This page goes from the smallest change to a full mapping, in the order you should reach for them.

Change the message in place

For most HL7 receivers, the message you send is the one you received with a few fields changed. Change it and return it:

// interfaces/adt-in/route.ts, in part
  msg.PV1[2] = translate(PATIENT_CLASS, msg.PV1[2], "patient class");
  for (const name of new Set(msg.segmentNames)) {
    if (name.startsWith("Z")) msg.remove(name); // the EHR's own segments mean nothing to the lab
  }
  msg.set("MSH-5", "LAB");
  return msg;

• Only what you touch changes. Segments your code doesn't change are written out exactly as they arrived, character for character, so a receiver's validation sees what the sender sent.
• Two ways to name a field: by position, msg.PV1[2] (with completion in the editor), or by path, msg.get("PV1-2") / msg.set("PV1-2", "IN"). Paths are handy when the field is data (a table, below); positions read better in code.
• No copy needed. Each destination's function already gets its own message as received, so changing it can't affect another destination (nuance).
• The reference for every way to read and write HL7 (components, repetitions, adding and removing segments, escaping) is Formats: HL7 v2.

A mapping as a table

When the receiver wants a different shape (JSON for an API, a flat record for a file), most of the work is "this field goes there". Write that part as a table: one row per field, which an analyst can read, review in a pull request and check against the receiver's spec line by line.

// interfaces/adt-to-billing/billing.ts, in part
/** The encounter the billing API takes, field by field. */
const ENCOUNTER: FieldMap = [
  ["PID-3.1", "patient.mrn"],
  ["PID-5.1", "patient.family"],
  ["PID-5.2", "patient.given"],
  ["PID-7", "patient.birthDate"],
  ["PV1-19", "visit.number"],
  ["PV1-3.1", "visit.unit"],
  ["PV1-44", "visit.admitted"],
  ["PV1-45", "visit.discharged"],
];

export async function transform(msg: Message, meta: Meta) {
  const encounter = mapFields(msg, ENCOUNTER);
  encounter.event = msg.MSH[9][2] === "A03" ? "discharge" : "admit";

The table is plain data; the few lines that apply it are a helper in the project's lib/, shared by every interface that maps this way:

// lib/mapping.ts, in part
export function mapFields(msg: Message, map: FieldMap): Record<string, unknown> {
  const out: Record<string, unknown> = {};
  for (const [from, to] of map) {
    const value = msg.get(from);
    if (value !== "") place(out, to, value);
  }
  return out;
}

• The table does the predictable part; code does the rest: the event that depends on the trigger, the NPI looked up in a database (lookups), the id that makes a repeat harmless. Nothing forces every field through the table.
• Empty fields are left out (mapFields skips ""), so a receiver sees a missing discharge time as missing, not as an empty string. If your receiver wants null or "" instead, change the helper once.
• HL7 to HL7 works the same way: copyFields(from, to, [["PID-3.1", "PID-2.1"]]), also in lib/mapping.ts, copies between two messages by path.
• Test the table with a whole message and compare the whole result (testing): a wrong row is then a failing test, not a receiver's complaint.

Ductus has no mapping file or mapping screen of its own: the table is code, kept with the interface in Git. That's deliberate for now (one place to look, one way to test, type-checked by ductus validate); typed message definitions that underline a wrong field in the editor are on the roadmap (v1.5).

Codes from one system to another

A sending system's codes are rarely the receiver's: patient classes, sexes, result statuses, units, locations. Keep each translation as a code table and translate through one function that quarantines a code it doesn't know instead of passing it on or guessing:

// lib/codes.ts, in part
/** PV1-2 patient class, from the EHR's codes to the lab system's. */
export const PATIENT_CLASS: Readonly<Record<string, string>> = {
  I: "IN",
  O: "OUT",
  E: "ER",
  P: "PRE",
  R: "OUT", // recurring patients are outpatients to the lab
};

A new code at the sender then shows up as quarantined messages with "unknown patient class "B"", the day it starts, instead of as wrong data in the receiver weeks later. Add the row, deploy, and resend the quarantined messages. Where tables live (in code, or in a database the business maintains) and how to keep a database lookup fast: Lookups and code tables.

A new message from scratch

When the receiver's message has little in common with the sender's (a CSV row becoming HL7, an HL7 result becoming FHIR), build the new one and return it:

// interfaces/census-in/census.ts, in part
export function transform(row: CsvRecord, meta: Meta) {
  if (!row.mrn) return kill("a row without an MRN");
  const now = hl7Time(new Date(meta.receivedAt));
  const msg = parseHl7(`MSH|^~\\&|BEDBOARD|HOSP|EHR|HOSP|${now}||ADT^A08^ADT_A01|BB${meta.id}|P|2.5`);
  msg.add("EVN", ["A08", now]);
  msg.add("PID");
  msg.set("PID-3.1", row.mrn);
  msg.set("PID-5.1", row.last);
  msg.set("PID-5.2", row.first);

• Start an HL7 message from its MSH with parseHl7 (in a TypeScript string, the escape character \ is written \\), then add segments and set fields; a field set past the end of a segment pads it with empty fields.
• Times and ids from meta, not the clock: meta.receivedAt and meta.id are the same every time this message is processed, so a retry, a resend or a replay produces the same message (and MSH-10 BB501 the second time lets a receiver that dedupes drop the repeat). new Date() would give a different message each time.
• An object returned to a JSON destination is the request body as it is; there's an example in Formats: FHIR.

When it's more than a mapping

Everything above is ordinary TypeScript, so anything a mapping table can't say is a few lines more: a loop over every OBX, a value computed from two fields, a lookup, a decision on today's date. Keep these rules:

• Same message, same answer. No Math.random, no new Date() in what you send, nothing kept from one message to the next except a bounded cache (nuance).
• Say why a message can't be handled: throw quarantine("…") with what's wrong and which value, never return null for a broken message (routes and filters).
• Ask once for many values: a lookup per OBX in a loop is a round trip per OBX (nuance).
• Big messages: build objects with every property in the same order, and let per-message objects go at the end (nuance).
• Keep it fast: a function running past timeout (5 s by default) without waiting on anything is stopped and its message quarantined. For most HL7 transforms 500ms is plenty.

Formats: HL7 v2, FHIR, X12, CSV, JSON and XML

An interface's format says what its messages are, so your code gets them parsed; output (on the interface or on one destination) says what your code's results are written as. Both default to hl7.

A string returned is always sent as it is, whatever the output format: the way out when a receiver wants something no format writes. An MLLP source always carries HL7.

Converting is the two settings together: an HL7 feed to an API is format: hl7 (the default) with output: json; a CSV file to HL7 is format: csv with output: hl7. Each destination can set its own output, so one HL7 feed can go out as HL7 to the lab and JSON to billing.

FHIR and X12 have no format of their own yet: FHIR is JSON, and X12 is read as text with a few lines of code. Both are below, with what's planned.

HL7 v2

// interfaces/adt-in/route.ts, in part
export function toLab(msg: Message) {
  if (!LAB_EVENTS.has(msg.MSH[9][2])) return null; // not for the lab: filtered, no reason needed
  if (msg.PID[3][1].startsWith("TEST")) return kill("test patient");

Numbering is HL7's, 1-based everywhere: msg.PID[3][1] is PID-3.1, msg.MSH[9][2] is MSH-9.2 (the trigger event). MSH is counted as the standard counts it: MSH-1 is the field separator itself, so MSH-9 is the message type, as in every spec.

• Indexing goes deeper, iterating goes across: msg.OBX[5] is OBX-5 of the first OBX; for (const obx of msg.OBX) visits each one. Most "only the first result was mapped" bugs are this.
• Missing reads as "" and never throws: a segment, field or component that isn't there is "", so code needs no null checks, but a wrong position is a quiet "". Test the fields you map (nuance).
• Writing into a segment the message doesn't have throws ("message has no PV1 segment; add it first with msg.add("PV1")"): a missing segment is usually a mistake worth seeing.
• Simple fields read as text, composite ones as fields. PV1-2 (a code) is a string; PID-5 (a name, with components) is a field you index into. The editor knows which is which for the common segments of HL7 2.5 (MSH, EVN, PID, NK1, PV1, ORC, OBR, OBX, NTE, AL1, DG1, MRG, MSA, ERR) and shows each field's name; any other segment, Z segments included, works the same way with numbers.
• Escaping is done for you. Reading decodes \F\, \S\, \T\, \R\ and \E\; writing a component or simple field encodes the delimiters in it. A composite field set as a whole (msg.PID[5] = "SMITH^ANNA") is taken as HL7, separators and all.
• MSH-1 and MSH-2 are read-only: the delimiters are the sender's, and the message is written with them.
• Untouched segments go out exactly as they came, so a message relayed with one field changed differs from the original in that field only.
• The ACK isn't your code's job. The engine answers the sender (AA once the message is stored) and reads the receiver's ACK (AA delivered, AE retried, AR quarantined by default: destinations).
• Batch files (many messages in one file) are split at each MSH into one message each by a file source with split: records (sources).

FHIR

FHIR R4 resources are JSON, so a FHIR interface is a JSON one: format: json for resources coming in, output: json for resources going out, and an HTTP destination (or api()) to the FHIR server. Here, each lab result (an HL7 ORU) becomes a FHIR transaction Bundle with one Observation per OBX:

# interfaces/results-to-fhir/interface.yaml
name: results-to-fhir
about:
  description: Lab results as FHIR R4 Observations, posted to the FHIR server as one transaction each
  owner: integration-team
  from: { application: Lab system }
  to: { application: FHIR server }
source: { mllp: { port: 6662 } }
script: to-fhir.ts
ordering: { key: PID-3.1 }
output: json
destination:
  http: { api: fhir, path: / }      # a transaction Bundle is posted to the server's base URL

// interfaces/results-to-fhir/to-fhir.ts, in part
/** An ORU^R01 as a FHIR R4 transaction Bundle: one Observation per OBX, for the patient by MRN. */
export function transform(msg: Message) {
  const mrn = msg.get("PID-3.1");
  if (!mrn) throw quarantine("a result without an MRN (PID-3.1)");
  const order = msg.get("OBR-3.1"); // the lab's own order number
  const subject = { reference: `Patient?identifier=${MRN_SYSTEM}|${mrn}` };

  const entry = [];
  for (const obx of msg.OBX) {
    const id = `${order}-${obx[1]}`; // the same on every resend, so the server can tell
    entry.push({
      resource: observation(obx, id, subject),
      request: { method: "POST", url: "Observation", ifNoneExist: `identifier=${RESULT_ID_SYSTEM}|${id}` },
    });
  }
  if (entry.length === 0) throw quarantine("a result with no OBX segments");
  return { resourceType: "Bundle", type: "transaction", entry };
}

What it shows, and why each part is there:

• A delivery can happen twice (a lost answer, a resend), so each Observation carries an identifier made from the result itself and is created with ifNoneExist: the server creates it once and answers the repeat with the one it has (nuance). Every FHIR create should have such a key.
• The patient is found by MRN with a conditional reference (Patient?identifier=…|123456), which the server resolves inside the transaction: no lookup from your code first.
• A transaction is all or nothing on the server: if one Observation is refused, none is stored, and the server's 4xx quarantines the message with its answer.
• FHIR's times need an offset whenever they have a time of day. HL7 v2 times often have none; guessing the server's offset is wrong half the year, so the example quarantines a time without one ("ask the sender to send offsets" is the fix). If your sender can't, convert with the site's time zone rules deliberately, in one helper, and test both sides of a daylight-saving change.
• The answer: a 2xx is delivered; 408, 429 and 5xx are tried again; other 4xx quarantine the message with the server's answer (an OperationOutcome saying which element was wrong) for a person to read (destinations).
• Content type: JSON goes out as application/json, which FHIR servers accept. One that insists on application/fhir+json gets it from the destination's headers, which win over the default: http: { api: fhir, path: /, headers: { Content-Type: application/fhir+json } }.
• Tokens: the FHIR server's base URL and token live on its apis: entry in resources.yaml, never in code (resources). Servers that want OAuth2 client credentials or SMART backend services (Epic, Cerner): the engine doesn't fetch those tokens yet (planned with OAuth2 for outbound APIs); until then, a token your organization issues and renews into the secret.

Reading FHIR from a server is api("fhir").get("/Patient?identifier=…") in any interface's code (lookups); receiving FHIR (a subscription's notifications, an app's POST) is an HTTP source with format: json. There are no FHIR typings or resource validation in the SDK yet: your code works with plain objects, and the server's validation is the check. FHIR typings and helpers are planned for v1.5, a FHIR client and server connector for v1.6.

X12 (EDI)

X12 files (835 remittances, 837 claims, 270/271 eligibility) are read as format: text, and a few lines in lib/ split them into segments and elements:

// lib/x12.ts, in part
/** Split an X12 interchange into its segments, with the separators its ISA segment declares. */
export function parseX12(text: string): X12Segment[] {
  const data = text.replace(/^/, "").trimStart();
  if (!data.startsWith("ISA") || data.length < 106) throw quarantine("not an X12 interchange: it must start with ISA");
  const element = data[3]; // the character right after "ISA"
  const segment = data[105]; // ISA is fixed-length: its terminator is the 106th character

// interfaces/remits-in/remits.ts, in part
  for (const seg of parseX12(text)) {
    if (seg.id === "TRN") trace = seg.e(2);
    else if (seg.id === "N1" && seg.e(1) === "PR") payer = seg.e(2);
    else if (seg.id === "CLP") {

• Read the separators from ISA, never assume * and ~: trading partners use |, ^ and others, and ISA is fixed-length precisely so its characters can be found by position.
• Elements are numbered as X12 numbers them: seg.e(1) is CLP01.
• Loops are code: X12's structure (the 2100 claim loop, its NM1 patient) is found by walking the segments in order, as the example does for each claim's patient.
• What isn't there yet: acknowledgements (TA1, 999) aren't produced for you; an X12 format with loops and typed segments is planned (v1.6). Until then, send a 999 back as its own destination if your trading partner needs one.

The whole interface turns an 835 into one CSV row per claim, for the billing office: examples/guides/interfaces/remits-in (CSV below for the output).

CSV and fixed-width

# interfaces/census-in/interface.yaml
name: census-in
about:
  description: The bed board's nightly census file, each row an ADT A08 for the EHR
  owner: integration-team
  from: { application: Bed board }
  to: { application: EHR, vendor: Epic }
source:
  file: { dir: inbound, pattern: "census_*.csv", split: records }   # each row is a message
format: csv
script: census.ts
output: hl7
destination: { file: { dir: out } }

• Records are objects by column name (row.mrn), from the header line. Without one, format: { csv: { header: false } } gives arrays of strings; another delimiter is format: { csv: { delimiter: "|" } }. Quoted fields with delimiters, quotes and line breaks inside are read as RFC 4180 says.
• One message per file, or per row: without split, the whole file is one message and transform gets every record (an array); with split: records, each row is a message of its own (the header kept with each), with its own status, retries and quarantine, and transform gets one record. Split files whose rows are separate things (patients, orders); keep whole files that only mean something together (a batch the receiver must get complete).
• Writing CSV: return a record or a list of records with output: csv; columns are written in the order first seen, with a header row. toCsv and parseCsv from the SDK do the same in your own code.
• Fixed-width layouts name each field's start column (0-based) and width (format: { fixed: { fields: [{ name: mrn, start: 0, width: 10 }, …] } }); reading trims the padding unless trim: false, writing pads (align: right, pad: "0" for numbers) and cuts values that are too long. Cutting is silent: check lengths in code when a cut value would be wrong (a name is fine shortened, an account number isn't).

JSON

format: json gives your code the parsed value; return an object (or array) and it's sent as JSON. A file of JSON Lines (one document per line) with split: records is one message per line. Numbers are JavaScript's: integers past 2^53 (some systems' ids) lose digits when parsed, so ask for them as strings or read them with format: text and parse yourself.

XML

// interfaces/orders-xml/orders.ts
import { quarantine, type XmlElement } from "@ductus/sdk";

/** A home-care equipment order for the supplier: the patient's MRN checked, the SSN taken out. */
export function transform(doc: XmlElement) {
  const mrn = doc.get("Patient/Identifiers/MRN"); // text at a path, "" if missing
  if (!mrn) throw quarantine("an order without Patient/Identifiers/MRN");
  doc.find("Patient")?.remove("SSN"); // remove child elements by name
  doc.set("Patient/Status", "verified"); // creates missing elements
  doc.set("@source", "ductus"); // an attribute, on the root
  const second = doc.get("Items/Item[2]/@sku"); // the second Item's attribute, 1-based
  if (second) doc.add("Note", `more than one item; the second is ${second}`);
  return doc;
}

• Paths are relative to the element, with / between names, [n] for the n-th of a name (1-based) and @name for an attribute as the last step. find gives the element, elements(name) all children with a name, add(name, text) a new child.
• Namespaces are part of the name: an element written <hl7:patient> is hl7:patient in a path, prefix and all. The prefix is the document's, so if a sender can change its prefixes, compare local names (el.name.split(":").pop()).
• The <?xml …?> declaration is kept; comments and processing instructions are dropped, and whitespace between elements is formatting, not content.
• An XML file is always one message: split: records doesn't split XML.

Text

format: text gives the payload as a string and sends a string back as it is: for formats Ductus doesn't parse (X12 above, a vendor's own layout), and for relaying anything unchanged.

Lookups and code tables

Interface code often needs something the message doesn't carry: a provider's NPI, a patient's identifier in another system, the code a receiver uses for a location. This page is how to look those up, where code tables should live, and what happens when the thing you ask is down.

Interface code has no network of its own. Everything goes through five SDK functions, so the engine can time each call, retry what's temporary, mock it in tests and list it in ductus impact:

Write the name out: db("provider_db"), not db(name). ductus validate then checks the database exists, and ductus impact knows this interface uses it; a name in a variable gets a warning (DX4) because neither can see it (nuance).

A database lookup

// interfaces/adt-to-billing/billing.ts, in part
// Providers' NPIs by the EHR's id, kept for an hour: they rarely change, and messages come often.
const npis = new Map<string, { npi: string; until: number }>();

async function npiOf(ehrId: string): Promise<string> {
  const known = npis.get(ehrId);
  if (known && known.until > Date.now()) return known.npi;
  const row = await db("provider_db").one<{ npi: string }>("SELECT npi FROM providers WHERE ehr_id = $1", [ehrId]);
  if (!row) throw quarantine(`provider ${ehrId} has no NPI in provider_db`);
  npis.set(ehrId, { npi: row.npi, until: Date.now() + 60 * 60 * 1000 });
  return row.npi;
}

• query(sql, params) gives rows as objects, one the first row or null, execute the number of rows a statement changed.
• Always parameters, never text pasted into SQL: $1, $2… on Postgres and SQL Server alike. Values from a message are whatever the sender sent.
• SQL Server sends text parameters as nvarchar: compared with an indexed varchar column, that can turn an index seek into a scan. Cast it: WHERE ehr_id = CAST($1 AS varchar(20)). One statement and one result set per call (nuance).
• Not found is a decision. Here an unknown provider is quarantined, because billing can't bill without the NPI. For an optional field, leave it out instead. Never send a guess.
• Many values, one query: looking up every OBX's code one at a time is a round trip each. Collect them and ask once (an array parameter is jsonb in Postgres) (nuance).
• Writes must be safe to repeat. A message's code can run again (a retry, a resend), so an execute that inserts uses the message's meta.id as a key, or an upsert.
• The login, host and TLS of the database are in resources.yaml, with the password in a secret; the code never sees them.

Caching

The lookup above keeps each answer for an hour in a Map at the top of the file. Do that for anything that changes rarely and is asked often (providers, locations, a code list); a feed of 100,000 messages a day then makes a few hundred queries, not 100,000.

• A cache lives in one isolate: each worker thread has its own copy, and it's empty after a restart (nuance). Fine for a cache; never use it for anything that must be counted or remembered.
• Bound it: keyed by something with a known number of values (providers), with a time limit. A cache keyed by patient grows with every patient until the engine restarts and counts toward the isolate's memory (nuance).
• Choose the time by how wrong a stale answer is: an NPI an hour old is fine; a patient's current insurance probably isn't.

An API lookup

// lib/empi.ts, in part
/** The patient's enterprise id from the EMPI, by MRN. */
export async function enterpriseId(mrn: string): Promise<string> {
  const r = await api("empi").get(`/patients/${encodeURIComponent(mrn)}`);
  if (r.status === 404) throw quarantine(`the EMPI doesn't know MRN ${mrn}`);
  return r.json().enterpriseId;
}

• api("name") sends to a path under the API's URL from resources.yaml, with its headers (the token) and TLS added by the engine: credentials never pass through your code. Prefer it to fetch() for anything with a token (nuance).
• fetch(url, options) for a URL of its own: method, headers, body (an object is sent as JSON), query, timeout (ms, default 30 s).
• Temporary failures wait and retry by default: a network error, a timeout, 408, 429 or a 5xx makes the message wait (in order) and try again later, so an API that's down for ten minutes costs ten minutes, not a quarantine (nuance). Every other status comes back to your code to decide: check r.status or r.ok.
• retry: false gives you every status and turns network errors into ordinary errors (which quarantine unless you catch them): for a call whose failure shouldn't hold the message up, such as an optional enrichment.
• r.json() and r.text() need no await: the body is already read.

Variables and secrets in code

env("FACILITY") reads a variable from the environment's file, and secret("NAME") a secret; both throw if it isn't defined (so a missing value is a quarantine with its name, not an empty string sent on). Prefer settings to code: a URL, a host or a token used by a connection belongs in resources.yaml or interface.yaml (${NAME}, ${secret:NAME}), where ductus validate checks each environment has it. Use env() for values only code uses (a facility code to stamp, a cut-off date).

Code tables: where they should live

A code table translates one system's codes into another's. There are three good places for one, and the right one depends on who changes it and how often:

// lib/codes.ts, in part
/**
 * The code's translation in `table`, or quarantine the message: an unknown code is for a person
 * to look at (add it to the table, or fix it at the source), never something to guess.
 *
 *   msg.PV1[2] = translate(PATIENT_CLASS, msg.PV1[2], "patient class");
 */
export function translate(table: Readonly<Record<string, string>>, code: string, what: string): string {
  const value = table[code];
  if (value === undefined) throw quarantine(`unknown ${what} "${code}"`);
  return value;
}

Whichever you choose:

• An unknown code quarantines. It's the first sign a sender added a code, and a person fixes it in minutes: add the row, then resend the quarantined messages. Passing the code on unchanged, or a default, puts wrong data in the receiver without anyone knowing.
• A table in code is immutable: Readonly<…> and const, in a shared file every interface imports (shared code).
• A table in a database needs the cache's time to match how fast a change must show: an hour, or a restart, after someone edits it. Say so to the team that maintains it.
• Test the table: one test per interesting code, and one for an unknown one (testing).

There's no code-table screen or upload in Ductus today; a shared reference-data cache across interfaces is planned (v1.4). Until then the table in Git, or a database table the business already maintains, is the way.

Lookups in respond mode

An HTTP source in respond mode answers the caller from your code, so lookups there are on the caller's clock. Calls that don't depend on each other should run together: const [a, b] = await Promise.all([db("x").one(…), api("y").get(…)]), not one after the other (nuance).

Testing with fixtures

ductus test runs every interface's *.test.ts files inside the real engine's JavaScript runtime, with the SDK the engine uses, and no database, network or running engine. Tests are how you know a mapping does what the receiver's spec says before a message reaches them, and how the next person changing it knows they didn't break it.

$ ductus test
adt-in
  ✓ the lab gets visits, with its own patient class and no Z segments (1ms)
  ✓ the lab doesn't get patient updates (A31) (0ms)
  ✓ test patients are dropped with a reason (0ms)
  ✓ an unknown patient class is quarantined, not guessed (1ms)
  ✓ billing gets admissions and discharges only (0ms)
adt-to-billing
  ✓ an admission becomes an encounter, with the attending's NPI (1ms)
  ✓ the NPI is looked up once an hour, not once a message (0ms)
  ✓ a provider the database doesn't know is quarantined (1ms)
…
17 passed, 0 failed

It exits 1 when a test fails, so it belongs in the checks every change goes through (a pull request's CI, a script before deploying), next to ductus validate --env prod, which type-checks the code and checks the configuration against that environment. --interface adt-in runs one interface's tests.

A test

// interfaces/adt-in/route.test.ts, in part
test("the lab gets visits, with its own patient class and no Z segments", () => {
  const out = toLab(hl7("a01.hl7")) as Message;
  expect(out.get("PV1-2")).toBe("IN");
  expect(out.get("MSH-5")).toBe("LAB");
  expect(out.segmentNames).toEqual(["MSH", "EVN", "PID", "PV1"]);
});

• A test file sits next to the code (route.test.ts beside route.ts), imports the functions it tests, and registers tests with test(name, fn). The name is a sentence saying what should happen: it's what a failing run prints.
• Tests run one at a time, in the order written; an async test is awaited.
• Name what the receiver cares about: a test per rule in the spec ("the lab doesn't get A31s"), not one per function.

Fixtures: real messages, without the patient

A test's messages are files in the interface's fixtures/ folder:

• Start from real messages. Take a few of each kind the sender sends (each event type, the odd ones: a merge, a result with 40 OBX, a name with an apostrophe) from the sender's test system or a message the engine has stored.
• Never commit a real patient. Fixtures live in Git, read by everyone with the repository. Replace names, MRNs, dates of birth, addresses and identifiers with made-up ones before saving the file; keep the structure (how many repetitions, which fields are empty, the sender's quirks), which is what the test needs.
• Fixtures can be read only inside a test, not when the test file loads: put hl7("a01.hl7") inside test(...), or in a function the test calls (nuance).
• With split: records, input() gives the fixture's first record, as the engine gives transform one record per message. For a file of several rows, parse it yourself and pick: parseCsv(fixture("census_20261031.csv"))[1] (nuance).
• Change a fixture in the test for a variation instead of keeping a file per case:

// interfaces/adt-in/route.test.ts, in part
test("an unknown patient class is quarantined, not guessed", () => {
  const msg = hl7("a01.hl7");
  msg.set("PV1-2", "X");
  expect(() => toLab(msg)).toThrow('unknown patient class "X"');
});

Checking results

An HL7 field passed to expect is compared by its text, so expect(msg.PID[5]).toBe("SMITH^ANNA^M") works.

Compare the whole result

For a mapping, compare the whole output with toEqual, not a field or two: a row added to the mapping, or a field that's suddenly empty, then fails the test.

// interfaces/adt-to-billing/billing.test.ts, in part
test("an admission becomes an encounter, with the attending's NPI", async () => {
  const calls = mock.db("provider_db", () => [{ npi: "1093817465" }]);
  const out = await transform(hl7("a01.hl7"), meta);
  expect(out).toEqual({
    patient: { mrn: "123456", family: "SMITH", given: "ANNA", birthDate: "19800214" },
    visit: { number: "V9001", unit: "4W", admitted: "20261031083000-0500" },
    event: "admit",
    attendingNpi: "1093817465",
    sourceMessage: 41,
  });
  expect(calls[0].params).toEqual(["1234"]);
});

Filters, kills and quarantines

• Filtered: expect(toLab(hl7("a31.hl7"))).toBeNull().
• Killed with a reason: the result is a Kill; check its reason:

// interfaces/adt-in/route.test.ts, in part
test("test patients are dropped with a reason", () => {
  const out = toLab(hl7("test-patient.hl7"));
  expect(out instanceof Kill && out.reason).toBe("test patient");
});

• Quarantined: toThrow (or toReject for async code) with part of the reason, as above. Test every quarantine you wrote: it's the behaviour someone relies on at 3 a.m.

meta

A function that reads meta gets one from the test: a plain object with the fields it uses.

// interfaces/adt-to-billing/billing.test.ts, in part
const meta: Meta = { id: 41, receivedAt: "2026-10-31T13:30:00Z", destination: "default", origin: 7 };

Mocks: databases, APIs, variables and secrets

Tests never reach a real database or API. A call your test didn't mock fails with what to add ("db("provider_db") is not mocked in this test: add mock.db("provider_db", (sql, params) => [/* rows */])"), so a test can't quietly depend on a system being up.

• Mocks last one test: each test starts with none, so tests don't depend on each other's. What your own code keeps (a module-level cache) does last, as it would in the engine: the caching test above uses a provider id no other test uses.
• Check what was asked, not only what came back: calls[0].params is the MRN your code sent; calls.length shows a lookup made once, not per message.
• Answer like the real system, including its failures: a 404, an empty result, the odd shape it sends for a merged patient.

// interfaces/adt-to-billing/empi.test.ts, in part
test("an MRN the EMPI doesn't know is quarantined", async () => {
  mock.api("empi", { status: 404 });
  await expect(enterpriseId("999")).toReject("the EMPI doesn't know MRN 999");
});

A mocked temporary failure (a 503) makes your code's call throw the same temporary error the engine would wait on and retry; tests usually check the answers your code decides on instead.

Beyond unit tests

• ductus validate type-checks every interface and checks its configuration and resources for one environment: run it for each environment you deploy to.
• ductus replay runs a real period's stored messages through a changed version of the code and compares every destination's output with what the installed version sent, before you deploy: the test against real traffic that fixtures can't be (replay).
• ductus gen and ductus sink send synthetic ADT and receive it, to try a whole project on your laptop with ductus dev (first interface).

Running Ductus day to day

These pages are for whoever keeps the interfaces running: the on-call analyst, the integration team, the operations center. They assume the engine is installed (install) and the project deployed; what each setting means is in Configure.

Where to look

The console, the CLI and the API are three ways into the same engine: a pause made in one shows in the others, and every change, whoever makes it and however, is in the audit log with who made it.

What a normal day looks like

• The Overview is green, or shows exactly what isn't: a destination behind, a quarantined message, a certificate expiring. Start every shift there, or let alerts tell you.
• Quarantined messages get looked at the same day. Each one is a message a receiver didn't get. Read the reason, fix what caused it (the sender, the code, a code table), then resend, correct or dismiss it (resend and correct).
• A receiver's planned downtime is a pause, made before it starts, with the ticket as the reason, and a resume after (pause).
• Backlogs drain on their own. A receiver that was down gets everything it missed, in order, when it's back; nothing needs resending.
• Nothing to clean up. Old messages go by themselves, partition by partition, at the retention you set; there's no purge job and no vacuum to schedule (retention and storage).

When something's wrong

What each state means exactly, and when it shows: the states.

Before you change anything

• Upgrades: ductus upgrade --check with the new version first, then ductus replay for the evidence a change board asks for (resend and replay).
• A project change (new code, a new destination): ductus test and ductus validate --env prod in CI, then deploy. On Linux, systemctl reload ductus deploys without refusing a single connection (high availability).
• A shorter retention deletes messages, so it waits for a confirmation (retention).

The console (web UI)

The engine serves its web UI on the same address as its REST API: api.listen, http://127.0.0.1:7700/ by default. Nothing is installed on anyone's desktop, and there's no second port: whoever can reach the API can open the console in a browser.

Getting to it

• On the server itself it works as installed: http://127.0.0.1:7700/.
• From people's desks it needs the API on the network, which means TLS: api.listen: 0.0.0.0:7700 with api.tls (the browser's sign-in cookie is only sent over https, or to localhost) and an allow: list of the networks your team works from (nuance, the API on the network).
• Signing in is with a username and password (local accounts, or the directory with auth.ldap), or with an API token (people). What you see and may do follows your role: viewer reads; operator also pauses, resends, edits, dismisses, drains and runs connection checks; admin also manages users and tokens. Reading message payloads is a permission of its own (payloads), and so is the audit log (audit): a role alone doesn't give them (nuance).
• Live: the header shows "Live" while the console gets the engine's updates as they happen; "Offline" means it lost the engine (stopped, or the network), and it reconnects by itself.

The pages

The menu has three groups: the day to day first, then reports, then the engine itself.

Overview

The engine at a glance: what needs attention (the worst first), open alerts, certificates expiring, and other engines sharing the database. Start a shift here. "Cut off from the database" across the top means the engine lost its database connection: it's still receiving, and sending waits until it's back (nuance).

Interfaces

Every interface with its health, source, how many messages wait and are quarantined, when it last received, and its owner; worst first. Mine shows the interfaces you own (an operator who owns something starts there). Filter by name, owner, host or port, tick Only those needing attention, or pick a documentation field's value.

An interface

Overview: who sends and who receives (about:), each destination's card (health, waiting, quarantined, last sent, its pause and reason, and Pause / Resume), the interface's open alerts, counts and timings since the engine started, and its documentation fields.

Messages: the interface's messages, newest first, filtered by status. Each row opens the message. Finding a message by its content (an MRN, a name) isn't there yet: the message store and search.

A message

What was received and what each destination did with it:

• Payload: the message as received, with each HL7 field's path shown as you point at it (PID-3.1). Needs the payloads permission, and opening it is logged before it's shown (nuance).
• Destinations: per destination, its status, attempts, error and when it was sent, and what it was sent (recomputed by running its code again, unless it has save_output: true). Resend (a quarantined delivery), Replay (a sent or filtered one), Dismiss.
• Edit and resend: a corrected copy, sent to the destinations you choose.
• Who viewed the payload and History: every read, resend, edit and dismissal, with who, when, and the reason.

What each action does exactly: resend, replay and correct.

Quarantine

Every quarantined delivery across the interfaces you can see, filtered by interface and destination, with the reason. Select many and resend or dismiss them in one go, after fixing what caused them.

Alerts

What's open (acknowledge it to stop the reminders; silence it for a while), what's silenced, the history, and each channel: where it sends, how many were sent or failed, the last error, and Test (alerts in practice).

Hotspots

Where each destination's backlog sits: the lanes messages wait in, each lane's first message (its key, age, attempts and last error), and what it means ("blocked by message 1234: look at this message", "every worker blocked: the receiver is down") (hotspots).

Reports: Impact, Owners, Documentation

• Impact: which interfaces a vendor, application or host touches, and who owns them: the first question when a vendor reports a breach or a VPN changes.
• Owners: who owns what, and whether each owner reaches a person; Update previews moving ownership from one person to another (a patch for the project, not a change in place).
• Documentation: the company's own documentation fields for every interface, destination and resource, and which are missing.

Engine: Retention, Drain, Connections, Users, Tokens, Audit

• Retention: each interface's retention, changes in progress, shorter values waiting for a confirmation (type the interface's name to confirm), and what the held area keeps (retention and storage).
• Drain: stop taking new messages and deliver what's stored, before a server move or a change that needs empty queues; Stop when drained (drain).
• Connections: ductus doctor from the engine's own server (Check now), the last scheduled checks, and every certificate the engine presents or connects to, with its expiry.
• Users and Tokens (admins): accounts, roles, permissions, lockouts; API tokens.
• Audit (the audit permission): who did what, filtered by person, action, interface or message.
• API reference: every REST operation, with who may call it.

Your account

Your role and permissions, your password (local accounts), My activity (your own audit rows), and the appearance: light, dark, high contrast, or your own colors.

The theme and colors are saved with your account, so they follow you to any browser (nuance); a palette of your own may be hard to read, and the check warns but never stops you (nuance). A session acts as whoever signed in, ends after 12 hours unused or 7 days, and at once when the account is disabled (nuance).

What the console can't do

Change the project: interfaces, code and settings live in Git and are deployed (the project). The console operates what's deployed; it doesn't edit it. That keeps every change reviewed and versioned, and is why ownership moves are previews.

The CLI day to day

Everything the console does, the ductus command does from a terminal, for runbooks, scripts and the moments you're already on the server. Every command's options are in the command-line reference; this page is which one to reach for, and what it tells you. The outputs below are from a real engine.

Which engine, as whom

Commands that act on the running engine talk to its REST API (--api, DUCTUS_API_URL, http://127.0.0.1:7700 by default) with a token (--token, DUCTUS_API_TOKEN). Commands that read the database directly need its URL (DUCTUS_DATABASE_URL). On the server, the service's environment file already has the URL; give the token yourself:

$ ductus token create ops-probe --role operator --permission payloads
dux_12b318581a5c9072d1603a927b0fd7c869aea43730050f668b48a94597568401
token `ops-probe` (operator) created. This is the only time it's shown: keep it in a secret store.
Use it as `Authorization: Bearer <token>`, or DUCTUS_API_TOKEN for `ductus drain`.

• One token per person or tool, named after it (ops-desk, monitoring, ci): every action made with it is in the audit log under that name, and revoking one (ductus token revoke ops-desk) stops only it.
• The smallest role that does the job: viewer for status, hotspots and metrics; operator to pause, resend, drain. A token reads payloads only with --permission payloads.
• ductus token create and ductus user work on the database directly (that's how the first admin is made), so they need DUCTUS_DATABASE_URL, not a token.

How things are: ductus status

$ ductus status
adt-in               sent         300

by destination:
adt-in → archive     sent         300
adt-in → lab         sent         300

health:
adt-in               ok, received 3s ago
  → archive          ok, sent 3s ago
  → lab              ok, sent 3s ago

The counts come from the database, so they work while the engine is down (on the server, after a crash, to see what's waiting). The health lines come from the running engine and need a token: each interface and destination in the words the console uses. When something's wrong it says what and how much:

health:
adt-in               ok, received 1 min ago
  → archive          ok, sent 1 min ago
  → lab              stalled: nothing sent for 1 min, 30 waiting; retrying: Connection refused (os error 111)

Pause and resume a destination

For a receiver's planned downtime (an upgrade, a weekend cut-over), pause its destination before it goes down: nothing is sent or retried, messages keep arriving and wait in order, and a paused destination never alerts (nuance).

$ ductus pause adt-in lab
`adt-in` destination `lab` paused by ops-probe at 2026-10-31 06:34: nothing is sent until `ductus resume adt-in lab` (0 waiting)
no reason yet: add one when you can with `ductus pause adt-in lab --reason "..."`
$ ductus pause adt-in lab --reason "lab system upgrade until 14:00, ticket CHG0042"
`adt-in` destination `lab` paused by ops-probe at 2026-10-31 06:34: nothing is sent until `ductus resume adt-in lab` (0 waiting)
reason: lab system upgrade until 14:00, ticket CHG0042

• Act first, explain after: in an incident, pause without a reason and add it when you can. The reason shows on the destination's card, in status and in the audit log, so the next shift knows why and until when.
• The interface keeps receiving: senders get their ACKs as usual. Only this destination waits; the interface's other destinations carry on.
• Resume sends the backlog in order:

$ ductus resume adt-in lab
`adt-in` destination `lab` resumed (paused since 2026-10-31 06:34): sending 80 waiting, in order

When not to pause: for an unplanned outage you haven't seen yet, nothing is needed: the destination retries by itself and sends everything when the receiver is back. A pause is for when you know it's down and want it quiet.

Where a backlog sits: ductus hotspots

$ ductus hotspots adt-in --lanes 3
adt-in → lab: 30 waiting in 8 of 8 lanes; blocked by message 381 (key 102001, lane 1, 7 attempts, waiting 1 min): Connection refused (os error 111); all 29 behind it wait (one worker: order)
  lane   worker    waiting  first msg    key                     age attempts  last error
  1      0               6  381          102001                1 min        7  Connection refused (os error 111)
  6      0               6  388          102008                1 min        0
  0      0               3  382          102002                1 min        0
  (5 more lanes waiting: --lanes N lists more)

The first line is the diagnosis. Here every message waits behind message 381, which can't reach the lab: the receiver is down, and nothing is to be done on this side. When instead the others were sent and one lane is stuck, it says "look at this message": one message the receiver keeps refusing with a temporary error, holding up its patient's lane (monitoring).

Drain before a change

ductus drain stops taking new messages (MLLP listeners close, queued HTTP answers 503, file pickup and schedules pause) and delivers everything stored. Use it before moving the engine to another server, or a change that needs empty queues (fewer ordering.partitions).

$ ductus drain --wait
not taking new messages; delivering what's stored
draining: 31 pending (adt-in)
drained
$ ductus resume
accepting: taking new messages

• --stop exits the engine once drained (the service manager then sees a clean stop).
• Senders are refused while it drains, on purpose: they hold their messages and resend once the engine takes them again (nuance).
• A drain isn't needed for an upgrade or a project deploy on Linux: systemctl reload ductus hands over without refusing anything (high availability).
• A drain finishes only when every destination has caught up, so a receiver that's down keeps it draining (nuance). Drain mode isn't remembered across a restart.

What a drain does to each kind of interface: MLLP listeners close (a message half received when it started is the sender's to send again: no AE is answered, nuance); respond-mode HTTP keeps answering, since nothing waits in a queue for it (nuance); and a drain isn't remembered across a restart: a restarted engine takes messages again (nuance).

Alerts

$ ductus alerts
CRITICAL  adt-in → lab: stalled since 2026-10-31 06:35  [alert 3]
          stalled: nothing sent for 1 min, 30 waiting; retrying: Connection refused (os error 111)

What's open now. ductus alerts test chat sends a test notification to the chat channel straight from ductus.yaml, without an engine: try every channel before going live (alerts in practice).

Check every connection: ductus doctor

Run on the engine's server (where the firewall rules apply), it checks every connection the project needs: DNS, TCP from which address, TLS and the certificate, then each system's own login. It never sends a message to a destination.

$ ductus doctor
ductus doctor · environment dev · 2026-10-31 06:35:42 -04:00
on DESKTOP-GCT6E6V · ductus 0.6.0-dev
checked 5 connection(s) in 0.0 s: 4 ok, 1 warning(s), 0 failed

✓ engine database  127.0.0.1:41293 Postgres
    used by  the engine (DUCTUS_DATABASE_URL)
    address  127.0.0.1 (an address, nothing to look up)
    connect  127.0.0.1:41293 from 127.0.0.1 in 0.2 ms
    TLS      none: the server doesn't offer TLS (sslmode=prefer), the connection is plain
    login    as `postgres`, database `ductus`: PostgreSQL 18.6; plain, no TLS: the server doesn't offer it (sslmode=prefer)
    commits  0.1 ms median of 20 (each waits for the disk): strict ingest ≈ 10000 msg/s per sender connection, one commit per ACK; more connections share commits. A 1M/day ADT feed averages ~12 msg/s.

✓ MLLP destination  127.0.0.1:6670 MLLP
    used by  adt-in → lab
…
⚠ listening port  0.0.0.0:6661 MLLP
    used by  adt-in
  ⚠ port     in use: by a running engine, or another program
  → If the engine isn't running on this server, another program holds port 6661: find it with `ss -ltnp` (Linux) or `netstat -ano` (Windows).

• Run it after any network change, a firewall request, a certificate renewal, and when a destination is stalled and you want to know whether the receiver is reachable at all.

• A pass means "can connect and log in", not "will accept your messages" (nuance).

• The commits line is the store's disk: on a laptop (here) it's quick; on a server, what it says per sender connection is the ceiling for strict senders. A port "in use" while the engine runs is expected, as here.

• The engine runs the certificate part of it on its own every 6 hours (doctor in ductus.yaml); the console's Connections page shows the last runs and has Check now.

• It checks from where it runs. DNS, routes, firewalls and the source address in a firewall request are this machine's: a laptop on the VPN and the engine's server can get opposite answers. Run it on the server (nuance).

• A name with several addresses (IPv4 and IPv6, a cluster) passes when one of them works, with a warning: the engine may try a failing one first and be slow (nuance).

Who did what: ductus audit export

$ ductus audit export --since 2026-10-31
5 entries
{"id":1,"at":"2026-10-31T10:34:05.714979Z","actor_kind":"system","actor":"todd","via":"cli","ip":null,"action":"token.create","subject":"ops-probe","interface":null,"message":null,"outcome":"ok","detail":{"role":"operator","permissions":["payloads"]}}
{"id":2,"at":"2026-10-31T10:34:06.045144Z","actor_kind":"system","actor":"engine","via":"engine","ip":null,"action":"engine.start","subject":null,"interface":null,"message":null,"outcome":"ok","detail":{"host":"DESKTOP-GCT6E6V","version":"0.6.0-dev","instance":"18e396c42b8b8823-620","previous":"first","environment":"dev","previous_start":null}}
{"id":3,"at":"2026-10-31T10:34:10.284626Z","actor_kind":"token","actor":"ops-probe","via":"token","ip":"127.0.0.1","action":"destination.pause","subject":"adt-in/lab","interface":"adt-in","message":null,"outcome":"ok","detail":{"reason":null}}
…

One JSON object per line, oldest first: for a SIEM, or an auditor's question ("who resent messages for this patient last month?" is --action message --interface adt-in --since …). Filters: --since, --until, --actor, --action (an action or a group: message, user, session), --interface, --message. The export is itself logged. Every action is listed in audit actions.

Retention

$ ductus retention
adt-in                       7w

Each interface's retention, and, when there are any, changes in progress, shorter values waiting for ductus retention confirm <interface>, and expired partitions that had to stay because they held messages someone still needs (retention and storage).

On the server: the service

Linux, Windows, logs.

The message store and search

Every message the engine accepts is stored in its Postgres database before the sender gets its ACK, as it was received, and kept for the interface's retention. That record is how you answer "did we get it, when, and what did each receiver get?", and it's where resend, replay and correction start from.

What's kept

Only the original is stored by default. "What did the lab get?" is answered by running the lab's code on the stored original again, which gives what was sent as long as the code hasn't changed since. Where you must be able to prove what went out (a receiver that disputes it, a regulator), turn on save_output for that destination; it costs about 1.15 GB per million deliveries (nuance).

An HTTP source in respond mode stores nothing: its requests are answered from code, not queued (nuance).

Statuses

A message's status is the worst of its destinations' (quarantined if any is), so "quarantined" on a message means some receiver didn't get it; the message's page says which.

Finding a message

In the console: an interface's Messages tab lists its messages newest first, filtered by status; a message's page shows the payload (with the payloads permission), each destination's outcome and the history. The Quarantine page lists every quarantined delivery across interfaces.

From the API: the same list, a page at a time:

$ curl -s -H "Authorization: Bearer $DUCTUS_API_TOKEN" \
    "http://127.0.0.1:7700/api/v1/interfaces/adt-in/messages?status=quarantined&limit=2"
{"messages":[{"id":2,"received_at":"2026-10-31T10:49:28.372890Z","interface":"adt-in","status":"quarantined","ordering_key":"QUARANTINE-ME2","control_id":"QQUARANTINE-ME2","size":117,"last_error":"lab: rejected (AR): rejected by the receiver","updated_at":"2026-10-31T10:49:28.382056Z","origin":null},{"id":1,…}],"next_before":1}

next_before is the next page's before. One message, with its payload and every destination's outcome, is GET /api/v1/messages/{id} (messages API).

Searching by content: not yet

There's no search by MRN, name or field in the console or the API yet. The message list filters by status and pages by time; a search across messages by patient and date is planned (v1.3, with long-term storage). Until then:

• By the ordering key: each message keeps its ordering key's value (ordering_key, the MRN for ordering.key: PID-3.1) and MSH-10 (control_id) in their own columns. A DBA, or you with a read-only login, can find them in the database:

-- psql, with a read-only login (pg_read_all_data); one interface, one day
SELECT id, received_at, status, control_id
FROM messages
WHERE interface = 'adt-in'
  AND received_at >= '2026-10-30' AND received_at < '2026-10-31'
  AND ordering_key = '123456'
ORDER BY id;

The time range matters: messages are partitioned by time, and with it Postgres reads only that day's partitions (checked with EXPLAIN); without one it reads every partition. Then open the message in the console by its id (/messages/<id>). The tables are described in the database schema.

• Know what that bypasses: a payload read through Ductus is recorded in its audit log before it's shown; a query in psql isn't. Give such logins only to people your privacy officer would let read every message, and prefer selecting ids and keys (as above) over payload.

Why not yet, and what's planned: nuance.

Who can see what

• Everyone signed in sees messages' metadata (ids, statuses, errors, keys).
• Payloads need the payloads permission, and every read is logged before it's shown, once per reader per message per 10 minutes (nuance). The message's page lists who viewed the payload.
• Errors can hold data: a receiver's AR text or a quarantine reason may quote a field. Write quarantine reasons that name the field and the problem, not the patient.

Resend, replay and correct

Four things can be done with a message that's already been handled, from the message's page in the console, the Quarantine page (many at once), or the API. Each is an operator's action, recorded in the audit log with who, when and the reason. A fifth, ductus replay, is different: it runs a period's messages through code to compare, and sends nothing.

Resend

After fixing what made a destination quarantine a message, send it again:

$ curl -s -X POST -H "Authorization: Bearer $DUCTUS_API_TOKEN" -H "X-Ductus: 1" \
    -H "content-type: application/json" -d '{"reason":"lab says fixed"}' \
    http://127.0.0.1:7700/api/v1/messages/2/resend
{"id":2,"interface":"adt-in","destinations":[{"destination":"lab","previous_status":"quarantined","previous_attempts":0,"previous_error":"rejected (AR): rejected by the receiver"}]}

• The same message goes again, in place: first in its lane, not behind everything that came since, and with the same id, so a receiver that dedupes still recognizes it (nuance). It runs the destination's code as deployed now: a fixed mapping or code table applies.
• Without destinations it resends to every destination that quarantined it; the others aren't touched.
• Fix first, then resend. Resending a message the receiver still refuses quarantines it again (the probe above did: the lab still answered AR). The history shows each try.
• Many at once: select them on the Quarantine page, or POST /api/v1/messages/resend with their ids, in one transaction.

Replay

The same call naming a destination whose delivery was sent (or filtered) sends it again:

$ curl -s -X POST -H "Authorization: Bearer $DUCTUS_API_TOKEN" -H "X-Ductus: 1" \
    -H "content-type: application/json" \
    -d '{"destinations":["lab"],"reason":"lab lost it in their upgrade"}' \
    http://127.0.0.1:7700/api/v1/messages/3/resend
{"id":3,"interface":"adt-in","destinations":[{"destination":"lab","previous_status":"sent","previous_attempts":0,"previous_error":null}]}

The receiver gets a message it already had: tell them, or be sure they dedupe on MSH-10 (or the id your code sends). A replay is the same message, not a new one (nuance).

Edit and resend

When the message is wrong (an MRN mistyped at registration, a segment the receiver can't take) and the sender can't send a corrected one, correct a copy:

$ curl -s -X POST -H "Authorization: Bearer $DUCTUS_API_TOKEN" -H "X-Ductus: 1" \
    -H "content-type: application/json" \
    -d '{"payload":"MSH|^~\\&|EHR|HOSP|LAB|HOSP|20261031120000||ADT^A08|QFIXED|P|2.5\rPID|1||777001^^^HOSP^MR||SMITH^ANNA\r","reason":"MRN corrected from the EHR"}' \
    http://127.0.0.1:7700/api/v1/messages/2/edit
{"id":3,"original":2,"interface":"adt-in","destinations":["lab"],"replaced":["lab"]}

• The copy is a new message (id 3 here) going to the destinations that quarantined the original (or the ones you name), through their code, as if it had just arrived. The original stays exactly as received, and its quarantined deliveries become "replaced" (filtered), each pointing at the copy.
• In the console, Edit and resend opens the payload in an editor, with the destinations to send to.
• A message still pending can't be edited (it's being retried): wait until it's quarantined, or set retry.max_attempts on that destination so it gets there (nuance).
• Fix the source too: tell the sending system's team, or the same mistake comes back tomorrow.

Dismiss

$ curl -s -X POST -H "Authorization: Bearer $DUCTUS_API_TOKEN" -H "X-Ductus: 1" \
    -H "content-type: application/json" -d '{"reason":"test message sent by mistake"}' \
    http://127.0.0.1:7700/api/v1/messages/1/dismiss
{"dismissed":[{"id":1,"interface":"adt-in","destinations":[{"destination":"lab","previous_status":"quarantined","previous_attempts":0,"previous_error":"rejected (AR): rejected by the receiver"}]}],"skipped":[]}

A dismissed delivery leaves the quarantine (the destination's error state clears) and becomes filtered, "dismissed by NAME: reason". Nothing is deleted: the message and its history stay for the retention window (nuance). A reason is required. Dismiss what nobody should get; don't dismiss what you just haven't had time for: an error that stays visible is the reminder.

What each leaves in the audit log

$ ductus audit export --since 2026-10-31 --action message
5 entries
{"id":3,…,"actor":"ops-probe2","via":"token","ip":"127.0.0.1","action":"message.payload.read","subject":null,"interface":"adt-in","message":2,"outcome":"ok","detail":null}
{"id":4,…,"action":"message.release","interface":"adt-in","message":2,"outcome":"ok","detail":{"reason":"lab says fixed","messages":[2],"deliveries":1,"destinations":["adt-in/lab"]}}
{"id":5,…,"action":"message.edit","interface":"adt-in","message":2,"outcome":"ok","detail":{"copy":3,"reason":"MRN corrected from the EHR","messages":[2],"destinations":["adt-in/lab"]}}
{"id":6,…,"action":"message.dismiss","interface":"adt-in","message":1,"outcome":"ok","detail":{"reason":"test message sent by mistake","messages":[1],"deliveries":1,"destinations":["adt-in/lab"]}}
{"id":7,…,"action":"message.replay","interface":"adt-in","message":3,"outcome":"ok","detail":{"reason":"lab lost it in their upgrade","messages":[3],"deliveries":1,"destinations":["adt-in/lab"]}}

A resend of quarantined deliveries is message.release, a replay of sent ones message.replay. Each message's page shows the same as its History.

Replay a period before a change

ductus replay answers a different question: what would this change do to real traffic? It runs a period's stored messages through the code in front of it (a new version, a changed mapping) and compares what each destination's code decides (sent and what, filtered, quarantined) with what was recorded, with kept outputs, and with the installed version run beside it. It sends nothing and writes nothing: a read-only login, or a replica, is enough.

$ ductus replay --since 1h
adt-in: 300 message(s) replayed in 0.1 s
ductus replay: version 0.6.0-dev against the record and kept outputs, 2026-10-31 15:25 to 2026-10-31 16:25 UTC, environment dev (nothing was sent)
note: no installed version found (/usr/bin/ductus): outputs compared with kept outputs only; name a binary with --against

adt-in: 300 message(s) in 0.1 s
  ✓ archive: 300 compared, 300 same decision (outputs not compared)
  ✓ lab: 300 compared, 300 same decision (outputs not compared)

300 message(s), 600 comparison(s): 0 identical, 600 same decision (outputs not compared), 0 different, 0 drift, 0 not comparable, 0 waiting (0.1 s)
⚠ outputs not compared: 600 comparison(s) matched by decision only (both sent), not by what was sent, because no version ran beside this one and no destination keeps its outputs; to compare outputs, run it on the server, where the installed version runs beside it, or name the installed binary with --against PATH

That lab had no installed version and kept no outputs, so it could only say each message was still sent; identical is claimed only where an output was compared (or a filter or quarantine, which have none). What each count means.

• Before an upgrade, run the new version's binary on the server: ductus replay --since 24h compares against the installed version and exits 1 on any difference. With --report replay.html it writes the evidence a change board asks for (--hide-values to leave patient data out of it) (nuance).
• Before a code change, --baseline <git revision> compares the project as it is with an earlier one: every message whose output the change alters is listed, field by field.
• It can't compare what code writes (db().execute, an API POST): those calls are refused in a replay, and the message is marked not comparable (nuance).
• The clock is each message's arrival and Math.random repeats, so outputs that use them still compare (nuance).
• What it costs: measured at 100,000 messages × 3 destinations in 7.5 s.

ductus replay re-runs code to compare; resend and replay in the console send. Don't confuse the two when talking to a change board.

Monitoring and metrics

The engine judges its own health: every interface and destination has a state, worked out from what it expects (monitor) and what it sees. The console, ductus status, the API and Prometheus all show the same states, and alerts are made from them. This page is what each state means, how to look deeper, and what to graph.

Health: what each state means

A destination shows the worst state that applies, worded with the numbers ("behind 1,240 (oldest 12 min), catching up ~6 min"), and once a few weeks of history are in, why: a burst of arrivals, or a receiver slower than usual for this hour of the week (nuance). Nothing is expected until you say so: without monitor:, a quiet interface is ok, and a destination is behind only past a limit you set (nuance).

Set behind with both a count and an age on every feed someone would phone about: a count alone misses an overnight stall when little arrives, an age alone misses a daytime pile-up that still moves. Set active_within on steady feeds and active_by on scheduled ones.

Where a backlog sits: hotspots

A count says that a queue is stuck; hotspots says which message or patient. Messages with an ordering key wait in lanes (the key's hash), each read by one worker, so a backlog always sits behind one message per worker.

$ ductus hotspots adt-in --lanes 3
adt-in → lab: 30 waiting in 8 of 8 lanes; blocked by message 381 (key 102001, lane 1, 7 attempts, waiting 1 min): Connection refused (os error 111); all 29 behind it wait (one worker: order)
  lane   worker    waiting  first msg    key                     age attempts  last error
  1      0               6  381          102001                1 min        7  Connection refused (os error 111)
  6      0               6  388          102008                1 min        0
  0      0               3  382          102002                1 min        0
  (5 more lanes waiting: --lanes N lists more)

How to read the first line (nuance):

• "blocked by message M" while others were sent since it arrived: the receiver is up and takes other messages; this one is the trouble (its content, or the receiver's data for that patient). Look at the message and its last error; set retry.max_attempts on that destination if such messages should be quarantined instead of holding their lane.
• "every worker is retrying its first message", or blocked with a connection error as above: the receiver, or something the code calls, is down. Nobody's message is to blame; call the vendor, and ductus pause stops the retries meanwhile.
• "lane L has 73% of the backlog" or one worker with most of it: one patient (or a few) arriving faster than one worker sends. Order keeps a key on one worker, so more workers don't help that key. A hot key that clears by itself is a burst; one that's hot every day is a feed to discuss with the sender.
• "a lane no worker reads": ordering.partitions was lowered while messages waited (nuance).

After a receiver comes back, the first message is tried at its next retry: the wait between tries doubles up to retry.max_backoff_ms (60 s by default), so a destination can show stalled for up to a minute after the receiver is back, then drains in order (nuance).

Prometheus

GET /api/v1/metrics is in Prometheus's format, for Prometheus, Grafana Agent, Datadog or anything that scrapes. It needs a token, a viewer one:

# prometheus.yml, in part
scrape_configs:
  - job_name: ductus
    metrics_path: /api/v1/metrics
    scheme: https                      # with api.tls; http on the engine's own server
    authorization: { credentials_file: /etc/prometheus/ductus-token }
    static_configs:
      - targets: ["ductus1.hospital.org:7700"]

(ductus token create prometheus --role viewer makes the token. List Prometheus's address in api.allow.) What one destination's lines look like, from a real engine:

ductus_messages_received_total{interface="adt-in"} 411
ductus_deliveries_total{interface="adt-in",destination="lab",outcome="sent"} 380
ductus_deliveries_total{interface="adt-in",destination="lab",outcome="quarantined"} 0
ductus_delivery_retries_total{interface="adt-in",destination="lab"} 7
ductus_delivery_latency_seconds_bucket{interface="adt-in",destination="lab",le="0.01"} 165
…

Every metric, with its labels: Prometheus metrics.

What to graph

• Counters start at zero when the engine starts; rate() and increase() handle it. Graph rates, not totals (nuance). Queue sizes come from the database at each scrape and are always current.
• Timings are histogram buckets: a p95 is good to about a bucket's width (nuance).
• About 65 series per destination: on a hosted service that bills per series, drop what you don't graph with metric_relabel_configs (nuance).

Alerting from Prometheus

The engine alerts by itself (alerts), and that's the simpler place. If your operations team alerts from Prometheus instead (one pager for everything), use the states the engine has already judged rather than re-deriving them:

# ductus-rules.yml, in part (a Prometheus rule file)
groups:
  - name: ductus
    rules:
      - alert: DuctusDestinationStalled
        expr: ductus_destination_state == 3
        for: 2m
      - alert: DuctusDown
        expr: up{job="ductus"} == 0
        for: 2m

ductus_destination_state is 0 ok, 1 behind, 2 error, 3 stalled, 4 paused; ductus_interface_state is 0 ok, 1 silent, 2 missed, 3 stopped. Watch the engine itself too: an engine that's down reports nothing, so up == 0 (or the engine's heartbeat) is the alert that covers it. And ductus_alert_send_failures_total going up means the engine's own alerts aren't getting through.

In the API

• GET /api/v1/interfaces: every interface and destination with its health, counts since the start, and timings (what the console's lists show).
• GET /api/v1/engine: the engine itself: version, start, database, other engines on the database, and each database's and API's call times (resources), to see which lookup is slow.
• GET /api/v1/events: the same changes as they happen (what keeps the console live).

REST API reference.

Logs

The engine writes one line per thing that happened worth knowing: a start, a pause, a delivery that failed and when it will be tried again, a refused connection. States and counts are in the console and Prometheus; the log is the why, in order, with times.

Where they are

RUST_LOG is info by default: every line on this page. info,ductus_runtime=debug adds detail for the engine (each start phase and its time, each claim), for a support case; don't leave it on, it's many lines per message. A restart applies a change (Linux: systemctl restart ductus).

Lines are plain text with fields (interface=adt-in destination=lab), so grep finds them: the message comes first, then the fields (grep 'delivery failed.*interface=adt-in'). Under ductus run in a terminal they're colored; elsewhere plain.

What the lines mean

From a real engine, a start, a pause and resume, then a receiver that went down and a drain:

INFO ductus_runtime: the engine's database database=plain, no TLS: the server doesn't offer it (sslmode=prefer)
INFO ductus_runtime::alerts: alerts on channels=1 rules=2 heartbeat=false
INFO ductus_runtime::certificates: no connection has a certificate: scheduled checks have nothing to do
INFO ductus_runtime::api: REST API listening (/api/v1) host=127.0.0.1 port=7700 https=false
INFO ductus_runtime: started interfaces=1 mllp=1 http=0 files=0 schedules=0 internal=0 shared=1 pool_threads=8 dedicated_threads=0 took=299.820201ms
WARN ductus_runtime::api::status: destination paused interface=adt-in destination=lab by="ops-probe" reason="lab system upgrade until 14:00, ticket CHG0042"
INFO ductus_runtime::worker: paused: waiting to be resumed interface=adt-in destination=lab slot=0
WARN ductus_runtime::api::status: destination resumed interface=adt-in destination=lab by="ops-probe"
WARN ductus_runtime::worker: delivery failed; retrying in 1s interface=adt-in destination=lab id=381 attempts=1 error=connection closed before ACK
WARN ductus_runtime::worker: delivery failed; retrying in 2s interface=adt-in destination=lab id=381 attempts=2 error=Connection refused (os error 111)
…
WARN ductus_runtime::worker: delivery failed; retrying in 60s interface=adt-in destination=lab id=381 attempts=7 error=Connection refused (os error 111)
INFO ductus_runtime::alerts: alert alert=dest:adt-in/lab kind="fired" state="stalled" severity="critical" notifications=1
WARN ductus_runtime::control: draining: not taking new messages; delivering what's stored stop=false
WARN ductus_runtime::worker: destination rejected; quarantined interface=adt-in destination=lab id=411 error=rejected (AR): rejected by the receiver
WARN ductus_runtime::control: drained: nothing left to deliver stop=false
INFO ductus_runtime: stopping: taking no new messages, finishing the ones in hand why="Ctrl-C" limit=30s
INFO ductus_runtime: stopped took=2.217222ms

(Each line starts with its time in UTC, 2026-10-31T10:34:06.069632Z, left out here.)

A start logs one line, not one per listener (nuance); problems with one interface are their own lines. Things that could repeat many times a second are said once a minute per sender or kind instead: a refused TLS handshake (nuance), a connection refused by allow: (nuance), so a misconfigured sender can't flood the log.

When a sender says "connection refused"

The engine logs every refusal it makes, with the reason. If there's no line for the sender's address, the engine never saw the connection: a firewall between, the wrong port, or the engine not running (a stopped service, a drain, and a failed start close the ports on purpose: nuance). ductus ports lists what the project listens on, and ductus doctor checks the ports from the server.

What not to look for in the logs

• Message content. The log names messages by id (id=381) and never prints a payload. Error texts are the receiver's answer and your code's quarantine reasons, which can quote a value: write reasons that name the field, not the patient. Look a message up by its id in the console.
• Who read what. That's the audit log, kept six years, not the service log, kept days.

Alerts in practice

How to set alerts up is Alerts and owners; this page is living with them: what arrives, what to do with it, and how to keep alerts trusted, which is the only way they keep working. An alert people learn to ignore is worse than none.

What an alert is

The engine judges every interface and destination every 15 seconds. When one enters a state a rule in ductus.yaml names, an alert opens and its channels are told once. While it stays open and nobody has acknowledged it, they're reminded every repeat (4 h by default); if it gets worse (a warning becomes critical) they're told again; once the state has been fine for settle (2 min), it recovers and they're told that too (nuance).

What a webhook receives, from a real engine whose lab receiver went down (format: json; Teams and Slack get the same as a card or message):

{"kind":"fired","alert":3,"key":"dest:adt-in/lab","interface":"adt-in","destination":"lab","state":"stalled","severity":"critical","since":"2026-10-31T10:35:21.087109+00:00","at":"2026-10-31T10:35:21.093252+00:00","summary":"stalled: nothing sent for 1 min, 30 waiting; retrying: Connection refused (os error 111)","environment":"dev","title":"[Ductus dev] CRITICAL adt-in → lab is stalled"}

The title says the environment, so a test engine's alert can't be taken for production's.

When one arrives

1. Open it in the console's Alerts page (or the interface it's about): the summary says what and how much ("30 waiting; retrying: Connection refused").
2. Acknowledge it once someone is on it: the reminders stop, and the others on the channel see who took it. Acknowledging isn't fixing: the alert stays open until the state is fine (nuance).
3. Find the cause: stalled → hotspots (the receiver, or one message); error → the quarantine; behind → the receiver's speed and the catch-up time; silent/missed → the sender; stopped → the code (the log and the quarantined messages say which message); certificate → renew it (renewing).
4. If it's planned downtime you forgot to announce, pause the destination with the reason: a paused destination doesn't alert, and its open alert waits quietly until it's resumed (nuance).
5. It recovers by itself when the state is fine, and the channel is told. Nothing to close.

Silence, pause, mute: which one

Silencing hides; pausing stops. Pause what's down on purpose, silence only what someone is already fixing, and never mute a live interface to stop noise: fix the rule instead.

Keeping alerts trusted

• Every alert should need a person. If people get alerts they do nothing about, change the rule: a state that isn't worth a call at 3 a.m. is a warning (held during quiet hours), and a feed nobody watches doesn't need monitor:.
• Tune monitor: from what happened. A silent that fires every Sunday morning means active_within doesn't fit the feed's quiet hours: add between:. A behind that fires during every nightly batch means the limit is below a normal night.
• Send to teams, not people: a team's list or chat channel survives holidays and job changes (nuance).
• Test every channel before you rely on it, and after the mail team changes anything:

$ ductus alerts test chat
sent a test notification to `chat`

It sends a test notification straight from ductus.yaml, without an engine (the webhook got "kind":"test", "title":"[Ductus dev] test notification"). The Alerts page's channel table also has Test, and shows each channel's sent, failed and last error: a channel that's failing can't tell you so itself (nuance). Prometheus's ductus_alert_send_failures_total is the other way to hear about it.

• Quiet hours hold warnings, never critical alerts (nuance).
• After a restart, states the engine hasn't watched for a minute yet don't alert, so a restart doesn't raise a wave of alerts for things that were already known (nuance).

An engine that's down tells nobody: the heartbeat

Alerts come from the engine, so an engine that's stopped, crashed or cut off from its database can't send one. Set alerts.heartbeat in production: the engine calls a URL every every (say 5m) while it runs, and a service that expects the calls (healthchecks.io, Better Stack, Uptime Kuma, your monitoring's own push check), running somewhere else, raises the alarm when they stop. Give that service a grace period of about twice every (nuance).

High availability and lanes

What keeps interfaces running through a restart, an upgrade, a dead server or a database failover, and how the work inside an interface is spread so one slow patient or one bad message doesn't hold up the rest. Read the first section before you plan a second server: what Ductus does today is one engine per interface, a standby you start, and no automatic failover yet.

What protects what

Nothing acknowledged is lost in any of these: a message is stored in the database before its sender gets the ACK, and stays until every destination has finished with it. What a sender experiences is the column on the right; senders resend by themselves when refused, as MLLP senders do.

One engine per interface

Several engines can share one database, each running different interfaces: that's how you spread a large project over servers (feeds on one, APIs on another). An interface runs on one engine at a time: an engine started with an interface another live engine is running refuses to start, exit 3, naming the other engine's host and pid (nuance). Two engines on one interface would both send each message, and one patient's order would break.

The Overview's other engines on this database card, and GET /api/v1/engine others, list them. A refused start is in the audit log (engine.refused).

The exception is an HTTP source in respond mode: it has no queue, so several engines can answer it behind your load balancer.

A standby server (disaster recovery)

Keep a second server installed and configured with the same project and database, with the service stopped. When the first server is gone:

1. Make sure the first engine is really down (or its server is): if it's alive and cut off from you but not from the database, two engines would fight, and the second is refused anyway.
2. Start the standby's service. It claims the interfaces as soon as the first engine's database session is gone: at once if the process ended (a crash, kill -9: measured 0.3 s), within about 25 s if the whole host vanished (the database's keepalive probes notice) (nuance).
3. Point senders at it: a DNS name or a virtual IP that moves, so nobody reconfigures an EHR during an outage. Its allow: lists and certificates must cover the names senders use.
4. It sends what was waiting, in order, then everything new.

Practice it: start the standby against a test database with the same project, and time the switch.

Not yet: a standby that takes over by itself. An automatic failover, where a waiting engine claims a dead one's interfaces with nobody starting it, is planned with clustering (v1.2). Until then the switch is a person's (or your cluster manager's) decision, on purpose: a split network that started a second engine automatically could make the duplicates the one-owner rule prevents.

The database is the single point to protect

Everything the engine knows is in Postgres: the messages, the queues, the audit log, the people. Its availability is the engine's, so it needs what your DBA gives any critical database:

• Backups with point-in-time recovery (pgBackRest or WAL archiving). The engine needs no cron job of its own (backups).
• A replica to fail over to. The engine reconnects after a failover by itself. With asynchronous replication, messages acknowledged in the last moments before the failover can be missing on the replica: if no acknowledged message may be lost, the DBA's answer is synchronous replication to the standby.
• A direct connection, or a pooler in session mode: PgBouncer in transaction mode can't hold the engine's lock (nuance).

Deploying and upgrading without a gap

• Linux: systemctl reload ductus for a project change, and the package's upgrade for a new version, both hand over: the new engine starts on the same listening sockets, takes over once it's ready, and the old one stops. Receiving never stops. A new version or project that can't start leaves the running one serving, and the reload says why (nuance). No drain, no maintenance window for the senders.
• Windows: a restart or the MSI is a short gap (above): pick a quiet moment. Zero-downtime Windows upgrades are planned for v1.0.
• Before every upgrade: ductus upgrade --check with the new version, and ductus replay for what it does to real traffic (replay).

Lanes: how an interface spreads its work

Inside an interface, each destination is its own queue. With an ordering.key (the MRN), the queue is split into lanes by the key's hash (64 by default, ordering.partitions); each lane is read by one worker at a time, so one patient's messages stay in order while different patients go in parallel. Without a key, the whole destination is one lane: strict arrival order, one message at a time (order).

What that means when you run it:

• A message that keeps failing holds up its lane, and the lanes its worker reads; the other workers' lanes go on (nuance). ductus hotspots shows which message. For receivers that refuse a message for good with errors that look temporary, retry.max_attempts quarantines it so its lane moves.
• More workers help only with a key, and only up to about the number of cores; they need isolation: dedicated (dedicated threads).
• One hot patient (a batch load sent as one patient's updates) goes as fast as one worker, whatever the number of workers: that's what order per patient costs.
• Change partitions only with an empty queue: lowering it with messages waiting is refused at start, raising it can reorder a patient's waiting and new messages (nuance). ductus drain --wait, change, deploy, ductus resume.
• Several destinations are independent: one receiver down holds up only its own queue (nuance).

Retention and storage

How long messages stay, where the space goes, and what an operator does about it, which is mostly nothing: old messages go by themselves, a partition at a time, with no purge job, no vacuum to schedule and no slow delete in the middle of the day. The settings are in ductus.yaml (the project's default) and interface.yaml (one interface's).

How it works

• Three windows: 1w, 3w or 7w (the default). Each is a set of partitions of the message tables, a day each for 1w and a week each for 3w and 7w (nuance).
• Old messages go by dropping a whole partition once all of it is older than the window, checked hourly. Dropping a partition costs the same whether it held ten messages or ten million, and blocks nothing: no row-by-row delete, no table bloat, nothing to vacuum.
• So a message is kept at least its window, and gone within a day (1w) or a week (3w, 7w) after: a 7w message lives 49 to 56 days (nuance). Treat the window as "at least".
• Nothing anyone still needs is deleted. When a partition expires, its messages still pending for a destination, and its quarantined ones (with quarantined: keep, the default), move to the held area first, with their deliveries, and stay in their queue and in the console (nuance). A month of the held area is dropped once it's over and nothing in it is waiting.
• The audit log is kept apart, six years by default (audit.keep), and trimmed hourly.

Looking at it

$ ductus retention
adt-in                       7w

Each interface's retention; when there are any, also changes in progress (with what's left to move), shorter values waiting for a confirmation (with how many messages they'd delete), and what the held area keeps, by month and interface. The console's Retention page shows the same. In Prometheus, ductus_held_messages (by interface and status) and ductus_held_bytes are counted at each hourly run.

When to act: held quarantined messages growing is a question nobody has answered: look at them (resend, correct or dismiss). Held pending messages are a destination that's been down past the window: they'll go when it's back, but the receiver's team should know.

Changing retention

Change retention: in interface.yaml (or ductus.yaml) and deploy.

• Longer applies at once: everything not yet dropped is kept to the new window.
• Shorter deletes messages, so it waits for a confirmation. The engine logs a warning, keeps the old value, and ductus retention shows how many messages the new one would delete. ductus retention confirm <interface> (an operator token), or typing the interface's name on the Retention page, applies it: finished messages past the new window are deleted in the background, the rest move into the new window's partitions (nuance). Confirm on each database the project runs on.
• One change at a time per interface: a change works in batches, at most half the time busy (8,000 messages took 1 to 1.5 s on a dev machine), and a different value deployed meanwhile waits (nuance). Don't flip it back and forth.
• Shortening frees disk later, as the old partitions are dropped; to free it now, a DBA can VACUUM FULL the affected old partitions off-hours (nuance).

Choose the window your team looks back over when a problem surfaces: 1w for busy feeds nobody looks back on, 7w where problems show up weeks later (billing, a monthly reconciliation). Longer history belongs in the source system or an archive; the engine isn't a data warehouse, and a long-term store with search is planned (v1.3).

How much space

Roughly: messages a day × their average size × the days kept, plus a small row per destination per message, the indexes, outputs kept with save_output (about 1.15 GB per million deliveries, measured), and whatever is held. A 7w window keeps up to 56 days, so size for that.

Measure it on your own traffic after a week rather than estimating for long:

-- psql: the whole database, then the message tables by window (t7 = 1w, t21 = 3w, t49 = 7w)
SELECT pg_size_pretty(pg_database_size(current_database()));
SELECT substring(relname from '^(?:messages|deliveries)_(t\d+|held)_') AS tier,
       pg_size_pretty(sum(pg_total_relation_size(oid))) AS size
FROM pg_class
WHERE relkind = 'r' AND relname ~ '^(messages|deliveries)_(t\d+|held)_\d+$'
GROUP BY 1 ORDER BY 1;

On a development database it printed:

 tier |  size
------+---------
 held | 8888 kB
 t21  | 54 MB
 t49  | 19 MB
 t7   | 832 kB

Space comes back as partitions are dropped, a day or a week at a time; between drops the database grows by about a day's worth. Leave the disk room for the largest window's worth plus a held backlog (a receiver down for a week keeps a week of its messages longer).

Sizing the database has the rest: the commit time that decides the speed, and keeping long queries off the message tables.

What's not there

• No archive or export of old messages yet: past the window they're gone (planned, with long-term storage, v1.3).
• No per-day windows (30 days, 90 days): only the three, for the reason above.
• No legal hold on a single patient's messages: if one is needed, copy them out (the schema describes the tables) before their partition goes.

Upgrade

How to move a running Ductus to a new version: what the version numbers promise, how to find out what the new version will do before you install it, the steps for each way of installing, what happens to the database, and how to go back. For deploying a change to your own project (new interface code, a new interface), the same tools apply: see Deploy a change.

The short version, for a server installed from the Linux package:

1. Unpack the new version's portable archive anywhere on the server.
2. ./ductus upgrade --check --replay 1d --env-file /etc/ductus/ductus.env --root /var/lib/ductus/project: it changes nothing and exits 1 when something would go wrong.
3. Install the new package. It hands the running engine over to the new version: receiving never stops.
4. Look: systemctl status ductus, then the console's Overview ("took over from …").

Versions and what's supported

What: a version is MAJOR.MINOR.PATCH (semver): 0.6.0, 0.6.1, 1.2.0. ductus --version prints it; the console's Overview, GET /api/v1/engine and ductus upgrade --check show the version of every engine on the database.

What each kind of version promises:

Why: a hospital upgrades rarely and needs to know an upgrade won't strand a project or a database. Deprecating in minors and removing only at majors means a project that validates without warnings on 4.9 starts on 5.0. A database that only grows within a major is what makes going back a matter of installing the older program.

What's supported today: Ductus is pre-release. There are no released versions yet, only development builds of the latest commit, and only that is supported (SECURITY.md). Long-term-support versions, for sites that stay on one version for a year or more, are planned once clustering (v1.2) is stable; until then, follow the latest version.

Several engines on one database may run different versions while you upgrade them one at a time: each runs its own interfaces, and the older ones keep working on what the newer one added. Don't leave them on different versions for longer than the upgrade: a fix you upgraded for isn't on the ones you haven't.

Before every upgrade: check, then replay

ductus upgrade --check

What: the new version's program reads the database and the project and says what its first start would do. It changes nothing: not the database, not the project folder.

Why: otherwise the first place you learn that a database change works through a 50 GB table, or that your tests fail under the new SDK, is the restart itself, in the change window.

When: before every upgrade, from the new version's portable archive (no install needed), on the server or any machine that reaches the database, with the service's settings. A read-only login is enough (GRANT pg_read_all_data TO ductus_check). Run it again on the day: the database and the project may have changed since.

Default: it checks everything below; --replay 1d adds a replay of a day's traffic; --json gives one document (for a change ticket). Exit 0: nothing in the way. Exit 1: something would refuse the start or fail the project.

From a lab engine (DESKTOP-GCT6E6V is the machine it ran on):

$ ./ductus upgrade --check
ductus upgrade --check: what version 0.6.0-dev would do here (nothing is changed)

The database
✓ migrations: nothing to apply

Engines on this database
✓ 18e39e5085ef2289-3ee on DESKTOP-GCT6E6V (pid 1006): version 0.6.0-dev, environment dev, since 2026-10-31 12:52 UTC, runs 1 interface(s)

The project (., environment dev)
✓ validate: valid
✓ test: 1 passed, 0 failed

Between versions
✓ required stops: none (versions before 1.0 upgrade to any later one)
✓ deprecated settings: none used

ready to upgrade to 0.6.0-dev (checked in 0.6 s)

What each section means, and what to do when it isn't ✓:

A failing check, here a test the new version fails, ends with the count and exits 1:

$ ./ductus upgrade --check --replay 1h
…
The project (., environment dev)
✓ validate: valid
✗ test: 0 passed, 1 failed
    adt-in
      ✗ sets the receiving application and keeps the patient
          expected "RECEIVER", got "LAB"
…
The last 1h replayed through 0.6.0-dev (nothing is sent)
✓ replay against the record and kept outputs: 300 message(s), 600 comparison(s), 0 identical, 600 same decision (outputs not compared), 0 different
    note: no installed version found (/usr/bin/ductus): outputs compared with kept outputs only; name a binary with --against
⚠ replay: outputs not compared: 600 comparison(s) matched by decision only (both sent), not by what was sent, because no version ran beside this one and no destination keeps its outputs; to compare outputs, run it on the server, where the installed version runs beside it, or name the installed binary with --against PATH

1 problem(s): fix them before upgrading (checked in 0.7 s)

The replay there passed although the code now sets MSH-5 to "LAB" in every message: this lab has no installed version and keeps no outputs, so all it could compare was that each message was still sent. That's what the ⚠ is for. "Identical" is only ever claimed for an output that was compared.

On a database no version has run on yet, the first section says so instead: "a new database: the first start creates every table (25 migrations)".

--json gives the same as one document: ok and problems, every check with its section, level and text, and the details (migrations.pending with each one's tables, engines, the project's validate and test output):

$ ./ductus upgrade --check --json
{
    "version": "0.6.0-dev",
    "ok": true,
    "problems": 0,
    "seconds": 0.61,
    "checks": [
        {
            "section": "database",
            "level": "ok",
            "text": "migrations: nothing to apply",
            "more": []
        },
…
    "migrations": {
        "new_database": false,
        "pending": [],
        "newer": [],
        "edited": [],
        "failed": []
    },
…

ductus replay: what it does to real traffic

What: ductus replay --since 24h runs a period's stored messages through the new version's engine and your project's code, sends nothing and writes nothing, and compares what each destination's code decides (sent and exactly what, filtered and why, quarantined and why) with the installed version run beside it on the same messages, with what was recorded, and with kept outputs. ductus upgrade --check --replay 1d runs it as part of the check.

Why: "what will change?" is the question a change board asks, and the usual evidence is weeks of testing by hand on a test system. A replay of production's own traffic answers it in minutes, message by message, field by field.

When not: it isn't a load test (one thread, one message at a time: measured), and it can't compare code that writes (db().execute, a POST): those messages are reported as not comparable (why).

What it compares with, by default: the version installed on the server (/usr/bin/ductus, C:\Program Files\Ductus\ductus.exe), run beside it; that needs the installed one to be 0.6.6.4 or later (why). Without an installed version, outputs are compared only where they were kept (save_output: true). Every other destination is compared by its decision alone, and the report counts what it could compare separately:

When no output was compared at all, the report ends with ⚠ outputs not compared and the fix, in the terminal, the --json document (outputs_not_compared) and the HTML page alike; upgrade --check --replay shows it as a warning (exit 0: nothing differed, but little was checked). On a machine without the installed version, name it: --against /path/to/old/ductus.

A difference the installed version shares with the new one isn't the upgrade's: the data it looks up changed since the message was sent. The report calls it drift and doesn't fail for it (nuance).

For a change board, --report replay.html writes a page with the versions, the period, the counts and a SHA-256 checksum. Examples in it quote field values from real messages: keep it as you keep patient data, or make it with --hide-values (nuance). The same tool compares a code change with --baseline <git revision>:

$ ductus replay --since 1h --baseline HEAD --report replay.html
adt-in: 300 message(s) replayed in 0.0 s
ductus replay: version 0.6.0-dev against the project at HEAD, 2026-10-31 15:25 to 2026-10-31 16:25 UTC, environment dev (nothing was sent)

adt-in: 300 message(s) in 0.0 s
  ✗ archive: 300 compared, 300 different
      vs the project at HEAD: MSH-5 differs in 300 message(s) (e.g. message 406: "RECEIVER" → "LAB")
  ✗ lab: 300 compared, 300 different
      vs the project at HEAD: MSH-5 differs in 300 message(s) (e.g. message 406: "RECEIVER" → "LAB")

300 message(s), 600 comparison(s): 0 identical, 600 different, 0 drift, 0 not comparable, 0 waiting (0.1 s)
report: …/replay.html (sha256:794fea6a30fe050d12f3253ba1240f39f40f5219f780783cb192b06c3d20489f)

Here the earlier project ran beside this one, so every output was compared: the change shows field by field, on the same messages that passed by decision above.

More on replay, with what the clock and Math.random do in it: Replay a period before a change.

Upgrading, by how you installed

Linux packages: the handover

sudo apt install ./ductus_0.6.1_amd64.deb          # Ubuntu, Debian
sudo dnf install ./ductus-0.6.1-1.x86_64.rpm       # RHEL, Rocky, Alma

What happens: the package replaces the program and runs systemctl reload ductus. The new version starts beside the running engine, on the same listening sockets: it reads the project, applies the database changes and loads every interface's code while the old one carries on. Once it's ready, the old engine stops cleanly, lets go of its interfaces, and the new one takes them and sends. Receiving never stops and sending pauses for a moment (measured under real systemd, 4 senders at 200 messages/s: 0 refused, longest wait for an ACK 0.17–0.22 s, nothing lost: nuance).

The handover's own lines, from an engine handing over to the program now on disk (ductus handover is what systemctl reload runs, with the service's control socket; these are the lines journalctl -u ductus shows):

$ ductus handover --control …/control
started …/ductus (pid 1312): it reads the project, migrates the database and loads every interface's code while this one runs
ductus 0.6.0-dev (pid 1312) is ready after 104 ms and receives beside this one: stopping this one
this engine stopped in 1 ms and let go of its interfaces: the new one takes them
handed over to ductus 0.6.0-dev (pid 1312): receiving never stopped, sending paused 41 ms

• A new version that can't start (a broken project, a failed database change, another engine owning an interface) leaves the old one serving: the install says so, journalctl -u ductus says why, and senders notice nothing. Fix what it names and sudo systemctl reload ductus (nuance).
• A start longer than 5 minutes (a large database change) is given up by the handover: sudo ductus handover --timeout 30m instead of the reload.
• For a few seconds two engines run, so the server needs memory for both.
• From a development build before 0.6.6.3 (no released version is), the package restarts instead of handing over; from one before 0.6.6.2 that restart refuses connections for a moment (nuance).
• A drain refuses a handover: resume first, or stop and start.

Windows installer

msiexec /i ductus-0.6.1-x64.msi /qn          # as an administrator; no need to uninstall first

What happens: the installer stops the service cleanly, replaces the program and starts it again; C:\ProgramData\Ductus (settings, project, logs) is never touched. The engine is down for a few seconds (an MSI upgrade measured 3.1–3.5 s: a few connections refused or reset, nothing lost), so choose a quiet moment (nuance). A handover like Linux's is planned for v1.0. Details (a service stopped on purpose, a password account, Restart Manager): Windows installer.

The portable archive

Replace the program and hand over, as the package would:

sudo install -m 0755 ductus-0.6.1/ductus /usr/bin/ductus
sudo systemctl reload ductus

Without the package's service (ductus run under your own supervisor), stop and start it with the new program; the senders see a short gap. To hand over there too, start the engine with DUCTUS_CONTROL=/some/dir/control and run ductus handover --control /some/dir/control after replacing the program (nuance).

Docker

Build the new version's image (docker build -t ductus:dev . in the new version's source, as in Docker), then docker compose -f packaging/docker/compose.yaml up -d: Compose sees the new image and replaces the container. Its ports close while the new one starts, so senders are refused for a moment. Docker is for trying and developing until v1.0.

ductus dev on a laptop

Replace the program and start ductus dev again. Its Postgres (in .ductus/postgres) is kept and updated by the new version like any other database; ductus dev --reset starts from an empty one.

Database changes (migrations)

What: a new version may change the database: a new table, a new column, a new index. Each change is a numbered migration, applied once, in order, by the first start of the version that carries it, and recorded in the database (_sqlx_migrations). ductus upgrade --check lists the ones still to apply, each with a sentence saying what it does, for example 0025: "keeps the names accounts had before, and the directory's id for each directory account".

They only add. Within a major version, a migration never renames or removes anything the version before uses: the old program keeps working on the new tables. That's what makes going back safe, and what lets several engines on one database be upgraded one at a time. A store test, on every push, starts the engine on a database that has a migration it doesn't know, as an older program meets a newer one's database.

Heavy ones. Most take milliseconds. One that works through an existing table row by row (an index built, a column filled) is marked ⚠ with the table and its size now: while it runs, the start waits, and storing or sending waits as it says (nuance). Under the Linux package's service senders wait in the queue rather than being refused, for as long as it takes. Plan the window by the size shown, and time it on a restored backup first. There have been none since v0.5 (0011, the queue index, was the last).

When one fails, the start refuses with the database's reason, and so does every later start of that version until the cause is fixed. Each migration runs in one transaction, so a failure leaves nothing half done: the database is as it was before it, and the version before still runs on it (going back keeps you running meanwhile). A migration recorded as failed (upgrade --check shows it ✗ with its number) is one that couldn't run in a transaction; none so far does. Then every version refuses to start until it's repaired: don't edit _sqlx_migrations by hand, ask for help.

Backups are the DBA's. The engine needs no backup of its own before an upgrade, but your usual point-in-time recovery (pgBackRest, WAL archiving) is what undoes anything at all (PostgreSQL).

Going back (rollback)

What: installing the previous version again. The database stays as the new version left it; the previous version runs on it, ignoring what it doesn't know.

When: the new version misbehaves in a way upgrade --check and replay didn't show. Decide by what you see: a project problem is fixed in the project and deployed (systemctl reload); a problem in Ductus itself is a rollback, and an issue.

What going back doesn't undo: whatever the new version wrote stays in the database: tables and columns the older one doesn't read. Messages received meanwhile are kept and sent as usual; features only the new version had stop until you upgrade again. Going back across a major version isn't possible (from 1.0 on: the major may have removed what the older one needs).

What going back does to a database the new version migrated: nothing. The older program doesn't undo the new version's migrations, can't, and needn't: they only added. At its start it sees migrations it doesn't know in _sqlx_migrations, leaves them as they are, and runs on the tables it knows; what the new version added (a table, a column) stays, unused, until you upgrade again, when the new version finds its migrations already applied and applies nothing twice. ductus upgrade --check lists them as "applied by a newer version than this one" (✓, not a problem). Two things to know:

• Data written only into what the new version added isn't read by the older one while it runs. Messages themselves are in tables every version reads, so none is lost or left waiting.
• A migration that failed left nothing behind: each runs in one transaction, so a failure undoes all of it. The new version refuses to start (each time, with the reason) and the older one runs on the database as it was (Database changes).

Both are shown on a server, under traffic and tested on every push: a store test starts the engine on a database with a migration it doesn't know, and the v0.6 demo goes back with a package while a sender sends.

On a server: the v0.6 demo

What: an upgrade and a downgrade of the Linux package, each while a sender sends, on the server Your first interface set up (Ubuntu 24.04, MLLP over TLS with client certificates, Postgres on another machine). The commands and outputs are from scripts/demo-v06.sh, which runs them under real systemd on every push.

Why: "receiving never stops" is a claim; this is what a sender saw while it happened, and what the database looks like after going back.

The demo's two packages (0.6.0~dev.1 and ~dev.2) hold the same program, so they show the mechanics of an upgrade, not a version's changes; a release's files are named 0.6.0, 0.6.1.

Check first, from the new version's archive

The new version's portable archive, unpacked anywhere, checks the running server with the service's settings, and replays the last day's traffic through the new version beside the installed one:

$ cd /tmp
$ tar -xzf ductus-0.6.0-dev.2-linux-x64.tar.gz
$ sudo -u ductus ./ductus-0.6.0-dev.2/ductus upgrade --check --replay 1d --env-file /etc/ductus/ductus.env --root /var/lib/ductus/project
ductus upgrade --check: what version 0.6.0-dev would do here (nothing is changed)

The database
✓ migrations: nothing to apply

Engines on this database
✓ 18e3c2043934cad1-14e on ductus1 (pid 334): version 0.6.0-dev, environment prod, since 2026-10-31 23:46 UTC, runs 1 interface(s)

The project (/var/lib/ductus/project, environment prod)
✓ validate: valid
    ⚠ database: plain, no TLS: the server doesn't offer it (sslmode=prefer); whoever answers at its address is believed: add sslmode=verify-full to DUCTUS_DATABASE_URL (sslrootcert=/path/ca.pem for a private CA)
✓ test: 1 passed, 0 failed

Between versions
✓ required stops: none (versions before 1.0 upgrade to any later one)
✓ deprecated settings: none used

The last 1d replayed through 0.6.0-dev (nothing is sent)
✓ replay against version 0.6.0-dev (installed): 100 message(s), 100 comparison(s), 100 identical, 0 different

ready to upgrade to 0.6.0-dev (checked in 0.9 s)

On the server, the replay found the installed version (/usr/bin/ductus) and ran it beside the new one, so every output was compared whole: "100 identical", where the lab example above could only compare decisions. (The database warning is the demo's database server offering no TLS; a real one should.)

The upgrade, under traffic

A sender keeps 4 connections open and sends 1,000 messages at 200 a second, over TLS with its client certificate; 1.5 s in, the package is installed:

$ sudo apt install ./ductus_0.6.0~dev.2_amd64.deb
…
Unpacking ductus (0.6.0~dev.2) over (0.6.0~dev.1) ...
Setting up ductus (0.6.0~dev.2) ...
$ journalctl -u ductus -o cat --no-pager | grep "^handed over"
handed over to ductus 0.6.0-dev (pid 637): receiving never stopped, sending paused 180 ms

What the sender saw (ductus gen's last line), and what the receiver got:

retries 4, rejected 0, refused 0, dropped 4, longest wait 504 ms
unique 1000 | duplicates 0 | out of order 0 | patients 100

• Refused 0: no connection was turned away at any moment.
• Dropped 4, retries 4: each of the 4 open connections was closed once, by the old engine as it let go, with no message lost on it, and opened again at once on the new one. A sender's MLLP client does this by itself; to its operators it looks like nothing.
• Longest wait 504 ms: the slowest ACK, from first try to ACK, for a message that waited out the close, the reconnect and the TLS handshake.
• Every message once and in order at the receiver; sending paused 180 ms while the new engine took the interfaces.

Over four runs of the demo on a laptop (an upgrade and a downgrade each), the longest wait was 0.34–0.52 s and sending paused 26–180 ms; on CI's runner 143 ms (upgrade) and 157 ms (downgrade), paused 14 and 18 ms. Refused was 0 and the receiver complete, with no duplicate and nothing out of order, every time. Without TLS, in the same kind of container, demo-packages.sh measured a reload at 75–81 ms longest wait and a package upgrade at 0.30–0.45 s.

Going back, under traffic

To see what an older version does on a database a newer one migrated, the demo first records one migration the older program doesn't know (number 9999, with a table of its own), as if the newer version had added it; both packages hold one program, so this stands in for a real one. Then the same sender sends while the older package goes in:

$ sudo apt install -y --allow-downgrades ./ductus_0.6.0~dev.1_amd64.deb
…
The following packages will be DOWNGRADED:
  ductus
…
dpkg: warning: downgrading ductus from 0.6.0~dev.2 to 0.6.0~dev.1
…
Unpacking ductus (0.6.0~dev.1) over (0.6.0~dev.2) ...
Setting up ductus (0.6.0~dev.1) ...
$ journalctl -u ductus -o cat --no-pager | grep "^handed over" | tail -1
handed over to ductus 0.6.0-dev (pid 832): receiving never stopped, sending paused 26 ms
$ cd /var/lib/ductus/project
$ sudo -u ductus ductus upgrade --check --env-file /etc/ductus/ductus.env
ductus upgrade --check: what version 0.6.0-dev would do here (nothing is changed)

The database
✓ migrations: nothing to apply
✓ applied by a newer version than this one: 9999 (fine: migrations only add, so this version runs on them)
…
ready to upgrade to 0.6.0-dev (checked in 0.5 s)

retries 4, rejected 0, refused 0, dropped 4, longest wait 349 ms
unique 1000 | duplicates 0 | out of order 0 | patients 100

A downgrade is a handover like an upgrade: refused 0, every message once and in order. The newer version's migration and its table are still there, and the older engine runs beside them. (upgrade --check with the installed program describes this version's view: nothing to apply, and one migration it doesn't know.)

On Windows: the same, with a gap

demo-windows.ps1 (every push, on CI's Windows runner) upgrades the MSI while a sender sends over TLS with its client certificate: 4 connections, 200 messages a second for a minute, the installer started 3 s in and done 3.4 s later. What the sender and the receiver saw:

sent 12000 messages over 4 connection(s) in 60.0s (200/s), retries 12, rejected 0, refused 8, dropped 4, longest wait 3291 ms
received 12200 | unique 12200 | duplicates 0 | out of order 0 | patients 300

(the receiver's count includes the 200 sent before). While the installer stopped, replaced and started the engine, the sender's connections were closed (dropped 4) and its new ones refused for a moment (refused 8); its slowest message waited 3.3 s for its ACK. Nothing was lost, sent twice or out of order: the sender resent on its own, as MLLP senders do. That's the Windows gap until the handover comes to Windows (v1.0): pick a quiet moment.

Going back on Windows is uninstall then install: the demo upgrades to a build numbered as the next minor version, tries the older MSI over it (refused, exit 1603: "A newer version of Ductus is already installed. Uninstall it first to go back."), uninstalls, installs the older one, and starts it on the same settings, project and database (Windows installer).

After the upgrade: confirm

• systemctl status ductus shows "running N interfaces" (Windows: Get-Service Ductus is Running, and the Event Log has the start, ID 1000).
• The console's Overview says the version, and "took over from v…" after a handover; the audit log has an engine.handover entry (audit actions).
• ductus status --token … shows every interface ok within a minute; ductus doctor checks every connection from the server.
• If you kept the replay's report, file it with the change: it's the record of what the upgrade was expected to change.

Troubleshooting by symptom

Find what you're seeing below. Each section shows the command that tells you what's going on, what its output means, and what fixes it. Every output on this page is from a real engine, in a lab with a lab system played by ductus sink and a sender by ductus gen.

Four commands answer most questions, in this order:

The same answers are in the console: the Overview and the interface pages show health, the Hotspots page the backlog, Connections the doctor's checks.

By workload: a feed (MLLP or HTTP in, stored, sent on) is what most of this page is about. An API in respond mode stores nothing, so nothing waits: its caller gets the error instead, and the log says why. A batch (files, SFTP, schedules) shows its problems at pickup, in the log, and its messages afterwards behave like a feed's.

Messages are stuck

You see: a receiving system says it isn't getting messages; the console shows a destination behind or stalled; an alert about a backlog.

$ ductus status
adt-in               pending      100
adt-in               sent         300

by destination:
adt-in → archive     sent         400
adt-in → lab         pending      100
adt-in → lab         sent         300

health:
adt-in               ok, received 8s ago
  → archive          ok, sent 8s ago
  → lab              ok, sent 1 min ago

The lab destination has 100 waiting and sent nothing for a minute. Health says stalled only after a minute without progress (what each state means), so early on, the counts and hotspots say more:

$ ductus hotspots
adt-in → lab: 100 waiting in 20 of 64 lanes; blocked by message 301 (key 100001, lane 23, 4 attempts, waiting 8s): Connection refused (os error 111); all 99 behind it wait (one worker: order)
  lane   worker    waiting  first msg    key                     age attempts  last error
  7      0               5  312          100012                   8s        0  
  8      0               5  317          100017                   8s        0  
…
  23     0               5  301          100001                   8s        4  Connection refused (os error 111)
…

And the log, one line per failed try, each with when it tries again:

WARN ductus_runtime::worker: delivery failed; retrying in 4s interface=adt-in destination=lab id=301 attempts=3 error=Connection refused (os error 111)
WARN ductus_runtime::worker: delivery failed; retrying in 8s interface=adt-in destination=lab id=301 attempts=4 error=Connection refused (os error 111)

What it means, by what hotspots says:

Once the receiver is back, the destination waits out its current retry (doubling up to retry.max_backoff_ms, 60 s by default), then sends everything, in order. In the lab, the 100 waiting were sent 8 s after the receiver came back (up to a minute). Temporary failures are retried forever unless you say otherwise (nuance).

When the engine itself can't reach its database, receiving goes on and sending pauses until it's back (nuance): see Postgres connection errors.

A partner answers AE or AR

You see: a destination in error with quarantined messages, or messages waiting with a receiver's text as their reason.

The two answers mean different things, and the engine treats them differently:

An HTTP receiver's answers map the same way: 408, 429 and 5xx are tried again, any other 4xx is quarantined.

$ ductus status
adt-in               sent         390
adt-in               quarantined  10

by destination:
adt-in → archive     sent         400
adt-in → lab         sent         390
adt-in → lab         quarantined  10

health:
adt-in               ok, received 15s ago
  → archive          ok, sent 15s ago
  → lab              error: 10 quarantined (last error: rejected (AR): rejected by the receiver)

The text after rejected (AR): is the receiver's own (MSA-3): the first thing to read, and what to quote when you ask its owner. Each one is logged:

WARN ductus_runtime::worker: destination rejected; quarantined interface=adt-in destination=lab id=395 error=rejected (AR): rejected by the receiver

The quarantine lists them (the console's Quarantine page, or the API; ordering_key and control_id find the message on the receiver's side):

$ curl -s -H "Authorization: Bearer $DUCTUS_API_TOKEN" 'http://127.0.0.1:7700/api/v1/quarantine?limit=2'
{
    "deliveries": [
        {
            "id": 395,
            "received_at": "2026-10-31T12:53:43.447943Z",
            "interface": "adt-in",
            "destination": "lab",
            "attempts": 0,
            "last_error": "rejected (AR): rejected by the receiver",
            "quarantined_at": "2026-10-31T12:53:58.559595Z",
            "ordering_key": "100015",
            "control_id": "95",
            "size": 150
        },
…

Do:

1. Read the receiver's reason, and the message (the console's message page shows it field by field).
2. If the data is wrong at the source, the sender fixes it and sends again; dismiss the rejected one with a reason (Dismiss).
3. If your mapping is wrong, fix the code, replay to see what changes, deploy, then resend the quarantined ones: they go through the new code (Resend). One message by hand: Edit and resend.
4. If the receiver rejected it by mistake (it was down for maintenance and said AR), resend once it's back.

An AE that never clears looks like stuck messages: hotspots shows AE: and the receiver's text on the first message. If the receiver will never take that message, give the destination a limit (retry: { max_attempts: 10 }) so it's quarantined instead of holding up its lane (delivery).

A TLS handshake fails

You see: messages waiting for a receiver with a TLS reason; a sender who says they can't connect since TLS was turned on; doctor ✗ on a TLS step.

Outbound (the engine to a receiver): each try fails with the reason, and what to change. Here the receiver's certificate comes from a CA this server doesn't trust (a self-signed lab certificate):

WARN ductus_runtime::worker: delivery failed; retrying in 4s interface=adt-in destination=lab id=401 attempts=3 error=TLS: the receiver's certificate isn't from a CA we trust (name its CA in `tls.ca`, or install it on this server)

ductus doctor shows the same at the step that fails, with what the certificate says:

$ ductus doctor --interface adt-in
…
✗ MLLP destination  127.0.0.1:6663 MLLP over TLS
    used by  adt-in → lab
    address  127.0.0.1 (an address, nothing to look up)
    connect  127.0.0.1:6663 from 127.0.0.1 in 0.2 ms
  ✗ TLS      not trusted: its issuer isn't one this server trusts (self-signed, or a private CA); certificate for 127.0.0.1, self-signed, valid until 2027-10-31 (364 days)
  → Name the issuing CA in `ca` (a `certificates:` entry in resources.yaml), or add it to this server's trusted certificates, or ask for a certificate from a public CA.
…

The fix, as it says: name the receiver's CA (here, its own certificate) in resources.yaml, and point the destination's tls: at it:

# resources.yaml
certificates:
  lab-ca:
    ca_file: tls/lab.pem

# interfaces/adt-in/interface.yaml, in part
destinations:
  lab: { mllp: { host: 127.0.0.1, port: 6663, tls: { ca: lab-ca } } }
  archive: { file: { dir: out/archive } }

Deployed with a handover (systemctl reload ductus on a server; the lab ran ductus handover, which is what the reload runs), the waiting messages went out within 2 s, and doctor agrees:

✓ MLLP destination  127.0.0.1:6663 MLLP over TLS
    used by  adt-in → lab
    address  127.0.0.1 (an address, nothing to look up)
    connect  127.0.0.1:6663 from 127.0.0.1 in 0.3 ms
    TLS      TLS 1.3, post-quantum key exchange (X25519MLKEM768), trusted; certificate for 127.0.0.1, self-signed, valid until 2027-10-31 (364 days)

Other reasons, and their fixes:

Inbound (a sender to the engine): the engine logs each refused sender once a minute with the reason ("the sender isn't speaking TLS", "no client certificate", "a subject not on the list"). No line at all means the connection never reached the engine: a firewall, or the wrong port. The full table is in TLS: when it doesn't work (why once a minute). With TLS 1.3 a sender whose client certificate is refused learns it at its first read, not when it connects, so its own error may say the connection closed rather than the certificate.

Postgres connection errors

You see: the engine won't start and names the database; or it runs, but sending has paused; or messages wait on a lookup (db()).

The engine's own database, at start. The start says what's wrong with the URL in DUCTUS_DATABASE_URL (on a server, in /etc/ductus/ductus.env or C:\ProgramData\Ductus\ductus.env):

$ ductus run
Error: connecting to PostgreSQL at 127.0.0.1:40713: the password for `postgres` was refused: check the password in the database URL (DUCTUS_DATABASE_URL; characters like @ or / in it are written %40, %2F)

$ ductus run
Error: connecting to PostgreSQL at 127.0.0.1:40713: there's no database `nosuch` on that server: create it (the PostgreSQL install guide says how) or correct its name in the database URL

$ ductus run
Error: connecting to PostgreSQL: no host is named `db1.hospital.invalid` (DNS has no address for it): check the host in the database URL (DUCTUS_DATABASE_URL)

A refused connection is tried again for 30 s before the start gives up, so a database that's starting at the same time as the engine (a server's boot) doesn't fail it (nuance):

$ ductus run
Error: connecting to PostgreSQL at 127.0.0.1:5999: refused (tried for 30 s): nothing accepts connections there. Postgres is down, listens on another port, or a firewall rejects it: check the host and port in the database URL (DUCTUS_DATABASE_URL), and that Postgres is running

ductus doctor checks the same connection step by step (address, connect, TLS, login, how fast a commit is), and is the quicker way to try a fix:

$ ductus doctor
…
✗ engine database  127.0.0.1:5999 Postgres
    used by  the engine (DUCTUS_DATABASE_URL)
    address  127.0.0.1 (an address, nothing to look up)
  ✗ connect  127.0.0.1:5999 from 127.0.0.1: refused
  → 127.0.0.1 is reachable but nothing accepts connections on port 5999: the service is down, listens on another port, or a firewall at the host rejects it. Ask its owner whether it's up on 5999.

While it runs. If the database goes away, the engine keeps receiving (acknowledging only what it could store) and pauses sending; it carries on by itself when the database is back (nuance). Behind PgBouncer in transaction mode the engine can't hold its lock (nuance).

A database your code looks things up in (db("patient_db")): a failed login or an unreachable server makes the message wait and retry, not fail (nuance); the reason is on the destination (hotspots) and in the log, and doctor checks every database in resources.yaml the same way as the engine's.

The service won't start

You see: systemctl start ductus fails or the service keeps restarting; on Windows the service stops right after starting.

Where the reason is:

systemctl status ductus                 # the last lines, and the exit code
journalctl -u ductus -n 50              # the start's lines

Get-Service Ductus
Get-WinEvent -FilterHashtable @{ LogName = 'Application'; ProviderName = 'Ductus' } -MaxEvents 10 | Format-List TimeCreated, Id, Message

On Windows, a failed start is Event Log 1003 with the reason; the full start is in C:\ProgramData\Ductus\logs\ (Event Log IDs). Before deploying, ductus validate finds most of what would stop a start (a configuration error, a type error, a certificate, the lanes), without starting anything.

The reasons a start refuses, as it says them:

A mistake in a configuration file names the file, the key and what's allowed:

$ ductus run
Error: in …/interfaces/adt-in/interface.yaml

Caused by:
    unknown field `destinatons`, expected one of `name`, `source`, `script`, `format`, `output`, `ordering`, `isolation`, `workers`, `dedicated_pools`, `durability`, `destination`, `destinations`, `retry`, `dedupe`, `rate_limit`, `timeout`, `about`, `retention`, `quarantined`, `monitor`, `alert`, `custom`

A port another program holds names the port and how to find who:

$ ductus run
Error: interface `adt-copy`

Caused by:
    0: binding 0.0.0.0:6661
    1: port 6661 is in use: another program listens on it (another engine, or a test tool); find it with `sudo ss -ltnp 'sport = :6661'`, or give this one another port
$ sudo ss -ltnp 'sport = :6661'
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      128          0.0.0.0:6661      0.0.0.0:*    users:(("ductus",pid=1696,fd=28),("ductus",pid=1696,fd=17))

Another engine already running one of these interfaces on the same database is refused on purpose, with exit code 3:

$ ductus run
error: another engine is running one of this engine's interfaces on this database:
  adt-in
    owned by: DESKTOP-GCT6E6V, pid 1696, ductus 0.6.0-dev, started 2026-10-31 12:54:28 UTC (instance 18e39e6d3fcb787f-6a0)
Two engines running one interface would both send its messages: duplicates, and per-patient order breaks.
Stop that engine first, or remove these interfaces from this one's project.

And the database errors above.

How the service manager retries:

On Windows the service's recovery settings restart it only when a running engine dies (after a minute, twice a day); a start that fails waits for you.

While a start fails, the ports are closed, so senders are refused: that's on purpose, so they don't wait on an engine that isn't there (nuance). On Windows, a service account with a password leaves the service stopped after an install or upgrade until ductus service account NAME --start (event 1005); a missing "Log on as a service" right is a Group Policy matter.

Other symptoms

Ductus architecture

How the engine works, and the rules that keep it correct. Read this before changing the runtime. What interface developers need to know about the resulting behaviour is in the nuances; the tables are described in schema.md (for transparency, not an API).

The shape of it

 sources                 store (Postgres)              workers                       destinations
 ───────                 ────────────────              ───────                       ────────────
 MLLP listener ─┐                                      pool thread 0..N  ─┐          MLLP client
 HTTP (queue)  ─┤   Intake          messages table ─► (shared V8 isolate) ├─► Sender ─ HTTP client
 File / SFTP   ─┼─► persist, then ─► (day partitions,   dedicated threads ─┘          File
 Schedule      ─┘   acknowledge      = the queue)      (own isolate)                 SFTP
                                                        │
 HTTP (respond) ───────────────► channel ──────────────►┘ transform(req) → HTTP response (no queue)
 Internal ◄──────────────────────────────────────────────── Route (destination `interface: x`): stored in
                                                            x's queue in the transaction that marks it sent

One binary (ductus) plus PostgreSQL. No other services. The engine also serves its REST API (/api/v1, default 127.0.0.1:7700), which is how the UI, scripts and ductus drain see and steer it, and the web UI itself (embedded in the binary) on the same listener. Every listener can take TLS (see "TLS on our listeners").

Invariants (never break these)

1. Persist before acknowledge. A source only ACKs (MLLP AA, HTTP 202) after the message row is committed. If storing fails, the sender gets a retryable error (AE, 503). Nothing acknowledged is ever lost by an engine crash. For files, "acknowledge" is moving or deleting the file: that happens only after every record in it is committed.
2. At-least-once delivery. A delivery stays pending until its destination confirms. A crash after sending but before recording sent re-sends it on restart; dedupe at ingest and idempotent receivers cover this. Sent statuses are flushed in batches of ≤50 and always before a retry back-off, so the re-send window is small.
3. One owner per lane and destination = per-key order. The ordering key (e.g. MRN) hashes (FNV-1a, stable forever: ductus_core::partition_of) to one of N virtual partitions ("lanes"). For each destination, each lane is owned by exactly one worker (slot), which processes its pending deliveries strictly by message id. A failed message blocks its slot's lanes for that destination until it succeeds or is quarantined: that's the price of ordering. Across engines sharing a database, each interface has one engine (interface_owners, see "Engines on one database"): nothing may read an interface's queue or change its rows before the engine has claimed it. Storing new messages isn't owned: any engine may (a takeover receives before it claims).
4. Migrations are expand-only. Never drop or rename a column/table in a migration that an older binary still uses. Older binaries tolerate unknown newer migrations (set_ignore_missing), so rollback = reinstall the previous binary. Every migration file from 0025 on starts with its header (-- Upgrade: what it does, -- Heavy: each existing table it works through, or none), which ductus upgrade --check shows with the table's size; 0001–0024 are described in ductus_store::migrations (an applied file can't change: sqlx checksums its whole text).
5. No credentials in interface code. Secrets come from secrets/<env>.json (encrypted) via ${secret:NAME} in YAML or secret() in code; connections are named resources (resources.yaml).
6. Interface code is sandboxed from the network and filesystem except through engine services (db(), fetch(), env(), secret()). Remote module imports are refused.

Crates

Message lifecycle

Statuses: 0 pending, 1 sent, 2 filtered (dropped on purpose: return null / kill(reason), over a rate limit with drop, or its destination was removed), 3 quarantined (needs a human), 4 duplicate (dedupe hit; stored for audit, never queued). last_error holds the reason.

Destinations and deliveries

• An interface has one destination (destination:, named default) or several (destinations: by name). Each destination has its own function (transform:, default transform), output format, retry, rate_limit (its own allowance) and save_output.
• deliveries (day-partitioned like messages): one row per queued message and destination, inserted in the same statement as the message (so still one commit per ACK). Its partial index deliveries_queue (destination_id, message_id, partition_key) WHERE status = 0 is each destination's queue, in message order: a worker's read of 50 stops after 50 matching rows, so it costs the same with 1k or 500k waiting (~1.3 ms). With the lane second (deliveries_pending, before migration 0011), Postgres would have had to sort a lane set's whole backlog, so it walked each partition's primary key instead, past every delivered row: ~300 ms a read with 1M delivered messages ahead of a 10k backlog, i.e. a destination back from an outage drained at ~170 messages/s. Rows carry status, attempts, last_error and, with save_output, output.
• One worker per (slot, destination). Each runs the destination's function on the original message (with meta: id, receivedAt, destination, and origin for a routed message), then delivers. A destination that's down blocks only its own lanes; the others keep going. Ingest wakes every destination's worker for the lane's slot (Wake).
• messages.status is the overall status: pending until no delivery is pending; then quarantined if any is, else sent if any is, else filtered. Store::settle sets it in a statement of its own after the delivery update commits: of two destinations finishing a message at once, the later always sees both done (in one statement, each could see the other pending: write skew). Settle locks the message rows in id order (ORDER BY id FOR UPDATE, a statement of its own) and only then reads their deliveries (a second statement, so a fresh snapshot): destinations settle overlapping sets at once, and left to the plan the lock order could differ between two of them (a deadlock, seen once with routes); and a resend moves a delivery from final back to pending while holding the message's lock, so a settle that had read the deliveries before waiting would mark the message done with a delivery pending again (a_resend_while_another_destination_settles_leaves_the_message_pending fails without it). A worker whose statuses committed but whose settle failed retries the settle alone (settle_saved), never the status write: by then a delivery may have been resent. Delivery updates only change deliveries that are still pending, so a retried flush changes nothing twice. Older binaries read only messages.status, so rolling back still works.
• Reconcile at startup (Store::reconcile, three queries for all interfaces): deliveries for pending messages that lack one (stored by an older version, or a new destination); pending deliveries to removed destinations → filtered; pending deliveries whose message an older version finished → its status. Only interfaces with something waiting are reconciled (Store::with_waiting: two loose index scans over the pending indexes, ~20 ms at 1,001 interfaces with or without a 300,000 backlog); the others have nothing for it to change. Messages left pending with no pending delivery (Store::settle_finished) are settled right after the start: workers read deliveries, never messages.status.
• The script host loads an interface module once and binds one entry per destination (Isolate::add_functions → FunctionIds); runaways are attributed per interface (InterfaceId), so every in-flight call of that interface takes the blame.

Routes between interfaces

• A destination interface: <name> routes to another interface, which has source: internal (no listener; only routes fill it). ductus_core::config::check_routes (validate and startup) requires the target to exist with that source and to read the format the destination outputs, and refuses loops.
• The routing worker has no Sender. A routed message joins the worker's flush batch (≤50, or after 20 ms, or at the end of a read), and Intake::accept_routed on the target's intake stores the batch: Store::finish_routed updates this destination's deliveries (pending ones only, RETURNING which) and inserts, for exactly those, the target's messages and deliveries (its keys, dedupe and destinations; origin_id = the source message) in one transaction. Then settle, then the target's lanes are woken. So each hop is exactly once: a crash before the commit routes again; a retry after it finds the deliveries done and routes nothing.
• That commit skips the flush wait (synchronous_commit = off): nobody outside the database has been told anything, so a lost commit is lost whole and the deliveries are still pending.
• Routed inserts don't pass the target's drain gate (they were received before the drain); the drain waits for them through the pending counts.
• Startup builds every queued interface's Intake before any worker, since a route's worker holds its target's.

Worker per delivery: function → outcome:

• deliver → rate limit → send → sent (batched)
• Filter(reason) → filtered
• Quarantine(reason) (transform threw, quarantine(), AR, 4xx) → quarantined
• Retry (error text contains [ductus:transient], e.g. DB unreachable) → back off 1s→60s, retry the same message (order kept)
• Runaway (the code ran past its timeout without pausing, or filled the isolate's heap) → quarantined; three timeouts in a row or one heap overflow stop the interface (health.rs): its messages stay pending until the code is fixed and the engine restarted; respond-mode requests get 503 Delivery failures: transient → back off and retry (forever unless retry.max_attempts); permanent → quarantine.

Files (file and SFTP sources)

• One FileWatcher per interface over a Backend (local folder or SFTP connection). Each poll lists the folder; a file is ready when its size and time are unchanged since the last poll (or its marker exists). Dot-files are skipped. Ready files go oldest first, one at a time; a failure stops the poll so later files wait (file order kept).
• split: records streams the file through split::Records (256 KB buffer, never the whole file): HL7 per MSH, CSV per record with the header prepended, others per line. Records go to Intake::accept_batch 500 at a time (or 4 MB): one transaction stores the records and the file's progress (source_state: size, time, byte offset, records, done).
• After a crash, the saved offset is where reading resumes (same size and time) or the file is read again from the start (it changed). done: true is saved with the last batch, so a crash before the move doesn't read the file again. Then move/delete, then the progress row is deleted.
• Unsplit files (one message each) are capped at 256 MB; bigger ones are left in place with an error.

Schedules

• schedule::run sleeps until the next due time (re-checking the clock every minute), then Intake::accept_scheduled stores a message {"due", "previous"} and the schedule state (anchor, previous) in one transaction, unless a run of this interface is still pending (then the run is skipped). The worker runs transform on it like any message; the transform's input format is JSON and format/output describe what it produces.
• ductus_core::schedule::Timing: every runs at start then every interval; cron (croner, server local time) waits for its next time. Missed runs collapse into one run on start (anchor_after).

Engines on one database (0.6.2)

• One engine per interface. Each engine run (instance, the id engine.start logs) holds a session advisory lock with a random key on a connection of its own for its whole life (Store::lock_engine, ownership::Owner; engines table). Alive = the lock is held (pg_locks): Postgres frees it however the process ends, so there's no heartbeat and no timeout to wait out after a crash. The lock connection sets TCP keepalives (10 s, 3 × 5 s) so a vanished host's session (and lock) goes within ~25 s instead of the OS default ~2 h, and idle_session_timeout = 0.
• The start sequence: load and check the project (nothing touches the database) → connect and migrate → lock → claim every queued interface in one transaction (Store::claim: delete owner rows whose engine isn't alive, INSERT … ORDER BY interface ON CONFLICT DO NOTHING, read who owns the rest; all or none, rolled back if any is owned) → the lanes guard (Owner::guard_lanes) → then everything that changes an interface's rows (docs snapshot, reconcile, retention, workers, listeners). A refused start writes engine.refused, deletes its engines row, closes the lock connection and exits 3 (ownership::Refused; systemd RestartPreventExitStatus=3). Claims are sorted, so two engines claiming overlapping sets at once wait for each other and the later one sees the earlier's rows (two_engines_claiming_the_same_interfaces_at_once_one_wins). Respond-mode interfaces aren't claimed (no queue). A clean stop deletes its claims and sets stopped_at after the workers have finished. Measured at 1,001 interfaces: claiming ~16 ms (fresh or from a dead engine), ~3 ms for its own rows again.
• Listening sockets outlive the process under systemd (0.6.6.2): listeners are opened last in the start, through systemd::listen: the package's unit is Type=notify with a file descriptor store, so each new listening socket is handed to systemd (FDSTORE=1, named by address) and the next start, after a restart or a crash, listens on the same socket (LISTEN_FDS) instead of binding. Connections arriving in between wait in the kernel's backlog: measured under real systemd 255 and 252 (4 senders, 200 messages/s), a restart refused none (longest ACK wait 0.2–0.3 s; without the store 4 refused, 0.6 s), a kill -9 none (5.4 s: RestartSec). A drain, a stop of the service and a failed start close the sockets for real (FDSTOREREMOVE). READY=1 follows the started line; STOPPING=1 comes before the gates close. Nothing runs in the background: one notify datagram per listener at start.
• The handover (0.6.6.3, Linux): systemctl reload runs ductus handover, which asks the running engine over its control socket ($RUNTIME_DIRECTORY/control, 0600, the asker's uid checked: handover::Control). The engine starts the binary at the path it was started from (argv[0], remembered at start: /proc/self/exe would be the deleted old file) with its arguments, a copy of every listening socket (systemd::handover_sockets, named in DUCTUS_LISTEN_FDS) and a socket pair (DUCTUS_TAKEOVER; JSON lines: From, Ready, Released, Running, Failed). The new engine runs the start with Owner::prepare instead of the claim (its own lock and engines row; refused, as a start would be, when a third live engine owns one of its interfaces, and when lanes go down: the old engine stores into every lane until it lets go, so the guard can only judge after), reads the tiers without applying retention (Retention::new), builds intakes and workers with Ownership not held, loads the code, listens on the inherited sockets (both engines accept on them now; any number may store), starts the API, says Ready. The old one stops HTTP sources first (in-flight requests answered there, not a 503 from a closed gate), then stops as a clean stop does (systemd::handing_over: no STOPPING=1, the sockets stay), releases, writes engine.stop, sends MAINPID=<new> (systemd follows the new process from here; NotifyAccess=main ignored the new one until then: systemd::promote then stores its new sockets and removes the dropped ones), says Released. The new one claims (claim_now), runs the lanes guard, reconciles (take_charge), re-reads pauses, holds: sending resumes; then the docs snapshot and retention (settle_in), audit/alerts/certificates/LDAP (two engines would judge alerts twice), engine.handover. Anything failing before Released ends the new process (Failed with the reason; its sockets are the old one's, so abandon does nothing) and the old one goes on. Measured: one interface at 300 msg/s, sending paused 80–170 ms, 0 refused; 1,001 interfaces 0.6 s (scripts/bench-handover.sh; ~40 ms per statement with 1,001-element arrays is sqlx 0.9's missing TCP_NODELAY). Background: one idle Unix socket listener.
• The replay (0.6.6.4): ductus replay reads a period's originals per interface in id order (Store::originals, 500 a statement, with each destination's record: Store::recorded) over a read-only connection and runs them on one thread with one isolate (replay.rs): the clock pinned to each message's received_at and Math.random seeded per message and destination (Isolate::pin_clock/set_clock, installed before any module loads), meta and the decision exactly as the worker makes them (worker::meta, Outcome::of), services that refuse writes (db().execute, non-GET fetch) and mark the call not comparable, Postgres pools with default_transaction_read_only. Nothing is connected or stored. The other version runs as ductus replay --emit (JSON lines: start, interface, row, done) and the CLI merges both streams by (interface, id, destination) (both produce that order), comparing each row with the other version's, the record's decision and the kept output (replay/report.rs: HL7 by field, JSON by path, text by line; drift when the other version shares a difference). Measured: 100,000 messages × 3 destinations in 7.5 s alone, 8.6 s beside a second version (scripts/bench-replay.sh). Background: nothing; it runs when asked.
• Losing the lock (Owner::watch: the lock connection is checked every 2 s with a 3 s timeout): Ownership turns false, so workers (in hold_while_paused), file watchers and schedules wait; intake keeps storing. The engine takes the same lock key again (retrying while the server still holds the old session) and claims again; if another engine owns an interface by then, it stops with exit 3. The ping is well inside the keepalive window, so an engine cut off from the database pauses before another can claim its interfaces.
• The lanes guard: interface_applied keeps each interface's lanes (partition_count(): partitions with an ordering key, else 1). Lowered or never recorded → Store::stranded counts waiting deliveries in lanes ≥ the new count (the waiting rows of those destinations read once from the queue index in a materialized CTE, filtered at the smallest limit, then matched to each limit; a per-destination LATERAL count let the plan seq-scan a big partition once per destination: 18 s at 1,001 destinations with 300,000 waiting, against 18–55 ms now); any → refused. Raised with messages waiting → warning + interface.partitions_raised. Only changed values are saved. ductus validate runs the same check read-only (Store::connect_read_only: no migrations, since it may run with a newer binary than the engines there), plus retention changes running, waiting for a confirmation or starting at the next start.
• Visibility: GET /engine has holds_interfaces and others (live engines on the database with what they own); the Overview shows both.

TLS on our listeners (0.6.3)

• Certificates are resources.yaml entries (certificates:; ductus_core::resources::CertificateEntry): a pair from files (cert_file/key_file, relative to the project) or from PEM text (usually secrets), or a CA alone (ca/ca_file). Listeners name them: tls: { certificate, clients: { ca, subjects } } on MLLP and HTTP sources and on api in ductus.yaml (ductus_core::tls: the shape, subject patterns, and check for names and kinds). HTTP interfaces on one port share its tls (check_endpoints).
• Key exchange (0.6.4.1): rustls's prefer-post-quantum feature (its own default, which default-features = false had dropped): X25519MLKEM768 first, both as client and server, for every rustls user in the binary (features unify: reqwest, lettre, ldap3, tiberius too). A client sends the hybrid and X25519 key shares together, so a peer without the hybrid costs no retry. +0.15 ms per full handshake (bench-tls.sh); doctor's TLS step names the group (doctor::negotiated).
• Loaded before the database (ductus-runtime/src/tls.rs, Certificates::load): every entry, each pair's key checked against its certificate (CertifiedKey::from_der), expired ones refused, so a start with a broken certificate changes nothing. ductus validate runs the same per entry.
• One rustls ServerConfig per listener (TLS 1.2 and 1.3, aws-lc, the same stack as the outbound connections), whose certificate comes from a resolver (Pair, a ResolvesServerCert behind a lock): reload swaps what the resolver hands out, so new connections get a renewed pair without rebuilding anything. One task checks every file-backed pair every minute, by a SHA-256 of both files' contents (a renewal of the same key type is often the same size, and copies can keep the old time); no database work. A changed pair that doesn't load is logged once per distinct content and the old one stays.
• Client certificates: rustls's WebPKI client verifier against the named CA, wrapped (ClientCheck) to match the leaf's subject (CN, O, OU, C, L, ST, DC by OID) and DNS names against subjects.
• Handshakes never hold up the accept loop: MLLP runs it on each connection's own task (ductus_mllp::serve_with: an open step before the first frame; the listener still closes at once on a drain). HTTP sources and the API use tls::Listener (axum's Listener): TLS connections are accepted and handshaken on their own tasks, finished ones handed to the server through a channel; the API wraps it in tap_io so axum's ConnectInfo<SocketAddr> (the audit log's address) still works. Handshakes time out after 10 s. Refusals are logged with a plain reason at most once a minute per listener and address (Refusals).
• The API over HTTPS sends HSTS (a year). api.listen off loopback without tls warns at start and in validate (the session cookie is Secure).
• Outbound (0.6.3.2): one way to verify (tls::Trust): the platform verifier (rustls_platform_verifier: the system's roots and its checks, the name against the host) plus the CA ca: names, and the pair certificate: names as the client certificate through a ResolvesClientCert (the same Pair, so a renewal applies to client certificates too). Names are ConnectTls { ca, certificate } (ductus_core::tls: tls: true | { ca, certificate } on MLLP destinations, flat ca/certificate on apis:, HTTP destinations with a url, alerts.smtp, webhooks and the heartbeat; LDAP's ca is a name); tls::outbound + check_outbound refuse wrong names and kinds in validate and at start. Certificates::client builds one ClientConfig per distinct ConnectTls and shares it (MLLP destinations naming the same CA share settings and session cache). Users: ductus_mllp::Client::with_connect (the client's connect step: a tokio_rustls handshake within the send's timeout, its refusals turned into a temporary error's words by tls::outbound_failure); reqwest via tls_backend_preconfigured (ALPN h2 + http/1.1 set by us) for HTTPS destinations, api() (a client per API that names TLS) and webhooks; ldap3's set_config; lettre takes certificates rather than settings (extra roots + identity from the pair's PEM; same platform verifier; the transport is rebuilt when the client pair's hash changes). Doctor builds its handshake from the same Trust with its recording verifier around the platform one, and after a finished handshake waits up to 500 ms for a TLS alert (TLS 1.3's refusal of a client certificate comes then).
• Tools (0.6.3.4), all in the binary, none running in the engine: ductus cert request (rcgen: a new RSA 2048 or ECDSA P-256 key, written 0600 with create_new, and a CSR with the names, the subject given and server + client authentication; tls::request), ductus cert show (cert_show.rs: reads a file or entry and judges it with doctor's own pieces: doctor::judge_chain for order and completeness, tls::check_pair for the key, the same 30-day warning), ductus cert self-signed. gen --tls and sink --tls build certificates: entries in memory from their files and go through Certificates like the engine (client/server, tls::mllp_client, tls::handshake), so they verify, refuse and log in the engine's words.
• Doctor's own connections to our listeners aren't refusals: every TCP connection doctor makes registers its local address (Refusals::our_own, kept 20 s; since 0.6.3.5 before connecting, tls::connect_own) and the listener's handshake from it isn't logged; otherwise the full check of a listener that wants client certificates logged a refusal from loopback and used up that address's once-a-minute line. An MLLP sender closing without a TLS close_notify is an ordinary close (debug, not a warning).
• (0.6.3.3, our certificates seen: see Certificates below.)