Troubleshooting by symptom

Find what you're seeing below. Each section shows the command that tells you what's going on, what its output means, and what fixes it. Every output on this page is from a real engine, in a lab with a lab system played by ductus sink and a sender by ductus gen.

Four commands answer most questions, in this order:

The same answers are in the console: the Overview and the interface pages show health, the Hotspots page the backlog, Connections the doctor's checks.

By workload: a feed (MLLP or HTTP in, stored, sent on) is what most of this page is about. An API in respond mode stores nothing, so nothing waits: its caller gets the error instead, and the log says why. A batch (files, SFTP, schedules) shows its problems at pickup, in the log, and its messages afterwards behave like a feed's.

Messages are stuck

You see: a receiving system says it isn't getting messages; the console shows a destination behind or stalled; an alert about a backlog.

$ ductus status
adt-in               pending      100
adt-in               sent         300

by destination:
adt-in → archive     sent         400
adt-in → lab         pending      100
adt-in → lab         sent         300

health:
adt-in               ok, received 8s ago
  → archive          ok, sent 8s ago
  → lab              ok, sent 1 min ago

The lab destination has 100 waiting and sent nothing for a minute. Health says stalled only after a minute without progress (what each state means), so early on, the counts and hotspots say more:

$ ductus hotspots
adt-in → lab: 100 waiting in 20 of 64 lanes; blocked by message 301 (key 100001, lane 23, 4 attempts, waiting 8s): Connection refused (os error 111); all 99 behind it wait (one worker: order)
  lane   worker    waiting  first msg    key                     age attempts  last error
  7      0               5  312          100012                   8s        0  
  8      0               5  317          100017                   8s        0  
…
  23     0               5  301          100001                   8s        4  Connection refused (os error 111)
…

And the log, one line per failed try, each with when it tries again:

WARN ductus_runtime::worker: delivery failed; retrying in 4s interface=adt-in destination=lab id=301 attempts=3 error=Connection refused (os error 111)
WARN ductus_runtime::worker: delivery failed; retrying in 8s interface=adt-in destination=lab id=301 attempts=4 error=Connection refused (os error 111)

What it means, by what hotspots says:

Once the receiver is back, the destination waits out its current retry (doubling up to retry.max_backoff_ms, 60 s by default), then sends everything, in order. In the lab, the 100 waiting were sent 8 s after the receiver came back (up to a minute). Temporary failures are retried forever unless you say otherwise (nuance).

When the engine itself can't reach its database, receiving goes on and sending pauses until it's back (nuance): see Postgres connection errors.

A partner answers AE or AR

You see: a destination in error with quarantined messages, or messages waiting with a receiver's text as their reason.

The two answers mean different things, and the engine treats them differently:

An HTTP receiver's answers map the same way: 408, 429 and 5xx are tried again, any other 4xx is quarantined.

$ ductus status
adt-in               sent         390
adt-in               quarantined  10

by destination:
adt-in → archive     sent         400
adt-in → lab         sent         390
adt-in → lab         quarantined  10

health:
adt-in               ok, received 15s ago
  → archive          ok, sent 15s ago
  → lab              error: 10 quarantined (last error: rejected (AR): rejected by the receiver)

The text after rejected (AR): is the receiver's own (MSA-3): the first thing to read, and what to quote when you ask its owner. Each one is logged:

WARN ductus_runtime::worker: destination rejected; quarantined interface=adt-in destination=lab id=395 error=rejected (AR): rejected by the receiver

The quarantine lists them (the console's Quarantine page, or the API; ordering_key and control_id find the message on the receiver's side):

$ curl -s -H "Authorization: Bearer $DUCTUS_API_TOKEN" 'http://127.0.0.1:7700/api/v1/quarantine?limit=2'
{
    "deliveries": [
        {
            "id": 395,
            "received_at": "2026-10-31T12:53:43.447943Z",
            "interface": "adt-in",
            "destination": "lab",
            "attempts": 0,
            "last_error": "rejected (AR): rejected by the receiver",
            "quarantined_at": "2026-10-31T12:53:58.559595Z",
            "ordering_key": "100015",
            "control_id": "95",
            "size": 150
        },
…

Do:

1. Read the receiver's reason, and the message (the console's message page shows it field by field).
2. If the data is wrong at the source, the sender fixes it and sends again; dismiss the rejected one with a reason (Dismiss).
3. If your mapping is wrong, fix the code, replay to see what changes, deploy, then resend the quarantined ones: they go through the new code (Resend). One message by hand: Edit and resend.
4. If the receiver rejected it by mistake (it was down for maintenance and said AR), resend once it's back.

An AE that never clears looks like stuck messages: hotspots shows AE: and the receiver's text on the first message. If the receiver will never take that message, give the destination a limit (retry: { max_attempts: 10 }) so it's quarantined instead of holding up its lane (delivery).

A TLS handshake fails

You see: messages waiting for a receiver with a TLS reason; a sender who says they can't connect since TLS was turned on; doctor ✗ on a TLS step.

Outbound (the engine to a receiver): each try fails with the reason, and what to change. Here the receiver's certificate comes from a CA this server doesn't trust (a self-signed lab certificate):

WARN ductus_runtime::worker: delivery failed; retrying in 4s interface=adt-in destination=lab id=401 attempts=3 error=TLS: the receiver's certificate isn't from a CA we trust (name its CA in `tls.ca`, or install it on this server)

ductus doctor shows the same at the step that fails, with what the certificate says:

$ ductus doctor --interface adt-in
…
✗ MLLP destination  127.0.0.1:6663 MLLP over TLS
    used by  adt-in → lab
    address  127.0.0.1 (an address, nothing to look up)
    connect  127.0.0.1:6663 from 127.0.0.1 in 0.2 ms
  ✗ TLS      not trusted: its issuer isn't one this server trusts (self-signed, or a private CA); certificate for 127.0.0.1, self-signed, valid until 2027-10-31 (364 days)
  → Name the issuing CA in `ca` (a `certificates:` entry in resources.yaml), or add it to this server's trusted certificates, or ask for a certificate from a public CA.
…

The fix, as it says: name the receiver's CA (here, its own certificate) in resources.yaml, and point the destination's tls: at it:

# resources.yaml
certificates:
  lab-ca:
    ca_file: tls/lab.pem

# interfaces/adt-in/interface.yaml, in part
destinations:
  lab: { mllp: { host: 127.0.0.1, port: 6663, tls: { ca: lab-ca } } }
  archive: { file: { dir: out/archive } }

Deployed with a handover (systemctl reload ductus on a server; the lab ran ductus handover, which is what the reload runs), the waiting messages went out within 2 s, and doctor agrees:

✓ MLLP destination  127.0.0.1:6663 MLLP over TLS
    used by  adt-in → lab
    address  127.0.0.1 (an address, nothing to look up)
    connect  127.0.0.1:6663 from 127.0.0.1 in 0.3 ms
    TLS      TLS 1.3, post-quantum key exchange (X25519MLKEM768), trusted; certificate for 127.0.0.1, self-signed, valid until 2027-10-31 (364 days)

Other reasons, and their fixes:

Inbound (a sender to the engine): the engine logs each refused sender once a minute with the reason ("the sender isn't speaking TLS", "no client certificate", "a subject not on the list"). No line at all means the connection never reached the engine: a firewall, or the wrong port. The full table is in TLS: when it doesn't work (why once a minute). With TLS 1.3 a sender whose client certificate is refused learns it at its first read, not when it connects, so its own error may say the connection closed rather than the certificate.

Postgres connection errors

You see: the engine won't start and names the database; or it runs, but sending has paused; or messages wait on a lookup (db()).

The engine's own database, at start. The start says what's wrong with the URL in DUCTUS_DATABASE_URL (on a server, in /etc/ductus/ductus.env or C:\ProgramData\Ductus\ductus.env):

$ ductus run
Error: connecting to PostgreSQL at 127.0.0.1:40713: the password for `postgres` was refused: check the password in the database URL (DUCTUS_DATABASE_URL; characters like @ or / in it are written %40, %2F)

$ ductus run
Error: connecting to PostgreSQL at 127.0.0.1:40713: there's no database `nosuch` on that server: create it (the PostgreSQL install guide says how) or correct its name in the database URL

$ ductus run
Error: connecting to PostgreSQL: no host is named `db1.hospital.invalid` (DNS has no address for it): check the host in the database URL (DUCTUS_DATABASE_URL)

A refused connection is tried again for 30 s before the start gives up, so a database that's starting at the same time as the engine (a server's boot) doesn't fail it (nuance):

$ ductus run
Error: connecting to PostgreSQL at 127.0.0.1:5999: refused (tried for 30 s): nothing accepts connections there. Postgres is down, listens on another port, or a firewall rejects it: check the host and port in the database URL (DUCTUS_DATABASE_URL), and that Postgres is running

ductus doctor checks the same connection step by step (address, connect, TLS, login, how fast a commit is), and is the quicker way to try a fix:

$ ductus doctor
…
✗ engine database  127.0.0.1:5999 Postgres
    used by  the engine (DUCTUS_DATABASE_URL)
    address  127.0.0.1 (an address, nothing to look up)
  ✗ connect  127.0.0.1:5999 from 127.0.0.1: refused
  → 127.0.0.1 is reachable but nothing accepts connections on port 5999: the service is down, listens on another port, or a firewall at the host rejects it. Ask its owner whether it's up on 5999.

While it runs. If the database goes away, the engine keeps receiving (acknowledging only what it could store) and pauses sending; it carries on by itself when the database is back (nuance). Behind PgBouncer in transaction mode the engine can't hold its lock (nuance).

A database your code looks things up in (db("patient_db")): a failed login or an unreachable server makes the message wait and retry, not fail (nuance); the reason is on the destination (hotspots) and in the log, and doctor checks every database in resources.yaml the same way as the engine's.

The service won't start

You see: systemctl start ductus fails or the service keeps restarting; on Windows the service stops right after starting.

Where the reason is:

systemctl status ductus                 # the last lines, and the exit code
journalctl -u ductus -n 50              # the start's lines

Get-Service Ductus
Get-WinEvent -FilterHashtable @{ LogName = 'Application'; ProviderName = 'Ductus' } -MaxEvents 10 | Format-List TimeCreated, Id, Message

On Windows, a failed start is Event Log 1003 with the reason; the full start is in C:\ProgramData\Ductus\logs\ (Event Log IDs). Before deploying, ductus validate finds most of what would stop a start (a configuration error, a type error, a certificate, the lanes), without starting anything.

The reasons a start refuses, as it says them:

A mistake in a configuration file names the file, the key and what's allowed:

$ ductus run
Error: in …/interfaces/adt-in/interface.yaml

Caused by:
    unknown field `destinatons`, expected one of `name`, `source`, `script`, `format`, `output`, `ordering`, `isolation`, `workers`, `dedicated_pools`, `durability`, `destination`, `destinations`, `retry`, `dedupe`, `rate_limit`, `timeout`, `about`, `retention`, `quarantined`, `monitor`, `alert`, `custom`

A port another program holds names the port and how to find who:

$ ductus run
Error: interface `adt-copy`

Caused by:
    0: binding 0.0.0.0:6661
    1: port 6661 is in use: another program listens on it (another engine, or a test tool); find it with `sudo ss -ltnp 'sport = :6661'`, or give this one another port
$ sudo ss -ltnp 'sport = :6661'
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      128          0.0.0.0:6661      0.0.0.0:*    users:(("ductus",pid=1696,fd=28),("ductus",pid=1696,fd=17))

Another engine already running one of these interfaces on the same database is refused on purpose, with exit code 3:

$ ductus run
error: another engine is running one of this engine's interfaces on this database:
  adt-in
    owned by: DESKTOP-GCT6E6V, pid 1696, ductus 0.6.0-dev, started 2026-10-31 12:54:28 UTC (instance 18e39e6d3fcb787f-6a0)
Two engines running one interface would both send its messages: duplicates, and per-patient order breaks.
Stop that engine first, or remove these interfaces from this one's project.

And the database errors above.

How the service manager retries:

On Windows the service's recovery settings restart it only when a running engine dies (after a minute, twice a day); a start that fails waits for you.

While a start fails, the ports are closed, so senders are refused: that's on purpose, so they don't wait on an engine that isn't there (nuance). On Windows, a service account with a password leaves the service stopped after an install or upgrade until ductus service account NAME --start (event 1005); a missing "Log on as a service" right is a Group Policy matter.

Other symptoms