Ductus architecture

How the engine works, and the rules that keep it correct. Read this before changing the runtime. What interface developers need to know about the resulting behaviour is in the nuances; the tables are described in schema.md (for transparency, not an API).

The shape of it

 sources                 store (Postgres)              workers                       destinations
 ───────                 ────────────────              ───────                       ────────────
 MLLP listener ─┐                                      pool thread 0..N  ─┐          MLLP client
 HTTP (queue)  ─┤   Intake          messages table ─► (shared V8 isolate) ├─► Sender ─ HTTP client
 File / SFTP   ─┼─► persist, then ─► (day partitions,   dedicated threads ─┘          File
 Schedule      ─┘   acknowledge      = the queue)      (own isolate)                 SFTP
                                                        │
 HTTP (respond) ───────────────► channel ──────────────►┘ transform(req) → HTTP response (no queue)
 Internal ◄──────────────────────────────────────────────── Route (destination `interface: x`): stored in
                                                            x's queue in the transaction that marks it sent

One binary (ductus) plus PostgreSQL. No other services. The engine also serves its REST API (/api/v1, default 127.0.0.1:7700), which is how the UI, scripts and ductus drain see and steer it, and the web UI itself (embedded in the binary) on the same listener. Every listener can take TLS (see "TLS on our listeners").

Invariants (never break these)

1. Persist before acknowledge. A source only ACKs (MLLP AA, HTTP 202) after the message row is committed. If storing fails, the sender gets a retryable error (AE, 503). Nothing acknowledged is ever lost by an engine crash. For files, "acknowledge" is moving or deleting the file: that happens only after every record in it is committed.
2. At-least-once delivery. A delivery stays pending until its destination confirms. A crash after sending but before recording sent re-sends it on restart; dedupe at ingest and idempotent receivers cover this. Sent statuses are flushed in batches of ≤50 and always before a retry back-off, so the re-send window is small.
3. One owner per lane and destination = per-key order. The ordering key (e.g. MRN) hashes (FNV-1a, stable forever: ductus_core::partition_of) to one of N virtual partitions ("lanes"). For each destination, each lane is owned by exactly one worker (slot), which processes its pending deliveries strictly by message id. A failed message blocks its slot's lanes for that destination until it succeeds or is quarantined: that's the price of ordering. Across engines sharing a database, each interface has one engine (interface_owners, see "Engines on one database"): nothing may read an interface's queue or change its rows before the engine has claimed it. Storing new messages isn't owned: any engine may (a takeover receives before it claims).
4. Migrations are expand-only. Never drop or rename a column/table in a migration that an older binary still uses. Older binaries tolerate unknown newer migrations (set_ignore_missing), so rollback = reinstall the previous binary. Every migration file from 0025 on starts with its header (-- Upgrade: what it does, -- Heavy: each existing table it works through, or none), which ductus upgrade --check shows with the table's size; 0001–0024 are described in ductus_store::migrations (an applied file can't change: sqlx checksums its whole text).
5. No credentials in interface code. Secrets come from secrets/<env>.json (encrypted) via ${secret:NAME} in YAML or secret() in code; connections are named resources (resources.yaml).
6. Interface code is sandboxed from the network and filesystem except through engine services (db(), fetch(), env(), secret()). Remote module imports are refused.

Crates

Message lifecycle

Statuses: 0 pending, 1 sent, 2 filtered (dropped on purpose: return null / kill(reason), over a rate limit with drop, or its destination was removed), 3 quarantined (needs a human), 4 duplicate (dedupe hit; stored for audit, never queued). last_error holds the reason.

Destinations and deliveries

• An interface has one destination (destination:, named default) or several (destinations: by name). Each destination has its own function (transform:, default transform), output format, retry, rate_limit (its own allowance) and save_output.
• deliveries (day-partitioned like messages): one row per queued message and destination, inserted in the same statement as the message (so still one commit per ACK). Its partial index deliveries_queue (destination_id, message_id, partition_key) WHERE status = 0 is each destination's queue, in message order: a worker's read of 50 stops after 50 matching rows, so it costs the same with 1k or 500k waiting (~1.3 ms). With the lane second (deliveries_pending, before migration 0011), Postgres would have had to sort a lane set's whole backlog, so it walked each partition's primary key instead, past every delivered row: ~300 ms a read with 1M delivered messages ahead of a 10k backlog, i.e. a destination back from an outage drained at ~170 messages/s. Rows carry status, attempts, last_error and, with save_output, output.
• One worker per (slot, destination). Each runs the destination's function on the original message (with meta: id, receivedAt, destination, and origin for a routed message), then delivers. A destination that's down blocks only its own lanes; the others keep going. Ingest wakes every destination's worker for the lane's slot (Wake).
• messages.status is the overall status: pending until no delivery is pending; then quarantined if any is, else sent if any is, else filtered. Store::settle sets it in a statement of its own after the delivery update commits: of two destinations finishing a message at once, the later always sees both done (in one statement, each could see the other pending: write skew). Settle locks the message rows in id order (ORDER BY id FOR UPDATE, a statement of its own) and only then reads their deliveries (a second statement, so a fresh snapshot): destinations settle overlapping sets at once, and left to the plan the lock order could differ between two of them (a deadlock, seen once with routes); and a resend moves a delivery from final back to pending while holding the message's lock, so a settle that had read the deliveries before waiting would mark the message done with a delivery pending again (a_resend_while_another_destination_settles_leaves_the_message_pending fails without it). A worker whose statuses committed but whose settle failed retries the settle alone (settle_saved), never the status write: by then a delivery may have been resent. Delivery updates only change deliveries that are still pending, so a retried flush changes nothing twice. Older binaries read only messages.status, so rolling back still works.
• Reconcile at startup (Store::reconcile, three queries for all interfaces): deliveries for pending messages that lack one (stored by an older version, or a new destination); pending deliveries to removed destinations → filtered; pending deliveries whose message an older version finished → its status. Only interfaces with something waiting are reconciled (Store::with_waiting: two loose index scans over the pending indexes, ~20 ms at 1,001 interfaces with or without a 300,000 backlog); the others have nothing for it to change. Messages left pending with no pending delivery (Store::settle_finished) are settled right after the start: workers read deliveries, never messages.status.
• The script host loads an interface module once and binds one entry per destination (Isolate::add_functions → FunctionIds); runaways are attributed per interface (InterfaceId), so every in-flight call of that interface takes the blame.

Routes between interfaces

• A destination interface: <name> routes to another interface, which has source: internal (no listener; only routes fill it). ductus_core::config::check_routes (validate and startup) requires the target to exist with that source and to read the format the destination outputs, and refuses loops.
• The routing worker has no Sender. A routed message joins the worker's flush batch (≤50, or after 20 ms, or at the end of a read), and Intake::accept_routed on the target's intake stores the batch: Store::finish_routed updates this destination's deliveries (pending ones only, RETURNING which) and inserts, for exactly those, the target's messages and deliveries (its keys, dedupe and destinations; origin_id = the source message) in one transaction. Then settle, then the target's lanes are woken. So each hop is exactly once: a crash before the commit routes again; a retry after it finds the deliveries done and routes nothing.
• That commit skips the flush wait (synchronous_commit = off): nobody outside the database has been told anything, so a lost commit is lost whole and the deliveries are still pending.
• Routed inserts don't pass the target's drain gate (they were received before the drain); the drain waits for them through the pending counts.
• Startup builds every queued interface's Intake before any worker, since a route's worker holds its target's.

Worker per delivery: function → outcome:

• deliver → rate limit → send → sent (batched)
• Filter(reason) → filtered
• Quarantine(reason) (transform threw, quarantine(), AR, 4xx) → quarantined
• Retry (error text contains [ductus:transient], e.g. DB unreachable) → back off 1s→60s, retry the same message (order kept)
• Runaway (the code ran past its timeout without pausing, or filled the isolate's heap) → quarantined; three timeouts in a row or one heap overflow stop the interface (health.rs): its messages stay pending until the code is fixed and the engine restarted; respond-mode requests get 503 Delivery failures: transient → back off and retry (forever unless retry.max_attempts); permanent → quarantine.

Files (file and SFTP sources)

• One FileWatcher per interface over a Backend (local folder or SFTP connection). Each poll lists the folder; a file is ready when its size and time are unchanged since the last poll (or its marker exists). Dot-files are skipped. Ready files go oldest first, one at a time; a failure stops the poll so later files wait (file order kept).
• split: records streams the file through split::Records (256 KB buffer, never the whole file): HL7 per MSH, CSV per record with the header prepended, others per line. Records go to Intake::accept_batch 500 at a time (or 4 MB): one transaction stores the records and the file's progress (source_state: size, time, byte offset, records, done).
• After a crash, the saved offset is where reading resumes (same size and time) or the file is read again from the start (it changed). done: true is saved with the last batch, so a crash before the move doesn't read the file again. Then move/delete, then the progress row is deleted.
• Unsplit files (one message each) are capped at 256 MB; bigger ones are left in place with an error.

Schedules

• schedule::run sleeps until the next due time (re-checking the clock every minute), then Intake::accept_scheduled stores a message {"due", "previous"} and the schedule state (anchor, previous) in one transaction, unless a run of this interface is still pending (then the run is skipped). The worker runs transform on it like any message; the transform's input format is JSON and format/output describe what it produces.
• ductus_core::schedule::Timing: every runs at start then every interval; cron (croner, server local time) waits for its next time. Missed runs collapse into one run on start (anchor_after).

Engines on one database (0.6.2)

• One engine per interface. Each engine run (instance, the id engine.start logs) holds a session advisory lock with a random key on a connection of its own for its whole life (Store::lock_engine, ownership::Owner; engines table). Alive = the lock is held (pg_locks): Postgres frees it however the process ends, so there's no heartbeat and no timeout to wait out after a crash. The lock connection sets TCP keepalives (10 s, 3 × 5 s) so a vanished host's session (and lock) goes within ~25 s instead of the OS default ~2 h, and idle_session_timeout = 0.
• The start sequence: load and check the project (nothing touches the database) → connect and migrate → lock → claim every queued interface in one transaction (Store::claim: delete owner rows whose engine isn't alive, INSERT … ORDER BY interface ON CONFLICT DO NOTHING, read who owns the rest; all or none, rolled back if any is owned) → the lanes guard (Owner::guard_lanes) → then everything that changes an interface's rows (docs snapshot, reconcile, retention, workers, listeners). A refused start writes engine.refused, deletes its engines row, closes the lock connection and exits 3 (ownership::Refused; systemd RestartPreventExitStatus=3). Claims are sorted, so two engines claiming overlapping sets at once wait for each other and the later one sees the earlier's rows (two_engines_claiming_the_same_interfaces_at_once_one_wins). Respond-mode interfaces aren't claimed (no queue). A clean stop deletes its claims and sets stopped_at after the workers have finished. Measured at 1,001 interfaces: claiming ~16 ms (fresh or from a dead engine), ~3 ms for its own rows again.
• Listening sockets outlive the process under systemd (0.6.6.2): listeners are opened last in the start, through systemd::listen: the package's unit is Type=notify with a file descriptor store, so each new listening socket is handed to systemd (FDSTORE=1, named by address) and the next start, after a restart or a crash, listens on the same socket (LISTEN_FDS) instead of binding. Connections arriving in between wait in the kernel's backlog: measured under real systemd 255 and 252 (4 senders, 200 messages/s), a restart refused none (longest ACK wait 0.2–0.3 s; without the store 4 refused, 0.6 s), a kill -9 none (5.4 s: RestartSec). A drain, a stop of the service and a failed start close the sockets for real (FDSTOREREMOVE). READY=1 follows the started line; STOPPING=1 comes before the gates close. Nothing runs in the background: one notify datagram per listener at start.
• The handover (0.6.6.3, Linux): systemctl reload runs ductus handover, which asks the running engine over its control socket ($RUNTIME_DIRECTORY/control, 0600, the asker's uid checked: handover::Control). The engine starts the binary at the path it was started from (argv[0], remembered at start: /proc/self/exe would be the deleted old file) with its arguments, a copy of every listening socket (systemd::handover_sockets, named in DUCTUS_LISTEN_FDS) and a socket pair (DUCTUS_TAKEOVER; JSON lines: From, Ready, Released, Running, Failed). The new engine runs the start with Owner::prepare instead of the claim (its own lock and engines row; refused, as a start would be, when a third live engine owns one of its interfaces, and when lanes go down: the old engine stores into every lane until it lets go, so the guard can only judge after), reads the tiers without applying retention (Retention::new), builds intakes and workers with Ownership not held, loads the code, listens on the inherited sockets (both engines accept on them now; any number may store), starts the API, says Ready. The old one stops HTTP sources first (in-flight requests answered there, not a 503 from a closed gate), then stops as a clean stop does (systemd::handing_over: no STOPPING=1, the sockets stay), releases, writes engine.stop, sends MAINPID=<new> (systemd follows the new process from here; NotifyAccess=main ignored the new one until then: systemd::promote then stores its new sockets and removes the dropped ones), says Released. The new one claims (claim_now), runs the lanes guard, reconciles (take_charge), re-reads pauses, holds: sending resumes; then the docs snapshot and retention (settle_in), audit/alerts/certificates/LDAP (two engines would judge alerts twice), engine.handover. Anything failing before Released ends the new process (Failed with the reason; its sockets are the old one's, so abandon does nothing) and the old one goes on. Measured: one interface at 300 msg/s, sending paused 80–170 ms, 0 refused; 1,001 interfaces 0.6 s (scripts/bench-handover.sh; ~40 ms per statement with 1,001-element arrays is sqlx 0.9's missing TCP_NODELAY). Background: one idle Unix socket listener.
• The replay (0.6.6.4): ductus replay reads a period's originals per interface in id order (Store::originals, 500 a statement, with each destination's record: Store::recorded) over a read-only connection and runs them on one thread with one isolate (replay.rs): the clock pinned to each message's received_at and Math.random seeded per message and destination (Isolate::pin_clock/set_clock, installed before any module loads), meta and the decision exactly as the worker makes them (worker::meta, Outcome::of), services that refuse writes (db().execute, non-GET fetch) and mark the call not comparable, Postgres pools with default_transaction_read_only. Nothing is connected or stored. The other version runs as ductus replay --emit (JSON lines: start, interface, row, done) and the CLI merges both streams by (interface, id, destination) (both produce that order), comparing each row with the other version's, the record's decision and the kept output (replay/report.rs: HL7 by field, JSON by path, text by line; drift when the other version shares a difference). Measured: 100,000 messages × 3 destinations in 7.5 s alone, 8.6 s beside a second version (scripts/bench-replay.sh). Background: nothing; it runs when asked.
• Losing the lock (Owner::watch: the lock connection is checked every 2 s with a 3 s timeout): Ownership turns false, so workers (in hold_while_paused), file watchers and schedules wait; intake keeps storing. The engine takes the same lock key again (retrying while the server still holds the old session) and claims again; if another engine owns an interface by then, it stops with exit 3. The ping is well inside the keepalive window, so an engine cut off from the database pauses before another can claim its interfaces.
• The lanes guard: interface_applied keeps each interface's lanes (partition_count(): partitions with an ordering key, else 1). Lowered or never recorded → Store::stranded counts waiting deliveries in lanes ≥ the new count (the waiting rows of those destinations read once from the queue index in a materialized CTE, filtered at the smallest limit, then matched to each limit; a per-destination LATERAL count let the plan seq-scan a big partition once per destination: 18 s at 1,001 destinations with 300,000 waiting, against 18–55 ms now); any → refused. Raised with messages waiting → warning + interface.partitions_raised. Only changed values are saved. ductus validate runs the same check read-only (Store::connect_read_only: no migrations, since it may run with a newer binary than the engines there), plus retention changes running, waiting for a confirmation or starting at the next start.
• Visibility: GET /engine has holds_interfaces and others (live engines on the database with what they own); the Overview shows both.

TLS on our listeners (0.6.3)

• Certificates are resources.yaml entries (certificates:; ductus_core::resources::CertificateEntry): a pair from files (cert_file/key_file, relative to the project) or from PEM text (usually secrets), or a CA alone (ca/ca_file). Listeners name them: tls: { certificate, clients: { ca, subjects } } on MLLP and HTTP sources and on api in ductus.yaml (ductus_core::tls: the shape, subject patterns, and check for names and kinds). HTTP interfaces on one port share its tls (check_endpoints).
• Key exchange (0.6.4.1): rustls's prefer-post-quantum feature (its own default, which default-features = false had dropped): X25519MLKEM768 first, both as client and server, for every rustls user in the binary (features unify: reqwest, lettre, ldap3, tiberius too). A client sends the hybrid and X25519 key shares together, so a peer without the hybrid costs no retry. +0.15 ms per full handshake (bench-tls.sh); doctor's TLS step names the group (doctor::negotiated).
• Loaded before the database (ductus-runtime/src/tls.rs, Certificates::load): every entry, each pair's key checked against its certificate (CertifiedKey::from_der), expired ones refused, so a start with a broken certificate changes nothing. ductus validate runs the same per entry.
• One rustls ServerConfig per listener (TLS 1.2 and 1.3, aws-lc, the same stack as the outbound connections), whose certificate comes from a resolver (Pair, a ResolvesServerCert behind a lock): reload swaps what the resolver hands out, so new connections get a renewed pair without rebuilding anything. One task checks every file-backed pair every minute, by a SHA-256 of both files' contents (a renewal of the same key type is often the same size, and copies can keep the old time); no database work. A changed pair that doesn't load is logged once per distinct content and the old one stays.
• Client certificates: rustls's WebPKI client verifier against the named CA, wrapped (ClientCheck) to match the leaf's subject (CN, O, OU, C, L, ST, DC by OID) and DNS names against subjects.
• Handshakes never hold up the accept loop: MLLP runs it on each connection's own task (ductus_mllp::serve_with: an open step before the first frame; the listener still closes at once on a drain). HTTP sources and the API use tls::Listener (axum's Listener): TLS connections are accepted and handshaken on their own tasks, finished ones handed to the server through a channel; the API wraps it in tap_io so axum's ConnectInfo<SocketAddr> (the audit log's address) still works. Handshakes time out after 10 s. Refusals are logged with a plain reason at most once a minute per listener and address (Refusals).
• The API over HTTPS sends HSTS (a year). api.listen off loopback without tls warns at start and in validate (the session cookie is Secure).
• Outbound (0.6.3.2): one way to verify (tls::Trust): the platform verifier (rustls_platform_verifier: the system's roots and its checks, the name against the host) plus the CA ca: names, and the pair certificate: names as the client certificate through a ResolvesClientCert (the same Pair, so a renewal applies to client certificates too). Names are ConnectTls { ca, certificate } (ductus_core::tls: tls: true | { ca, certificate } on MLLP destinations, flat ca/certificate on apis:, HTTP destinations with a url, alerts.smtp, webhooks and the heartbeat; LDAP's ca is a name); tls::outbound + check_outbound refuse wrong names and kinds in validate and at start. Certificates::client builds one ClientConfig per distinct ConnectTls and shares it (MLLP destinations naming the same CA share settings and session cache). Users: ductus_mllp::Client::with_connect (the client's connect step: a tokio_rustls handshake within the send's timeout, its refusals turned into a temporary error's words by tls::outbound_failure); reqwest via tls_backend_preconfigured (ALPN h2 + http/1.1 set by us) for HTTPS destinations, api() (a client per API that names TLS) and webhooks; ldap3's set_config; lettre takes certificates rather than settings (extra roots + identity from the pair's PEM; same platform verifier; the transport is rebuilt when the client pair's hash changes). Doctor builds its handshake from the same Trust with its recording verifier around the platform one, and after a finished handshake waits up to 500 ms for a TLS alert (TLS 1.3's refusal of a client certificate comes then).
• Tools (0.6.3.4), all in the binary, none running in the engine: ductus cert request (rcgen: a new RSA 2048 or ECDSA P-256 key, written 0600 with create_new, and a CSR with the names, the subject given and server + client authentication; tls::request), ductus cert show (cert_show.rs: reads a file or entry and judges it with doctor's own pieces: doctor::judge_chain for order and completeness, tls::check_pair for the key, the same 30-day warning), ductus cert self-signed. gen --tls and sink --tls build certificates: entries in memory from their files and go through Certificates like the engine (client/server, tls::mllp_client, tls::handshake), so they verify, refuse and log in the engine's words.
• Doctor's own connections to our listeners aren't refusals: every TCP connection doctor makes registers its local address (Refusals::our_own, kept 20 s; since 0.6.3.5 before connecting, tls::connect_own) and the listener's handshake from it isn't logged; otherwise the full check of a listener that wants client certificates logged a refusal from loopback and used up that address's once-a-minute line. An MLLP sender closing without a TLS close_notify is an ordinary close (debug, not a warning).
• (0.6.3.3, our certificates seen: see Certificates below.)

Who may connect (0.6.3.5)

• allow: on MLLP/HTTP sources and api (ductus_core::allow: addresses, subnets, host names; parsed with about::Network). Each listener with a list has an allow::Admission, checked right after accept(): in serve_with's open for MLLP (on the connection's own task) and in tls::Listener's accept loop for HTTP and the API, before the handshake task. A refused connection is dropped unread, counted, and logged through tls::Refusals (once a minute per listener and address). HTTP interfaces on one port share one admission (check_endpoints refuses different lists).
• Host names: one allow::Names for the engine, every distinct name across all lists once. Resolved while the database connects (awaited before anything listens), then by one task every 5 minutes, only when some list names a host; a refusal at a listener with names looks them up again at once, at most once a minute per name. Accepted connections never look anything up (an in-memory compare). A failed lookup keeps the last good addresses. Background cost: nothing without names; with N names, N lookups every 5 minutes on tokio's blocking pool. No database work.
• The engine's own checks are let in: doctor connects with tls::connect_own, which binds to the listener's own address first and registers it (Refusals::our_own) before connecting, so the admission (which runs before any handshake) can see it. allow::on_port finds a port's admission in this process (the API's /ports, the interface detail, doctor).
• TCP keepalive on every accepted connection (allow::keepalive, socket2: 15 s idle, 5 s between probes, 3 probes ≈ 30 s), on MLLP, HTTP and the API. Probes are the kernel's.

Retention

• Each interface keeps its messages for one of three windows (retention: 1w/3w/7w, i.e. 7/21/49 days; default from ductus.yaml, else 7w): its tier. Longer history belongs to the source systems or an archive (optional cold storage is planned); the engine isn't a data warehouse. Tiers and periods are chosen for partition count (statements Postgres can't prune pay for every partition): a weekly tier costs about window ÷ 7 partitions, the daily 1w tier one per day, about 24 in all. Shifting a window by a few days changes that by a fraction of a partition, so windows are whole weeks. A message and its deliveries are stored in the interface's tier and always share it (tier column, part of both primary keys). Old messages go by dropping whole partitions (Store::drop_expired, hourly, 5 s after startup, and after each retention change): a partition goes once its whole period is older than its tier's days.
• The held area (migration 0014): an expired partition holding messages someone still needs (pending: a destination down or paused for months; quarantined, unless the interface says quarantined: purge) doesn't stay: those messages move, each with all its deliveries, to tier 0 (messages_held/deliveries_held, partitioned by month), and then the partition drops like any other. Without it, one waiting message kept its whole period for every interface in the tier (a receiver held three months ≈ double the disk; unresolved quarantines, growth without bound). The move (Store::hold_step, from the hourly run in ductus-runtime/src/retention.rs) walks the partition's primary key in batches of 1,000, busy at most half the time, locking messages then deliveries in id order. Held rows are still partitions of messages/deliveries: they stay in their destination's queue (same index, same order) and in the message browser; nothing reads or writes by tier. A worker whose update meets a row moved under it gets a serialization failure (40001) and runs the statement again (retrying in the store: every statement that updates messages or deliveries goes through it, deadlocks too), and finds the row where it is now; the stress test fails without it. A month that's over is dropped whole once nothing in it is unfinished (no row deletes; the disk goes back at once); the current month isn't, since more can still come. Held tables are created when first needed, one table per transaction. Counted hourly (every held row, so not per API call): held in GET /retention, ductus retention, ductus_held_messages/ductus_held_bytes in /metrics. Measured (HISTORY=100000 scripts/demo-held.sh, WSL): 300,000 pending messages with 2 deliveries each moved in 58 s (three partitions of 100,000, 18–22 s each, ~5,000/s with the throttle), then delivered in order in 174 s once the destination was resumed (~1,700/s, a single destination's usual rate). Its effect on ingest during a move isn't measured yet.
• Kept instead of dropped: a partition with rows of an interface whose retention is now longer than the tier (a change still moving them) waits for the change. These are reported (kept in GET /retention, ductus retention).
• Drops don't block: each partition is detached with DETACH PARTITION CONCURRENTLY (no lock that conflicts with inserts or reads; it waits for queries that started earlier instead), checked again (re-attached if it got unfinished messages meanwhile), then dropped. deliveries first. An interrupted run is finished by the next (pending detaches FINALIZEd, detached leftovers dropped).
• Changing retention applies to old messages too (ductus_store::retention, ductus-runtime/src/retention.rs). The retention table holds each interface's applied tier and the running change. One change at a time per interface; a shorter value deletes messages, so it waits for ductus retention confirm (API: POST /retention/{name}/confirm). When a change starts, new messages go to the new tier at once (intake reads the tier from a shared atomic), and a background task works through the interface's rows in other tiers in batches of 1,000, busy at most half the time: finished messages past a shorter window are deleted with their deliveries; the rest move into the new tier with their deliveries, in one statement. Pending messages aren't touched (a worker is updating them) and move once delivered; the change ends when none are left. Quarantined messages move but are never deleted; past the window they stay in their old tier (and go to the held area when it expires). Held messages within the new window move back like any other.
• Moving rows and settling: a message another destination just finished can be moved while a settle statement has it in view; Postgres can't re-check a row that moved partitions and raises a serialization failure (40001), so settle retries (it's idempotent). Found by the stress test.
• Why tier first, then periods: with days × tiers (~690 leaves per table), statements Postgres can't prune by partition paid for every leaf: the worker's queue read went 8.8 → 20 ms, its delivery update 3.9 → 9.6 ms, the message browser 0.2 → 9.6 ms; at ~100 leaves (the first five-tier design, 7/30/90/180/365) they were back to 5.9, 4.8 and 1.4 ms (pgbench on WSL, prepared statements); three tiers (1w/3w/7w) bring it to ~24. And statements that find messages by (id, received_at) pairs carry the batch's received_at range (span): without it, Postgres priced looking up each message in all ~100 partitions above scanning them, and settling 50 messages read every messages partition (84 ms instead of 1.8 ms on 375k rows, and growing with the table).
• Rows stored by engines that don't know tiers go to the 7w tier (the longest) (the column default); the first start of an interface moves its rows to its own tier (that's how migration 0010's copy of older data gets its tiers).

REST API and drain mode

• /api/v1 (ductus-runtime/src/api/): axum, one listener (api.listen in ductus.yaml, default 127.0.0.1:7700, off to disable; --api-listen overrides). Routes are declared with utoipa_axum::routes!, which builds the router and the OpenAPI document (/api/v1/openapi.json) from the same handler annotations: the spec can't drift from the code. The UI will use only this API.
• Who calls (Caller, a Principal): a user (signed in with a password) or a token. Roles: viewer (read), operator (+ drain, pauses, resend/edit, connection checks), admin (+ users and tokens, and every permission). Two permissions go beyond a role (users.permissions, api_tokens.permissions): payloads (read message payloads and saved outputs; editing needs it too) and audit (the audit log, below). A token that belongs to a user (api_tokens.user_id) acts as that user at most: role = the lower of the two, permissions = the intersection, and it stops when the user is disabled. Handlers call caller.require(Role) / require_permission(..); api/mod.rs has one table (ACCESS) of who may call every operation, and a test fails for an operation without a row.
• Tokens: Authorization: Bearer dux_<64 hex>. Stored as SHA-256 hashes (api_tokens); every call looks the hash up (an index hit) joined to the owner, so revoking or disabling works at once. Service tokens (no owner) are made by admins or on the server with ductus token create; users make their own under /me/tokens (listed as <user>/<name>). Migration 0018 revoked every token made before users existed.
• Users (users, ductus_core::password, api/users.rs, api/me.rs): local accounts with argon2id hashes (hashing runs on blocking threads, at most 4 at once, so sign-ins can't take intake's cores); wrong passwords count per username (failed_logins, lockout window and length in auth.lockout); a password an admin chose sets must_change_password, and then only GET/DELETE /session and POST /me/password work (the MustChange extractor; everything else answers 403 password_change_required). The first admin is made on the server (ductus user create; ductus dev makes a dev admin).
• Directory sign-in (auth.ldap; ductus-runtime/src/ldap.rs, ductus_core::ldap, ductus_store directory.rs, migration 0022): POST /session checks a name a local account holds locally and only locally; any other name goes to the directory named by auth.ldap.resource (an ldap: entry in resources.yaml, so doctor, ductus ports, documentation fields and the certificate table cover it). One connection per sign-in (ldap3 over our rustls and platform verifier, plus the entry's ca): bind as the service account, find the person (user_filter, the username escaped per RFC 4515; more than one match is refused), refuse an Active Directory account disabled in userAccountControl, bind as them with the password (never an empty one: that would be an unauthenticated bind), bind back as the service account, read the groups (a base search per configured group; with nested, Active Directory's in-chain rule when the root entry says it's AD, else a walk up the groups, at most 8 levels), then highest role group wins. The account (users.source = 'ldap', no hash) is made or updated in one transaction with its audit row (user.create, user.directory_change). One task (ldap::Round), started with the background work (after a handover's takeover), runs a round at once and then every refresh, on one connection: it checks again, by directory id, everyone with a live session or token and every directory account an owner text of this engine's project reaches (owners::reach); a refusal sets users.directory_refused_at (the identity join treats it like disabled_at, so sessions and tokens stop; the sessions are deleted), logged once (user.directory_refused); an account the directory says nothing new about is read without a lock and not written (check_directory_user), and the round notes when it asked in one statement (directory_checked); a directory that can't be reached changes nothing, and never delays a start. Then (0.6.7.3) owner emails that reach nobody and owner texts naming a disabled account are looked up by attributes.email: one entry, in a role group, gets its account (save_directory_user, audit user.create by the engine with "reason": "owner"); two, or none in a group, are warned about once per run. Two engines saving one new person at once: the second insert hits users_username (or users_directory_id) and is retried once, finding the account. Measured on loopback slapd and WSL's disk (measure_a_refresh_round, measure_an_owners_round, ignored tests): 200 accounts 0.41 s a round (was 1.25 s with a commit per account), 200 owners' accounts 0.38 s, 200 owner emails nobody has 75 ms, none 0.3 ms (no connection). A wrong directory password counts toward the local lockout once the account exists. No AuthProvider trait: SSO (OIDC/SAML) is a redirect flow, not a password check, so its extension point is designed with it.
• Browser sessions (api/session.rs, migrations 0015, 0018): POST /session with {username, password} or {token} (the latter unless auth.token_sign_in is off) sets the cookie ductus_session=dus_<64 hex> (HttpOnly, Secure, SameSite=Strict, Path=/api), stored hashed in api_sessions with the user's or the token's id. Caller takes a bearer token first, else the cookie: a session acts as its user or token (one join, checked per call) and ends after 12 h idle, 7 days (auth.session), sign-out (DELETE /session), sign-out everywhere else (DELETE /me/sessions), a password change (every other session of that user), or when the user is disabled or the token revoked. Requests that change something with the cookie must carry X-Ductus: 1: a header other origins can't send without a CORS preflight the engine never grants, so the cookie can't be used cross-site (SameSite=Strict alone is the first line). GET /session says who is calling and with which permissions.
• The audit log (audit_log, migration 0019; ductus_store::audit, ductus-runtime/src/audit.rs, api/audit.rs): one row per thing done, with who (By: a user, a token, system, anonymous), how (session, token, cli, engine), the connection's address (into_make_service_with_connect_info), what, to what, the outcome and a detail document. Rows are written with the change they describe: a resend, edit or dismissal inserts its audit row first in its own transaction (its message_actions point at it with audit_id, and a request that finds nothing to do rolls the row back); account, token and session changes take an &Audit and insert it in the same transaction; a sign-in, a failed password (counted and logged together) and a sign-out likewise. Actions with no transaction of their own (pause, resume, drain, retention confirm, connection checks, an export of the log) write the row first (record_first) and are refused with 503 if it can't be written. A payload read is written before the payload is returned (record_payload_read: one row per actor and message per audit.payload_read_window; refused with 503 if the row can't be written). The engine writes engine.start (with how the previous run on the host ended: an engine.start without an engine.stop of its instance is a crash), engine.stop, interface.stopped (the runaway guard tells Health, which tells a channel, which a task drains), and retention.change. An hourly job (60 s after start, then every hour) sets message_actions.previous_output NULL once an action is older than its interface's retention (a copy of a payload must not outlive the message), and deletes message_actions and audit_log rows older than audit.keep (default 6 years, at least 1) in batches of 5,000. GET /audit needs the audit permission (format=ndjson is an oldest-first export, itself logged); GET /me/activity is the person's own rows (audit_log.user_id: the user, or the owner of a token).
• Dismissing (Store::dismiss, POST /messages/{id}/dismiss, /messages/dismiss): a quarantined delivery becomes filtered ("dismissed by NAME: reason", reason required), the message is set pending and settled again like after an edit, in the held area as anywhere. It locks the messages first in id order and moves deliveries final to final, so it adds no backward move to the locking rules.
• Appearance (user_settings, GET/PUT /me/settings): a user's theme and custom colors follow them to any browser; the UI keeps a browser copy for the first paint and for token sessions.
• The web UI (ui/, React; api/ui.rs): crates/ductus-runtime/build.rs embeds ui/dist (each file with a gzip copy when that's a tenth smaller) as a sorted static table; every path outside /api is served from it, unknown ones as index.html (the app's router shows the page) except under /assets/, whose content-hashed names are cached for a year (immutable); everything else is no-cache, so an upgrade shows at once. Every UI response carries a CSP (own scripts only, connect-src 'self', no framing). Cargo never runs npm: without ui/dist a "build the UI" page is embedded, and DUCTUS_REQUIRE_UI=1 (CI, check.sh) makes that an error. The UI's API types are generated from ductus api-spec (the OpenAPI document, without an engine) and committed as ui/openapi.json + ui/src/api/schema.d.ts; check.sh and CI fail when they don't match the binary. Operation ids must be unique (utoipa takes the handler's name; every_operation_has_its_own_id). Size: a first visit loads what index.html references, at most 300 KB gzipped (182 KB at step 7c); pages used rarely load on demand in chunks of their own (≤ 50 KB each), enforced by ui/scripts/budget.mjs. The API reference is one: our own renderer of /api/v1/openapi.json (3.7 KB gzipped, the document fetched when opened; no CDN, which hospital networks often block). Off-the-shelf renderers measured 128–1,256 KB gzipped (RapiDoc 128, Redoc 320, Swagger UI 447, Stoplight Elements 650, Scalar 1,256), wouldn't follow the UI's themes or its AAA high-contrast check, and Redoc and Swagger UI call eval/new Function, which the CSP forbids.
• What it serves: engine info and /health (no token: for load balancers); interfaces with state (running/stopped by the runaway guard), drain gate and each destination's pending/quarantined counts; the message browser (an interface's messages newest first, paged by id; one message with payload and deliveries); drain status, start and stop; pausing and resuming a destination; retention (each interface's, changes in progress, partitions kept) and confirming a shorter one; impact, owners and doctor as JSON (the same code as the CLI; doctor skips listening ports, which the engine itself holds); owners with who each is (GET /owners: an account, a team, an address or nobody, and whether it is the caller, which is the UI's "Mine"; GET /owners/people, the accounts that can own things); a preview of moving ownership (POST /owners/reassign/preview: ductus_core::owners::reassign on the project's interface.yaml files and resources.yaml as they are on disk, a warning when the new owner isn't an account, each file's changed lines, a unified diff for git apply and the CLI command; it writes nothing, since the project is the repository's and a server-side edit would be undone by the next deploy); token management; /events, Server-Sent Events with the same counts and each interface's and destination's health, sent whenever a count or a health state changes or a delivery finishes (the workers' in-memory counters, so a message resent and quarantined again between two counts still shows); /metrics for Prometheus and /hotspots (see Monitoring).
• Message actions (api/actions.rs, ductus_store::actions, migration 0016; operator): resend puts finished deliveries back to pending in place (Store::resend: locks the messages in id order, resets the chosen deliveries, records each in message_actions with what it was before, marks the messages pending; the lane's workers are woken through the interface's Intake). The message keeps its id, so its worker takes it next in its lane. Quarantined deliveries by default; named destinations whatever they ended as (a replay); /messages/resend takes up to 500 ids in one transaction. Edit (Intake::copy_edited → Store::copy_edited) inserts a corrected copy with keys read from the new payload (no dedupe, the original's origin_id), filters the original's quarantined deliveries to the copy's destinations ("replaced by edited copy N") and records the link (copy_id), in one transaction; refused for a destination where the original is still pending (a worker may hold it). GET /quarantine reads quarantined deliveries across interfaces from deliveries_quarantined.
• Queue counts come from the partial indexes (deliveries_queue, deliveries_quarantined): their cost grows with the backlog, not with what's kept (about 10 ms for 15k pending here). One monitor task (control.rs) counts every 2 s, only while someone is subscribed to /events or a drain is running, and shares the result with every subscriber.
• Drain: each interface has a Gate (gate.rs). Draining closes them all: Intake refuses to store (gate::Closed), MLLP listeners close their socket and connections (ductus_mllp::serve_until), queued HTTP answers 503 + Retry-After, file watchers and schedules wait for the gate. Every store holds a counted Pass; the monitor declares the engine drained once every gate is closed, no pass is in flight, and none of this engine's destinations has a pending delivery. With stop, the engine then exits (Engine::shutdown). Resume opens the gates; MLLP listeners bind again.
• Stop (ductus_runtime::stop; on Ctrl-C, SIGTERM, or drained with stop): close every gate and wait until no pass is in flight (each message being stored gets its ACK; an MLLP message being handled is still answered), then cancel the workers' stopping token. A worker treats it like a pause (a transform or send under way finishes; it's checked before each message, before each send and in every wait: back-off, rate limit, pause, idle), saves its finished statuses and ends; a thread ends when its workers have (respond-mode workers once their HTTP server has answered what was in flight). Then last activity is saved once more and the API answers its requests in flight. All within --stop-timeout (30 s); past it, or on a second signal, the process exits and what was in hand is processed again at the next start (at-least-once, as after a crash). Before this, SIGTERM killed the process outright, as kill -9 does.

Monitoring

• Counts and timings (stats.rs) are atomics in memory, one set per interface, destination and resource, created at startup and shared by intake, the workers and the API: an increment per event, no lock, no write. They start at zero with the engine (Prometheus expects that). Intake counts received / duplicates / store failures; each destination's worker counts sent / filtered / quarantined / retries and times its function, each successful send, and receipt → sent (received_at to the moment it's handed over; routed messages when they're routed); respond-mode requests are timed in the responder; db()/api()/fetch() calls are timed per resource around the call on the main runtime (ResourceCalls: fixed at startup, so no lock).
• Histograms have 17 buckets (1, 2.5, 5 per decade from 0.5 ms to 10 s, then 1 min, 5 min, 1 h) plus a sum; quantiles are interpolated inside a bucket like Prometheus's histogram_quantile, means are exact. Few buckets because each is a series per destination.
• Exposed as since_start in GET /interfaces (counts, and mean/p50/p95 in ms), resources in GET /engine, and GET /metrics (Prometheus text format, viewer token: ductus_messages_received_total, ductus_deliveries_total{outcome}, ductus_delivery_retries_total, ductus_delivery_latency_seconds, ductus_send_seconds, ductus_transform_seconds, ductus_http_request_seconds, ductus_resource_call_seconds, queue gauges from the database at each scrape, ductus_interface_stopped, ductus_draining).
• Expectations (monitor: in interface.yaml, ductus_core::monitor): active_within (with optional between hours), active_by (a daily, weekly or cron deadline, its window from midnight or from), behind (messages waiting, the oldest one's age, or both; a destination may set its own). Pure functions of the times below, unit-tested with local times.
• Last activity survives a restart or a failover (activity.rs, migration 0012): intake and the workers mark last_received, last_sent and last_error in memory (stats::Moment: an atomic fetch_max, no lock), loaded from interface_activity/destination_activity at startup; one task saves the ones that changed every 5 s (two statements, whatever the volume; a time never moves back, so a second engine's older write can't undo a newer one). watched_since is when an engine first ran the interface: deadlines before it don't count. Rates and timings stay in memory.
• Pausing a destination (ductus pause, POST /interfaces/{name}/destinations/{d}/pause): written to destination_activity first, then a watch switch shared by the destination's workers (every slot). A worker checks it before each message, before each send (also after a rate-limit wait) and after each retry back-off; while paused it saves its finished statuses and waits on the switch, so it costs nothing. A send under way finishes. Pausing again with a reason adds it, keeping the original time and author (act first, explain after). The drain's "nothing pending" leaves paused destinations out. Resuming (DELETE, ductus resume <interface> [destination]) returns the pause it ended.
• Evaluation (ductus_core::monitor::diagnose and interface_state, pure; Engine::diagnose gathers the facts): per destination the waiting count and the oldest waiting message's age (Store::oldest_pending: one probe of the queue index per destination with a backlog, added to every snapshot), quarantined count, last sent and last error, pause, stopped interface, the last 5 minutes and the normal for this hour of the week. States, worst wins: paused > stalled (waiting over STALL_AFTER = 1 min and nothing finished for as long; the last error if it's newer than the stall) > error (quarantined) > behind (past behind) > ok. For a backlog over a minute old: the trend (catching up with an ETA = waiting ÷ (out − in), or not) and causes (burst: over 2× normal inflow and ≥ 5/min more; slow receiver: over 2× normal send and ≥ 5 ms more). Shown as health in GET /interfaces, ductus_destination_state/ductus_interface_state/ductus_queue_oldest_seconds in /metrics, and in ductus status with a token.
• Rates and normal (activity::Trends): the activity task samples each destination's counts every 5 s (received, finished, sends and their total time): a 5-minute window of samples, and when it last finished something. When the hour of the week changes, the hour just ended is learned if at least 15 minutes of it were watched (Baseline::learn: each hour moves 30% of the way to the new value), and the changed baselines are saved in one statement (destination_activity.baseline_*, migration 0013, 168 values per array). The window starts over with the engine; the baseline doesn't.
• Hotspots (ductus_core::hotspots, pure; GET /hotspots, ductus hotspots): for each destination with a backlog, Store::lanes counts the waiting deliveries per lane from the queue index and looks up each lane's first one (its key, attempts, last error). Since a worker handles its lanes one message at a time, a lane whose first message has failed attempts is the one its worker is retrying, and every lane of that worker waits behind it. From that: blocked workers (and whether the receiver took other messages since the blocking one arrived, i.e. the message is the trouble, not the receiver), every worker blocked (the receiver looks down), a worker or lane with most of the backlog (a hot key; 100 waiting at least), and lanes no worker owns (ordering.partitions lowered with messages waiting). On demand only: 300,000 waiting for one destination over 64 lanes takes ~40 ms (WSL), about what counting the queue costs.
• Alerts (ductus_core::alerts, pure; ductus-runtime/src/alerts.rs; ductus_store::alerts, migration 0020; api/alerts.rs): alerts: in ductus.yaml (the only section that may use ${...}: it is read per environment, Settings::alerts) names an SMTP server, channels (email addresses, a webhook in json, slack or teams format) and rules (the first naming a state gives its severity and recipients; owner resolves to the email of the one account it names (owners::find; not a disabled one), else the owners: directory, else the owner text if it is an address). Three tasks run while the engine does, whoever watches: judging every 15 s takes one queue snapshot, interface_state and diagnose for everything (the same as the API), and alerts::step on each stored alert (alert_state, one row per interface and destination, made at start): fire, escalate (higher severity), change quietly, reopen (bad again within settle), recovering, recover, remind (every repeat until acknowledged). Paused destinations hold; states that need watching wait until the engine has run a minute (error and stopped don't). A change is written with its event and its notifications in one transaction, compare-and-set on rev, so two engines on one database record it once. Sending claims due rows of alert_deliveries (the outbox) with FOR UPDATE SKIP LOCKED and a two-minute lease, sends each on its own task (lettre over rustls with the platform verifier, as the HTTP client; reqwest for webhooks), and retries with back-off (30 s doubling to an hour, ten tries); at least once, like deliveries. Quiet hours make a notification due when they end; a recovery cancels what still waits and is sent only to an alert someone was told about. The heartbeat calls a URL every every. History is kept alerts.keep (a year). Acknowledge and silence are operator actions, written with their audit row. Measured (1,001 interfaces, WSL): 35 ms a round, +0.1% of a core, +7 MB.
• Certificates (ductus-runtime/src/certificates.rs, doctor.rs; ductus_store certificates.rs, migration 0021; api/certificates.rs): every doctor.schedule (6 h; the first 60–120 s after start) the engine runs doctor's checks in certificates mode: DNS, TCP and the TLS handshake (after STARTTLS for the mail server) of what has a certificate: HTTPS APIs and destinations, the directory (ldaps://, or StartTLS written by hand like SMTP's: a 31-byte extended request), the alerts' SMTP server, webhooks and heartbeat, and Postgres connections the engine verifies (verify-full/verify-ca: the engine's own database as "engine database", and resources.yaml entries; the 8-byte SSLRequest written by hand, then the same handshake; prefer/require ones are described by doctor, not watched), and SQL Servers the engine verifies (no trust_server_certificate; 0.6.4.3: a PRELOGIN packet asking for encryption written by hand, then the handshake with its records framed in PRELOGIN packets (doctor::Prelogin), or plain TLS with ALPN tds/8.0 for encrypt: strict; a named instance's port from the SQL Server Browser). The verifier records the chain before judging it, so a certificate is read even when it isn't trusted. Each server's certificate goes to certificate_status (a check that couldn't read one keeps the earlier dates, read_at), each run to doctor_runs (the last 30; POST /doctor stores its run too). No code analysis in certificates mode (it only labels "used by"). Judging (the alerts task, every 15 s) reads the stored dates: the earliest expiry in the chain, whole UTC days, against alerts.certificates.warn_days (30/14/7): fire at the first step, escalate at each closer one and when expired (critical), no reminders before expiry, quiet change for a short renewal, recover after settle (ductus_core::alerts::certificate_step; alert_state.stage). GET /certificates, ductus_certificate_expiry_seconds{side,resource,host,port}. Measured: a check of 6 servers ~0.1 s. Ours (0.6.3.3, migration 0024 side): each certificates: pair the engine presents (a TLS listener's, the API's, a client certificate; ductus_core::tls::presented) is one target cert:<name>:<this host>:0 (port 0: no remote row can collide, and one pair served on many ports alerts once), owner = the entry's, else its one interface's. Its row is built from the loaded pair (certificates::our_row), so judging, the API and the gauge read memory, not the table; the table is written at start (one transaction) and when tls::Certificates::watch (the minute's file check) or a connection check finds a renewal or a renewal that can't be used. In full mode doctor also connects to each TLS listener (loopback for 0.0.0.0/::), records the chain served without verifying it, and judges the path (judge_chain: in order, ends at a root, a certificates: CA, or a root of the system's (rustls-native-certs)).
• Respond-mode profile (v0.4 demo, 2 queries + 1 HTTP call to another respond interface, local database): 2.1–3.0 ms in the engine on average; the three calls one after another ~0.5–0.7 ms each, the engine itself ~0.6 ms; curl sees ~0.4 ms more (HTTP both ways). The lever is in interface code (independent calls together): nuance.

Threads and isolates

• Shared pool (default isolation: shared): --threads (default = cores) OS threads, each a tokio current-thread runtime + LocalSet + one V8 isolate hosting many interfaces. Each interface is one async task (Worker); shared interfaces have one slot (all lanes). See docs/shared-pool.md for the scoping rules interface authors follow.
• Dedicated (isolation: dedicated, workers: N): N threads, each with its own isolate, lanes split round-robin.
• V8 isolates can't move between threads; the isolate is Rc<RefCell<Isolate>> per thread and is only borrowed for synchronous calls, never across an await.
• Async transforms: Isolate::begin returns Call::Done (sync transform) or Call::Pending (a future not borrowing the isolate). A per-thread drive_event_loop task polls the JS event loop when kicked, so many interfaces can await DB calls concurrently on one thread. settle converts the result.
• Wake-ups: ingest calls notify_one on the owning slot's Notify (permit kept if busy); a 30 s idle poll is only a safety net.
• Store calls from workers run on the main runtime (Worker::db). The Postgres pool is shared with intake, and a connection's socket is served by the runtime that opened it: a connection opened on a worker thread stalled intake's commits (the senders' ACKs) whenever that thread was busy in V8. Measured in the runaway demo: one sender's 200 ACKs took 4.0 s instead of 0.9 s; a pass-through run 3.3 s → 0.5 s.

Runaway scripts (ductus-script/src/guard.rs)

• Every call into V8 is armed with a deadline: the interface's timeout (default 5 s, 10 ms–10 min) for its transform, the module's for loading it. A single script-watchdog thread checks all isolates every 10 ms and, past a deadline, calls IsolateHandle::request_interrupt. The interrupt runs on the isolate's thread inside the running code: it checks the call is still the one armed (generation counter), reads the JS stack to find whose code it is (module URL, then folder), and calls terminate_execution. disarm then cancels the termination; the isolate stays usable. Terminating only from inside the interrupt means a call that finished in the meantime can never be hit.
• The event loop runs continuations of any interface, so it's armed with the shortest budget on the isolate; when it passes, the interrupt extends the deadline to the budget of the interface found on the stack (or the longest, if none is found).
• Heap: each isolate is created with heap_limits(0, isolates.heap_mb) (ductus.yaml, default 512). Near the limit V8 calls our callback: we raise the limit by a quarter (or the process dies) and stop the running call the same way, then run a full GC.
• A termination in the event loop drops every queued continuation (V8 clears the microtask queue), not just the runaway's: every async transform in flight on that thread will never settle. The worker's drive_event_loop signals them all (watch): the culprit gets the Runaway, the others a transient error and are retried.
• Limits: code in a shared module resumed after an await has no interface file on its stack, so it can't be attributed: all in-flight transforms are retried and it can recur. While runaway code runs, its thread's other interfaces wait up to timeout.

SDK ↔ engine protocol

• Each interface is loaded through a generated glue module: export const run = (raw) => __invoke(entry.transform, raw, formats) where formats = InterfaceConfig::format_spec() ({"input", "output", "http"}).
• __invoke decodes the payload by format, calls transform, serializes the result. Special results: null → filtered; kill(reason) → string prefixed ductus_script::KILL; errors containing QUARANTINE / TRANSIENT markers are classified by the worker.
• Respond-mode HTTP: the engine passes a JSON envelope (method, path, params, query, headers, body); the SDK returns a JSON envelope (status, headers, body).
• Native ops (ops.rs / ops.js): op_ductus_db_query/execute/fetch/variable/secret, exposed to the SDK as globalThis.__ductus_ops. The runtime implements the Services trait (services.rs); parameters are always bound.
  • Postgres: queries are wrapped as WITH ductus_q AS (<sql>) SELECT json_agg(...) so Postgres produces the JSON.
  • SQL Server (mssql.rs, tiberius): $1 → @P1 outside literals/quoted names/comments; rows are converted in Rust to match Postgres's JSON (numbers, ISO 8601 times, \x hex binary, upper-case GUIDs; unnamed columns column<N>); first result set only. Own small pool: a connection goes back only after a call that succeeded (a failed one may be mid-response or inside an open transaction); idle > 5 min is closed (firewalls drop quiet connections); Azure/AG routing is followed once; named instances via SQL Browser. TLS throughout (tiberius Required, or Strict for TDS 8), verified against the system's certificates plus ca: (trust_cert_ca_bundle) for host or name_in_certificate, unless trust_server_certificate; refusals worded by mssql::refusal.
  • fetch() (services.rs http_fetch, one shared reqwest client): the engine returns every status as a response and marks only network errors/timeouts transient; the SDK turns 5xx/408/429 into transient errors unless retry: false. Responses capped at 16 MB; messages never show the query string or URL password.
  • api(name) uses the same op with api set and a path as url; the engine (resolve_api) puts the path under the API's URL, adds its headers (the code's win) and its timeout, so tokens stay in resources.yaml and never enter V8. HTTP destinations with api: are resolved the same way when their sender is built.
  • Under ductus test the SDK swaps the ops for mocks (mock.db/fetch/env/secret, reset before each test; unmocked calls fail with how to mock them), and tests may be async (run_tests drives the event loop).
  • Pools: one shared pool per database resource (max_connections). A dedicated interface can have its own (dedicated_pools: { patient_db: 4 }), created once and shared by its worker threads; its db() calls use it before the shared one (a bulkhead: a busy interface can't starve the others). Shared-pool interfaces can't: the calling interface isn't known inside a shared isolate.
  • DB and HTTP calls are spawned on the main multi-thread runtime, not the worker thread's: a socket is served by the runtime that opened it, and a worker thread busy in V8 would delay its replies.
  • Temporary (message waits and retries): connection/TLS errors, pool timeouts, deadlocks, failovers, login failures and missing databases (fixed by fixing config; quarantining every message would be worse). Everything else (SQL errors) is permanent → quarantine.

Storage

• messages: partitioned by retention tier (tier, LIST: messages_t7 ... messages_t365), then by time (received_at, RANGE): a day for the 1w tier, a week for 3w/7w (messages_t21_20261005, tiers named in days, partitions by the period's first day, bounds at midnight UTC). About 24 leaves per table; see Retention below for why. Partial index (interface, partition_key, id) WHERE status = 0 (the queue of older versions; still used to find pending messages). deliveries has the same tiers and a partition for every messages partition, with the same bounds (copied from the catalog). Partitions created 3 days ahead, hourly, one table and tier per transaction (messages, then deliveries): one transaction holding both tables deadlocked with status settling, which reads deliveries then updates messages (migration 0006; reproduced 8/8 by partition_maintenance_never_deadlocks_with_settling), and one holding two tier tables deadlocks with queries, which lock every tier in creation order (found by the stress test, migration 0010). Each waits at most 2 s for its lock (lock_timeout), then gives up until the next hour: a partition DDL waiting behind a long query makes every later insert queue behind it (measured: an insert waited the whole 4 s of a "report"; partition_maintenance_never_holds_up_inserts). Startup retries for up to a minute. Last resort: when tomorrow's partitions are still missing (every hourly try for two days was blocked), maintenance waits for the lock however long it takes (lock_timeout = 0, logged as a warning): new messages wait behind it for as long as the blocking query runs, which beats having nowhere to store them at midnight (partition_maintenance_waits_when_the_day_is_close). With nothing to create, maintenance takes no lock at all. Locking rules and the stress test that checks them (everything_at_once_without_lock_errors: inserts with colliding dedupe keys, a batch writer, three destinations draining, partitions created, keys purged, all at once; it catches the old deadlock 3/3): see decisions.md "Locking" in the planning repo, and CLAUDE.md.
• dedupe_keys(interface, key, seen_at): checked and claimed in the same transaction as the message insert (one commit).
• durability: relaxed sets synchronous_commit = off for that insert only.
• source_state(interface, key, value jsonb): what sources remember (file progress file:<name>, schedule). Always written in the same transaction as the messages it accounts for.
• Batched inserts (Store::insert_batch) use one INSERT ... SELECT FROM unnest(...) WITH ORDINALITY ORDER BY n (ids keep the batch's order) and claim dedupe keys in bulk (repeats inside the batch are duplicates after the first).

The docs site and the generated references (0.6.8)

• What it is: docs/ is an mdBook (docs/book.toml, docs/SUMMARY.md), built into target/book. The references under docs/reference/ (CLI, REST API, configuration, Event Log IDs, metrics, audit actions) are written by ductus reference docs from the binary itself and committed, so a diff shows what a change did to them and the site's build needs no Rust compile. docs/reference/libraries.md comes from scripts/dev/dependencies.sh (Cargo.lock and the UI's lock file). The configuration schemas (docs/schema/*.json) are written by a ductus-core test (schemars is a dev-dependency only) and embedded in the binary as text, so ductus types gives the editor the schema of the installed version, offline.
• Two builds of one book (0.6.8.3): target/book (mdBook's own search) is what the links are checked on and what each release attaches as the offline docs zip; target/site (scripts/dev/docs-site.sh: the same book with mdBook's search turned off, then a Pagefind index of each page's <main>) is what docs.getductus.com serves. docs/site/search.js puts Pagefind's search modal in the menu bar (S, / and Ctrl+K open it) only where mdBook's search isn't on the page and the page isn't opened from disk, so the zip's pages never show it. Why two: mdBook's search downloads the whole index before its first result, and Pagefind downloads pieces but can't search from file:// (measurements below).
• Checks: scripts/dev/docs-check.sh builds the book and runs lychee offline, with anchors, over the built HTML. It runs in check.sh (when mdBook and lychee are installed) and in .github/workflows/docs.yml on every docs change; a weekly run also checks other sites' links and reports without failing. docs.yml then builds the site and runs scripts/dev/docs-search-check.cjs: both searches in a real browser (the runner's Chrome through playwright-core). Publishing (pages job, target/site) waits for the repository variable DOCS_PAGES=on.
• What it costs (measured 2026-10-30, 0.6.8.1, WSL on the D: disk, the repo on /mnt/d):

  • The binary: 142,961,384 bytes against 142,246,552 for 0.6.7.3, both from cargo build --release (check.sh's build step) in their own target dirs: +714,832 bytes (+0.50%). The five embedded schemas are 288,659 bytes of it (pretty-printed JSON, every setting's description included); the rest is the reference command, the tables it reads (metrics, audit actions, Event Log IDs) and 75 options' new help texts. Minified, the schemas are 236,160 bytes: rebuilt that way the binary is 142,911,792 bytes, 49,592 bytes (0.035%) smaller. Not done: the files people read in .ductus/schema/ stay readable, and 50 KB is noise next to V8. (Gzipped the schemas would be ~46 KB, at the price of a decompression step; also not worth it.) Note: the binary check.sh left in ~/target was 125,800 bytes larger (143,087,184): its test stage builds the binary again, likely with dev-dependency features unified in. Compare sizes from cargo build --release alone; the sizes in STATUS and history before 0.6.8.1 part C were read after check.sh, so they carry that difference.

  • check.sh: ductus reference docs 0.32–0.49 s (5 runs); docs-check.sh 5.98–6.59 s (5 runs), of which mdBook 1.74–1.78 s and lychee 4.32–4.38 s over 3,122 links (3 runs each). So ~6.5 s on a ~600 s check.sh (~1%). The design expected 1–2 s: lychee over the built HTML on /mnt/d (Windows' drive, slow from WSL) is most of it.

  • docs.yml on GitHub's runner (ubuntu-24.04): the build job 5–8 s from start to end, installing mdBook and lychee included (~1 s); building and checking the site ~1 s; the whole run 9–19 s from push to done (4 runs). The network check (links-online) adds a parallel job of ~7 s.

  • The site: 7,993,318 bytes, 109 files, 64 pages, from 61 markdown files (591,129 bytes, 95,396 words; the generated references are 291,955 bytes of it). The search index (searchindex.js, loaded by the browser on the first search) is 3,286,297 bytes, 465,479 gzipped (GitHub Pages serves it compressed). The theme (fonts, CSS, scripts) is 1,928,680 bytes; print.html (every page in one) 893,859 bytes.

  • After the first half of the guides (0.6.8.2 part A, measured 2026-10-30): 84 pages in the book, 770,058 bytes of markdown (120,558 words; the 25 new guide pages 178,123 bytes, 24,783 words); the site 9,482,184 bytes, 133 files, 88 HTML pages. The search index 4,036,253 bytes (+749,956), 589,073 gzipped (-9; 619,144 at -6): 5.2 bytes of index per byte of markdown. docs-check.sh 8.7–9.0 s (3 runs; 4,379 links, lychee most of it). Under both thresholds below, but the second half of the guides (writing interfaces, operate, upgrade, troubleshooting) is likely to bring the index to about 5 MB: measure again there.

  • After the guides (0.6.8.2 part C, measured 2026-10-31): 136 HTML pages, 1,010,951 bytes of markdown (the reference 291,907 of it); the site 11,587,479 bytes. The search index 5,004,962 bytes, 776,262 gzipped (-6; 739,527 at -9): past the ~5 MB line. It's loaded on the first search, not with the page. The first search (headless Chromium, the book served gzipped as GitHub Pages does, network throttled through DevTools, cache off, from typing "retention" to the first result shown, 3 runs each; search-time.cjs in history.md): unthrottled 0.14–0.33 s; 10 Mbit/s with 40 ms latency 0.90–1.04 s; 4 Mbit/s, 100 ms 1.87–1.91 s; 1.6 Mbit/s, 150 ms (3G) 4.45–4.47 s. What mdBook's settings can do (same book): no search on reference/libraries.md 749,205 gzipped (−3.5%); no search on it and reference/api/ 686,972 (−11.5%; 4 Mbit/s 1.87–1.89 s, 3G 3.89–3.91 s: latency and parsing count as much as bytes there); heading-split-level = 2 706,796 (−9%, but a setting's result would land on its page's top: the configuration reference has a ### per setting); heading-split-level = 1 561,707 (−28%, every result a page's top); no search on reference/ 551,797 (−29%), but mdBook 0.4.52 then finds nothing at all (no error; the same query on the full build finds 30). A split index (Pagefind 1.3.0, which loads its index in pieces): indexed in 2.5 s, 2,023,605 bytes in 167 files; a first search fetches ~220 KB in 11 requests: unthrottled 0.09–0.12 s, 10 Mbit/s 0.45–0.47 s, 4 Mbit/s 0.90–0.91 s, 3G 1.57–1.59 s. But Pagefind can't search from file:// (it loads its index as modules: "Failed to fetch dynamically imported module"), while mdBook's search works there: the release's docs zip, read offline, would lose search.

  • Pagefind on the site (0.6.8.3, measured 2026-10-31): Todd chose Pagefind for docs.getductus.com and mdBook's search for the zip. Pagefind 1.5.2 (pinned with its checksum in docs-tools.sh) indexes the 133 searchable pages in 0.75 s; docs-site.sh takes 7–10 s on /mnt/d. Its folder in the site is 1,423,654 bytes in 159 files. The first search, measured again with the search as people use it (scripts/bench-docs-search.cjs; a second run gave 4 Mbit/s 1.21–1.27 s): the page loaded, S pressed, "retention" typed, until the first result is in the page, timed by the page itself; the test server paces every response (a latency per request, one shared link at the profile's bandwidth, gzipped as Pages serves it), because Pagefind 1.5 searches in a Web Worker whose requests DevTools' throttling neither slows nor counts; 3 runs each, both builds of the same commit:

    	local	10 Mbit/s, 40 ms	4 Mbit/s, 100 ms	3G (1.6 Mbit/s, 150 ms)
    mdBook (the zip; the site before)	0.20–0.30 s	0.83–0.90 s	1.80–1.91 s	4.18–4.29 s
    Pagefind's modal (the site now)	0.45–0.53 s	0.65–0.72 s	1.09–1.21 s	1.82–1.89 s

    mdBook's first search downloads 772,889 bytes in one request; Pagefind's 143,455 bytes in 12, six of them in a chain (its script, its worker, the entry, the metadata, the 72 KB WebAssembly, then the 33 KB index chunk for the word, then a fragment per result shown). Slower on a fast connection: Pagefind's input waits 300 ms after the last key before it searches (its default; mdBook doesn't wait), which on slow connections overlaps the downloads and costs nothing. Each page load is 29,256 bytes more (399,515 vs 370,259: Pagefind's UI script and styles, 39 KB and 7 KB gzipped, in place of mdBook's search scripts). Part C's numbers were taken with DevTools throttling and Playwright's waitFor, which re-checks at growing intervals and rounds times up, and Pagefind 1.3's API without its UI: these replace them.

  • Ranking: guides first (Todd, 2026-10-31). Pagefind ranks short pages first, so for 8 of 12 everyday words (retention, alerts, owners, drain, handover, upgrade, quarantine, ldap) a reference page came first ("retention": the CLI page, then the API's, before operate/retention). docs-site.sh weighs each page's <main> (data-pagefind-weight, which scales every word in it, headings included): reference pages 0.1, nuances 0.6, guides Pagefind's 1. Then an explanation comes first for 11 of the 12 (the CLI's page stays first for "handover"), the reference still shows in the first five for the commands' names, and a word only the reference has (payload_read_window) still finds it first. Weighting guides up instead (2 or 5 on their <main>) lifted the long pages (schema.md first for "alerts"). docs-search-check.cjs checks "retention" opens with the guide. Pagefind's 300 ms wait after the last key stays (Todd, 2026-10-31).

  • The offline zip (target/book zipped: 3,212,622 bytes), unpacked into a folder whose name has a space and opened from file://: mdBook's search finds and opens pages, no errors, no Pagefind on the page (docs-search-check.cjs checks this on every docs change).

  • When to act: the site's search no longer loads one index of everything, so the ~5 MB line applies only to the zip's search now, read from disk where size costs little. Watch instead Pagefind's index chunk for common words (33 KB for "retention") and the page-load cost above. If docs-check.sh passes ~15 s in check.sh (10 s on 2026-10-31, 6,398 links), run lychee from WSL's own disk (target/book under $CARGO_TARGET_DIR) instead of /mnt/d.

Performance notes (measured, WSL, 2.5 ms fsync disk)

• One sender connection is bounded by one durable commit per ACK: ~443 msg/s strict, ~868/s relaxed; 16 connections ~2,100/s (group commit).

• SDK: ~14.8k realistic ORU transforms/s per thread. Embedded V8 call overhead is negligible.

• 1,001 interfaces: ~111 MB, 0 idle CPU. Ready (launch → started) in 0.69–0.83 s warm, 1.3 s with a cold file cache (0.6.2, WSL D: disk; it had grown to 4.9–5.2 s by v0.5, unmeasured since v0.3's 0.4 s). Where it went: retention applied in one transaction per interface (3.3 s; now one read, plus writes only for changes), every worker reading its queue at once through the 32-connection pool (startup tasks queued behind ~1,000 reads; now only interfaces with something waiting read at start), reconcile over the whole backlog for every interface (now only those with something waiting), destinations registered with a conflict check each (now read first). Each step's time: RUST_LOG=info,ductus_runtime=debug ("start: …"). Over a Unix socket (or with TCP_NODELAY) the start is 0.35 s: sqlx 0.9 doesn't set TCP_NODELAY, and statements with large parameters (1,001-element arrays) wait ~40 ms for a delayed ACK over TCP. Message flow is unaffected (fan-out benchmark within noise with and without it). Fixed upstream (transact-rs/sqlx#4336, after 0.9.0): the next sqlx release brings it.

• TLS against plain (0.6.3.4, scripts/bench-tls.sh, one engine, WSL on the D: disk: ~5 ms durable commits, so absolute rates sit below the 443/s above): strict MLLP ingest the same within noise: 1 connection 234–239 ACKs/s TLS vs 238–245 plain, 16 connections 2,690–2,780 vs 2,720–2,930 (two rounds each, alternated; one handshake per connection, then the commit flush bounds both). Handshake (a sender's connect to its end, 500 one after another): ECDSA P-256 0.56 ms p50 / 0.80 p95, RSA 2048 0.96 / 1.53, ECDSA with a client certificate checked 0.66 / 0.85, TLS 1.3 resumption 0.54 / 0.68 (it still does a key exchange); plain TCP connect 0.08. Engine memory per idle connection, 1,000 held (a fresh engine each): TLS ~22.6 kB, plain ~14.9 kB. Respond mode (a trivial JSON answer, 2,000 GETs one after another): HTTPS kept alive 0.29 ms p50 / 0.39 p95 (HTTP 0.27 / 0.36); a new connection and full handshake per request 1.2 / 3.0 (HTTP 0.51 / 0.73). Guideline: TLS on every feed that crosses a network not ours, no capacity planning needed for it; respond-mode callers keep connections alive (a new HTTPS connection per call ~4× the time); senders handshaking per message pay ~1 ms each with RSA (many at once not measured). the nuances ("What TLS costs").

• TLS to Postgres (0.6.4.2, scripts/bench-pgtls.sh, a throwaway Postgres 18 with its own CA on the D: disk, sslmode=disable against verify-full, three rounds alternated): strict ingest 1 connection 217–233 ACKs/s plain vs 224–241 TLS (noise), 16 connections 2,786–2,883 vs 2,560–2,739 (~7% lower); a durable commit 2.8 vs 3.0 ms. Opening a connection (TLS and login, 200 in a row): 5.2 ms p50 plain, 64 prefer, 79 verify-full. Nearly all of it is sqlx 0.9.0 not setting TCP_NODELAY (the handshake then the startup message wait on delayed ACKs): with an LD_PRELOAD shim setting it, opening is 8.6 / 10.6 / 22 ms and 16-connection ingest 2,821–2,834 vs 2,765–2,770 (~2%). verify-full's extra ~10 ms is sqlx reading the system's certificates at every new connection. The 1,001-interface start on a dev Postgres with ssl on (so prefer = TLS): 0.91–1.06 s vs 0.52–0.66 s with sslmode=disable; with the shim 0.63–0.72 vs 0.54–0.60. Background: nothing new runs; each verified database adds one handshake to the scheduled certificate check, and a renewed client certificate is put in its pool's options by the minute's file check that already runs. Binary +0.34 MB (141,809,832 bytes). the nuances ("Opening a database connection over TLS costs ~60 ms, for now".

• TLS to SQL Server (0.6.4.3, scripts/bench-mssqltls.sh, the test SQL Server 2025 in Docker on this machine, three rounds alternated, 100 connections in a row each): opening a connection as db()'s pool does (PRELOGIN, TLS, login, SELECT 1): 123–130 ms p50 trust_server_certificate, 120–141 verified, 116–141 strict verified: within the noise between rounds, so verification costs nothing measurable (tiberius sets TCP_NODELAY). Doctor's certificate check of it (DNS, TCP, PRELOGIN, the handshake): 33–36 ms p50 through PRELOGIN (TLS 1.2, so no wait for a late refusal), 545–564 ms strict (TLS 1.3: ~45 ms plus the 0.5 s wait every TLS 1.3 check makes for a refused client certificate). Doctor's TCP connections now set TCP_NODELAY and skip that wait after TLS 1.2 (it was 550 ms through PRELOGIN). Background: nothing new runs; each verified SQL Server adds one handshake to the 6-hourly certificate check.

• File ingest (split: records, durability: relaxed): ~100k CSV rows/s (300,000 rows, 13 MB, 500 per transaction); ~155k before status per destination (a delivery row costs about as much to insert as the message). Strict MLLP ingest (16 connections) unchanged: ~3,100/s either way, it's bound by the commit flush.

• Fan-out (scripts/bench-fanout.sh, 6,000 messages, 16 connections, strict, file destinations): 1 destination 3,125 ACKs/s, all delivered in 3.9 s; 5 destinations 3,081/s, 3.6 s (8,444 deliveries/s); 25 destinations 1,710/s, 8.0 s (18,776 deliveries/s). Ingest holds up to 5 destinations; at 25 the drain's status writes (~19k/s) compete with ingest for the same database. A single destination drains at ~1,500/s: a shared interface has one worker per destination (isolation: dedicated with workers: N for more). With retention tiers (v0.5 step 5; fresh databases, two runs each, before → after): 1 and 5 destinations unchanged within noise; 25 destinations ingest 1,663–1,751 → 1,394–1,445 ACKs/s and deliver 4–13% slower (each message writes 25 delivery rows, whose key now includes the tier). A fresh database has ~15 partitions per table; after 7 weeks, ~24 (see Retention). At steady state (every partition a database has once its longest window has passed; two runs each): five tiers (115 partitions per table) → four (46): everything delivered 1 destination 5.9–6.5 → 4.8–5.0 s, 5 destinations 5.3 → 4.1–4.2 s, 25 destinations 12.3–12.7 → 9.3–9.4 s (~12,000 → ~16,000 deliveries/s); ingest unchanged for 1 destination, ~8% higher with 5 and 25. 50 destinations (step 6, three tiers at steady state): ~500–700 ACKs/s, everything delivered in ~16 s (~19,000 deliveries/s): delivery throughput is flat from 25 destinations up, ingest falls with each delivery row the ACK waits for (guideline in the nuances). The queue index (migration 0011) changed none of these (fresh tables: no delivered rows ahead of the queue); it matters once a day of delivered traffic sits ahead of a backlog (queue read ~300 ms → ~1.3 ms).