Running Ductus day to day
These pages are for whoever keeps the interfaces running: the on-call analyst, the integration team, the operations center. They assume the engine is installed (install) and the project deployed; what each setting means is in Configure.
Where to look
| For | Page | |
|---|---|---|
| The web UI (the console) | Everything a person does: health at a glance, messages, quarantine, resend, pause, alerts | The console |
The ductus command | The same from a terminal or a script, and what needs the server (doctor, the audit export, users and tokens) | The CLI day to day |
| The REST API | Your own tools and runbooks; everything the console does | REST API reference |
| Prometheus | Graphs and your monitoring system's own alerts | Monitoring and metrics |
| Logs | What the engine did and why, line by line | Logs |
| Alerts | Being told, so nobody has to watch | Alerts in practice |
The console, the CLI and the API are three ways into the same engine: a pause made in one shows in the others, and every change, whoever makes it and however, is in the audit log with who made it.
What a normal day looks like
- The Overview is green, or shows exactly what isn't: a destination behind, a quarantined message, a certificate expiring. Start every shift there, or let alerts tell you.
- Quarantined messages get looked at the same day. Each one is a message a receiver didn't get. Read the reason, fix what caused it (the sender, the code, a code table), then resend, correct or dismiss it (resend and correct).
- A receiver's planned downtime is a pause, made before it starts, with the ticket as the reason, and a resume after (pause).
- Backlogs drain on their own. A receiver that was down gets everything it missed, in order, when it's back; nothing needs resending.
- Nothing to clean up. Old messages go by themselves, partition by partition, at the retention you set; there's no purge job and no vacuum to schedule (retention and storage).
When something's wrong
| You see | Look at |
|---|---|
| A destination stalled | Its receiver is down, or one message keeps failing: hotspots says which |
| A destination behind | It's sending, but slower than messages arrive: the receiver is slow, or a rate limit |
| A destination in error | Quarantined messages: the quarantine |
| An interface silent or missed | Nothing received when it should have been: the sender, or the network between |
| An interface stopped | Its code ran away or ran out of memory three times: the logs and the quarantined messages say which message (nuance) |
| A sender says connection refused | Is the engine running, draining, or refusing it by allow:? Logs, ductus doctor |
| A TLS handshake fails | The engine logs the reason once a minute per sender; ductus cert show and ductus doctor (TLS) |
What each state means exactly, and when it shows: the states.
Before you change anything
- Upgrades:
ductus upgrade --checkwith the new version first, thenductus replayfor the evidence a change board asks for (resend and replay). - A project change (new code, a new destination):
ductus testandductus validate --env prodin CI, then deploy. On Linux,systemctl reload ductusdeploys without refusing a single connection (high availability). - A shorter retention deletes messages, so it waits for a confirmation (retention).