Lucas RayanandClaude Opus 5
e5edee096c
Three things the console and the mail client were getting wrong. **Sent stayed empty and drafts would not save.** Both write through the same persistence path, and both failed on the same constraint: spf_result, dkim_result and dmarc_result are restricted to the RFC verdict vocabulary and are nullable, so "not evaluated" has to be NULL. The compose path left them at Go's zero value, which sends an empty string - satisfying neither the CHECK nor the meaning - and the insert was rejected. Filing the Sent copy is deliberately non-fatal, so it failed silently; saving a draft reported the error. Inbound mail was unaffected because it always supplies a real verdict, which is also why the existing test never caught it. **DNS reconciliation could not verify anything.** The console posted to the auth service, which has no resolver, so the endpoint recorded the request and re-read the status already in the database. The machinery to do it properly already existed - the background reconciler resolves every expected record through public resolvers - it just was not reachable from the console. Adds GET /api/v1/admin/dns/verify to the protocols service, which owns both the resolver and the record spec, and wires the domains screen to it: every expected record, its expected value, and whether it actually resolves. A proxied record is reported as present rather than matching, because behind a proxy the published answer is the provider's address and comparing values would always disagree. Reconciliation still runs afterwards so the stored status and the audit trail stay in step. The spec is read, never provisioned: opening a screen must not create DKIM keys as a side effect. Verified: the persistence test fails without the fix with the exact constraint violation (email_messages_dkim_result_check), 22 Go packages and 36 webmail tests green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
216 lines
8.7 KiB
Markdown
216 lines
8.7 KiB
Markdown
# Operating LambdaMail
|
||
|
||
What to watch, what the alerts mean, and how to recover. For the initial
|
||
install see [DEPLOYMENT.md](DEPLOYMENT.md).
|
||
|
||
## Health
|
||
|
||
`GET /health` on the `protocols` service answers three ways, and the difference
|
||
matters:
|
||
|
||
| Response | Meaning |
|
||
| :--- | :--- |
|
||
| `200`, `X-LambdaMail-Health: ok` | Healthy |
|
||
| `200`, `X-LambdaMail-Health: degraded` | Running, but something needs an operator. The body says what. |
|
||
| `503` | A dependency is down — usually Postgres. |
|
||
|
||
Degraded is deliberately not a failure: restarting cannot fix a certificate the
|
||
proxy never issued, and a restart loop would only hide it.
|
||
|
||
### Container health
|
||
|
||
Every long-running service carries a Docker healthcheck, so `docker compose ps`
|
||
is a real answer rather than a list of processes that have not exited:
|
||
|
||
```bash
|
||
docker compose ps --format 'table {{.Service}}\t{{.Status}}'
|
||
```
|
||
|
||
`migrate` is the one service without a probe, on purpose: it applies the
|
||
migrations and stops. Everything that needs it waits on
|
||
`service_completed_successfully`, which is a stronger guarantee than a probe —
|
||
a healthcheck there would report unhealthy exactly when the job had succeeded.
|
||
|
||
The probes are shallow by design. `web` checks only that Next can route a
|
||
request; it does not reach through to `auth` or `protocols`, because a blip in
|
||
one backend should not mark the frontend unhealthy and take the site down with
|
||
it. Those services have probes of their own.
|
||
|
||
## Certificates
|
||
|
||
Under `TLS_MODE=traefik` the mail listeners read the certificate Coolify's proxy
|
||
manages. Two failure modes are worth understanding.
|
||
|
||
**No certificate for the mail host.** The proxy only holds certificates for
|
||
hosts it routes, so `mail.example.com` needs a router even though nothing is
|
||
served over HTTP there. `docker-compose.yaml` declares one; if you removed it,
|
||
the listeners fall back to a self-signed certificate and `/health` reports
|
||
degraded.
|
||
|
||
**A certificate that stopped renewing.** The watcher re-reads the store every
|
||
60 seconds and reports:
|
||
|
||
- `WARNING` when the soonest expiry is under 7 days
|
||
- `CRITICAL` under 24 hours, or when it has already expired
|
||
|
||
Watch for these in `docker compose logs protocols`. The admin console's TLS
|
||
panel shows the same state, including when the watcher last managed to read the
|
||
store — a timestamp that stops advancing means it has gone blind, which is the
|
||
failure that shows up 90 days later as an expired certificate.
|
||
|
||
## Backups
|
||
|
||
Two things must be backed up, and **they must not live in the same place**:
|
||
|
||
1. **The database.** `pg_dump` covers mailboxes, messages metadata, DKIM keys
|
||
(encrypted), sessions and the queue.
|
||
2. **`LAMBDAMAIL_MASTER_KEY`.** Every DKIM private key, TOTP secret and provider
|
||
token in that dump is encrypted with it. A backup you cannot decrypt is not a
|
||
backup.
|
||
|
||
Message bodies live in the `mail_storage` volume (or S3, with
|
||
`STORAGE_DRIVER=s3`). The database references them by hash; restoring one
|
||
without the other leaves rows pointing at blobs that are not there.
|
||
|
||
```bash
|
||
# Database
|
||
docker compose exec -T db pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > lambdamail-$(date +%F).sql.gz
|
||
|
||
# Message bodies
|
||
docker run --rm -v lambdamail_mail_storage:/data -v "$PWD:/backup" \
|
||
alpine tar czf /backup/mail-$(date +%F).tar.gz -C /data .
|
||
```
|
||
|
||
Restoring has not been rehearsed in this repository. Before relying on it,
|
||
restore into a scratch environment and confirm a message opens — an untested
|
||
restore is a hypothesis.
|
||
|
||
## Accounts
|
||
|
||
```bash
|
||
make create-admin EMAIL=… PASSWORD='…' # first admin, or --force for another
|
||
make reset-password EMAIL=… PASSWORD='…' # recovers a lockout, ends its sessions
|
||
make reset-mfa EMAIL=… # clears every second factor
|
||
```
|
||
|
||
An account locks itself for 15 minutes after 5 failed passwords. That is the
|
||
lockout working, not a fault; `reset-password` clears it.
|
||
|
||
`reset-mfa` is the lost-phone command, and the answer to a subtler case: an
|
||
enrolment can be confirmed against a secret the authenticator no longer holds,
|
||
and the account is then locked behind codes nobody can produce. It removes the
|
||
authenticator secrets and the recovery codes, and signs the account's sessions
|
||
out - a live session would otherwise keep the console open without the factor
|
||
that was just removed. The password is untouched, so this alone grants nobody
|
||
access; the next sign-in enrols a new factor.
|
||
|
||
Do not delete and recreate the mailbox for this. That takes its mail, folders
|
||
and aliases with it, and none of those are the problem.
|
||
|
||
The equivalent by hand, if you are on an image that predates the command -
|
||
note that clearing `mfa_totp` alone leaves the recovery codes valid:
|
||
|
||
```sql
|
||
DELETE FROM mfa_totp WHERE mailbox_id = (SELECT id FROM mailboxes WHERE email_address = 'them@example.com');
|
||
DELETE FROM mfa_recovery_codes WHERE mailbox_id = (SELECT id FROM mailboxes WHERE email_address = 'them@example.com');
|
||
```
|
||
|
||
## Delivery
|
||
|
||
The outbound queue lives in Postgres, so it survives a restart of anything. The
|
||
retry ladder is 5 min, 15 min, 1 h, then 2/4/8/12 h with jitter, giving up after
|
||
5 days with a bounce. A delay notice goes out at the fourth attempt.
|
||
|
||
```sql
|
||
-- What is stuck and why
|
||
SELECT destination_domain, status, attempt, last_smtp_code, left(last_error, 80)
|
||
FROM outbound_jobs WHERE status IN ('DEFERRED','BOUNCED')
|
||
ORDER BY created_at DESC LIMIT 20;
|
||
```
|
||
|
||
Reading the failures:
|
||
|
||
- **`4.7.5` with a TLS message** — the destination published a DANE or MTA-STS
|
||
policy that could not be satisfied. The message is deferred rather than sent
|
||
in the clear, which is the point. If it persists, the destination's
|
||
certificate is broken, not yours.
|
||
- **`5.1.1`** — no such recipient. Permanent, bounced immediately.
|
||
- **A dial or connection error with no code** — the destination is unreachable;
|
||
it will retry.
|
||
|
||
The admin console's Queue page shows the same data with retry and cancel.
|
||
|
||
## DNS drift
|
||
|
||
With a Cloudflare token, the zone is reconciled on start and every six hours.
|
||
It is idempotent, and records it did not create are reported as conflicts
|
||
rather than overwritten.
|
||
|
||
```bash
|
||
docker compose logs protocols | grep -E "DNS reconcile|DNS conflict"
|
||
```
|
||
|
||
`PARTIAL` or `DRIFT` in that output means the desired state and the zone
|
||
disagree — most often because a record was edited by hand.
|
||
|
||
## Spam filtering
|
||
|
||
Rspamd holds its state in Redis and its thresholds in `system_config`, editable
|
||
from the admin console. The defaults reject at 15, tag at 6 and greylist at 4.
|
||
|
||
A message the filter routes to `Junk` is filed there; one over the reject
|
||
threshold is refused during the SMTP session, so the sender is told. Nothing is
|
||
accepted and silently discarded — a silent discard loses legitimate mail with
|
||
nobody aware of it.
|
||
|
||
## Rotating a DKIM key
|
||
|
||
From the admin console, or:
|
||
|
||
```bash
|
||
curl -X POST https://<host>/api/v1/admin/dkim/rotate \
|
||
-H 'Content-Type: application/json' -b 'lm_admin_session=…' \
|
||
-d '{"domain":"example.com","selector":"lmail2","algorithm":"rsa2048"}'
|
||
```
|
||
|
||
The previous key is retired with a 7-day overlap rather than deleted: mail
|
||
already sent under it is still being verified by receivers who may take a while
|
||
to stop resolving the old selector. **Publish the new selector's TXT record
|
||
before the overlap ends**, or signatures start failing when the old key goes.
|
||
|
||
## Rotating the master key
|
||
|
||
There is no automated path today. It requires decrypting every sealed secret
|
||
with the old key and resealing with the new one, and getting it wrong destroys
|
||
them. Plan a maintenance window and rehearse against a copy first.
|
||
|
||
## Upstream images
|
||
|
||
Two images in the stack are upstream projects with our files layered on top:
|
||
`migrations` (golang-migrate plus the schema) and `rspamd` (Rspamd plus its
|
||
configuration). Vulnerability scanning in CI covers the four images whose
|
||
contents this project authors; findings inside those two belong to the upstream
|
||
project and cannot be fixed here.
|
||
|
||
That is a decision, not an oversight: watch their releases and bump the pinned
|
||
tags when they publish. `docker run --rm aquasec/trivy image rspamd/rspamd:3.9`
|
||
shows where they stand at any point.
|
||
|
||
## What is not automated
|
||
|
||
Being explicit so nothing is assumed:
|
||
|
||
- Backups are documented, not scheduled. Set up a cron job.
|
||
- Restores are not rehearsed.
|
||
- There is no metrics endpoint or alerting yet; the signals above are log lines
|
||
and the health endpoint.
|
||
- Master key rotation is manual.
|
||
- Log retention is whatever Docker's json-file driver is configured for
|
||
(10 MB × 3 per service, in `docker-compose.yaml`).
|
||
- **Attachments are not uploaded.** The compose screen lists the files you pick
|
||
and sends their names only; there is no upload endpoint behind the picker
|
||
yet. The screen says so when you attach something, rather than letting the
|
||
message leave without them.
|
||
- The vacation responder, the signature and the Sieve rules are stored, but
|
||
nothing consumes them during delivery yet.
|