LambdaMail/docs/OPERATIONS.md

# Operating LambdaMail

What to watch, what the alerts mean, and how to recover. For the initial
install see [DEPLOYMENT.md](DEPLOYMENT.md).

## Health

`GET /health` on the `protocols` service answers three ways, and the difference
matters:

| Response | Meaning |
| :--- | :--- |
| `200`, `X-LambdaMail-Health: ok` | Healthy |
| `200`, `X-LambdaMail-Health: degraded` | Running, but something needs an operator. The body says what. |
| `503` | A dependency is down — usually Postgres. |

Degraded is deliberately not a failure: restarting cannot fix a certificate the
proxy never issued, and a restart loop would only hide it.

### Container health

Every long-running service carries a Docker healthcheck, so `docker compose ps`
is a real answer rather than a list of processes that have not exited:

```bash
docker compose ps --format 'table {{.Service}}\t{{.Status}}'
```

`migrate` is the one service without a probe, on purpose: it applies the
migrations and stops. Everything that needs it waits on
`service_completed_successfully`, which is a stronger guarantee than a probe —
a healthcheck there would report unhealthy exactly when the job had succeeded.

The probes are shallow by design. `web` checks only that Next can route a
request; it does not reach through to `auth` or `protocols`, because a blip in
one backend should not mark the frontend unhealthy and take the site down with
it. Those services have probes of their own.

## Certificates

Under `TLS_MODE=traefik` the mail listeners read the certificate Coolify's proxy
manages. Two failure modes are worth understanding.

**No certificate for the mail host.** The proxy only holds certificates for
hosts it routes, so `mail.example.com` needs a router even though nothing is
served over HTTP there. `docker-compose.yaml` declares one; if you removed it,
the listeners fall back to a self-signed certificate and `/health` reports
degraded.

**A certificate that stopped renewing.** The watcher re-reads the store every
60 seconds and reports:

- `WARNING` when the soonest expiry is under 7 days
- `CRITICAL` under 24 hours, or when it has already expired

Watch for these in `docker compose logs protocols`. The admin console's TLS
panel shows the same state, including when the watcher last managed to read the
store — a timestamp that stops advancing means it has gone blind, which is the
failure that shows up 90 days later as an expired certificate.

## Backups

Two things must be backed up, and **they must not live in the same place**:

1. **The database.** `pg_dump` covers mailboxes, messages metadata, DKIM keys
   (encrypted), sessions and the queue.
2. **`LAMBDAMAIL_MASTER_KEY`.** Every DKIM private key, TOTP secret and provider
   token in that dump is encrypted with it. A backup you cannot decrypt is not a
   backup.

Message bodies live in the `mail_storage` volume (or S3, with
`STORAGE_DRIVER=s3`). The database references them by hash; restoring one
without the other leaves rows pointing at blobs that are not there.

```bash
# Database
docker compose exec -T db pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > lambdamail-$(date +%F).sql.gz

# Message bodies
docker run --rm -v lambdamail_mail_storage:/data -v "$PWD:/backup" \
  alpine tar czf /backup/mail-$(date +%F).tar.gz -C /data .
```

Restoring has not been rehearsed in this repository. Before relying on it,
restore into a scratch environment and confirm a message opens — an untested
restore is a hypothesis.

## Accounts

```bash
make create-admin EMAIL=… PASSWORD='…'     # first admin, or --force for another
make reset-password EMAIL=… PASSWORD='…'   # recovers a lockout, ends its sessions
make reset-mfa EMAIL=…                     # clears every second factor
```

### App passwords for mail clients

A mail client cannot answer a second factor, so an account signs a client in
with an **app password**: create one in the webmail (Settings, app passwords),
copy it once, and give it to the client as the account password. It is shown a
single time and stored only as an Argon2id hash.

An app password works for SMTP submission, IMAP and POP3, as its `scopes`
column allows (the default is all three). A scope limits it: an app password
with only `imap` can read mail and cannot send. ManageSieve takes the mailbox
password only. The mailbox password keeps working everywhere, and a login with
an app password stamps its `last_used_at`, which the settings screen shows.

Until this was wired in, the protocols compared every password only against the
mailbox password, so an app password was created, shown and then refused by
SMTP, IMAP and POP3 with `535 Authentication credentials invalid`.

An account locks itself for 15 minutes after 5 failed passwords. That is the
lockout working, not a fault; `reset-password` clears it.

`reset-mfa` is the lost-phone command, and the answer to a subtler case: an
enrolment can be confirmed against a secret the authenticator no longer holds,
and the account is then locked behind codes nobody can produce. It removes the
authenticator secrets and the recovery codes, and signs the account's sessions
out - a live session would otherwise keep the console open without the factor
that was just removed. The password is untouched, so this alone grants nobody
access; the next sign-in enrols a new factor.

Do not delete and recreate the mailbox for this. That takes its mail, folders
and aliases with it, and none of those are the problem.

The equivalent by hand, if you are on an image that predates the command -
note that clearing `mfa_totp` alone leaves the recovery codes valid:

```sql
DELETE FROM mfa_totp WHERE mailbox_id = (SELECT id FROM mailboxes WHERE email_address = 'them@example.com');
DELETE FROM mfa_recovery_codes WHERE mailbox_id = (SELECT id FROM mailboxes WHERE email_address = 'them@example.com');
```

### Sessions

A webmail session lasts 30 days and is renewed once it is a day old; the
console's lasts 8 hours and is never renewed. The sessions a person holds are
listed in Settings, where each can be signed out; a password change or reset
revokes every other one, and a reset revokes all of them. To end every session
of an account from the database:

```sql
UPDATE web_sessions SET revoked_at = NOW()
 WHERE mailbox_id = (SELECT id FROM mailboxes WHERE email_address = 'user@example.com')
   AND revoked_at IS NULL;
```

Renewal retires the old token after a two-minute grace rather than at once
(its `revoked_at` is briefly in the future), so a session list can show two rows
for a moment after a renewal. Rotating `JWT_SECRET` signs everyone out, webmail
and console alike.

Each account signed in on a browser also has a `lm_acct_<mailbox id>` cookie.
They are only the browser's own copy of its sessions: revoking a session on the
server makes that account drop out of the switcher the next time it is listed.

### Trusted senders

`trusted_senders (mailbox_id, address)` holds what each person has trusted. It
is part of the database backup and nothing else needs backing up for it. To see
or remove a mailbox's entries:

```sql
SELECT address, created_at FROM trusted_senders
 WHERE mailbox_id = (SELECT id FROM mailboxes WHERE email_address = 'user@example.com');
DELETE FROM trusted_senders WHERE mailbox_id = '<id>' AND address = 'x@example.org';
```

A trusted sender's mail is rescued from Junk only when it proves where it came
from (DMARC passed, or SPF passed for the From domain); see
[docs/WEBMAIL.md](WEBMAIL.md#trusted-senders). If a trusted sender's mail still
lands in Junk, look at the `Authentication-Results` header on it: a message
without that proof is deliberately left in Junk.

## Delivery

The outbound queue lives in Postgres, so it survives a restart of anything. The
retry ladder is 5 min, 15 min, 1 h, then 2/4/8/12 h with jitter, giving up after
5 days with a bounce. A delay notice goes out at the fourth attempt.

```sql
-- What is stuck and why
SELECT destination_domain, status, attempt, last_smtp_code, left(last_error, 80)
  FROM outbound_jobs WHERE status IN ('DEFERRED','BOUNCED')
 ORDER BY created_at DESC LIMIT 20;
```

Reading the failures:

- **`4.7.5` with a TLS message** — the destination published a DANE or MTA-STS
  policy that could not be satisfied. The message is deferred rather than sent
  in the clear, which is the point. If it persists, the destination's
  certificate is broken, not yours.
- **`5.1.1`** — no such recipient. Permanent, bounced immediately.
- **A dial or connection error with no code** — the destination is unreachable;
  it will retry.

The admin console's Queue page shows the same data with retry and cancel.

## DNS drift

With a Cloudflare token, the zone is reconciled on start and every six hours.
It is idempotent, and records it did not create are reported as conflicts
rather than overwritten.

```bash
docker compose logs protocols | grep -E "DNS reconcile|DNS conflict"
```

`PARTIAL` or `DRIFT` in that output means the desired state and the zone
disagree — most often because a record was edited by hand.

## Spam filtering

Rspamd holds its state in Redis and its thresholds in `system_config`, editable
from the admin console. The defaults reject at 15, tag at 6 and greylist at 4.

A message the filter routes to `Junk` is filed there, unless the mailbox trusts
the sender and the message proves it came from them (see
[Trusted senders](#trusted-senders)); one over the reject threshold is refused
during the SMTP session, so the sender is told. Nothing is accepted and
silently discarded — a silent discard loses legitimate mail with nobody aware
of it.

"Not spam" in the webmail trusts the sender; it does not train Rspamd. To teach
the filter, use Rspamd's own learning (`rspamc learn_ham`) or its web interface.

## Rotating a DKIM key

From the admin console, or:

```bash
curl -X POST https://<host>/api/v1/admin/dkim/rotate \
  -H 'Content-Type: application/json' -b 'lm_admin_session=…' \
  -d '{"domain":"example.com","selector":"lmail2","algorithm":"rsa2048"}'
```

The previous key is retired with a 7-day overlap rather than deleted: mail
already sent under it is still being verified by receivers who may take a while
to stop resolving the old selector. **Publish the new selector's TXT record
before the overlap ends**, or signatures start failing when the old key goes.

## Rotating the master key

There is no automated path today. It requires decrypting every sealed secret
with the old key and resealing with the new one, and getting it wrong destroys
them. Plan a maintenance window and rehearse against a copy first.

## Upstream images

Two images in the stack are upstream projects with our files layered on top:
`migrations` (golang-migrate plus the schema) and `rspamd` (Rspamd plus its
configuration). Vulnerability scanning in CI covers the four images whose
contents this project authors; findings inside those two belong to the upstream
project and cannot be fixed here.

That is a decision, not an oversight: watch their releases and bump the pinned
tags when they publish. `docker run --rm aquasec/trivy image rspamd/rspamd:3.9`
shows where they stand at any point.

## What is not automated

Being explicit so nothing is assumed:

- Backups are documented, not scheduled. Set up a cron job.
- Restores are not rehearsed.
- There is no metrics endpoint or alerting yet; the signals above are log lines
  and the health endpoint.
- Master key rotation is manual.
- Log retention is whatever Docker's json-file driver is configured for
  (10 MB × 3 per service, in `docker-compose.yaml`).
- **Attachments are not uploaded.** The compose screen lists the files you pick
  and sends their names only; there is no upload endpoint behind the picker
  yet. The screen says so when you attach something, rather than letting the
  message leave without them.
- The vacation responder, the signature and the Sieve rules are stored, but
  nothing consumes them during delivery yet.