Step-by-step procedures for incident response, recovery, and common operational tasks

Operational Runbooks

Mailbox version boundary: the hosted mailbox service uses Stalwart Community 0.16.23 with native account administration. The application sending instance has a separate configuration and lifecycle. Older SQL-directory and container examples below must not be applied to the production mailbox; see Hosted email access and the platform infra/mailbox-next deployment source.

Step-by-step procedures for diagnosing and resolving operational issues.

Service Restart Order

When restarting the entire stack, follow this order to respect service dependencies:

1. PostgreSQL  (database - no dependencies)
2. Redis       (cache - no dependencies)
3. Stalwart    (mail server - depends on PostgreSQL for SQL directory)
4. App         (Next.js - depends on PostgreSQL, Redis, Stalwart)
5. Cron        (scheduler - depends on App)
6. Nginx       (proxy - depends on App)

Commands

# Restart in order
docker compose restart postgres
docker compose restart redis
docker compose restart stalwart
docker compose restart app
docker compose restart cron
docker compose restart nginx

# Or restart the entire stack (Docker Compose handles ordering via depends_on)
docker compose down && docker compose up -d

Warning: docker compose down stops all services and removes containers. Data is preserved in volumes. If you only need to restart services, use docker compose restart instead.

Complete Outage Recovery

If all services are down (VPS reboot, power loss, etc.):

Step 1: Verify VPS is Accessible

ssh -i ~/.ssh/your_key -p 2222 ubuntu@YOUR_VPS_IP

Step 2: Check Docker

# Ensure Docker is running
sudo systemctl status docker

# If Docker is down
sudo systemctl start docker

Step 3: Start Services

cd /opt/northrelay
docker compose up -d

Step 4: Verify All Services

# Check container status
docker compose ps

# Check health endpoint
curl -s http://localhost:3000/api/health | jq

# Check PostgreSQL
docker compose exec postgres pg_isready -U north_user

# Check Stalwart
curl -s -u admin:TOKEN http://localhost:8080/api/healthz

# Check mail queue for stuck messages
curl -s -u admin:TOKEN http://localhost:8080/api/queue/messages | jq length

Step 5: Check WireGuard Tunnels (if applicable)

# Verify tunnels are up
sudo wg show wg0

# If tunnel is down, bring it up
sudo wg-quick up wg0

# Verify connectivity to the sending relay
ping -c 3 10.0.2.2

Step 6: Verify Email Flow

# Send a test email via API
curl -X POST http://localhost:3000/api/v1/emails/send \
  -H "Authorization: Bearer TEST_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"to": "test@example.com", "subject": "Recovery Test", "text": "Service recovered"}'

# Monitor delivery in Stalwart logs
docker compose logs -f stalwart --tail 20

Email Delivery Failure Diagnosis

When emails are not being delivered, work through these steps in order:

Step 1: Check the Mail Queue

# Queue size
curl -s -u admin:TOKEN http://localhost:8080/api/queue/messages | jq length

# Queue details
curl -s -u admin:TOKEN http://localhost:8080/api/queue/messages | jq '.[] | {id, from, to, status, next_retry}'

If the queue is growing, messages are being accepted but not delivered. If the queue is empty, messages may not be reaching Stalwart.

Step 2: Check Stalwart Logs

# Recent delivery attempts
docker compose logs stalwart --tail 200 2>&1 | grep -E "deliver|bounce|reject|defer"

Common error patterns:

Log Pattern	Meaning	Action
`connection refused`	Relay VPS unreachable	Check WireGuard tunnel
`TLS handshake failed`	Certificate issue on relay	Check relay TLS certs
`550 5.1.1 user unknown`	Recipient doesn't exist	Verify recipient address
`451 4.7.1 rate limit`	Rate limit hit	Check user quota
`blacklisted`	IP on blocklist	Check blocklist status

Step 3: Check DNS

# Verify MX records
dig MX yourdomain.com +short

# Verify SPF
dig TXT yourdomain.com +short | grep spf

# Verify DKIM
dig TXT default._domainkey.yourdomain.com +short

# Verify DMARC
dig TXT _dmarc.yourdomain.com +short

Step 4: Check the relay route

# Verify Stalwart's route points at the sending relay (DNS hostname, never a raw IP)
docker exec stalwart stalwart-cli \
  -u http://localhost:8080 -c admin:TOKEN \
  server list-config 'queue.route.relay'
# Expected: address = sovereign.north-pipe.ca

# Confirm the hostname resolves to the WireGuard address on Sanctuary
getent ahosts sovereign.north-pipe.ca
# Expected: 10.0.2.2

# Confirm the tunnel to the sending relay is up
ping -c 1 10.0.2.2  # Sovereign relay

# Check the latest delivery canary run
# Admin → System emails (/dashboard/admin/system-emails): last run should be PASS
# Alerts email_canary_failed / email_canary_stale mean the full path is broken

Step 5: Check Blocklists

# Check your IP against common blocklists
# Use the Blocklist status card on /dashboard/reputation
# Or check manually:
dig +short YOUR_IP.zen.spamhaus.org
# If it returns an IP (e.g., 127.0.0.2), you're listed
# If NXDOMAIN, you're clean

Stalwart Won't Start

Step 1: Check Logs

docker compose logs stalwart --tail 50

Step 2: Common Causes

Error	Cause	Fix
`address already in use`	Port 25/587 is occupied	`ss -tlnp \| grep :25` to find the process, then stop it
`permission denied`	Binding to port < 1024	Ensure host networking or proper port mapping
`config parse error`	Invalid TOML syntax	Validate config: `cat config.toml \| python3 -c "import sys,tomllib; tomllib.load(sys.stdin.buffer)"`
`certificate not found`	TLS cert path wrong	Verify cert paths in config match mounted volumes
`database locked`	RocksDB lock file stale	Remove lock file: `rm data/db/LOCK` then restart
`out of memory`	Insufficient RAM	Check `free -h`, increase VPS RAM or reduce max-connections

Step 3: Validate Configuration

# Check TOML syntax
docker compose run --rm stalwart /usr/local/bin/stalwart-mail --config /opt/stalwart/config.toml --validate

# If that fails, check for common TOML issues:
# - Missing quotes around string values
# - Incorrect indentation
# - Duplicate keys

Step 4: Start with Verbose Logging

# Temporarily increase log level
docker compose run --rm stalwart /usr/local/bin/stalwart-mail \
  --config /opt/stalwart/config.toml \
  --log-level trace

WireGuard Tunnel Down

Symptoms

Emails queue up but never deliver
Ping to 10.0.2.2 (sending relay) fails
wg show shows no recent handshake

Step 1: Check Tunnel Status

sudo wg show wg0

Look for:

latest handshake: Should be within the last 2 minutes. If blank or old, the tunnel is down.
transfer: Should show non-zero bytes. Zero rx bytes means no traffic is coming back.

Step 2: Restart the Tunnel

sudo wg-quick down wg0
sudo wg-quick up wg0

Step 3: Check Firewall

# Ensure UDP port is open (default WireGuard port: 51820)
sudo ufw status | grep 51820

# If not listed, allow it
sudo ufw allow 51820/udp

Step 4: Check Remote Peer

If the tunnel won't establish, the issue may be on the relay VPS side:

# SSH to relay VPS and check WireGuard there
ssh relay-vps "sudo wg show wg0"

# Restart WireGuard on relay
ssh relay-vps "sudo wg-quick down wg0 && sudo wg-quick up wg0"

Step 5: Check NAT / ISP Issues

If both sides show correct config but no handshake:

# Test UDP connectivity
nc -u -z RELAY_PUBLIC_IP 51820

# Check if NAT is blocking the return traffic
sudo tcpdump -i eth0 udp port 51820 -c 10

Database Issues

Connection Failures

# Check PostgreSQL is running
docker compose ps postgres

# Check connection count
docker compose exec postgres psql -U north_user -d northrelay \
  -c "SELECT count(*) FROM pg_stat_activity;"

# If max connections exceeded, find idle connections
docker compose exec postgres psql -U north_user -d northrelay \
  -c "SELECT pid, state, query_start, query FROM pg_stat_activity WHERE state = 'idle' ORDER BY query_start LIMIT 20;"

# Terminate idle connections older than 10 minutes
docker compose exec postgres psql -U north_user -d northrelay \
  -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND query_start < now() - interval '10 minutes';"

Migration Problems

# Check current migration status
docker compose exec app npx prisma migrate status

# If migrations are out of sync, reset (DESTRUCTIVE - development only)
docker compose exec app npx prisma migrate reset

# For production, push schema directly
docker compose exec app npx prisma db push

# Regenerate Prisma client
docker compose exec app npx prisma generate

Warning: prisma migrate reset drops and recreates the entire database. Never use this in production. Use prisma db push for non-destructive schema updates.

Backup and Restore

# Create a backup
docker compose exec postgres pg_dump -U north_user northrelay > backup_$(date +%Y%m%d).sql

# Restore from backup
docker compose exec -T postgres psql -U north_user northrelay < backup_20260319.sql

IP Blocked by Stalwart

Symptoms

SMTP connections close immediately without a banner
telnet your-server 25 connects then disconnects
No error appears in application logs

Diagnosis

# Check if the IP is in the blocked list
docker exec stalwart stalwart-cli \
  -u http://localhost:8080 -c admin:TOKEN \
  server list-config 'server.blocked-ip'

Fix

# Unblock the IP
docker exec stalwart stalwart-cli \
  -u http://localhost:8080 -c admin:TOKEN \
  server delete-config 'server.blocked-ip.THE_BLOCKED_IP'

Prevention

See Stalwart Setup - IP Blocking Prevention for configuration options to prevent future blocks.

DKIM Signing Failures

Symptoms

Outbound emails arrive without a DKIM-Signature header
DMARC alignment fails
Email clients show "not authenticated" warnings

Step 1: Check OpenDKIM on the Relay VPS

# SSH to the relay VPS
ssh relay-vps

# Check OpenDKIM container is running
docker ps | grep opendkim

# Check OpenDKIM logs for errors
docker logs shared-opendkim --tail 50 2>&1 | grep -iE "error|fail|key|sign"

Step 2: Verify Key Files Exist

# On the relay VPS
ls -la /opt/shared-relay/dkim/

# Each domain should have a .key file
# Example: yourdomain.com.default.key

Step 3: Verify KeyTable and SigningTable

# Check KeyTable has the domain
cat /opt/shared-relay/KeyTable | grep yourdomain.com

# Check SigningTable has the domain
cat /opt/shared-relay/SigningTable | grep yourdomain.com

Step 4: Verify DNS Record

# Check DKIM DNS record
dig TXT default._domainkey.yourdomain.com +short

# Should return something like:
# "v=DKIM1; k=rsa; p=MIIBIjANBg..."

Step 5: Restart OpenDKIM

# On the relay VPS
cd /opt/shared-relay
docker compose restart opendkim

# Verify it started cleanly
docker logs shared-opendkim --tail 10

Note: Remember that NorthRelay uses OpenDKIM on the relay VPSes, not Stalwart's built-in DKIM. Stalwart's DKIM signing is disabled (queue.outbound.sign.enable = false). Always check the relay VPS, not the Sanctuary VPS.

Backup and Recovery Reference

Backup Targets

Component	Method	Frequency	RPO
PostgreSQL	`pg_dump`	Daily	24h
Stalwart RocksDB	Directory snapshot	Daily	24h
Config files	File copy	On change	0
TLS certificates	File copy	On renewal	0
DKIM keys	File copy	On creation	0

Automated Backup Script

#!/bin/bash
# /opt/northrelay/scripts/backup.sh
DATE=$(date +%Y%m%d)
BACKUP_DIR="/opt/backups/${DATE}"
mkdir -p ${BACKUP_DIR}

# PostgreSQL
docker compose exec -T postgres pg_dump -U north_user northrelay \
  | gzip > ${BACKUP_DIR}/postgres.sql.gz

# Stalwart data
tar czf ${BACKUP_DIR}/stalwart-data.tar.gz /opt/northrelay/stalwart-data/

# Config files
tar czf ${BACKUP_DIR}/config.tar.gz /opt/northrelay/.env /opt/northrelay/config/

# DKIM keys (from relay VPS - copy separately)
# scp relay:/opt/shared-relay/dkim/* ${BACKUP_DIR}/dkim/

# Retention: keep 30 days
find /opt/backups -maxdepth 1 -mtime +30 -exec rm -rf {} \;

echo "Backup completed: ${BACKUP_DIR}"

Recovery Time Objectives

Scenario	Expected RTO
Single service restart	< 5 minutes
Full stack restart	< 15 minutes
Database restore from backup	30 min - 2 hours
Full VPS rebuild	2 - 4 hours

Deployment Guide - Initial setup and deployment
Configuration - Environment variables and service configuration
Stalwart Setup - Mail server configuration details
Monitoring - Health checks and alerting

Operational Runbooks

Operational RunbooksCopy Anchor Link

Service Restart OrderCopy Anchor Link

CommandsCopy Anchor Link

Complete Outage RecoveryCopy Anchor Link

Step 1: Verify VPS is AccessibleCopy Anchor Link

Step 2: Check DockerCopy Anchor Link

Step 3: Start ServicesCopy Anchor Link

Step 4: Verify All ServicesCopy Anchor Link

Step 5: Check WireGuard Tunnels (if applicable)Copy Anchor Link

Step 6: Verify Email FlowCopy Anchor Link

Email Delivery Failure DiagnosisCopy Anchor Link

Step 1: Check the Mail QueueCopy Anchor Link

Step 2: Check Stalwart LogsCopy Anchor Link

Step 3: Check DNSCopy Anchor Link

Step 4: Check the relay routeCopy Anchor Link

Step 5: Check BlocklistsCopy Anchor Link

Stalwart Won't StartCopy Anchor Link

Step 1: Check LogsCopy Anchor Link

Step 2: Common CausesCopy Anchor Link

Step 3: Validate ConfigurationCopy Anchor Link

Step 4: Start with Verbose LoggingCopy Anchor Link

WireGuard Tunnel DownCopy Anchor Link

SymptomsCopy Anchor Link

Step 1: Check Tunnel StatusCopy Anchor Link

Step 2: Restart the TunnelCopy Anchor Link

Step 3: Check FirewallCopy Anchor Link

Step 4: Check Remote PeerCopy Anchor Link

Step 5: Check NAT / ISP IssuesCopy Anchor Link

Database IssuesCopy Anchor Link

Connection FailuresCopy Anchor Link

Migration ProblemsCopy Anchor Link

Backup and RestoreCopy Anchor Link

IP Blocked by StalwartCopy Anchor Link

SymptomsCopy Anchor Link

DiagnosisCopy Anchor Link

FixCopy Anchor Link

PreventionCopy Anchor Link

DKIM Signing FailuresCopy Anchor Link

SymptomsCopy Anchor Link

Step 1: Check OpenDKIM on the Relay VPSCopy Anchor Link

Step 2: Verify Key Files ExistCopy Anchor Link

Step 3: Verify KeyTable and SigningTableCopy Anchor Link

Step 4: Verify DNS RecordCopy Anchor Link

Step 5: Restart OpenDKIMCopy Anchor Link

Backup and Recovery ReferenceCopy Anchor Link

Backup TargetsCopy Anchor Link

Automated Backup ScriptCopy Anchor Link

Recovery Time ObjectivesCopy Anchor Link

Related PagesCopy Anchor Link

On this page

Operational Runbooks

Service Restart Order

Commands

Complete Outage Recovery

Step 1: Verify VPS is Accessible

Step 2: Check Docker

Step 3: Start Services

Step 4: Verify All Services

Step 5: Check WireGuard Tunnels (if applicable)

Step 6: Verify Email Flow

Email Delivery Failure Diagnosis

Step 1: Check the Mail Queue

Step 2: Check Stalwart Logs

Step 3: Check DNS

Step 4: Check the relay route

Step 5: Check Blocklists

Stalwart Won't Start

Step 1: Check Logs

Step 2: Common Causes

Step 3: Validate Configuration

Step 4: Start with Verbose Logging

WireGuard Tunnel Down

Symptoms

Step 1: Check Tunnel Status

Step 2: Restart the Tunnel

Step 3: Check Firewall

Step 4: Check Remote Peer

Step 5: Check NAT / ISP Issues

Database Issues

Connection Failures

Migration Problems

Backup and Restore

IP Blocked by Stalwart

Symptoms

Diagnosis

Fix

Prevention

DKIM Signing Failures

Symptoms

Step 1: Check OpenDKIM on the Relay VPS

Step 2: Verify Key Files Exist

Step 3: Verify KeyTable and SigningTable

Step 4: Verify DNS Record

Step 5: Restart OpenDKIM

Backup and Recovery Reference

Backup Targets

Automated Backup Script

Recovery Time Objectives

Related Pages