Operators
Monitoring & Operations
Health checks, metrics, log analysis, and alerting for NorthRelay infrastructure
Monitoring & Operations
Monitor NorthRelay services, analyze logs, and configure alerts for production reliability.
Service Health Checks
Run these commands to verify each service is healthy:
All Services at a Glance
# Check all Docker containers
docker compose ps
# Expected: all services show "Up" with healthy statusIndividual Service Checks
# PostgreSQL
docker compose exec postgres pg_isready -U north_user
# Expected: "accepting connections"
# Redis
docker compose exec redis redis-cli -a ${REDIS_PASSWORD} ping
# Expected: "PONG"
# Stalwart (Relay)
curl -s -u admin:TOKEN http://localhost:8080/api/healthz
# Expected: 200 OK
# Next.js App
curl -s http://localhost:3000/api/health | jq .status
# Expected: "ok"
# WireGuard Tunnel (Sanctuary → sending relay)
wg show wg0
# Expected: "latest handshake" within last 2 minutes
# Nginx
curl -s -o /dev/null -w "%{http_code}" https://yourdomain.com
# Expected: 200Key Metrics and Thresholds
Monitor these metrics and alert when thresholds are exceeded:
| Metric | Warning | Critical | How to Check |
|---|---|---|---|
| Mail queue size | > 100 | > 500 | curl -u admin:TOKEN http://localhost:8080/api/queue/messages | jq length |
| Bounce rate | > 5% | > 10% | Dashboard reputation page or reputation_history table |
| Delivery time (p95) | > 30s | > 60s | Stalwart queue logs |
| SMTP connections | > 500 concurrent | > 800 | ss -tnp | grep :25 | wc -l |
| Disk usage | > 80% | > 90% | df -h / |
| PostgreSQL connections | > 80 | > 95 | SELECT count(*) FROM pg_stat_activity; |
| PostgreSQL query time (p95) | > 500ms | > 2s | pg_stat_statements extension |
| Redis memory | > 80% maxmemory | > 90% | redis-cli info memory |
| WireGuard last handshake | > 3 min | > 5 min | wg show wg0 latest-handshakes |
| CPU usage | > 70% sustained | > 90% | top or monitoring agent |
| Memory usage | > 80% | > 90% | free -h |
Log Locations and Access
All services log to Docker's logging driver. Access logs via docker compose logs:
# App logs (API errors, auth failures, business logic)
docker compose logs app --tail 200
# Stalwart logs (SMTP transactions, queue events, delivery)
docker compose logs stalwart --tail 200
# PostgreSQL logs (slow queries, connection issues)
docker compose logs postgres --tail 200
# Nginx logs (HTTP requests, TLS errors)
docker compose logs nginx --tail 200
# Cron logs (scheduled job execution)
docker compose logs cron --tail 50
# Follow all logs in real time
docker compose logs -f
# Filter by time range
docker compose logs --since "2026-03-19T10:00:00" --until "2026-03-19T11:00:00" stalwartLog Analysis Patterns
Delivery Events
# Successful deliveries
docker compose logs stalwart 2>&1 | grep "delivered"
# Failed deliveries
docker compose logs stalwart 2>&1 | grep -E "bounced|rejected|failed"
# Deferred messages (temporary failures)
docker compose logs stalwart 2>&1 | grep "deferred"DKIM Signing
Since DKIM signing happens on the relay VPSes (not Stalwart), check OpenDKIM logs:
# On relay VPS
docker logs shared-opendkim 2>&1 | grep -E "DKIM|signing"
docker logs sovereign-opendkim 2>&1 | grep -E "DKIM|signing"
# Look for signing failures
docker logs shared-opendkim 2>&1 | grep -i "error\|fail\|no key"Delivery canary
Every hour the cron worker calls /api/cron/email-canary, which sends a real message through the full path (app → Stalwart → sending relay → internet → MX → mailbox), reads it back over JMAP and records latency plus SPF/DKIM/DMARC results.
- Where to see it: Admin → System emails (
/dashboard/admin/system-emails) shows the latest run (PASS/FAIL, latency, authentication results). - Alerts:
email_canary_failed(Critical) is raised when a run fails;email_canary_stale(High) when no run has completed recently. Both appear in the admin alert list. - Run one now:
POST /api/cron/email-canarywith the cron bearer token, then check the admin page a few minutes later.
Authentication Failures
# Failed login attempts
docker compose logs app 2>&1 | grep -i "auth.*fail\|invalid.*credentials"
# Stalwart auth failures (may trigger IP blocking)
docker compose logs stalwart 2>&1 | grep -i "auth.*fail"Rate Limiting
# Rate limit violations
docker compose logs app 2>&1 | grep "rate.limit\|451 4.7.1"
# Check per-user quotas
docker compose exec postgres psql -U north_user -d northrelay \
-c "SELECT email, \"quotaLimit\", \"quotaUsed\" FROM users WHERE \"quotaUsed\" > \"quotaLimit\" * 0.8;"Alert Recommendations
Set up alerts for these critical conditions:
Tier 1: Page Immediately
- Health check endpoint returns non-200 for > 2 minutes
- Mail queue size > 500 and growing
- Disk usage > 90%
- PostgreSQL connection count > 95
- WireGuard handshake > 5 minutes old
Tier 2: Alert Within 15 Minutes
- Bounce rate > 10% over 1 hour
- Delivery time p95 > 60 seconds
- App error rate > 5% of requests
- Redis memory > 90% of maxmemory
Tier 3: Review Daily
- Bounce rate > 5% daily average
- Queue size consistently > 50
- Slow queries > 500ms appearing in PostgreSQL logs
- Disk usage > 80%
Monitoring Tools
NorthRelay does not include a built-in monitoring stack. Recommended external tools:
| Tool | Purpose | Integration |
|---|---|---|
| UptimeRobot / Pingdom | Uptime monitoring | Poll /api/health |
| Grafana + Prometheus | Metrics dashboards | Docker metrics exporter |
| Loki | Log aggregation | Docker logging driver |
| PagerDuty / OpsGenie | Alert routing | Webhook integrations |
Performance Monitoring
Database Performance
# Active connections and their state
docker compose exec postgres psql -U north_user -d northrelay \
-c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"
# Long-running queries (> 5 seconds)
docker compose exec postgres psql -U north_user -d northrelay \
-c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state != 'idle' AND now() - pg_stat_activity.query_start > interval '5 seconds';"
# Database size
docker compose exec postgres psql -U north_user -d northrelay \
-c "SELECT pg_size_pretty(pg_database_size('northrelay'));"
# Table sizes (top 10)
docker compose exec postgres psql -U north_user -d northrelay \
-c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;"Mail Queue Monitoring
# Queue size
curl -s -u admin:TOKEN http://localhost:8080/api/queue/messages | jq length
# Queue details (message IDs, recipients, status)
curl -s -u admin:TOKEN http://localhost:8080/api/queue/messages | jq '.[] | {id, from, to, status}'
# Flush stuck messages
curl -s -u admin:TOKEN -X DELETE http://localhost:8080/api/queue/messages/MESSAGE_IDResource Usage
# Docker container resource usage
docker stats --no-stream
# Disk usage by Docker volumes
docker system df -v
# Network connections per service
ss -tnp | grep -E ':25|:587|:443|:5432|:6379' | awk '{print $4}' | sort | uniq -c | sort -rnWireGuard Tunnel Health
Sanctuary reaches the sending relay over a single WireGuard tunnel (sovereign.north-pipe.ca resolves to 10.0.2.2 via /etc/hosts):
# Check tunnel status
wg show wg0
# Key fields to check:
# - "latest handshake": Should be within last 2 minutes
# - "transfer": Should show non-zero rx/tx bytes
# - "allowed ips": Should match the relay subnet
# Ping the sending relay through the tunnel
ping -c 3 10.0.2.2 # Sovereign relay
# Check routing table
ip route show | grep wgIf the tunnel is down, see Runbooks - WireGuard Tunnel Down.
Next Steps
- Runbooks - Step-by-step procedures for incident response
- Configuration - Adjust settings based on monitoring findings
- Stalwart Setup - Mail server tuning