Morphee Operations Runbook

Quick reference for common operational tasks and incident response.

Last Updated: February 13, 2026

---

Table of Contents

1. Daily Operations
2. Common Tasks
3. Incident Response
4. Troubleshooting
5. Emergency Procedures

---

Daily Operations

Morning Checklist

# 1. Check service status
docker compose -f docker-compose.prod.yml ps

# 2. View recent logs (last 100 lines)
docker compose -f docker-compose.prod.yml logs --tail=100

# 3. Check disk space
df -h

# 4. Check resource usage
docker stats --no-stream

# 5. Verify backups ran
ls -lht /backups/morphee/postgres/ | head -5

Health Check

# Run automated health check
./scripts/monitoring/health-check.sh

# Expected output: "All checks passed (X/X)"

---

Common Tasks

Restart Services

Restart All Services:

docker compose -f docker-compose.prod.yml restart

Restart Specific Service:

# Backend only
docker compose -f docker-compose.prod.yml restart backend

# Database only (WARNING: causes brief downtime)
docker compose -f docker-compose.prod.yml restart postgres

Graceful Restart (zero downtime):

# Scale up to 2 instances
docker compose -f docker-compose.prod.yml up -d --scale backend=2

# Wait 30 seconds for new instance to be healthy
sleep 30

# Scale back down (old instance removed)
docker compose -f docker-compose.prod.yml up -d --scale backend=1

View Logs

All Services:

docker compose -f docker-compose.prod.yml logs -f

Specific Service:

docker compose -f docker-compose.prod.yml logs -f backend

Filter by Time:

# Last 1 hour
docker compose -f docker-compose.prod.yml logs --since 1h backend

# Specific time range
docker compose -f docker-compose.prod.yml logs --since "2026-02-13T10:00:00" --until "2026-02-13T11:00:00" backend

Search Logs:

docker compose -f docker-compose.prod.yml logs backend | grep "ERROR"
docker compose -f docker-compose.prod.yml logs backend | grep -i "user_id=123"

Database Operations

Connect to Database:

docker compose -f docker-compose.prod.yml exec postgres psql -U morphee -d morphee

Run Query:

docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "SELECT COUNT(*) FROM users;"

Vacuum Database (reclaim space):

docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "VACUUM ANALYZE;"

Check Database Size:

docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "
SELECT pg_size_pretty(pg_database_size('morphee')) AS db_size;
"

Check Table Sizes:

docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "
SELECT tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC
LIMIT 10;
"

Backup & Restore

Manual Backup:

# All data
./scripts/backup/backup-all.sh

# Database only
./scripts/backup/backup-database.sh

# Redis only
./scripts/backup/backup-redis.sh

Restore Database:

# From latest backup
./scripts/backup/restore-database.sh

# From specific backup
./scripts/backup/restore-database.sh /backups/morphee/postgres/morphee_20260213_120000.sql.gz

⚠️ WARNING: Restore destroys all existing data!

User Management

List Active Users:

docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "
SELECT id, email, name, created_at
FROM auth.users
ORDER BY created_at DESC
LIMIT 20;
"

Delete User (GDPR):

# Via API (recommended)
curl -X POST https://yourdomain.com/api/auth/delete-account \
  -H "Authorization: Bearer USER_JWT_TOKEN"

# Direct database (last resort)
docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "
DELETE FROM auth.users WHERE id = 'user-uuid-here';
"

Clear Redis Cache

# Flush all keys
docker compose -f docker-compose.prod.yml exec redis redis-cli FLUSHALL

# Flush specific database
docker compose -f docker-compose.prod.yml exec redis redis-cli -n 0 FLUSHDB

---

Incident Response

Service Down

1. Check Service Status:

docker compose -f docker-compose.prod.yml ps

2. View Recent Logs:

docker compose -f docker-compose.prod.yml logs --tail=200 [service-name]

3. Restart Service:

docker compose -f docker-compose.prod.yml restart [service-name]

4. If Restart Fails:

# Stop service
docker compose -f docker-compose.prod.yml stop [service-name]

# Remove container
docker compose -f docker-compose.prod.yml rm -f [service-name]

# Start fresh
docker compose -f docker-compose.prod.yml up -d [service-name]

5. Check Health:

./scripts/monitoring/health-check.sh

Database Connection Failures

Symptoms: Backend logs show "connection refused" or "too many connections"

1. Check PostgreSQL Status:

docker compose -f docker-compose.prod.yml ps postgres
docker compose -f docker-compose.prod.yml logs --tail=100 postgres

2. Check Active Connections:

docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "
SELECT COUNT(*) FROM pg_stat_activity;
"

3. Kill Idle Connections:

docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle' AND state_change < NOW() - INTERVAL '1 hour';
"

4. Restart PostgreSQL (if needed):

docker compose -f docker-compose.prod.yml restart postgres

Out of Disk Space

Symptoms: Services fail to start, logs show "no space left on device"

1. Check Disk Usage:

df -h
du -sh /var/lib/docker
du -sh /backups

2. Clean Docker:

# Remove unused containers, images, networks
docker system prune -a -f

# Remove dangling volumes
docker volume prune -f

3. Clean Logs:

# Truncate Docker logs
truncate -s 0 /var/lib/docker/containers/*/*-json.log

# Clean old backups (keep last 7 days)
find /backups/morphee -name "*.gz" -mtime +7 -delete

4. Clean Application Data:

# Clean old logs
find ./logs -name "*.log" -mtime +30 -delete

# Clean temporary files
rm -rf ./data/files/tmp/*

Memory Issues

Symptoms: Services slow, OOM killer, high swap usage

1. Check Memory Usage:

free -h
docker stats --no-stream

2. Identify Memory Hogs:

docker stats --no-stream --format "table {{.Container}}\t{{.MemUsage}}\t{{.MemPerc}}" | sort -k 3 -h -r

3. Restart High-Memory Service:

docker compose -f docker-compose.prod.yml restart [service-with-high-memory]

4. Increase Swap (if needed):

# Create 4GB swap file
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

High CPU Usage

1. Check CPU Usage:

docker stats --no-stream
top -bn1 | head -20

2. Identify CPU Hogs:

docker compose -f docker-compose.prod.yml exec backend ps aux --sort=-%cpu | head -10

3. Check for Runaway Processes:

# Kill runaway process
docker compose -f docker-compose.prod.yml exec backend kill -9 [PID]

SSL Certificate Expiring

1. Check Certificate Expiry:

echo | openssl s_client -servername yourdomain.com -connect yourdomain.com:443 2>/dev/null | openssl x509 -noout -dates

2. Renew Let's Encrypt Certificate:

sudo certbot renew

3. Reload Nginx:

docker compose -f docker-compose.prod.yml restart nginx

---

Troubleshooting

"Connection Refused" Errors

Cause: Service not listening on expected port

Solution:

# Check port bindings
docker compose -f docker-compose.prod.yml ps

# Check if port is listening
netstat -tlnp | grep :8000

# Check firewall
sudo ufw status

"Unhealthy" Service Status

Cause: Health check failing

Solution:

# Check health check logs
docker inspect morphee-backend --format='{{json .State.Health}}' | jq

# Check service logs
docker compose -f docker-compose.prod.yml logs --tail=100 backend

WebSocket Connection Failures

Cause: Nginx not configured for WebSocket upgrade

Solution:

# Verify nginx config
docker compose -f docker-compose.prod.yml exec nginx nginx -t

# Check logs
docker compose -f docker-compose.prod.yml logs nginx | grep "upgrade"

# Restart nginx
docker compose -f docker-compose.prod.yml restart nginx

Slow Performance

Diagnostic Steps:

1. Check System Resources:

   top
   df -h
   free -h

2. Check Database Performance:

   docker compose -f docker-compose.prod.yml exec -T postgres psql -U morphee -d morphee -c "
   SELECT query, calls, mean_exec_time
   FROM pg_stat_statements
   ORDER BY mean_exec_time DESC
   LIMIT 10;
   "

3. Check Redis Performance:

   docker compose -f docker-compose.prod.yml exec redis redis-cli INFO stats

4. Check Network Latency:

   ping -c 10 yourdomain.com
   curl -o /dev/null -s -w "Time: %{time_total}s\n" https://yourdomain.com/api/health

---

Emergency Procedures

Full Service Outage

1. Stop All Services:

docker compose -f docker-compose.prod.yml down

2. Check System Resources:

df -h
free -h
docker ps -a

3. Clean and Restart:

# Clean stopped containers
docker system prune -f

# Start services
docker compose -f docker-compose.prod.yml up -d

4. Monitor Recovery:

./scripts/monitoring/health-check.sh
docker compose -f docker-compose.prod.yml logs -f

Data Breach (GDPR)

Within First Hour:

1. Isolate Systems:

   # Take services offline
   docker compose -f docker-compose.prod.yml down

2. Preserve Logs:

   # Copy all logs
   tar czf breach-logs-$(date +%Y%m%d-%H%M%S).tar.gz /var/lib/docker/containers/*/\*-json.log
   cp -r ./logs ./logs-backup-$(date +%Y%m%d-%H%M%S)

3. Notify DPO:

   # Send email
   echo "Data breach detected at $(date). All services offline. Logs preserved." | mail -s "URGENT: Data Breach" dpo@morphee.app

Within 72 Hours:

1. Investigate Root Cause: Check logs, access logs, database audit trail

2. Notify Affected Users: Via email, in-app notification

3. File Breach Report: Follow docs/legal/BREACH_RESPONSE_PLAN.md

4. Implement Fixes: Patch vulnerabilities, rotate secrets

Rollback Deployment

1. Stop Current Version:

docker compose -f docker-compose.prod.yml down

2. Checkout Previous Version:

git log --oneline | head -10  # Find previous good commit
git checkout [previous-commit-hash]

3. Restore Previous Database (if needed):

./scripts/backup/restore-database.sh [previous-backup-file]

4. Deploy Previous Version:

./scripts/deploy.sh

5. Verify:

./scripts/monitoring/health-check.sh

---

On-Call Contacts

Role	Contact	Escalation
Primary On-Call	ops@morphee.app	15 minutes
Secondary On-Call	devops@morphee.app	30 minutes
Engineering Lead	eng@morphee.app	1 hour
DPO (Data Breach)	dpo@morphee.app	Immediate
Legal (Compliance)	legal@morphee.app	Immediate

---

Useful Commands Reference

Docker Compose

# Start services
docker compose -f docker-compose.prod.yml up -d

# Stop services
docker compose -f docker-compose.prod.yml down

# Restart services
docker compose -f docker-compose.prod.yml restart

# View logs
docker compose -f docker-compose.prod.yml logs -f [service]

# Check status
docker compose -f docker-compose.prod.yml ps

# Execute command in container
docker compose -f docker-compose.prod.yml exec [service] [command]

# Build and restart
docker compose -f docker-compose.prod.yml up -d --build

PostgreSQL

# Connect to database
psql -U morphee -d morphee

# List tables
\dt

# Describe table
\d [table-name]

# Run query from file
\i /path/to/query.sql

# Copy data to file
\copy (SELECT * FROM users) TO '/tmp/users.csv' CSV HEADER;

Monitoring

# Health check
./scripts/monitoring/health-check.sh

# Backup
./scripts/backup/backup-all.sh

# Resource usage
docker stats --no-stream

---

For More Help:

• Deployment Guide
• Troubleshooting Guide
• Backup Scripts