runbook-creation – Documents & Office skill for AI agents

Runbook Creation

Create effective operational runbooks, standard operating procedures, and

troubleshooting guides that any on-call engineer can follow under pressure.

Runbook Template — Full Structure

# Runbook: [Service / Process Name]

**Owner:** [Team or individual]
**Last Reviewed:** YYYY-MM-DD
**Version:** X.Y
**Severity if unavailable:** SEV[1-4]

---

## Overview

Brief description of the service, why this runbook exists, and when to
use it.

## Prerequisites

- [ ] Required access / IAM role: [details]
- [ ] Tools installed: [kubectl, aws-cli, psql, etc.]
- [ ] VPN connected to [environment]
- [ ] Communication channel open: [Slack #channel]

## Procedure

### Step 1 — [Action Name]

[Explanation of what this step does and why.]

command here

**Expected output:** [describe what success looks like]

### Step 2 — [Action Name]

command here

**Expected output:** [description]

*(Continue with numbered steps...)*

## Verification

How to confirm the procedure succeeded:

- [ ] [Check 1 — e.g., health endpoint returns 200]
- [ ] [Check 2 — e.g., no errors in logs for 5 minutes]
- [ ] [Check 3 — e.g., metrics return to baseline]

## Rollback

If the procedure fails or causes unexpected issues:

### Rollback Step 1

rollback command

### Rollback Step 2

rollback command

## Troubleshooting

| Symptom | Likely Cause | Resolution |
|---------|-------------|------------|
| [symptom 1] | [cause] | [fix] |
| [symptom 2] | [cause] | [fix] |

## Escalation

If unresolved after [X] minutes:
- **Primary:** @[team-lead] — [phone/Slack]
- **Secondary:** @[manager] — [phone/Slack]

## Related Runbooks

- [Link to related runbook 1]
- [Link to related runbook 2]

## Change Log

| Date | Author | Change |
|------|--------|--------|
| YYYY-MM-DD | [Name] | Initial version |

Overview

Failover the primary PostgreSQL instance to the synchronous replica when

the primary is unreachable or degraded. This runbook covers both planned

(maintenance) and unplanned (emergency) failover.

Prerequisites

• [ ] DBA or SRE-level access to primary and replica hosts
• [ ] psql client installed (v14+)
• [ ] VPN connected to production network
• [ ] Slack channel #db-ops open
• [ ] Confirm replica is in sync: replication lag < 1 MB

Procedure

Step 1 — Verify Replica Health

psql -h replica.db.internal -U dba -d postgres -c \
  "SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn();"

Expected output: pg_is_in_recovery = t, LSN advancing.

Step 2 — Stop Application Writes

kubectl scale deployment api-server --replicas=0 -n production
kubectl scale deployment worker --replicas=0 -n production

Expected output: Deployments scaled to 0 pods.

Step 3 — Fence the Old Primary and Confirm Write Quiescence

Before promotion, use the documented failover manager/fencing mechanism to ensure the old primary cannot accept writes, including after network recovery. Scaling two deployments and observing zero active queries is not proof of fencing or absence of other writers. If the primary is unreachable and fencing cannot be confirmed, stop and escalate; do not risk two primaries. Capture original replica counts/configuration for restoration.

psql -h primary.db.internal -U dba -d postgres -c \
  "SELECT count(*) FROM pg_stat_activity WHERE state = 'active' AND query !~ 'pg_stat';"

Expected output: Count = 0 (no active queries).

Step 4 — Promote Replica

psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_promote();"

Wait up to 30 seconds, then confirm:

psql -h replica.db.internal -U dba -d postgres -c "SELECT pg_is_in_recovery();"

Expected output: pg_is_in_recovery = f (no longer a replica).

Step 5 — Update DNS

aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "db.internal.example.com",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [{"Value": "replica.db.internal"}]
      }
    }]
  }'

Step 6 — Restart Application

kubectl scale deployment api-server --replicas=6 -n production
kubectl scale deployment worker --replicas=4 -n production

Verification

• [ ] psql -h db.internal.example.com -c "SELECT 1;" returns successfully
• [ ] Application logs show successful DB connections (no errors for 5 min)
• [ ] Transaction throughput returns to baseline on Grafana dashboard
• [ ] No replication-lag alerts firing

Rollback

Promotion is not undone by restoring a backup. Establish the authoritative primary, fence other writers and obtain a DBA-approved data-recovery/rejoin plan. The following restores a separate RDS instance and may lose writes since the snapshot; it is not an automatic rollback for the self-managed PostgreSQL example:

# Restore latest automated snapshot (RDS example)
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier prod-db-restored \
  --db-snapshot-identifier prod-db-latest-snapshot

Escalation

If unresolved after 15 minutes:

• Primary: @dba-lead — +1-555-0101
• Secondary: @platform-oncall — +1-555-0102

The shell fragments assume Linux/GNU date, curl, jq/openssl and authorized cluster/cloud access. Use bounded timeouts for network probes and protect collected logs/queries; a fresh temporary directory does not make raw incident data safe to publish.

## Automation Scripts for Common Operations

### Service Health Check

#!/usr/bin/env bash

health-check.sh — Check health of critical services

set -euo pipefail

SERVICES=(

"https://api.example.com/healthz"

"https://app.example.com/healthz"

"https://admin.example.com/healthz"

)

EXIT_CODE=0

for url in "${SERVICES[@]}"; do

if ! HTTP_CODE=$(curl -so /dev/null -w '%{http_code}' --max-time 5 "$url" 2>/dev/null); then

HTTP_CODE=000

fi

if [ "$HTTP_CODE" -eq 200 ]; then

printf " OK %s\n" "$url"

else

printf " FAIL %s (HTTP %s)\n" "$url" "$HTTP_CODE"

EXIT_CODE=1

fi

done

exit $EXIT_CODE

### Log Collection for Incident Investigation

#!/usr/bin/env bash

collect-logs.sh — Gather logs from multiple sources for incident review

set -euo pipefail

INCIDENT_ID="${1:?Usage: collect-logs.sh <incident-id>}"

[[ "$INCIDENT_ID" =~ ^[A-Za-z0-9_-]+$ ]] || exit 2

umask 077

OUTDIR=$(mktemp -d "${TMPDIR:-/tmp}/incident-${INCIDENT_ID}.XXXXXX")

echo "Collecting logs for incident $INCIDENT_ID..."

Kubernetes pod logs (last 30 min)

kubectl logs -l app=api-server -n production --since=30m \

> "${OUTDIR}/api-server-pods.log" 2>&1

CloudWatch Logs (last 30 min)

aws logs filter-log-events \

--log-group-name /ecs/production/api \

--start-time "$(date -d '30 minutes ago' +%s)000" \

--output text > "${OUTDIR}/cloudwatch-api.log" 2>&1

Database slow query log

psql -h db.internal -U dba -d postgres -c \

"SELECT * FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start;" \

> "${OUTDIR}/db-active-queries.log" 2>&1

System resource snapshot

kubectl top pods -n production > "${OUTDIR}/pod-resources.log" 2>&1

echo "Logs saved to $OUTDIR"

tar czf "${OUTDIR}.tar.gz" -C "$(dirname "$OUTDIR")" "$(basename "$OUTDIR")"

echo "Archive: ${OUTDIR}.tar.gz"

### Certificate Expiry Check

#!/usr/bin/env bash

cert-check.sh — Warn if TLS certificates expire within 30 days

set -euo pipefail

DOMAINS=(

"api.example.com"

"app.example.com"

"admin.example.com"

)

WARN_DAYS=30

TODAY=$(date +%s)

EXIT_CODE=0

for domain in "${DOMAINS[@]}"; do

if ! EXPIRY=$(echo | openssl s_client -servername "$domain" -connect "${domain}:443" 2>/dev/null \

| openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2); then

printf " FAIL %s certificate unavailable\n" "$domain"

EXIT_CODE=1

continue

fi

EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s 2>/dev/null || echo 0)

DAYS_LEFT=$(( (EXPIRY_EPOCH - TODAY) / 86400 ))

if [ "$DAYS_LEFT" -lt "$WARN_DAYS" ]; then

printf " WARN %s expires in %d days (%s)\n" "$domain" "$DAYS_LEFT" "$EXPIRY"

EXIT_CODE=1

else

printf " OK %s — %d days remaining\n" "$domain" "$DAYS_LEFT"

fi

done

exit $EXIT_CODE

### Disk Space Cleanup

#!/usr/bin/env bash

disk-cleanup.sh — Free disk space on a host

set -euo pipefail

echo "=== Disk Usage Before ==="

df -h /

Inventory first; no deletion in this diagnostic example.

journalctl --disk-usage

docker system df

find /var/log -type f -name "*.gz" -mtime +7 -print

Use service-specific approved retention and reviewed exact paths for cleanup.

Do not prune volumes or globally delete /tmp; backups/log evidence may be needed.

echo "=== Disk Usage After ==="

df -h /

## Runbook Review Checklist

Use this checklist every time a runbook is created or updated.

content_review:

• [ ] Title clearly identifies the s

… (full content is in the download package)