incident-response – Security skill for AI agents

Incident Response

Handle security incidents effectively with structured response procedures.

Prerequisites

• IR team roster with on-call rotation and escalation paths
• Secure communication channel (separate from production systems)
• Forensic workstation with analysis tools installed
• Evidence storage with chain-of-custody controls
• Legal counsel contact information
• Pre-authorized incident response actions documented

Incident Response Phases

phases:
  1_preparation:
    - IR team roster and 24/7 contact info
    - Tools and privileged access ready
    - Playbooks documented and tested
    - Evidence collection kit prepared
    - Communication templates drafted

  2_detection:
    - Alert triage and validation
    - Initial assessment and scoping
    - Severity classification
    - Incident ticket creation

  3_containment:
    - Short-term containment (stop bleeding)
    - Evidence preservation (before changes)
    - System isolation (network/host level)
    - Credential rotation if needed

  4_eradication:
    - Root cause analysis
    - Remove threat actor access
    - Patch exploited vulnerabilities
    - Clean compromised systems

  5_recovery:
    - System restoration from clean backups
    - Enhanced monitoring deployment
    - Phased return to production
    - Business continuity verification

  6_lessons_learned:
    - Post-incident review (within 72 hours)
    - Timeline reconstruction
    - Documentation update
    - Process and detection improvements

Severity Classification

The targets below (and the 72-hour retrospective target) are examples to agree with

the organization, not notification deadlines or guarantees. Classify phishing by

actual access/data/business impact, not automatically Medium.

| Level | Impact | Response Time | Examples |

|-------|--------|---------------|----------|

| Critical (P1) | Active data breach, full outage, ransomware | Immediate (< 15 min) | Data exfiltration in progress, ransomware spreading |

| High (P2) | Service degraded, potential breach | < 1 hour | Unauthorized admin access, malware detected |

| Medium (P3) | Limited impact, contained | < 4 hours | Phishing compromise (single user), policy violation |

| Low (P4) | Minimal impact | Next business day | Failed brute force, blocked scanning activity |

Evidence collection and containment

Live collection changes a host and may run attacker-controlled binaries. Use a

trusted forensic workstation/toolkit, approved scope and a private encrypted evidence

volume; record collector identity, UTC time/clock offset, source, tool/version,

commands, errors and custody transfers. Hashes detect later byte changes but do not

prove an honest source, completeness or chain of custody. Active containment may take

priority over preservation to prevent ongoing harm; record that decision.

Bounded Linux collection example

On an approved Linux test host with GNU tools, a minimal metadata collection is:

set -euo pipefail
umask 077
# EVIDENCE_PARENT must already be an approved private evidence volume.
: "${EVIDENCE_PARENT:?Set approved evidence volume}"
EVIDENCE_DIR=$(mktemp -d "$EVIDENCE_PARENT/collection.XXXXXXXX")
date -u +%Y-%m-%dT%H:%M:%SZ > "$EVIDENCE_DIR/collection-time.txt"
uname -a > "$EVIDENCE_DIR/uname.txt"
ps -eo pid,ppid,user,comm > "$EVIDENCE_DIR/process-names.txt"
# Write the final log before hashing it; never append to hashed evidence afterward.
for name in collection-time.txt uname.txt process-names.txt; do
  (cd "$EVIDENCE_DIR" && sha256sum "$name")
done > "$EVIDENCE_DIR/checksums.sha256"
(cd "$EVIDENCE_DIR" && sha256sum -c checksums.sha256)

This omits memory, full command lines, credential databases, logs, containers and

remote clusters deliberately: each requires explicit data scope, retention/access

controls and an independently tracked outcome. A missing tool/error is partial

collection, not a clean host. Avoid unlimited whole-filesystem walks or collecting

/etc/shadow by default. Bound disk/time and do not hide permission failures.

Memory acquisition

Use a reviewed kernel-compatible acquisition tool only under the IR plan: loading

LiME modifies kernel state and may crash a system. /proc/kcore is not always

available and copying its enormous virtual address space is not a reliable generic

memory dump. Record hashes, kernel/tool version, errors and approved size/time bounds;

no successful memory capture is implied by a printed message.

AWS evidence and isolation runbook

1. Confirm account, region, instance and every attached volume/ENI from a read-only

inventory. Preserve that inventory and request IDs in private evidence storage.

2. Snapshot the approved volume set consistently (multi-volume/application consistency

needs planning), record IDs and wait for completed/error states. Creation is a

billable write; an accepted request is not a completed forensic image.

3. Export the relevant log data and query/time coverage. describe-flow-logs exports

configuration, not packets or historical flow records. CloudTrail event history

lookup is not complete data-event or organization-wide incident coverage.

4. Separately approve containment and preserve original SG/ENI configurations plus

rollback/recovery path. A newly created SG has default outbound allow rules:

attaching it alone does not isolate egress. Security groups are stateful and

existing connections may persist; use the organization’s tested quarantine controls

across every ENI/route/connection path, preserving authorized evidence access.

5. Verify observed isolation from controlled test points and monitor before declaring

containment. Store evidence in an existing restricted encrypted retention-controlled

destination; never derive arbitrary public bucket names or reuse predictable /tmp

evidence files.

Forensics Commands Reference

# --- Disk forensics ---
# Create forensic image of a disk
dd if=/dev/sda of=/evidence/disk.img bs=4M status=progress
sha256sum /evidence/disk.img > /evidence/disk.img.sha256

# Mount forensic image read-only
# Mount a working copy with filesystem-specific no-journal-replay options
# and a write blocker where applicable; ro alone may permit journal recovery.
# Do not mount an untrusted image on the production host.

# Find recently modified files
find /mnt/forensic -type f -mtime -3 -ls | sort -k11

# Find files by owner
find /mnt/forensic -user www-data -type f -newer /tmp/reference-time -ls

# --- Log analysis ---
# Search auth logs for brute force
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -20

# Search for privilege escalation
grep -E "(sudo|su\[)" /var/log/auth.log | grep -v "session opened"

# Search web logs for attack patterns
grep -iE "(union.*select|<script|\.\.\/|%00)" /var/log/nginx/access.log

# Timeline analysis with find
find / -newermt "2025-01-15 00:00" ! -newermt "2025-01-16 00:00" -ls 2>/dev/null | sort -k9

# --- Network forensics ---
# Capture network traffic
tcpdump -i eth0 -w /evidence/capture.pcap -c 100000

# Analyze pcap for suspicious connections
tcpdump -r /evidence/capture.pcap -nn 'dst port 4444 or dst port 8888 or dst port 1337'

# Check for DNS tunneling
tcpdump -r /evidence/capture.pcap -nn 'udp port 53' | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20

# --- Malware analysis ---
# Check file for known malware hashes
sha256sum suspicious_file
# Query only a hash if disclosure is approved; never upload confidential samples automatically.

# Strings analysis
strings suspicious_file | grep -iE "(http|ftp|ssh|password|key|token)"

# Check for packed/obfuscated binaries
file suspicious_file
readelf -h suspicious_file 2>/dev/null

Communication Templates

Initial Notification (Internal)

## Security Incident Notification

**Incident ID:** INC-YYYY-NNNN
**Severity:** [Critical/High/Medium/Low]
**Status:** Active - Investigating
**Time Detected:** YYYY-MM-DD HH:MM UTC
**Reported By:** [Name/System]

### Summary
[1-2 sentence description of what was detected]

### Impact Assessment
- **Systems affected:** [list]
- **Data at risk:** [type and scope]
- **Users impacted:** [count/scope]
- **Business impact:** [description]

### Current Actions
- [ ] Evidence preservation in progress
- [ ] Containment measures being applied
- [ ] IR team assembled

### Next Update
Expected at: YYYY-MM-DD HH:MM UTC

### Incident Commander
[Name] - [Contact info]

Stakeholder Update

## Incident Update - INC-YYYY-NNNN

**Update #:** N
**Time:** YYYY-MM-DD HH:MM UTC
**Severity:** [unchanged/upgraded/downgraded]
**Status:** [Investigating/Contained/Eradicating/Recovering/Resolved]

### Progress Since Last Update
- [Bullet points of actions taken]

### Current Understanding
- **Root cause:** [Known/Under investigation]
- **Scope:** [Expanded/Unchanged/Reduced]
- **Threat actor:** [If applicable]

### Active Containment Measures
- 

… (full content is in the download package)