incident-management – skill Viết lách & Nội dung cho agent AI

Incident Management

Implement effective incident management processes including severity definitions, escalation matrices, war room procedures, and blameless post-mortem templates.

When to Use

• Establishing incident management processes for production systems
• Defining severity levels and escalation procedures
• Running war rooms and coordinating incident response
• Conducting blameless post-incident reviews
• Building on-call schedules and notification workflows
• Meeting compliance requirements for incident response (SOC 2, HIPAA, PCI DSS)

Severity/timing/staffing numbers are example policy, not universal compliance requirements. Classify security/privacy impact independently of availability and invoke the authorized security/legal response path when needed. Do not wait for a service-wide outage or a timer before escalating suspected compromise.

Creating channels, paging people, publishing status/customer notices and changing production require the user's authorized incident scope and designated owner. Drafts are not sent notifications. Preserve evidence and restricted/redacted logs before destructive recovery when feasible; containment and safety may take priority over restoration. State confirmed impact separately from hypotheses, use UTC plus dates, and verify both service recovery and security containment before closure.

Severity Levels

severity_definitions:
  SEV1_critical:
    impact: "Complete service outage or data breach affecting all/most customers"
    examples:
      - Production site completely down
      - Data breach confirmed or suspected
      - Complete loss of a critical business function
      - Security incident with active exploitation
    response_time: "Immediate (within 5 minutes)"
    update_frequency: "Every 15-30 minutes"
    who_is_paged: "On-call engineer, engineering manager, incident commander, executive on-call"
    communication: "Status page update, customer email, executive notification"
    resolution_target: "< 1 hour to mitigate"

  SEV2_major:
    impact: "Major feature broken or severe degradation affecting many customers"
    examples:
      - Key feature completely non-functional
      - Significant performance degradation (>5x latency)
      - Data processing pipeline completely stalled
      - Partial outage affecting a region or segment
    response_time: "Within 15 minutes"
    update_frequency: "Every 30-60 minutes"
    who_is_paged: "On-call engineer, engineering manager"
    communication: "Status page update if customer-facing"
    resolution_target: "< 4 hours to mitigate"

  SEV3_moderate:
    impact: "Minor feature impaired or degradation affecting some customers"
    examples:
      - Non-critical feature broken
      - Moderate performance degradation
      - Elevated error rate (below threshold for SEV2)
      - Single-customer impact on non-critical function
    response_time: "Within 1 hour during business hours"
    update_frequency: "Every 2-4 hours"
    who_is_paged: "On-call engineer"
    communication: "Internal only unless customer inquires"
    resolution_target: "< 1 business day"

  SEV4_low:
    impact: "Cosmetic issue, minor inconvenience, or non-customer-facing problem"
    examples:
      - UI cosmetic bug
      - Non-critical monitoring gap
      - Internal tool degradation
      - Documentation inaccuracy in production
    response_time: "Next business day"
    update_frequency: "As needed"
    who_is_paged: "None (ticket created)"
    communication: "None"
    resolution_target: "Within sprint planning cycle"

Escalation Matrix

escalation_matrix:
  tier_1_on_call_engineer:
    reached_via: "PagerDuty / OpsGenie alert"
    responsibilities:
      - Acknowledge alert within 5 minutes
      - Assess severity and impact
      - Begin troubleshooting
      - Escalate to Tier 2 if unable to resolve within 30 minutes (SEV1/2)
    escalation_trigger: "Cannot resolve, needs additional expertise, or severity upgrade"

  tier_2_team_lead_or_sme:
    reached_via: "PagerDuty escalation or direct page"
    responsibilities:
      - Provide subject matter expertise
      - Assist with diagnosis and resolution
      - Coordinate with other teams if cross-service issue
      - Escalate to Tier 3 if broader coordination needed
    escalation_trigger: "Multi-service issue, needs executive decision, or customer-facing SEV1"

  tier_3_engineering_management:
    reached_via: "PagerDuty escalation or direct call"
    responsibilities:
      - Assign incident commander (if not already)
      - Allocate additional resources
      - Make business decisions (feature disable, rollback, etc.)
      - Coordinate external communication
    escalation_trigger: "Business impact decision, extended outage, or PR/legal concern"

  tier_4_executive:
    reached_via: "Direct phone call"
    responsibilities:
      - Authorize extraordinary measures
      - Manage board/investor communication
      - Approve public statements
      - Engage external resources (vendors, consultants)
    escalation_trigger: "Major breach, extended SEV1, regulatory or legal implication"

  time_based_escalation:
    sev1:
      "15 min no ack": "Re-page on-call + backup on-call"
      "30 min unresolved": "Page team lead"
      "1 hour unresolved": "Page engineering manager + executive on-call"
      "2 hours unresolved": "All-hands engineering involvement"
    sev2:
      "30 min no ack": "Re-page on-call + backup on-call"
      "1 hour unresolved": "Page team lead"
      "4 hours unresolved": "Page engineering manager"

War Room Procedures

war_room:
  activation: "Automatically for SEV1, on-demand for SEV2"

  setup:
    communication_channel:
      primary: "Dedicated Slack channel (#incident-YYYY-MM-DD-brief-name)"
      voice: "Zoom/Google Meet bridge (persistent link)"
      backup: "Phone conference bridge"
    channel_rules:
      - "Only incident-related communication in the channel"
      - "Use threads for side discussions"
      - "Prefix messages with role (IC:, COMMS:, ENG:)"

  roles:
    incident_commander:
      responsibilities:
        - Own the incident from declaration to resolution
        - Coordinate all response activities
        - Make decisions on response actions
        - Assign tasks to responders
        - Determine when incident is resolved
        - Schedule post-mortem
      selection: "On-call IC roster, or senior engineer who declares the incident"

    communications_lead:
      responsibilities:
        - Draft and publish status page updates
        - Coordinate customer notifications
        - Handle internal stakeholder updates
        - Manage executive communication
        - Document timeline in real-time
      selection: "Designated from on-call comms roster or engineering manager"

    technical_lead:
      responsibilities:
        - Lead technical diagnosis and troubleshooting
        - Coordinate technical responders
        - Recommend mitigation and resolution actions
        - Verify fix effectiveness
      selection: "Senior engineer with relevant system expertise"

    scribe:
      responsibilities:
        - Document all actions, decisions, and findings
        - Maintain real-time timeline
        - Record who did what and when
        - Capture screenshots and log excerpts
      selection: "Any available team member (can be rotated)"

  workflow:
    1_declare:
      - "IC declares incident with severity level"
      - "War room channel and bridge created"
      - "Roles assigned"
      - "First status update posted"

    2_assess:
      - "Determine scope and customer impact"
      - "Identify affected systems and services"
      - "Establish working hypothesis"

    3_mitigate:
      - "Focus on restoring service first, root cause second"
      - "IC approves all changes to production"
      - "Changes documented in real-time"
      - "Rollback if mitigation makes things worse"

    4_resolve:
      - "Confirm service restored to normal"
      - "Verify monitoring shows healthy metrics"
      - "IC declares incident resolved"
      - "Final status page update"

    5_follow_up:
      - "Schedule post-mortem within 48 hours"
      - "Assign action items from immediate findings"
      - "Send internal summary"

On-Call Configuration

on_call_schedule:
  rotation_structure:
    primary:
      rotation: "Weekly"
      handoff: "Monday 10:00 AM local time"
      team_size: "Minimum 5 engineers in rotation"
    secondary:
      rotation: "Weekly (offset from primary)"
      activation: "If primary does not acknowledge within 10 minutes"

  expectations:
    response_time: "Acknowledge alert within 5 minutes"
    availability: "Reachable by phone and laptop within 15 minutes"
    handoff: "Document any ongoing issues during handoff"
    compensation: "Per company on-call compensation policy"

  health:
    max_consecutive_weeks: 2
    minimum_gap_between_rotations: "2 weeks"
    post_in

… (nội dung đầy đủ có trong gói tải về)