mid-engagement-ir-detection – Security skill for AI agents

When to use this skill

Trigger when:

• Running active testing against a target with active SOC monitoring
• A confirmed-vulnerable finding stops reproducing on recheck
• Baseline timing shifts unexpectedly (3× slower, sudden errors, new headers)
• Response sizes change between test windows
• New WAF cookies or headers appear that weren't there at session start
• Lockout / error rates change between test windows (especially LOCKED count for credential attacks)
• Engagement is "assume breach" or "white box" — client knows you're testing

DO NOT use for:

• Bug bounty (client doesn't know you're there; no real-time IR)
• Pure recon (no state-change happening)
• One-off vulnerability scanning (no temporal dimension)

---

The core insight

In a real red-team engagement against a competent SOC, the security state of the target is not static. It changes during your test in response to your traffic. These state changes are:

1. Themselves valuable findings (positive operational observations about IR responsiveness)

2. Confirmation evidence (mid-engagement patch = the original vulnerability was real)

3. Classification signals (WAF rule deployment vs code fix — different remediation depth)

Anti-pattern: treating reproduction failure as evidence the original signal was a false positive. Original PoC artifacts captured before the change are still the vulnerability finding.

---

The discipline — capture before, diff after

Before any active test:

# Capture pre-test fingerprint of the target
fingerprint = {
    "ts_pre": time.time(),
    "ip_seen": "<operator-src-ip>",
    "baseline_response_time_ms": <measure>,
    "baseline_response_size_bytes": <measure>,
    "response_headers": <capture set>,
    "waf_cookies": <list>,
    "lockout_count_in_state": <count from o365_attempts.json>,
}

Persist to engagement_log/fingerprint_pre.json.

During the test:

Log every test result with full context (timestamp, IP, payload, response code, response size, response time, headers if relevant) to JSONL append-only.

After the test session OR on first failed-recheck:

fingerprint_post = same structure
delta = {
    "baseline_time_change_ms": post.time - pre.time,
    "baseline_size_change_bytes": post.size - pre.size,
    "new_headers_appeared": post.headers - pre.headers,
    "new_waf_cookies": post.cookies - pre.cookies,
    "new_lockouts": post.locked_count - pre.locked_count,
}

If any delta is significant — investigate, don't retract.

---

The three primary IR observations

Observation 1 — Mid-engagement WAF rule deployment

Symptoms:

• Original payloads return identical response → no signal at all on recheck
• Body size identical to baseline (login page reflection)
• Timing reverts to baseline regardless of payload
• New cookie or header in responses (e.g., cf-bm, __cf_bm, awselb)
• Specific keyword in URL/body now triggers different response code (403, 406, 429)

Confirmation: retry with WAF-evasion variants:

• URL-encode the payload differently (%27 vs %5cu0027)
• Change request method (POST → PUT, GET → POST)
• Different content-type (form-urlencoded → multipart)
• Slower pace (5s → 60s between requests)
• Mixed-case keywords (SLEEP → SlEeP)

If WAF-evasion variants restore the signal, the mitigation is at the WAF layer (bypassable).

If even WAF-evasion variants stay blocked, the mitigation is likely in code.

Finding template:

Subject: Mid-engagement mitigation deployed for <vulnerability X>
Observation: At engagement timestamp T0, vulnerability <X> on <endpoint> was
confirmed via <PoC>. At T0+<minutes>, recheck via the original payload no longer
reproduces the timing/error/size differential. <WAF-evasion variant> [does/does
not] restore the signal.

Description: This pattern is consistent with the client SOC observing engagement
traffic and deploying a mitigation in real time. Mitigation depth assessment:
[at-WAF, bypassable] vs [in-code, durable].

Impact (positive): Client SOC has both detection-grade visibility into application
traffic AND the authority to deploy mitigations within ~<minutes> of detection.

Impact (caveat): The original vulnerability did exist and was exploitable for at
least the engagement window before mitigation. If the mitigation is at the WAF
layer only, the underlying code-level flaw remains exploitable via alternative
payloads.

Recommendation:
1. Verify the mitigation is in code (parameterized queries, input sanitization),
   not just at the WAF layer.
2. Audit the codebase for the same root cause across sister applications.
3. (Positive) Document the IR responsiveness as a capability metric.

Observation 2 — Active concurrent attacker

Symptoms:

• Many AADSTS50053 (LOCKED) responses despite your 1-attempt-per-user discipline
• Lockouts cluster alphabetically or by some other sort key
• New lockouts appear DURING your engagement (diff before/after)
• LOCKED rate exceeds expected baseline (in our engagement: 11% of all attempts → red flag)

Math check:

• Your discipline: 1 attempt per user lifetime
• Smart Lockout default: lockout after 10 failed attempts; lockout duration starts at 60 seconds and grows with each subsequent lockout (not a flat 10-minute window)
• Therefore: IF command-line logs confirm exactly 1 attempt/user (no burst retries, no parallel-goroutine duplicate sends, no command-line misconfiguration), you cannot mathematically cause Smart Lockout — verify journal.jsonl shows 1 attempt/user before asserting this
• Therefore: every NEW AADSTS50053 accumulating during the window (per before/after diff) was caused by someone else; pre-existing locks may stem from the legitimate user's own failures or a prior test run, so attribute only the newly-accumulating locks to an external party

Confirmation:

• Sort locked accounts alphabetically; if they cluster, attacker is using sorted username list
• Compare pre-session lockout count vs post-session — new locks during your session = attacker is active *right now*
• Probe a known-active "system" account (noreply@, info@, oc@) — if it's locked, attacker is hitting service mailboxes too (typically MFA-exempt → high-value to attackers)

Finding template:

Subject: Active external password-spray campaign detected during engagement

Observation: During M365 ROPC validation against the <tenant> Entra tenant, <N>
unique principals returned AADSTS50053 (Smart Lockout) when probed with a single
password attempt at safe pace. Ourcommand-line journal (journal.jsonl) confirms exactly
1 attempt per user with no duplicate sends, so under the Smart Lockout default
(lockout after 10 failed attempts) we cannot have caused these lockouts. The
attribution below applies to the <K> NEW locks that accumulated during our
engagement window between <timestamp_start> and <timestamp_end> (per before/after
diff); pre-existing locks are not attributed to this campaign.

Description: The pattern (alphabetical clustering, real-time accumulation,
including system mailboxes) is consistent with an external attacker performing
a username-list-driven password spray attack against the tenant.

Impact: An external adversary is actively attempting to compromise corporate
M365 accounts. The attacker has knowledge of the user-email schema and a
password-guess wordlist. <List of locked accounts is now in attacker's hands>
(Smart Lockout differentiates valid from invalid usernames).

Recommendation (CRITICAL — within 24h):
1. Open a P1 incident with the SOC. Pull Entra sign-in logs for the <N> locked
   accounts over the last 30-60 days. Identify source IPs and time windows.
2. Apply Conditional Access rule blocking sign-ins from outside <expected geo>
   for non-admin accounts.
3. Enable Identity Protection's User Risk policy with auto-reset on high risk.
4. Force tenant-wide password reset for all <N> previously-locked accounts.
5. Audit service accounts for MFA exemptions; ensure all human-interactive
   accounts have phishing-resistant MFA.

Evidence: engagement_log/poc/m365/locked_accounts.txt (<N> entries with timestamps)

Observation 3 — Detection-induced rate limiting / IP blocks

Symptoms:

• Specific IP starts returning 403 / 429 / 451 after a window of normal responses
• Specific IP starts seeing dramatically slower responses (3x+ baseline)
• TLS handshake fails or RST mid-connection
• DNS suddenly returns NXDOMAIN for hosts that resolved before
• Some hosts work from one IP but not another

Confirmation:

• Rotate to a different IP and retry — if works, you got rate-limited/blocked
• Compare TTL on DNS responses — sudden short TTL = active mitigation deployed
• Check Server:, Via:, CF-Cache-Status: headers for CDN-introduced limits

Finding template (operational note, usually low/info severity):

Subject: Engagement traffic detected — IP <X> rate-limited at <timestamp>

Observation: After ~<N> requests in <window> f

… (full content is in the download package)