subdomain-enumeration – Security skill for AI agents

Subdomain Enumeration Skill

Comprehensive subdomain discovery using certificate transparency logs (crt.sh), DNS brute force, and passive sources. The first step in any recon pipeline — you can't attack what you don't know exists. Subdomain enumeration consistently reveals staging environments, internal admin panels, API gateways, and forgotten WordPress installs that are softer targets than the production site.

When to Use

• Starting recon on any target domain.
• Production site is well-secured — find softer entry points.
• After skill_view(name='wp-mass-recon') — enumerate subdomains for each WordPress target.
• Building a complete asset inventory for a target organization.

Prerequisites

• curl, httpx, dig, jq, dnsx, and subfinder.
• A DNS brute-force wordlist, supplied by the operator.
• Approval to disclose the domain to third-party passive sources such as crt.sh, crt.name, and subfinder providers.

How to Run

DOMAIN="${1:-example.com}"
OUTPUT_ROOT="${OUTPUT_DIR:-./output}"
OUTDIR="$OUTPUT_ROOT/subdomains/$DOMAIN"

if [[ ! "$DOMAIN" =~ ^[A-Za-z0-9.-]+$ ]]; then
  echo "Invalid domain: $DOMAIN" >&2
  exit 2
fi

mkdir -p "$OUTDIR"

# Passive: crt.sh (retry: it frequently returns 502 while its database refreshes)
for attempt in 1 2 3; do
  curl -fsS --max-time 30 --connect-timeout 10 \
    "https://crt.sh/?q=%25.$DOMAIN&output=json" -o "$OUTDIR/crtsh_raw.json" && break
  sleep 15
done
jq -r '.[].name_value' "$OUTDIR/crtsh_raw.json" 2>/dev/null \
  | sed 's/\*\.//g' \
  | sort -u > "$OUTDIR/crtsh.txt"
# If crt.sh stays down, crtsh.txt ends up empty: keep going with crt.name/subfinder.

# crt.name (historical CT data; resolve names before probing)
curl -fsS --max-time 30 --connect-timeout 10 "https://crt.name/v1/search?apex=$DOMAIN" \
  | sed 's/\r$//' \
  | sed 's/^\*\.//' \
  | sort -u > "$OUTDIR/crtname.txt"

# Passive: subfinder
subfinder -d "$DOMAIN" -silent -timeout 30 > "$OUTDIR/subfinder.txt"

# Merge and deduplicate
cat "$OUTDIR/crtsh.txt" "$OUTDIR/crtname.txt" "$OUTDIR/subfinder.txt" \
  | grep -E '^[A-Za-z0-9.-]+\.[A-Za-z]{2,}$' \
  | sort -u > "$OUTDIR/all_subs.txt"

# Probe live hosts
httpx -silent -l "$OUTDIR/all_subs.txt" -threads 50 -rate-limit 2 \
  -status-code -tech-detect -title -o "$OUTDIR/alive.txt"

Quick Reference

| Source | Method | Coverage | Speed |

|--------|--------|----------|-------|

| crt.sh | Certificate transparency | Excellent (most certs) | Fast (1-5s) |

| subfinder | Passive APIs (VirusTotal, Shodan, DNSdumpster, etc.) | Very good | Fast (30-60s) |

| dnsx | Bulk DNS A/AAAA/CNAME resolution (100x faster than dig) | Good (uncovers non-HTTP) | Fast (10-30s) |

| httpx probe | Live HTTP/HTTPS check | Best for web attack surface | Fast (30-60s) |

| Google dork | site:example.com | Supplemental | Manual |

Procedure

Step 1 — Passive Enumeration (crt.sh + subfinder)

DOMAIN="$1"
OUTPUT_ROOT="${OUTPUT_DIR:-./output}"
OUTDIR="$OUTPUT_ROOT/subdomains/$DOMAIN"

if [[ ! "$DOMAIN" =~ ^[A-Za-z0-9.-]+$ ]]; then
  echo "Invalid domain: $DOMAIN" >&2
  exit 2
fi

mkdir -p "$OUTDIR"

echo "[*] Passive enumeration for $DOMAIN..."

# crt.sh — certificate transparency logs (retry: it frequently returns 502 while its database refreshes)
echo "[*] crt.sh query..."
for attempt in 1 2 3; do
  curl -fsS --max-time 30 --connect-timeout 10 \
    "https://crt.sh/?q=%25.$DOMAIN&output=json" -o "$OUTDIR/crtsh_raw.json" 2>/dev/null && break
  sleep 15
done
jq -r '.[].name_value' "$OUTDIR/crtsh_raw.json" 2>/dev/null | \
  sed 's/\*\.//g' | \
  sed 's/^www\.//' | \
  sort -u > "$OUTDIR/crtsh.txt"

crt_count=$(wc -l < "$OUTDIR/crtsh.txt")
echo "  crt.sh: $crt_count entries"
if [[ ! -s "$OUTDIR/crtsh.txt" ]]; then
  echo "  [!] crt.sh unavailable or empty after retries; results are incomplete" >&2
fi

# crt.name — historical certificate-transparency data
echo "[*] crt.name query..."
curl -fsS --max-time 30 --connect-timeout 10 "https://crt.name/v1/search?apex=$DOMAIN" 2>/dev/null | \
  sed 's/\r$//' | \
  sed 's/^\*\.//' | \
  sort -u > "$OUTDIR/crtname.txt"
crtname_count=$(wc -l < "$OUTDIR/crtname.txt")
echo "  crt.name: $crtname_count entries (historical; resolve before probing)"

# Also query with %25. (wildcard)
for attempt in 1 2 3; do
  curl -fsS --max-time 30 --connect-timeout 10 \
    "https://crt.sh/?q=%25.%25.$DOMAIN&output=json" -o "$OUTDIR/crtsh_wildcard_raw.json" 2>/dev/null && break
  sleep 15
done
jq -r '.[].name_value' "$OUTDIR/crtsh_wildcard_raw.json" 2>/dev/null | \
  sed 's/\*\.//g' | \
  sort -u > "$OUTDIR/crtsh_wildcard.txt"

# subfinder — passive API aggregation
echo "[*] subfinder..."
subfinder -d "$DOMAIN" -silent -timeout 30 2>/dev/null | sort -u > "$OUTDIR/subfinder.txt"
subf_count=$(wc -l < "$OUTDIR/subfinder.txt")
echo "  subfinder: $subf_count entries"

# Merge all passive sources
cat "$OUTDIR"/crtsh.txt "$OUTDIR"/crtsh_wildcard.txt "$OUTDIR"/crtname.txt "$OUTDIR"/subfinder.txt 2>/dev/null | \
  sed 's/^www\.//' | \
  grep -E '^[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$' | \
  sort -u > "$OUTDIR/all_passive.txt"

total=$(wc -l < "$OUTDIR/all_passive.txt")
echo ""
echo "[+] Total unique subdomains (passive): $total"

Step 2 — DNS Resolution

DOMAIN="$1"
OUTPUT_ROOT="${OUTPUT_DIR:-./output}"
OUTDIR="$OUTPUT_ROOT/subdomains/$DOMAIN"

echo "[*] Resolving subdomains..."

# Batch resolve with dnsx (100x faster than dig loop)
dnsx -silent -l "$OUTDIR/all_passive.txt" -a -resp-only -o "$OUTDIR/resolved_raw.txt" 2>/dev/null
# Format output: subdomain => IP
while read -r line; do
  sub=$(echo "$line" | cut -d' ' -f1)
  ip=$(echo "$line" | cut -d' ' -f2)
  echo "$sub => $ip"
done < "$OUTDIR/resolved_raw.txt" > "$OUTDIR/resolved.txt"

resolved=$(wc -l < "$OUTDIR/resolved.txt")
echo "[+] $resolved subdomains resolved to IPs"

# Count unique IPs
unique_ips=$(awk '{print $3}' "$OUTDIR/resolved.txt" | sort -u | wc -l)
echo "[+] $unique_ips unique IPs"

# Identify shared hosting (many subdomains → same IP)
echo ""
echo "[*] Shared hosting clusters:"
awk '{print $3}' "$OUTDIR/resolved.txt" | sort | uniq -c | sort -rn | head -10 | while read -r count ip; do
  [[ "$count" -gt 1 ]] && echo "  $ip: $count subdomains"
done

Step 3 — Live Host Discovery

DOMAIN="$1"
OUTPUT_ROOT="${OUTPUT_DIR:-./output}"
OUTDIR="$OUTPUT_ROOT/subdomains/$DOMAIN"

echo "[*] Probing live hosts..."

# httpx with tech detection
httpx -silent -l "$OUTDIR/all_passive.txt" -threads 50 \
  -status-code -tech-detect -title -location \
  -o "$OUTDIR/alive.txt" 2>/dev/null

alive=$(wc -l < "$OUTDIR/alive.txt")
echo "[+] $alive live hosts"

# Categorize by status code
echo ""
echo "[*] By HTTP status:"
echo "  200: $(grep -c '\[200\]' "$OUTDIR/alive.txt")"
echo "  301/302: $(grep -cE '\[301\]|\[302\]' "$OUTDIR/alive.txt")"
echo "  403: $(grep -c '\[403\]' "$OUTDIR/alive.txt")"
echo "  404: $(grep -c '\[404\]' "$OUTDIR/alive.txt")"

# Categorize by technology
echo ""
echo "[*] By technology:"
grep -Eo '\[[a-z-]+\]' "$OUTDIR/alive.txt" | tr -d '[]' | sort | uniq -c | sort -rn | head -15

# WordPress subdomains
echo ""
echo "[*] WordPress subdomains:"
grep -i 'wordpress' "$OUTDIR/alive.txt" | awk '{print $1}' | head -10

# Non-HTTP services (from DNS resolution)
echo ""
echo "[*] Interesting non-standard ports from DNS (MX, NS, etc.):"
dig +short "$DOMAIN" MX 2>/dev/null | head -5
dig +short "$DOMAIN" NS 2>/dev/null | head -5
dig +short "$DOMAIN" TXT 2>/dev/null | grep -i 'spf\|v=spf' | head -3

Step 4 — Subdomain Categorization

DOMAIN="$1"
OUTPUT_ROOT="${OUTPUT_DIR:-./output}"
OUTDIR="$OUTPUT_ROOT/subdomains/$DOMAIN"

echo "[*] Categorizing subdomains..."

# Staging/Dev
echo ""
echo "=== STAGING / DEV ==="
grep -iE 'staging|stage|dev\.|development|test|uat|beta|sandbox|preview|qa' "$OUTDIR/all_passive.txt"

# Admin/Internal
echo ""
echo "=== ADMIN / INTERNAL ==="
grep -iE 'admin|portal|internal|dashboard|manage|cp\.|control|panel|cpanel|webmail|mail\.' "$OUTDIR/all_passive.txt"

# API
echo ""
echo "=== API ==="
grep -iE 'api|rest|graphql|ws\.|websocket' "$OUTDIR/all_passive.txt"

# Infrastructure
echo ""
echo "=== CDN / STATIC ==="
grep -iE 'cdn|static|assets|media|img|images|files|download|origin|proxy' "$OUTDIR/all_passive.txt"

# Email
echo ""
echo "=== EMAIL ==="
grep -iE 'mail\.|smtp|imap|pop|email|webmail|autodiscover' "$OUTDIR/all_passive.txt"

# Cloud
echo ""
echo "=== CLOUD ==="
grep -iE 'aws|azure|gcp|cloud|s3|bucket|firebase' "$OUTDIR/all_passive.txt"

# Legacy
echo ""
echo "=== LEGACY / OLD ==="
grep -iE 'old|old\.|v1|v2|legacy|archive|backup|bak' "$OUTDIR/all_passive.txt"

Step 5 — Subdomain Takeover Check

DOMAIN="$1"
OUTPUT_ROOT="${OUTPUT_DIR:-./output}"
OUTDIR="$OUTPUT_ROOT/subdomains/$DOMAIN"

echo "[*] Checking for subdomain takeover opportunities..."

# Check for dangling CNAMEs (subdomains pointing to non-existent services)
# For bulk CNAME check: dnsx -silent -l all_passive.txt -cname -resp-only
while read -r sub; do
  cnam

… (full content is in the download package)