recon-sector-expansion – Security skill for AI agents

RECON-SECTOR-EXPANSION — Batch Discovery Across Untapped Sectors

When to Use

Use within authorized engagements to expand coverage into fresh sectors — not dig deeper into a single known target, but find and test 20+ new company domains across 5-10 different sectors. Target list must be from authorized scope. Batch expansion is for scoped programs, not indiscriminate scraping.

Signals:

• "Discover 20 more domains in non-regulated sectors"
• "We need to expand coverage; what haven't we tested?"
• "Find fresh targets outside the usual sectors"

The Sector Expansion Workflow

Phase 1 — Coverage Audit

Before discovering new targets, know what's already been tested:

# 1. Check existing target list
ALL_TARGETS="${OUTPUT_DIR:-./output}/sector-expansion/all_targets.txt"
ALL_MASSIVE="${OUTPUT_DIR:-./output}/sector-expansion/all_massive.txt"

# Extract all domains from both files
grep -ohE '[a-zA-Z0-9][a-zA-Z0-9.-]*\.[a-z]{2,}' "$ALL_TARGETS" "$ALL_MASSIVE" | sort -u > /tmp/targets_deduped.txt
echo "Already tested: $(wc -l < /tmp/targets_deduped.txt) domains"

# 2. Check what findings files exist
ls "${OUTPUT_DIR:-./output}"/sector-expansion/*_findings.md 2>/dev/null | wc -l

# 3. Identify which sectors have been covered by inspecting findings files
# Check for sector-specific keywords
for sector in "dentist" "dental" "gym" "fitness" "baker" "auto_body" "carpet" "laundry" "daycare" "pest" "tree" "pet"; do
  count=$(grep -l "$sector" "${OUTPUT_DIR:-./output}"/sector-expansion/*_findings*.md 2>/dev/null | wc -l)
  echo "Sector '$sector': $count findings files"
done

Phase 2a — Source 1: crt.sh Certificate Transparency (Preferred for Local Business Sectors)

For local service sectors (roofing, landscaping, plumbing, pool services, HVAC) where no national chains exist, crt.sh is the primary discovery source. These businesses often have descriptive domain names like [name][SECTOR_EXAMPLE] or [city][SECTOR_EXAMPLE].

crt.sh Query Pattern

Use simple sector keywords with the HTML output mode (JSON API returns 502/503 on broad queries):

QUERY="roofing"
curl -s --max-time 40 --connect-timeout 10 \
  "https://crt.sh/?q=${QUERY}&excluded=expired&dedup=Y" \
  -H 'User-Agent: Mozilla/5.0' 2>/dev/null | \
  grep -oE '>[A-Za-z0-9][A-Za-z0-9.-]*\.com<' | \
  sed 's/^>//;s/<$//' | \
  sort -u >> /tmp/all_crt_domains.txt

Parameters explained:

• &excluded=expired — only current certificates
• &dedup=Y — deduplicate certificates
• -oE regex — BusyBox-compatible (no -P flag available)
• No &output=json — JSON API is unreliable for broad queries

Sector Keyword Catalog

Query ALL these sectors to gather candidates:

SECTORS="roofing landscaping pestcontrol dentist dentist fitness \
cleaningservice movingcompany photography vetclinic realtor hvac \
treeservice lawncare plumbing poolcleaning windowcleaning petsalon \
barbershop daycare carpetcleaning handyman lawfirm concrete \
autorepair petgrooming autobody remodeling"
for sector in $SECTORS; do
  curl -s --max-time 40 --connect-timeout 10 "https://crt.sh/?q=${sector}&excluded=expired&dedup=Y" \
    -H 'User-Agent: Mozilla/5.0' 2>/dev/null | \
    grep -oE '>[A-Za-z0-9][A-Za-z0-9.-]*\.com<' | \
    sed 's/^>//;s/<$//' >> /tmp/all_crt_domains.txt
  sleep 2  # avoid rate limiting
done

Filtering Noise from crt.sh Results

crt.sh returns many subdomains and infrastructure hosts. Strip these patterns:

import re

# Common noise patterns from crt.sh
skip_patterns = [
    '^autodiscover\\.', '^vpn\\.', '^api\\.', '^mail\\.', '^remote\\.',
    '^webmail\\.', '^crm\\.', '^ftp\\.', '^test\\.', '^dev\\.',
    '^exchange', '^hostmaster', '^owa\\.', '^smtp\\.',
    # Generic placeholder domains
    'roofingcompany\\.com$', 'landscapingcompany\\.com$', 'vetclinic\\.com$',
]

Then associate domains with sectors by matching their name:

SECTOR_KEYWORDS = {
    'roofing': ['roofing', 'roof', 'roofer'],
    'landscaping': ['landscaping', 'landscape', 'lawn'],
    'hvac': ['hvac', 'heating', 'cooling', 'air'],
    'dental': ['dentist', 'dental'],
    'pest-control': ['pest', 'mosquito'],
    'daycare': ['daycare'],
    'legal': ['lawfirm', 'lawfirm'],
    'fitness': ['gym', 'fitness'],
    'auto-repair': ['autorepair', 'auto', 'collision'],
    'cleaning': ['cleaning', 'maid'],
    'veterinary': ['vet', 'pet'],
    'pool-services': ['pool', 'spa'],
}

Noise Subdomain Filtering

When using grep on crt.sh HTML output, you'll get entries from large hosting providers. Exclude:

• autodiscover.*, vpn.*, api.*, mail.*, remote.* — infrastructure
• .*\.hvacrightnow\.com, .*\.hvacbrain\.com — SaaS platforms (not actual targets)
• .*\.theroofingprofessor\.com, .*\.bakerroofing\.com — large org subdomains
• .*\.singaporepools\.com, .*\.devsingaporepools\.com — non-US

Piping through grep -v with exclusion patterns before dedup saves time.

Phase 2b — Source 2: Google / Manual (Preferred for National Chains)

For sectors dominated by national/regional chains (dental, gyms, bakeries, auto body), use Google searches because chain domain names rarely contain sector keywords:

| Sector | How to Find Candidates | Example Chains |

|--------|----------------------|----------------|

| Dental chains | Google "top dental chains USA" | Aspen Dental, Gentle Dental, Coast Dental, DentalWorks, Heartland Dental, Pacific Dental |

| Gym/fitness | Google "largest fitness chains USA" | F45, Barry's, CrossFit, Gold's Gym, 24 Hour Fitness, LA Fitness, OrangeTheory |

| Bakery chains | Google "largest bakery chains USA" | Cinnabon, Crumbl, Insomnia Cookies, Sprinkles, Nothing Bundt Cakes |

| Auto body/collision | Google "auto body repair chains USA" | ABRA, Caliber, Gerber, Service King, Fix Auto, Sterling, Crash Champions |

| Carpet cleaning | Google "carpet cleaning companies USA" | Chem-Dry, ServPro, Stanley Steemer, Rainbow International, Heaven's Best |

| Laundry services | Google "laundry delivery service USA" | Poplin, Rinse, Washlava, LaundryHeap |

| Daycare/childcare | Google "largest daycare chains USA" | Bright Horizons, KinderCare, Goddard, Primrose, Children of America |

| Pet grooming | Google "pet grooming chains USA" | Petco, PetSmart, Camp Bow Wow, Dogtopia |

| Pest control | Google "pest control companies USA" | Terminix, Orkin, Ehrlich, Bulwark, Arrow |

| Tree services | Google "tree service companies USA" | Davey, SavATree, Arbor Care, Bartlett |

Phase 3 — Batch Alive-Check (Curl Probe)

Once you have 50-60 candidate domains, filter against already-tested domains, then quick-check HTTP:

# Filter against known targets
grep -v -f /tmp/targets_deduped.txt /tmp/candidates.txt | sort -u > /tmp/fresh_candidates.txt

# Quick alive check with curl (serial, ~2.5 min for 60 targets)
echo "=== Checking all candidates with curl ==="
for domain in $(cat /tmp/fresh_candidates.txt); do
  code=$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 5 --max-time 8 "https://${domain}" 2>/dev/null)
  echo "$domain => $code"
done > /tmp/curl_alive.txt

# Extract alive domains (anything not 000/timeout)
grep -v "=> 000" /tmp/curl_alive.txt | grep -v "^$" | cut -d' ' -f1 > /tmp/alive_domains.txt
echo "Alive: $(wc -l < /tmp/alive_domains.txt) / $(wc -l < /tmp/fresh_candidates.txt) candidates"

Phase 4 — Full Testing Pipeline

Then run the full 6-step pipeline on alive domains. The most efficient approach at scale (20+ targets) is a Python batch script:

#!/usr/bin/env python3
"""Batch test targets: httpx → WP REST → CORS → XMLRPC → ports"""
import subprocess, json, sys, os, re, time

OUTPUT_DIR = os.environ.get("OUTPUT_DIR", "./output") + "/sector-expansion"

def run(cmd, timeout=15):
    try:
        r = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=timeout)
        return r.stdout + r.stderr
    except subprocess.TimeoutExpired:
        return f"TIMEOUT ({timeout}s)"
    except Exception as ex:
        return f"ERROR: {ex}"

def test_domain(domain, sector):
    findings = {"domain": domain, "sector": sector, "wp": False, "wp_users": 0,
                "cors_reflected": False, "xmlrpc_open": False}
    
    # STEP 1: httpx probe
    r = run(f"httpx -sc -title -tech-detect -server -no-color -u https://{domain} -timeout 10")
    findings["http"] = r.strip()
    
    # STEP 2: WordPress REST API check (3 paths)
    for path in ["/wp-json/wp/v2/users", "/wp-json/", "/?rest_route=/wp/v2/users"]:
        r = run(f'curl -sk -o /dev/null -w "%{{http_code}}" --max-time 10 --connect-timeout 10 "https://{domain}{path}"')
        if r in ["200"]:
            findings["wp_rest_status"] = f"{path} => HTTP {r}"
            findings["wp"] = True
            # Enumerate users
            user_resp = run(f'curl -sk --max-time 10 --connect-timeout 10 "https://{domain}{path}"', timeout=12)
            try:
                users = json.loads(user_resp)
   

… (full content is in the download package)