reverse-engineering – Data & Science skill for AI agents

Reverse Engineering

When to Use

• Understanding how an unknown binary works before any further tasking.
• Choosing tools and workflow for static/dynamic analysis of a new target.

授权、隔离与执行边界

先依据用户提供的目标和授权范围做只读静态分诊，不把未知目标自动认作 CTF、已授权或已隔离环境。已有明确授权不需重复询问；缺失且影响下一步时只澄清必要范围。动态跟踪、模拟器和调试器会执行目标，必须限于获准的副本及专用可还原隔离环境，禁用外网、宿主凭据、共享目录和生产访问；普通 QEMU 用户态、Unicorn/Qiling 或 Node vm 不构成完整安全边界。内核/固件操作仅限隔离测试虚拟机或获准测试硬件。

不自动上传二进制、密钥、字符串或转储到 Dogbolt、云模型、在线沙箱；先取得对应数据外发授权。只保留必要证据并遮盖敏感值。不绕过第三方服务的访问控制、付费授权或风控来获取未授权访问。安装工具和修改样本是单独可见的操作，先检查来源、固定版本及输出路径，保留原件哈希。

Quick reference for RE challenges. For detailed techniques, see supporting files.

Prerequisites

Python packages (all platforms):

pip install frida-tools angr qiling uncompyle6 capstone lief z3-solver
# For Python 3.9+ bytecode: build pycdc from source
git clone https://github.com/zrax/pycdc && cd pycdc && cmake . && make

Linux (apt):

apt install gdb radare2 binutils strace ltrace apktool upx

macOS (Homebrew):

brew install gdb radare2 binutils apktool upx ghidra

radare2 plugins:

r2pm -ci r2ghidra   # Native Ghidra decompiler for radare2

Manual install:

• pwndbg — Linux: [GitHub](https://github.com/pwndbg/pwndbg), macOS: brew install pwndbg/tap/pwndbg-gdb

Additional Resources

• [tools.md](references/tools.md) - Static analysis tools (GDB, Ghidra, radare2, IDA, Binary Ninja, dogbolt.org, RISC-V with Capstone, Unicorn emulation, Python bytecode, WASM, Android APK, .NET, packed binaries)
• [tools-dynamic.md](references/tools-dynamic.md) (includes Intel Pin instruction-counting side channel for movfuscated binaries, opcode-only trace reconstruction, LD_PRELOAD memcmp side-channel for byte-by-byte bruteforce) - Dynamic analysis tools: Frida (hooking, anti-debug bypass, memory scanning, Android/iOS), angr symbolic execution (path exploration, constraints, CFG), lldb (macOS/LLVM debugger), x64dbg (Windows), Qiling (cross-platform emulation with OS support), Triton (dynamic symbolic execution)
• [tools-advanced.md](references/tools-advanced.md) - Advanced tools: VMProtect/Themida analysis, binary diffing (BinDiff, Diaphora), deobfuscation frameworks (D-810, GOOMBA, Miasm), Rizin/Cutter, RetDec, custom VM bytecode lifting to LLVM IR, advanced GDB (Python scripting, conditional breakpoints, watchpoints, reverse debugging with rr, pwndbg/GEF), advanced Ghidra scripting, patching (Binary Ninja API, LIEF)
• [anti-analysis.md](references/anti-analysis.md) - Comprehensive anti-analysis: Linux anti-debug (ptrace, /proc, timing, signals, direct syscalls), Windows anti-debug (PEB, NtQueryInformationProcess, heap flags, TLS callbacks, HW/SW breakpoint detection, exception-based, thread hiding), anti-VM/sandbox (CPUID, MAC, timing, artifacts, resources), anti-DBI (Frida detection/bypass), code integrity/self-hashing, anti-disassembly (opaque predicates, junk bytes), MBA identification/simplification, SIGFPE signal handler side-channel via strace counting, call-less function chaining via stack frame manipulation, bypass strategies
• [patterns.md](references/patterns.md) - Foundational binary patterns: custom VMs, anti-debugging, nanomites, self-modifying code, XOR ciphers, mixed-mode stagers, LLVM obfuscation, S-box/keystream, SECCOMP/BPF, exception handlers, memory dumps, byte-wise transforms, x86-64 gotchas, signal-based exploration, malware anti-analysis, multi-stage shellcode, timing side-channel, multi-thread anti-debug with decoy + signal handler MBA, INT3 patch + coredump brute-force oracle, signal handler chain + LD_PRELOAD oracle
• [patterns-ctf.md](references/patterns-ctf.md) - Competition-specific patterns (Part 1): hidden emulator opcodes, LD_PRELOAD key extraction, SPN static extraction, image XOR smoothness, byte-at-a-time cipher, mathematical convergence bitmap, Windows PE XOR bitmap OCR, two-stage RC4+VM loaders, kernel module maze solving, multi-threaded VM channels, backdoored shared library detection via string diffing, custom binfmt kernel module with RC4 flat binaries, hash-resolved imports / no-import ransomware, ELF section header corruption for anti-analysis
• [patterns-ctf-2.md](references/patterns-ctf-2.md) - Competition-specific patterns (Part 2): multi-layer self-decrypting brute-force, embedded ZIP+XOR license, stack string deobfuscation, prefix hash brute-force, CVP/LLL lattice for integer validation, decision tree function obfuscation, GF(2^8) Gaussian elimination, ROP chain obfuscation analysis (ROPfuscation)
• [patterns-ctf-3.md](references/patterns-ctf-3.md) - Competition-specific patterns (Part 3): Z3 single-line Python circuit, sliding window popcount, keyboard LED Morse code via ioctl, C++ destructor-hidden validation, syscall side-effect memory corruption, MFC dialog event handlers, VM sequential key-chain brute-force, Burrows-Wheeler transform inversion, OpenType font ligature exploitation, GLSL shader VM with self-modifying code, instruction counter as cryptographic state, batch crackme automation via objdump, fork+pipe+dead branch anti-analysis, TensorFlow DNN inversion via sigmoid layer inversion, BPF filter analysis via kernel JIT to x64 assembly
• [languages.md](references/languages.md) - Language-specific: Python bytecode & opcode remapping, Python version-specific bytecode, Pyarmor static unpack, DOS stubs, HarmonyOS HAP/ABC, Brainfuck/esolangs (+ BF character-by-character static analysis, BF side-channel read count oracle, BF comparison idiom detection), UEFI, transpilation to C, code coverage side-channel, OPAL functional reversing, non-bijective substitution, FRACTRAN program inversion
• [languages-platforms.md](references/languages-platforms.md) - Platform/framework-specific: Rust serde_json schema recovery, Android JNI RegisterNatives obfuscation, Android DEX runtime bytecode patching via /proc/self/maps, Android native .so loading bypass via new project, Frida Firebase Cloud Functions bypass, Verilog/hardware RE, prefix-by-prefix hash reversal, Ruby/Perl polyglot constraint satisfaction, Electron ASAR extraction + native binary analysis, Node.js npm runtime introspection
• [languages-compiled.md](references/languages-compiled.md) - Go binary reversing (GoReSym, goroutines, memory layout, channel ops, embed.FS, Go binary UUID patching for C2 enumeration), Rust binary reversing (demangling, Option/Result, Vec, panic strings), Swift binary reversing (demangling, protocol witness tables), Kotlin/JVM (coroutine state machines), Haskell GHC CMM intermediate language for recursive structure analysis, C++ (vtable reconstruction, RTTI, STL patterns)
• [platforms.md](references/platforms.md) - Platform-specific RE: macOS/iOS (Mach-O, code signing, Objective-C runtime, Swift, dyld, jailbreak bypass), embedded/IoT firmware (binwalk, UART/JTAG/SPI extraction, ARM/MIPS, RTOS), kernel drivers (Linux .ko, eBPF, Windows .sys), automotive CAN bus
• [platforms-hardware.md](references/platforms-hardware.md) - Hardware and advanced architecture RE: HD44780 LCD controller GPIO reconstruction, RISC-V advanced (custom extensions, privileged modes, debugging), ARM64/AArch64 reversing and exploitation (calling convention, ROP gadgets, qemu-aarch64-static emulation)
• [field-notes.md](references/field-notes.md) - Quick reference notes: binary types, anti-debugging bypass, specialized patterns, CTF case notes

---

When to Pivot

• Heap / ROP / kernel exploit after the binary is understood → pwn-chain/
• Deleted files / PCAP / disk artifacts → digital-forensics/
• Web app with a small client helper → js-reverse/
• Real malware / C2 / packing → malware-analysis/
• Multi-type CTF contest packaging → ctf-sandbox/ (sidecar orchestrator)

Problem-Solving Workflow

1. Start with strings extraction - many easy challenges have plaintext flags

2. Try ltrace/strace - dynamic analysis often reveals flags without reversing

3. Try Frida hooking - hook strcmp/memcmp to capture expected values without reversing

4. Try angr - symbolic execution solves many flag-checkers automatically

5. Try Qiling - emulate foreign-arch binaries or bypass heavy anti-debug without artifacts

6. Map control flow before modifying execution

7. Automate manual processes via scripting (r2pipe, Frida, angr, Python)

8. Validate assumptions by comparing decompiler outputs (dogbolt.org for side-by-side)

Quick Wins (Try First!)

# Plaintext flag extraction
strings binary | grep -E "flag\{|CTF\{|pico"
strings binary | grep -iE "flag|secret|password"
rabin2 -z binary | grep -i "flag"

# Dynamic analysis - often captures flag directly
ltrace ./binary
strace -f -s 500 ./binary

# Hex dump search
xxd binary | grep -i flag

# Run with test inputs
./binary AAAA
echo "test" | ./binary

Initial Analysis

file binary           # Type, architecture
checksec --file=binary # Security features (for pwn)
chmod +x binary       # Make executable

Memory Dumping Strategy

Key insight:

… (full content is in the download package)