autoskill – Data & Science skill for AI agents

autoskill

> Requires a running [screenpipe](https://github.com/screenpipe/screenpipe) daemon. This skill has no alternate data source — it reads exclusively from the local screenpipe HTTP API (default http://localhost:3030). If the daemon isn't running, run() raises ScreenpipeUnreachable with install instructions.

> Network access & environment variables. This skill makes authenticated HTTP requests to (a) the user's local screenpipe daemon on loopback, and (b) the user-configured LLM backend — one of http://localhost:1234/v1 (LM Studio, default), https://api.anthropic.com (opt-in Claude), or a user-supplied BYOK Foundry gateway. The adapters read SCREENPIPE_TOKEN, LM_API_TOKEN, ANTHROPIC_API_KEY, and FOUNDRY_API_KEY for the corresponding configured service. HTTPX also honors its standard proxy and CA environment settings; sentence-transformers uses Hugging Face cache/download settings. Opt-in cloud backends receive redacted cluster summaries and matched skill descriptions. Embedding-model installation may also download public model files; local inference does not imply zero network access.

Overview

Turn the user's own workflow history — captured passively by the local [screenpipe](https://github.com/screenpipe/screenpipe) daemon — into new skills. This skill is on-demand: the user invokes it with a time window, it queries screenpipe's local HTTP API, clusters repeated workflow patterns, compares each pattern against the existing skills in this repo, and produces a staged folder of proposals the user can review, edit, and promote.

When to Use This Skill

Invoke this skill when the user asks to:

• "Analyze my last 4 hours / day / week and propose new skills."
• "Look at what I've been doing and tell me what's not covered yet."
• "Draft a skill from my recent workflow."
• "Find composition recipes for workflows I repeat."

Do not invoke it for one-off questions about screenpipe itself, for real-time screen queries, or without an explicit user request — the skill analyzes sensitive local content and must stay explicitly user-triggered.

Privacy Posture

• Configure capture filtering in Screenpipe before collecting history. references/screenpipe-config.yaml is a checklist of literal, case-insensitive app/title substrings, not a Screenpipe-importable YAML file. Apply them in Screenpipe settings or as repeated --ignored-windows arguments; * is not a glob. Check exclusions with synthetic windows. Filtering cannot remove sensitive material already captured or guarantee complete exclusion.
• Raw OCR is not sent to the synthesis backend by this pipeline. scripts/fetch_window.py pulls data over localhost HTTP. scripts/cluster.py reduces the timeline to app/duration/title summaries. scripts/redact.py scrubs recognizable emails, API keys, bearer tokens, and selected phone formats as defense-in-depth before any cluster summary reaches the LLM.
• LLM backend defaults to local. Use an already installed chat model served by [LM Studio](https://lmstudio.ai/); its exact server model ID belongs in local.model. Summaries stay on the machine only while this endpoint is loopback. A remote HTTPS endpoint also sends summaries off-host. Cloud backends (claude, foundry) remain opt-in. Detection and embedding inference run locally regardless of backend choice.
• Dry-run mode (--dry-run) skips skill matching and LLM synthesis and writes a clustered plan.md. Review the retained app names and window titles before selecting a cloud backend; the regex scrubber does not guarantee anonymization or removal of unpublished research details.
• TLS for localhost (optional, for corporate policy): see references/https-proxy.md for the Caddy pattern.

Prerequisites

1. Screenpipe daemon

Either install the official release or build from source. Either way the daemon binds HTTP on localhost:3030 by default.

From source (recommended if you want the CLI daemon without the desktop GUI):

git clone --depth 1 https://github.com/screenpipe/screenpipe.git
cd screenpipe
cargo build -p screenpipe-engine --release
# System deps (macOS): cmake + full Xcode.app (not just Command Line Tools).
#   brew install cmake
#   # if xcodebuild plug-ins error: sudo xcodebuild -runFirstLaunch
./target/release/screenpipe doctor   # confirm permissions + ffmpeg
./target/release/screenpipe record --disable-audio --use-pii-removal \
  --ignored-windows "1Password" --ignored-windows "Bitwarden" \
  --ignored-windows "Private Browsing" --ignored-windows "Incognito"

This source-build example is illustrative and was not compiled in the API review. Check the installed screenpipe record --help; permissions and system dependencies vary by platform/release. On macOS, grant the requested Screen Recording/Accessibility permissions and relaunch. Review the full deny-list before real capture.

2. Screenpipe API token

Current Screenpipe enables API auth by default; protected routes such as /search require a bearer token even on loopback. /health is exempt, so a successful health check does not validate search authorization. For an authenticated instance, retrieve its local API token:

export SCREENPIPE_TOKEN="$(screenpipe auth token)"

(Or set screenpipe.token directly in config.yaml — env var is preferred since it keeps secrets out of version control.)

Screenpipe connections permit HTTP only on loopback; remote endpoints require HTTPS.

Generated draft names must be valid skill names, so model output cannot write outside the proposal directory.

3. Python environment

Create a separate environment; do not add scientific dependencies to the repository environment:

uv venv .venv-autoskill --python 3.12
uv pip install --python .venv-autoskill/bin/python httpx==0.28.1 pyyaml==6.0.3 sentence-transformers==6.1.0
source .venv-autoskill/bin/activate

The public sentence-transformers/all-MiniLM-L6-v2 model downloads on first use. For an existing cache, set embeddings.local_files_only: true to prevent download attempts. Its 384-dimensional embeddings truncate inputs beyond 256 word pieces; long summaries may lose detail. Similarity is a retrieval heuristic, not proof of workflow equivalence.

4. Local LLM (default path) — LM Studio

• Install [LM Studio](https://lmstudio.ai/).
• Choose a chat model already installed on your machine (lms ls). Match the context length to its supported limits and available memory; no particular GPU fit is assumed.
• Load it with a stable identifier and start the API server explicitly:

lms load <installed-model-key> --identifier autoskill-local
lms server start --port 1234
lms server status

This setup is illustrative; no model was downloaded or inferred during this review. lms load does not itself start the HTTP server. If LM Studio's Require Authentication option is enabled (0.4.0+), set LM_API_TOKEN to a token created in its server settings. doctor checks the configured ID against /v1/models; the list can include JIT-loadable models and does not prove inference succeeds.

5. Cloud LLM backends (optional, opt-in)

Only if you explicitly opt out of local:

• claude: set ANTHROPIC_API_KEY, flip backend: claude in config.yaml.
• foundry: set FOUNDRY_API_KEY, flip backend: foundry, and set foundry.endpoint to https://<resource>.services.ai.azure.com/anthropic (or a gateway with that same Messages contract). Set foundry.model to the deployment name. This adapter supports API-key auth, not Entra token acquisition; Entra-only deployments need a different client.

Architecture

screenpipe daemon (user-installed)
        │  HTTP on localhost:3030
        ▼
scripts/fetch_window.py    → normalized timeline events
scripts/redact.py          → regex scrub (defense-in-depth)
scripts/cluster.py         → sessions + clusters (local only)
scripts/match_skills.py    → top-k vs discovered skills (local embeddings)
scripts/synthesize.py      → LLM judge: reuse / compose / novel
        │
        ▼
~/.autoskill/proposed/<timestamp>/        (default; override with --out)
  ├── report.md
  ├── composition-recipes/<name>/SKILL.md
  └── new-skills/<name>/SKILL.md

scripts/promote.py         → user-approved proposal → skills/<name>/

Workflow

The skill ships a unified CLI at scripts/autoskill.py with three subcommands:

python skills/autoskill/scripts/autoskill.py doctor --config skills/autoskill/config.yaml --skills-dir skills
python skills/autoskill/scripts/autoskill.py run --start <ISO-start> --end <ISO-end> --config skills/autoskill/config.yaml
python skills/autoskill/scripts/autoskill.py promote --proposed <proposal-dir> --skills-dir skills --name <skill>

0. Preflight with doctor

Before a full run, check connectivity and backend configuration:

python skills/autoskill/scripts/autoskill.py doctor \
  --config skills/autoskill/config.yaml \
  --skills

… (full content is in the download package)