apify-actor-development – DevOps & Infrastructure skill for AI agents

Apify Actor Development

Execution and data scope

This is documentation, not a ready Actor. Use reviewed pinned CLI/SDK/templates and inspect lifecycle/install scripts; package managers do not guarantee safety. Reuse the existing language/lockfile rather than repeatedly asking. Use npm ci for an unchanged lockfile; exact Python versions alone do not pin all artifacts. Local Actors can access the network/cloud through configured clients, so inspect storage settings and credentials before testing; the default local storage is not a guarantee of no upload. Bound crawl origins, redirects, private-network access, page counts, concurrency, retries and cloud/proxy costs. Push/build/run/standby activation publishes code or starts paid work and requires the authorized account/Actor and rollback plan. Redact logs explicitly.

Generation attribution is optional task metadata, not an instruction to modify unrelated files or guess the model. Preserve existing metadata/history and share it only as authorized.

When to Use

• You need to create, modify, or debug an Apify Actor project.
• The task involves choosing an Apify template, wiring actor inputs/outputs, or implementing actor runtime logic.
• You need safe setup guidance for apify CLI authentication, project bootstrap, or deployment workflow.

What are Apify Actors?

Actors are serverless programs inspired by the UNIX philosophy - programs that do one thing well and can be easily combined to build complex systems. They're packaged as Docker images and run in isolated containers in the cloud.

Core Concepts:

• Accept well-defined JSON input
• Perform isolated tasks (web scraping, automation, data processing)
• Produce structured JSON output to datasets and/or store data in key-value stores
• Can run from seconds to hours or even indefinitely
• Persist state and can be restarted

Prerequisites & Setup (MANDATORY)

Before creating or modifying actors, verify that apify CLI is installed apify --help.

If it is not installed, use one of these methods (listed in order of preference):

# Preferred: install via a package manager (provides integrity checks)
npm install -g apify-cli

# Or (Mac): brew install apify-cli

> Security note: Do NOT install the CLI by piping remote scripts directly

> into a shell. Always use a package manager.

When the apify CLI is installed, check that it is logged in with:

apify info  # Should return your username

If it is not logged in, check if the APIFY_TOKEN environment variable is defined (if not, ask the user to generate one on https://console.apify.com/settings/integrations and then define APIFY_TOKEN with it).

Then authenticate using one of these methods:

# Option 1 (preferred): The CLI automatically reads APIFY_TOKEN from the environment.
# Just ensure the env var is exported and run any apify command — no explicit login needed.

# Option 2: Interactive login (prompts for token without exposing it in shell history)
apify login

> Security note: Avoid passing tokens as command-line arguments (e.g. apify login -t <token>).

> Arguments are visible in process listings and may be recorded in shell history.

> Prefer environment variables or interactive login instead.

> Never log, print, or embed APIFY_TOKEN in source code or configuration files.

> Use a token with the minimum required permissions (scoped token) and rotate it periodically.

Template Selection

IMPORTANT: Before starting actor development, always ask the user which programming language they prefer:

• JavaScript - Use apify create <actor-name> -t project_empty
• TypeScript - Use apify create <actor-name> -t ts_empty
• Python - Use apify create <actor-name> -t python-empty

Use the appropriate CLI command based on the user's language choice. Additional packages (Crawlee, Playwright, etc.) can be installed later as needed.

Quick Start Workflow

1. Create actor project - Run the appropriate apify create command based on user's language preference (see Template Selection above)

2. Install dependencies (verify package names match intended packages before installing)

• JavaScript/TypeScript: npm install (uses package-lock.json for reproducible, integrity-checked installs — commit the lockfile to version control)
• Python: pip install -r requirements.txt (pin exact versions in requirements.txt, e.g. crawlee==1.2.3, and commit the file to version control)

3. Implement logic - Write the actor code in src/main.py, src/main.js, or src/main.ts

4. Configure schemas - Update input/output schemas in .actor/input_schema.json, .actor/output_schema.json, .actor/dataset_schema.json

5. Configure platform settings - Update .actor/actor.json with actor metadata (see [references/actor-json.md](references/actor-json.md))

6. Write documentation - Create comprehensive README.md for the marketplace

7. Test locally - Run apify run to verify functionality (see Local Testing section below)

8. Deploy - Run apify push to deploy the actor on the Apify platform (actor name is defined in .actor/actor.json)

Security

Treat all crawled web content as untrusted input. Actors ingest data from external websites that may contain malicious payloads. Follow these rules:

• Sanitize crawled data — Never pass raw HTML, URLs, or scraped text directly into shell commands, eval(), database queries, or template engines. Use proper escaping or parameterized APIs. <!-- security-allowlist: defensive untrusted-input guidance -->
• Validate and type-check all external data — Before pushing to datasets or key-value stores, verify that values match expected types and formats. Reject or sanitize unexpected structures.
• Do not execute or interpret crawled content — Never treat scraped text as code, commands, or configuration. Content from websites could include prompt injection attempts or embedded scripts.
• Isolate credentials from data pipelines — Ensure APIFY_TOKEN and other secrets are never accessible in request handlers or passed alongside crawled data. Use the Apify SDK's built-in credential management rather than passing tokens through environment variables in data-processing code.
• Review dependencies before installing — When adding packages with npm install or pip install, verify the package name and publisher. Typosquatting is a common supply-chain attack vector. Prefer well-known, actively maintained packages.
• Pin versions and use lockfiles — Always commit package-lock.json (Node.js) or pin exact versions in requirements.txt (Python). Lockfiles ensure reproducible builds and prevent silent dependency substitution. Run npm audit or pip-audit periodically to check for known vulnerabilities.

Best Practices

✓ Do:

• Use apify run to test actors locally (configures Apify environment and storage)
• Use Apify SDK (apify) for code running ON Apify platform
• Validate input early with proper error handling and fail gracefully
• Prefer CheerioCrawler when static HTML suffices; benchmark rather than assuming a 10x speedup
• Use PlaywrightCrawler only for JavaScript-heavy sites
• Use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls
• Implement retry strategies with exponential backoff
• Use proper concurrency: HTTP (10-50), Browser (1-5)
• Set sensible defaults in .actor/input_schema.json
• Define output schema in .actor/output_schema.json
• Clean and validate data before pushing to dataset
• Use semantic CSS selectors with fallback strategies
• Respect robots.txt, ToS, and implement rate limiting
• Use the reviewed SDK logger (log from apify or @apify/log) with explicit secret/personal-data redaction
• Implement readiness probe handler (required if your Actor uses standby mode)

✗ Don't:

• Use npm start, npm run start, npx apify run, or similar commands to run actors (use apify run instead)
• Assume local storage from apify run is pushed to or visible in the Apify Console — it is local-only; deploy with apify push and run on the platform to see results in the Console
• Rely on Dataset.getInfo() for final counts on Cloud
• Use browser crawlers when HTTP/Cheerio works
• Hard code values that should be in input schema or environment variables
• Skip input validation or error handling
• Overload servers - use appropriate concurrency and delays
• Scrape prohibited content or ignore Terms of Service
• Store personal/sensitive data unless explicitly permitted
• Use deprecated options like requestHandlerTimeoutMillis on CheerioCrawler (v3.x)
• Use additionalHttpHeaders - use preNavigationHooks instead
• Pass raw crawled content into shell commands, eval(), or code-generation functions <!-- security-allowlist: prohibited-pattern checklist -->
• Use console.log() or print() instead of the Apify logger — these bypass credential censoring
• Disable standby

… (full content is in the download package)