references/ai-framework-package.md
# AI framework package integrations
Design guide for building a PyPI/npm package that exposes Apify to an AI/LLM framework - LangChain, LlamaIndex, Haystack, Vercel AI SDK, or similar. These are *client-side* integrations: code that calls Apify Actors from outside the Apify runtime, for applications, agents, and RAG pipelines. Apply the cross-cutting rules from `SKILL.md` on top.
## 1. Scope: client-side only, wrap apify-client, never the Actor SDK
This package is for applications that call Apify Actors from outside the Apify runtime. It is **not** for code running *inside* an Actor. Apify Actors should run with limited permissions and use scoped tokens via the Actor SDK's `Actor.open_dataset()`; importing a framework client that reconstructs its own `ApifyClient` from an env-var token would bypass that scoping and pull an unnecessary dependency into Actor images.
Dependency philosophy: wrap the official `apify-client` library, never the `apify` SDK. `apify` is for *building* Actors; `apify-client` is for *calling* them. Keep the runtime dependency surface minimal (`langchain-core`, `apify-client`, and a backport if needed) to minimize version conflicts and keep install time short in agent environments.
Stamp a custom `user-agent` suffix (e.g. `; Origin/langchain`) or the attribution header on the client so Apify can attribute traffic.
## 2. Layered architecture: client -> framework adapters -> public API
```
Public API curated exports
|
+--------------------+--------------------+
| | |
Tools Document loaders Retriever
(agents) (RAG ingestion) (RAG retrieval)
| | |
ApifyToolsClient (sync)
|
apify-client (sync + async)
|
Apify REST API
```
| Layer | Role |
|---|---|
| **Client** | Thin, synchronous wrapper over `apify-client`. One method per Actor operation. No framework types here. |
| **Tools** | Framework `BaseTool` subclasses for agent tool-calling. |
| **Document loaders** | `BaseLoader` implementations for RAG ingestion. |
| **Retriever** | `BaseRetriever` for RAG query-time retrieval. |
Framework types live only above the client layer. The client layer speaks pure Python/JS dicts and `apify-client` objects. This lets the client be unit-tested with no framework dependency, and lets the framework-facing layers focus exclusively on schema, tool semantics, and envelope formatting.
## 3. ApifyToolsClient: one sync gateway, typed method per Actor
All Actor interaction goes through a single synchronous client class with one convenience method per supported Actor (e.g. `google_search`, `instagram_scrape`, `crawl_website`). Each method:
- Builds the Actor-specific `run_input` dict, translating from the integration's normalized parameter names to the Actor's raw input schema. (Actor schemas are idiosyncratic - `searchStringsArray`, `directUrls`, `detailsUrls` vs `listingUrls`; the client absorbs that so the tool exposes clean names like `query`, `url`, `url_type`.)
- Calls `client.actor(id).call(...)` which blocks until the run finishes.
- Checks run status and raises if the run did not reach `SUCCEEDED` (a failed run must never silently return empty results).
- Returns a `(run_details, items)` tuple (or just one where appropriate).
**Why blocking?** Callers don't manage polling loops; the API stays simple. The async surface is handled at the framework layer (`asyncio.to_thread` / `Promise.resolve`) rather than duplicating every method in async form.
Adding a new Actor tool means adding one client method (input translation + status check) and one tool class (schema + `_run`), not wiring up polling, retries, or async variants.
## 4. Uniform JSON output envelope
All tools return a JSON string of one shape:
```json
{"run": {"run_id": "...", "status": "...", "dataset_id": "...",
"started_at": "...", "finished_at": "..."},
"items": [...]}
```
`run` is `null` for dataset-only tools. An optional `notice` key surfaces out-of-band hints (e.g. an Actor returned demo placeholder data on the free plan). Serialize with `default=str` so non-JSON-native types (datetimes from a `clean=True` deserialiser) never throw mid-tool-call.
A single predictable envelope lets agents parse results with one code path. The `run` metadata gives the agent enough to chain calls - run an Actor with one tool, then fetch the dataset with another using the returned `dataset_id`. End every tool description with "Use only the data returned; do not hallucinate missing fields."
## 5. Safety clamps - defense against LLM-requested extremes
An LLM invoking a tool can request absurd values: 10,000 results, 32 GB of memory, a 1-hour timeout. Clamp every request to **developer-controlled ceilings**:
| Clamp | Default ceiling | Developer max |
|---|---|---|
| `timeout_secs` | 600 s | - |
| `memory_mbytes` | 4,096 MB (snapped to nearest valid power-of-2) | 8,192 MB |
| `items` / `limit` | 1,000 | - |
| `max_crawl_depth` | 5 | - |
Memory is notable: Apify accepts memory only as a power-of-2 (128, 256, 512, ..., 32768). Snap an arbitrary LLM value to the nearest valid step at or below the developer's cap. The default ceiling of 4,096 MB (4 GB) is generous for most Actors but well below the platform max, so LLM-requested extremes are clamped. The developer can raise the ceiling up to 8,192 MB, but an LLM cannot widen it beyond the developer-set value.
Some Actors have runtime limits not declared in their input schema (e.g. a RAG web browser rejects `maxResults > 100` at runtime). These can't be derived by schema introspection - track them by hand as overrides on the specific tool so the clamp enforces the Actor's real ceiling.
The ceilings are *developer-controlled fields* on the tool instance - an application can tighten them further, but the LLM cannot widen them. This makes the integration safe to hand to an autonomous agent without risking runaway compute costs.
## 6. Curated tool subsets, not one monolithic list
Tools are grouped into convenience lists:
| List | Tools | Use case |
|---|---|---|
| Core | Run Actor, get dataset, run+get, web fetch, run task, run task+get | Generic platform primitives |
| Search | Google search, web crawler, RAG web browser, Google Maps, YouTube, e-commerce | Web search & content crawling |
| Social | Instagram, LinkedIn, Twitter/X, TikTok, Facebook | Social media scraping |
Warn explicitly: **don't bind all tools at once.** Most LLMs lose routing accuracy past ~8 tools, so pick the family the agent actually needs. Curated subsets let an agent built for social-media analysis avoid distinguishing among 19 tool descriptions.
## 7. Hand-written tools + dynamic schema for the long tail
Alongside hand-written tools (which get clean schemas and descriptions), ship one dynamic tool that takes an `actor_id` at construction, fetches the Actor's latest default build, and **generates an input model dynamically** from the build's input schema. Prune descriptions to a fixed length; limit properties to `type`, `default`, `prefill`, `enum`.
This covers the long tail of Actors without a dedicated wrapper - you don't need a hand-written tool for every one of Apify's thousands of Actors. The trade-off is a looser schema (the LLM sees the raw Actor input shape) and a network call at construction time.
## 8. Map to the framework's idiomatic surfaces
Implement the framework's actual extension points, all backed by the same client:
| Surface | Base class | Use case |
|---|---|---|
| **Tools** | `BaseTool` | Agent tool-calling (ReAct, LangGraph) |
| **Document loaders** | `BaseLoader` | Batch RAG ingestion (load -> split -> embed -> vector store) |
| **Retriever** | `BaseRetriever` | Query-time web retrieval for RAG chains |
- A **dataset loader** loads an existing dataset by ID and maps each item to a `Document` via a user-supplied mapping function (every Actor's output schema is different, so give the user full control). Implement both eager `load()` and streaming `lazy_load()`.
- A **crawl loader** is an *active* loader: it runs a content crawler on construction, then yields `Document`s with `page_content` (markdown) and `metadata` (`source`, `title`, `crawl_depth`).
- A **search retriever** wraps a search-and-crawl Actor for low-latency interactive RAG; the async path runs the synchronous client off the event loop via `to_thread`.
## 9. Token hygiene
One canonical token parameter/env var (e.g. `apify_token` / `APIFY_TOKEN`). If a legacy name exists (`APIFY_API_TOKEN`), honor it with a `DeprecationWarning` but reject new code that declares it. Centralize the policy in two helpers: one for explicit `__init__` signatures, one for Pydantic `model_validator(mode='before')` hooks. Store the token as a `SecretStr` (excluded from repr and serialization) and never log it.
## 10. Content extraction: markdown-first with defensive fallbacks
When extracting page content from crawling Actors, prefer `markdown` over `text`, with a trailing `or ''` to guarantee a string even when a key is present but null. Follow a fixed fallback order for the source URL: nested `metadata.url` -> `crawledUrl` -> top-level `url`. Tolerate a `metadata` field that is missing or not a dict (some Actor responses surface `null`). Actor output shapes are inconsistent across versions and configurations; centralize one canonical fallback order so the retriever, loaders, and tools all agree on what "the content", "the source URL", and "the title" mean.
**Single known URL -> Web Fetch.** For the "fetch this URL" tool, do not run a crawler with a page limit of 1 - call the Web Fetch Server Actor endpoint described in `SKILL.md`. It is the one client method that is a plain HTTPS request instead of `client.actor(id).call(...)`, so add the Bearer token and the attribution header by hand, and set `run` to `null` in the envelope. Take the content from `markdown`, the source URL from `fetch.loadedUrl`, and the title from `metadata.title`. For the LLM-facing tool: pin `formats` to `["markdown"]`, do not expose `headers` to the model (a fetched page could talk the agent into sending credentials to a host of its choosing), and fetch one URL per call with the content truncated to a developer-controlled cap.
## 11. Error mapping
- **Client layer** raises `RuntimeError` for failed/empty runs and `ValueError` for invalid input. Wrap transport errors in `RuntimeError`.
- **Tool layer** catches both and re-raises as the framework's tool-error type (e.g. `ToolException`) with `handle_tool_error = True`, which surfaces to the agent as a recoverable error message.
An agent that gets a `ToolException` can read the message and retry with corrected input. An unhandled `RuntimeError` would crash the agent loop. The boundary is clean: the client raises domain errors; the tool adapts them to the framework's tool-error protocol.
## 12. Packaging, release, and quality bar
- Minimal runtime deps; an explicit sdist allowlist so local-only paths (`dist/`, `.venv/`, `docs/`, test fixtures) never reach the registry.
- Release via conventional-commits-driven automation that reads commit-message prefixes to auto-generate the changelog and compute the version bump; a `BREAKING CHANGE:` footer triggers a major bump. Never hand-edit `version =` or `CHANGELOG.md` if the workflow manages them.
- Strict linting (`select = ["ALL"]` with a curated ignore list), strict typing (`disallow_untyped_defs`), and **socket-disabled unit tests** so the unit suite is truly unit - no hidden integration dependencies. Integration tests need a real token (CI only).
## 13. Position vs the Apify MCP server
The README's top banner should direct users to Apify's MCP server (`https://mcp.apify.com`) as a richer, more featureful alternative for interactive agent workflows that need dynamic Actor discovery. The package is not deprecated, but the MCP path is recommended for new interactive agent sessions.
The positioning: the package is the **programmatic, typed, registry-installable** option for code that outlives a single agent session (servers, scheduled jobs, pipelines); the MCP server is the **interactive, dynamic** option. Rather than compete, position them for their respective audiences.
## Definition-of-done checklist
- [ ] Package is client-side only; depends on `apify-client`, never `apify`.
- [ ] Layered: thin client (no framework types) -> framework adapters -> curated public API.
- [ ] One synchronous client with a typed method per supported Actor; input normalization centralized.
- [ ] All tools return the uniform JSON envelope; serialization never throws on non-native types.
- [ ] Developer-controlled safety clamps (timeout, memory power-of-2, items, depth) are in place; hand-tracked runtime limits override specific tools.
- [ ] Tools grouped into curated subsets; documentation warns against binding all at once.
- [ ] A dynamic-schema tool covers the long tail of Actors.
- [ ] Framework surfaces (tools / loaders / retriever) all backed by the same client.
- [ ] One canonical token name; legacy alias emits a deprecation warning; token is `SecretStr`, never logged.
- [ ] Content extraction is markdown-first with documented fallback order; the single-URL tool uses Web Fetch with `formats` pinned and no `headers` parameter.
- [ ] Client raises domain errors; tools adapt them to the framework's tool-error protocol.
- [ ] sdist allowlist excludes local paths; release automation drives versioning.
- [ ] Unit tests are socket-disabled; lint/typing are strict.
- [ ] README cross-references the MCP server for interactive/dynamic use.
- [ ] Attribution header / user-agent suffix is set on the client; skill-origin header included if built from this skill.references/ai-harness-plugin.md
# AI agent plugin integrations
Design guide for building an Apify plugin that gives an AI agent access to Actors. There are **two plugin shapes**, and which one you build depends on the host:
- **Approach A - Coding agent plugin (skills + MCP bundle):** for skills/MCP-aware coding assistants like Cursor, Claude Code, Codex, and GitHub Copilot. You assemble a small set of runtime artifacts the host already knows how to load, and the hosted Apify MCP server (`https://mcp.apify.com`) provides the tool surface. Minimal code.
- **Approach B - Harness / assistant plugin (custom tool registry):** for agent runtimes like OpenClaw-style runtimes and Hermes-style harnesses that have their own tool registry and config file. You build a small custom toolset (`discover` / `start` / `collect`) backed by the `apify-client` SDK, using a stored credential rather than per-session OAuth.
Apply the cross-cutting rules from `SKILL.md` on top of either approach.
## Which approach? Trade-offs
| Dimension | A - Coding agent plugin (skills + MCP) | B - Harness / assistant plugin (custom registry) |
|---|---|---|
| Target hosts | Cursor, Claude Code, Codex, GitHub Copilot | OpenClaw-style runtimes, Hermes-style harnesses, custom tool-calling agents |
| Tool surface | Hosted Apify MCP server - no tool code to write | You implement `discover` / `start` / `collect` yourself |
| Auth | OAuth (MCP) / `apify login` (CLI) / `APIFY_TOKEN` (SDK), per route | Stored API key resolved by the plugin, passed to `apify-client` |
| Build effort | Low - assemble artifacts, no HTTP/retry/registry code | Higher - tools, schema, error taxonomy, host gotchas |
| Capabilities | Run existing Actors **and** build/test/deploy new Actors **and** integrate an app | Broker the Store to the agent (run existing Actors) |
| Maintenance | The MCP server owns the runtime surface | You own the tool code plus host-SDK compatibility |
| Best when | The host supports skills/MCP and you want the fastest path | The host has its own registry and needs bespoke tools |
If the host is a skills/MCP-aware coding tool, prefer Approach A. If the host is a custom harness with its own registry (no MCP), use Approach B. A product can ship both over time - start with whichever matches the primary host.
---
# Approach A - Coding agent plugin (skills + MCP bundle)
For skills/MCP-aware coding assistants (Cursor, Claude Code, Codex, GitHub Copilot). This section describes the **installed plugin** - what the user gets and how the pieces interact at runtime - not how the bundle is produced.
## A.1 The four runtime artifacts
| Artifact | Role at runtime |
|---|---|
| **MCP server** | Registers `https://mcp.apify.com` with the host and exposes the callable tool surface (below). OAuth: the user signs in via browser on the first tool call that needs auth - no token in config. |
| **Skills** | On-demand `SKILL.md` instruction documents. The host matches each skill's `description` against user intent and loads the body into context only when relevant, keeping baseline context small. |
| **Subagent / router** | The entry point. Classifies the request into one of three routes, selects the transport (MCP vs CLI), and invokes the matching skill or tools. |
| **Slash commands** | User-invoked entry points (e.g. `/create-actor <description>`) that drive a guided end-to-end workflow. |
**MCP tool surface** once connected: `search-actors` (search the Store), `fetch-actor-details` (input schema, output format, pricing), `call-actor` (run with input JSON), `get-actor-run` (poll status), `get-dataset-items` (fetch results), `search-apify-docs` / `fetch-apify-docs` (docs). The discovery subset (`search-actors`, `fetch-actor-details`, `search-apify-docs`, `fetch-apify-docs`) works without an account.
## A.2 The three routes the plugin serves
The router classifies every request and routes it:
| Signal | Route | How it runs |
|---|---|---|
| Use existing Actors (search, run, get data) | 1 | MCP tools directly; the CLI is the fallback when MCP is unavailable |
| Build / test / deploy a custom Actor | 2 | Apify CLI (`apify create` / `run` / `push`) - local filesystem, no MCP equivalent |
| Add Apify to an existing app | 3 | `apify-client` over HTTPS - neither MCP nor CLI |
**MCP-vs-CLI selection (Route 1 only).** Detect transports once: MCP is available if a tool named `search-actors` is in the tool list; CLI is available if `apify --help` exits 0. Prefer MCP when present (no shell/install friction, OAuth auth); fall back to the CLI otherwise. Routes 2 and 3 are unaffected.
**Naming trap.** The `apify` npm package is the **SDK for building** Actors (Route 2). The `apify-client` package is the **API client for calling** Actors (Route 3). Never confuse them.
**Auth per route:** Route 1 (MCP) OAuth via browser prompt, never ask for a token; Route 1 CLI fallback + Route 2 `apify login --token <TOKEN>` once (the CLI ignores `APIFY_TOKEN`); Route 3 the `APIFY_TOKEN` env var.
## A.3 Definition of done (Approach A)
- [ ] MCP declared and reachable - `https://mcp.apify.com` registered and its tools appear in the tool list.
- [ ] OAuth works - the first auth-requiring MCP call prompts a browser sign-in; no token in config.
- [ ] Skills load by intent - each skill's `description` matches its requests; bodies load only when relevant.
- [ ] Router classifies correctly - requests land on Route 1 / 2 / 3; ambiguous ones ask the user to choose.
- [ ] Transport selection correct - Route 1 prefers MCP and falls back to CLI cleanly; Routes 2/3 use CLI/SDK.
- [ ] Auth wired per route; the `apify` vs `apify-client` distinction is never confused.
- [ ] Cost caps honored (`maxTotalChargeUsd` / `maxItems`) and attribution headers set (see `SKILL.md`).
- [ ] Slash command (e.g. `/create-actor`) runs end to end.
- [ ] Verified inside the actual target tool, not just in isolation.
---
# Approach B - Harness / assistant plugin (custom tool registry)
For agent runtimes with their own tool registry (OpenClaw-style runtimes, Hermes-style harnesses, or any custom tool-calling agent). The plugin brokers the entire Apify Store to the agent - it does not bundle scrapers.
The harness runs locally/persistently, has its own tool registry and config file, and calls Apify with a stored credential rather than per-session OAuth. So this shape borrows from the "API token + apify-client" path, not the MCP path.
## 1. Shape decision: few composable tools vs a dynamic tool list
Decide based on what the harness supports:
- **Static tool registry** (tools registered once at plugin load, no per-Actor materialization): register a **small, fixed set of composable tools** and let the LLM compose them. This keeps the prompt budget small and the call graph legible.
- **Dynamic tool registration** (the harness can materialize tools at runtime): you *can* expose a dynamic per-Actor tool list, but a fixed trio is still simpler and usually enough.
The MCP server's surface (search / inspect / call / poll / fetch as separate dynamic tools) is one shape. A harness plugin is a different shape - do not copy it blindly.
## 2. Canonical action set: discover / start / collect
Three tools cover the entire workflow and map cleanly to the asynchronous REST flow (`POST /runs` -> poll `GET /actor-runs/{id}` -> `GET /datasets/{id}/items`):
| Tool | Purpose | Why |
|---|---|---|
| **discover** | Search Apify Store by keyword, OR fetch a single Actor's input schema + README by `actorId` | Two modes in one tool: an LLM that just got a list of Actor IDs almost always wants to inspect one next; splitting would double round-trips |
| **start** | Fire-and-forget batch starts (cap batch size, e.g. 10 per call). Accepts cost limiting params (`maxTotalChargeUsd`, `maxItems`) sent as run options, never Actor input | Returns run references (`run_id`, `actor_id`, `default_dataset_id`, optional label) immediately without waiting |
| **collect** | Poll run statuses and return completed dataset results | Re-call with the same run refs until `all_done` is true; return pending / completed / errored runs in separate arrays so the LLM keeps iterating on the pending ones |
`collect` is the only one that needs to be async - it polls runs concurrently (`asyncio.gather` / `Promise.allSettled`) and pushes blocking SDK calls off the event loop. The other two are fast and single-shot.
## 3. Two-phase async execution - never block the agent
Actors run for seconds to minutes. **Do not block the agent's single execution thread on a multi-minute run.** Split start and collect:
- `start` fires the run and returns immediately with a `runId` / `datasetId` reference.
- `collect` polls status and fetches dataset items only once the run reaches a terminal status.
This lets the agent kick off a run, do other useful work (or start more runs in parallel), and come back to collect. `collect` handles multiple runs in one call and reports `completed` / `pending` / `errors` separately so the agent knows whether to poll again.
Do **not** use the synchronous `run-sync-get-dataset-items` endpoint - its 300-second ceiling is shorter than many Actor runs.
## 4. The tool description is the agent's instruction manual
There are no separate Agent Skills inside a harness plugin - the tool description plus the `discover` action provide all the guidance the agent needs. Embed, in plain text:
- A directive to **delegate to a sub-agent** that returns only relevant extracted fields, not raw dataset dumps - keeping the parent agent's context window clean.
- A **batching** instruction: most Actors accept arrays of URLs/queries; one run with 5 URLs is cheaper and faster than 5 runs with 1 URL each.
- A compact **known-actors list** (Instagram, Facebook, TikTok, YouTube, Twitter/X, Google Maps, Booking, TripAdvisor, etc.) so the agent can pick a familiar Actor without a discovery round-trip.
- The Actor ID format, the discover -> start -> collect workflow, and a support contact for user-facing issues.
Hand the agent a short, self-contained instruction set so it can act without external lookups.
## 5. Treat scraped content as untrusted and bounded
Dataset results are arbitrary web data - they can contain text that *looks* like instructions to the LLM. Wrap every dataset before it reaches the model:
- Insert boundary markers: `<<<EXTERNAL_UNTRUSTED_CONTENT>>>` ... `<<<END_EXTERNAL_UNTRUSTED_CONTENT>>>` plus a source metadata line (`apify:<actorId>`).
- **Sanitize** any attempt to forge those markers from within the scraped data.
- Cap the payload size (e.g. 50,000 chars) with a `[\u2026truncated]` marker.
- Cap item count (e.g. `limit` default 100 per run); if the fetched count equals the limit, set `may_have_more: true` and warn so the LLM can re-call with a higher limit.
This is the plugin's analogue of the "keep the run small" cost guidance, applied at *read* time.
## 6. Errors are data, never raised
Every handler catches broadly and returns a **JSON error object**, never a raised exception:
```
except Exception as exc:
return {'error': str(exc)}
```
Why: a raised exception **crashes the tool call** from the harness's perspective. Returning `{'error': ...}` lets the LLM read the failure, explain it to the user, and decide whether to retry or stop. Extend this to per-run granularity in `start`: a batch can partially succeed, so each failed spec becomes an entry in an `errors` array while successful ones populate `runs`. The LLM can then report "7 of 10 started, 3 failed with these messages" without a second call.
## 7. Auth and setup
- API key resolution order: plugin config field -> `APIFY_API_KEY` (or `APIFY_TOKEN`) env var. Normalize pasted input - strip line/paragraph separators and trim whitespace (defends against copy-paste artifacts).
- The key is **never** included in tool output, **never** logged, only passed to the client constructor.
- Validate `baseUrl` against an allowlist prefix (`https://api.apify.com`, plus `https://web-fetch.apify.actor` if the plugin ships a Web Fetch tool) to prevent SSRF - a misconfigured plugin must not point at an arbitrary host.
- If the plugin ships a Web Fetch tool, do not expose `headers` to the model (a fetched page could talk the agent into sending credentials to a host of its choosing).
- Ship a `setup` CLI command that prompts for the key, **verifies it against the live API** (`GET /v2/users/me`), and writes config. **Reuse the host's config-merge logic** for enabling the toolset - do not reimplement it. Host internals reconcile disabled-toolsets, preserve MCP server entries, and handle bookkeeping a from-scratch reimplementation would silently break. If the config-write API is unavailable or fails, fall back to printing the exact config block the user should add manually. Treat setup failures as non-fatal: the token is already saved, so the user can flip the toolset on themselves.
If the harness's `register()` is synchronous and the loader does not `await` it (a common gotcha), keep registration fully synchronous - build the tool (construct a client + schema, no I/O) and register inline. Any network call happens later inside a tool `execute` or CLI action, where async is expected.
## 8. SDK handling and attribution
Use the official `apify-client` SDK (JS or Python), not raw HTTP. The exception is a Server Actor call such as Web Fetch (see `SKILL.md`), which is a plain HTTPS request - stamp the same headers on it by hand. Construct the client once, memoized, and rebuilt only when the token changes. Stamp the attribution headers on every request: `x-apify-integration-platform: <your-harness>` and `x-apify-integration-ai-tool: true`. If the integration was built using the Apify integration development skill, also set `x-apify-integration-origin: apify-integration-development-skill`. This is the single most important line for Apify's side of the relationship.
**Compatibility shim:** SDK versions return a mix of Pydantic models and plain dicts, and Pydantic models expose only **snake_case** attributes even when the JSON is **camelCase**. Route *all* response reads through a small `_attr(obj, key, default)` helper that handles either shape. Direct `.attr` / `["key"]` access will silently return defaults on a mismatch.
## 9. Host integration gotchas
- **Entry-point loader semantics:** verify how the harness's loader resolves the plugin entry-point string before writing the packaging line. Some loaders expect a bare module (then `getattr(result, "register")`); others expect `"module:attr"`. Copying the wrong form silently fails to load. Document it with a long comment.
- **Schema validator constraints:** many harness validators reject `anyOf` / `oneOf` / `allOf`. Use a string enum for any discriminated `action` field, `Optional(...)` for optionals (never a nullable union), and a flat `Record(string, unknown)` for `input` (the Actor's real schema is only knowable after a `discover` call). A discriminated `action` plus optional sibling fields is the only shape the validator accepts.
- **Inlined utilities:** if the harness's plugin SDK does not export small helpers (error types, secret normalization, content wrapping), inline stable copies rather than deep-importing internals. Internal file layouts change frequently; deep imports couple the plugin to them. Accept the trade-off that upstream bug fixes won't track.
## 10. Actor ID format: tilde, not slash
Use `username~actor-name` everywhere an Actor ID appears: tool descriptions, `discover` results, `start`/`collect` payloads. The REST API uses `/` as a path delimiter, so a slash-separated ID in a URL is ambiguous. The tilde form is unambiguous and what the SDK and Store APIs accept directly. Build slugs in this form so the agent can pass them straight through to `start` without transformation.
## 11. Dependency injection for tests
The tool factory should accept an optional injected `client`. When omitted, construct a real client from the resolved key; when provided (in tests), bypass it entirely. This lets the test suite exercise every action and edge case - store search, schema fetch, run start, collect success/pending, unknown action, missing key - with no network access, using a hand-rolled mock shaped to the SDK's method-chain surface. No mocking framework needed.
## 12. Known gaps to design for
1. **Poll vs webhook.** `collect` is an LLM-driven poll loop; long-running Actors mean multiple round-trips. A webhook-backed `collect` would be cheaper but requires the harness to expose a callback surface.
2. **Account-free discovery.** If the harness's `check_fn` gates all tools on a token, `discover` requires an account even for research. Consider giving `discover` a separate, looser check so users can browse before connecting.
3. **Surface scope.** Only the basic run-start -> poll -> fetch-dataset flow is exposed. Standby runs, Tasks, and schedules may be out of scope for v0.1 - document the boundary.
## Definition-of-done checklist (Approach B)
- [ ] Fixed, small set of composable tools (`discover` / `start` / `collect`) registered; dynamic list only if the harness truly supports it.
- [ ] Two-phase async: `start` returns refs, `collect` polls; no blocking on long runs.
- [ ] Tool description carries known-actors list, batching instruction, and delegation directive.
- [ ] Dataset output is untrusted-content fenced, size-capped, and marker-sanitized.
- [ ] Errors are returned as data, never raised; partial batch failures are per-item.
- [ ] Setup command verifies the token, reuses host config-merge, and has a manual fallback.
- [ ] Attribution headers (`-platform`, `-ai-tool`, and `-origin`) are set on the client.
- [ ] All SDK response reads go through a compatibility shim.
- [ ] Entry-point loader semantics verified; `register()` is synchronous if the loader does not await.
- [ ] Schema uses string enums + `Optional`, no `anyOf`/`oneOf`; `input` is a record.
- [ ] Actor IDs use the tilde form in all user/agent-facing surfaces.
- [ ] Tool factory accepts an injected client; tests run with no network.
- [ ] Known gaps (webhook, account-free discovery) are documented, not hidden.
- [ ] Cost cap (`maxTotalChargeUsd` / `maxItems`) is plumbed through as run options on `start`, never Actor input.references/sdk-integration.md
# SDK integrations
Design guide for integrating Apify into an existing application by calling Actors directly with `apify-client` (JS/TS or Python) or the REST API. This is the lightest integration shape: no host platform, no tool registry, no plugin lifecycle - just your code calling Apify as a backend service. Apply the cross-cutting rules from `SKILL.md` on top.
## 1. Wrap apify-client, never the Actor SDK
- **`apify-client`** is the API client for **calling** Actors from your app.
- **`apify`** is the SDK for **building** Actors (wrong package for this use case).
Always install `apify-client`. Never install `apify` for integration work. Keep the dependency footprint small to minimize version conflicts and keep install time short.
Stamp a custom `user-agent` suffix or the attribution header (`x-apify-integration-platform: <your-app>`) on the client so Apify can attribute traffic. If the integration was built using the Apify integration development skill, also set `x-apify-integration-origin: apify-integration-development-skill`.
## 2. Token handling
Get an `APIFY_TOKEN` from **Console > Settings > Integrations** at `https://console.apify.com/settings/integrations`. Account sign-up: `https://console.apify.com/sign-up` (free, no credit card). Store the token in an environment variable or a secrets manager - never hardcoded, never in chat logs or command output, never in URLs (query-string tokens leak through browser history and server logs).
The token is a normal Bearer credential:
```
Authorization: Bearer <APIFY_TOKEN>
```
Use scoped tokens where possible and rotate them periodically.
## 3. Find the right Actor before writing code
Before writing integration code, find the Actor that fits the need. Use the MCP tools if available in your environment:
- `search-actors` - search the Apify Store by keyword (search by platform/product name, not end goal).
- `fetch-actor-details` - get the Actor's input schema, output format, and pricing.
Alternatively, browse `https://apify.com/store`. Append `.md` to any Actor's Store URL to get its docs in markdown (e.g. `https://apify.com/apify/web-scraper.md`). Build input from the schema rather than guessing field names.
## 4. JavaScript / TypeScript
### Install
```bash
npm install apify-client
```
### Synchronous execution (wait for results)
```typescript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('apify/web-scraper').call({
startUrls: [{ url: 'https://example.com' }],
maxPagesPerCrawl: 10,
}, { maxTotalChargeUsd: 5 });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```
`.call()` blocks until the Actor finishes. Use for short-running Actors (under a few minutes). Pass `maxTotalChargeUsd` in the **options** argument - never in the input.
### Asynchronous execution (start and poll)
```typescript
const run = await client.actor('apify/web-scraper').start({
startUrls: [{ url: 'https://example.com' }],
});
// Poll for completion
// Derive waitSecs from the run's timeoutSecs + a grace buffer, never unbounded.
const finishedRun = await client.run(run.id).waitForFinish({ waitSecs: 120 });
// Retrieve results
const { items } = await client.dataset(finishedRun.defaultDatasetId).listItems();
```
Use `.start()` + `.waitForFinish()` for long-running Actors or when you need the run ID immediately.
### Retrieving results
```typescript
// Dataset items (structured data from pushData)
const { items } = await client.dataset(run.defaultDatasetId).listItems({
limit: 100,
offset: 0,
});
// Key-value store (files, screenshots, etc.)
const record = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');
```
### Error handling
```typescript
try {
const run = await client.actor('apify/web-scraper').call(input);
if (run.status !== 'SUCCEEDED') {
const log = await client.log(run.id).get();
throw new Error(`Actor failed with status ${run.status}: ${log}`);
}
const { items } = await client.dataset(run.defaultDatasetId).listItems();
} catch (error) {
if (error.type === 'record-not-found') {
// Actor ID is wrong or Actor was deleted
} else if (error.statusCode === 401) {
// Invalid or missing APIFY_TOKEN
}
throw error;
}
```
## 5. Python
### Install
```bash
pip install apify-client
```
### Synchronous execution
```python
from decimal import Decimal
from apify_client import ApifyClient
import os
client = ApifyClient(token=os.environ['APIFY_TOKEN'])
run = client.actor('apify/web-scraper').call(
run_input={
'startUrls': [{'url': 'https://example.com'}],
'maxPagesPerCrawl': 10,
},
max_total_charge_usd=Decimal('5'),
)
items = client.dataset(run['defaultDatasetId']).list_items().items
```
### Asynchronous execution
```python
run = client.actor('apify/web-scraper').start(run_input={
'startUrls': [{'url': 'https://example.com'}],
})
# Poll for completion
finished_run = client.run(run['id']).wait_for_finish()
items = client.dataset(finished_run['defaultDatasetId']).list_items().items
```
### Async client (asyncio)
```python
from apify_client import ApifyClientAsync
client = ApifyClientAsync(token=os.environ['APIFY_TOKEN'])
run = await client.actor('apify/web-scraper').call(run_input={
'startUrls': [{'url': 'https://example.com'}],
})
items = (await client.dataset(run['defaultDatasetId']).list_items()).items
```
## 6. REST API (any language)
For languages without an official client, use the REST API directly.
### Start a run
```
POST https://api.apify.com/v2/actors/{actorId}/runs
Authorization: Bearer <APIFY_TOKEN>
Content-Type: application/json
{ "startUrls": [{ "url": "https://example.com" }] }
```
### Get run status
```
GET https://api.apify.com/v2/actor-runs/{runId}
Authorization: Bearer <APIFY_TOKEN>
```
### Get dataset items
```
GET https://api.apify.com/v2/datasets/{datasetId}/items?format=json
Authorization: Bearer <APIFY_TOKEN>
```
The cost rule applies on the HTTP paths too: `maxItems` (pay-per-result) and `maxTotalChargeUsd` (other pricing models) go as **query parameters**, never in the JSON body where they are read as Actor input.
For runs expected to finish within 300 seconds, the synchronous endpoint returns dataset items directly:
```
POST https://api.apify.com/v2/actors/{username}~{actor-name}/run-sync-get-dataset-items
```
Longer work must use the asynchronous flow: POST /runs -> poll GET /actor-runs/{id} -> GET /datasets/{id}/items.
REST reference: `https://docs.apify.com/api/v2`. OpenAPI spec: `https://apify.com/openapi.json`.
## 7. Best practices
- **Set timeouts:** pass `timeoutSecs` as a run option / query parameter on `.call()` or `.start()`, or use `waitSecs` on `.call()`. Never put `timeoutSecs` in Actor input — it is a run option and an Actor whose schema rejects unknown fields will fail on it.
- **Paginate large datasets:** use `limit` and `offset` when retrieving dataset items.
- **Reuse clients:** create one `ApifyClient` instance and reuse it across calls.
- **Handle Actor-specific input:** every Actor has its own input schema. Use `fetch-actor-details` MCP tool or append `.md` to the Actor's Store URL to get the schema before constructing input.
- **Bound polling:** never `while (true)` - use the run's own timeout + a grace buffer, with an absolute ceiling.
## 8. Documentation pointers
- API client for JS: `https://docs.apify.com/api/client/js`
- API client for Python: `https://docs.apify.com/api/client/python`
- REST API reference: `https://docs.apify.com/api/v2`
- Apify docs (LLM-friendly): `https://docs.apify.com/llms.txt`
- Apify docs (full): `https://docs.apify.com/llms-full.txt`
If the Apify MCP server is available, use `search-apify-docs` and `fetch-apify-docs` tools for contextual documentation lookups during development.
## Definition-of-done checklist
- [ ] Depends on `apify-client`, never `apify`.
- [ ] Token stored in env var / secret manager; never hardcoded or logged.
- [ ] Actor input built from the schema (MCP `fetch-actor-details` or `.md` URL), not guessed.
- [ ] Sync `.call()` used for short runs; async `.start()` + `.waitForFinish()` for long ones.
- [ ] Attribution header / user-agent suffix set on the client; skill-origin header included if built from this skill.
- [ ] Cost cap (`maxTotalChargeUsd` / `max_total_charge_usd`) passed in options, never input.
- [ ] Run status checked before consuming dataset; failed runs raise, not return empty.
- [ ] Dataset and KV store retrieval covered; pagination on large datasets.
- [ ] Errors mapped to actionable app-level messages.
- [ ] REST API fallback documented for languages without a client.references/workflow-automation.md
# Workflow automation integrations
Design guide for integrating Apify into a workflow-automation platform (Zapier, n8n, Make, Pipedream, Activepieces, or a similar trigger/action/search host). This file is **UX-design focused and platform-agnostic**: it describes the capability surface, the resource model, and the user-facing behavior, not the host platform's internal build mechanics, release pipeline, or secrets. Apply the cross-cutting rules from `SKILL.md` on top.
The host platform's model is rigid: every Apify capability must be expressed as one of **trigger**, **action (create)**, or **search (read)**. Decide the mapping before writing any code.
## 1. Map Apify to the host's trigger / action / search model
| Host type | Apify capability | Purpose |
|---|---|---|
| **Trigger** (hook) | Actor / Task run finished | Event-driven: start a workflow when a run reaches a terminal status |
| **Trigger** (hidden, polling) | List Actors / Tasks | Back dynamic dropdowns - not user-facing steps |
| **Action (create)** | Run Actor; Run Task; Run Actor + get dataset; Run Task + get dataset; Web Fetch; Set KV record | Synchronous operations that produce or store data |
| **Search (read)** | Get last run; Get run; List runs; Get dataset items; Get KV record | Find existing records, optionally branch on them |
Reads of the last run or stored data are **searches**, because that is the host's mechanism for "find an existing record." Run-finished is a **webhook trigger**. Actor/Task *selection* is a **hidden trigger** that feeds dropdowns - reusing one hidden trigger across multiple actions keeps the surface DRY.
## 2. Canonical capability matrix
The complete set of operations a workflow integration should surface. Treat this as the floor, not a menu to trim without reason.
**Triggers**
- Actor run finished (webhook; user picks terminal statuses: SUCCEEDED / FAILED / TIMED-OUT / ABORTED)
- Task run finished (same shape, scoped to a Task)
**Actions (creates)**
- Run Actor - fire an Actor run
- Run Actor and get dataset items - fire and return results inline
- Run Task - fire a saved Task configuration
- Run Task and get dataset items - fire and return results inline
- Web Fetch - curated URL-in, content-out action over the Web Fetch Server Actor (see section 13)
- Set key-value store record - write a file/string to a KV store
**Searches (reads)**
- Get last Actor/Task run - the most recent matching run
- Get run - by run ID
- List runs - filterable by status
- Get dataset items - by dataset ID, with offset/limit and optional fields/omit
- Get key-value store record - by store ID + record key
**Hidden selectors (back dropdowns, not user steps)**
- List Actors (recently used source + Store source)
- List Tasks
If the host platform cannot represent every operation, prioritize in this order: run-finished trigger, run Actor + get dataset, get dataset items, Web Fetch, run Task + get dataset, get KV record, set KV record.
## 3. Resource -> operation organization
Organize the action node's surface as **resource -> operation**, mirroring both the host platform's UX convention and Apify's domain model. A two-level router (resource, then operation) keeps adding an operation a copy-and-adapt task and co-locates each operation's parameters and I/O. Users navigate Actor -> "Run actor", Dataset -> "Get items", etc. Avoid a flat list of dozens of operations.
## 4. Actor / Task selection UX
IDs (Actor, Task, run, dataset, KV store, record key) should use the host's resource-locator property type with multiple modes: **From list**, **By URL**, **By ID**. Extract the ID from a pasted Console URL (e.g. `https://console.apify.com/actors/<id>/input` -> `<id>`).
Offer **two selection sources** in one selector:
- **Recently used** (default) - the user's own Actors/Tasks, sorted by last run started. This is what most users want.
- **Store** - browse the public marketplace by search term, sorted by popularity.
A toggle with "recompute fields when changed" lets users switch the dropdown source without leaving the node. Choose a pagination limit that balances completeness against load time (the host UI must support searching within dropdown results).
## 5. Dynamic input translation
The highest-leverage UX piece. When a user selects an Actor, fetch the Actor's build and translate its `inputSchema` into the host platform's form fields dynamically. Hardcoding fields per Actor is unmaintainable across thousands of Actors; a translator supports *any* Actor's input as a host form.
Handle the type/editor mappings that have no direct host equivalent, and degrade gracefully rather than crashing:
| Apify type/editor | Host field |
|---|---|
| `string` + `javascript`/`python` editor | code field |
| `string` + `textarea` | multiline text |
| `string` + `datepicker` | datetime |
| `string` + `select` / `enum` / `enumSuggestedValues` | choices dropdown |
| `isSecret: true` | password |
| `array` + `json`/`keyValue` editor | multiline text with a JSON.stringify default |
| `array` + `requestListSources`/`pseudoUrls`/`globs`/`stringList` | flat string list (re-expand at run time) |
| `object` + `proxy` editor | info box telling users to set it in Apify Console (host has no proxy UI) |
| `object` + `schemaBased` | nested children fields |
| `sectionCaption` | helpText/info box (host has no stackable sections) |
Two gotchas:
- On list fields, a `default` would pre-populate a non-removable first item and duplicate entries - move prefill values to `placeholder`.
- Mark the build selector so a build-tag change **recomputes the generated fields**, since input schemas can differ across builds.
## 6. Run UX: sync vs async, cost caps, bounded polling
Offer a **sync/async toggle** on run-start actions:
- Sync: block until the run finishes (document the host's hard timeout so users know when to choose async).
- Async: return immediately with the run ID for long-running jobs.
Use the asynchronous REST flow under the hood (POST /runs with `waitForFinish=0`, then poll). Polling must be **bounded** - use the run's own `timeoutSecs` + a grace buffer, with an absolute ceiling fallback (e.g. 24h). Poll at a fixed interval (e.g. 1s). "Run actor and get dataset" must additionally require `status === 'SUCCEEDED'` before fetching dataset items.
Every run-start action exposes a **Maximum Cost per Run** field (`maxTotalChargeUsd`), `min: 0`, default `null`. Send it as a query parameter only when non-null and `> 0`; `0`/empty means *no limit*. Never let this be an Actor input field.
## 7. Output normalization into one enriched shape
Bare Actor run objects contain only storage IDs, not results - useless downstream. Enrich every run once, centrally, into a single shape returned by all run-producing and run-finding actions:
- The run's `OUTPUT` from the default key-value store (and any user-selected store keys).
- `datasetItems` - up to a sensible cap (e.g. 100 items) from the default dataset.
- `datasetItemsFileUrls` - pre-built download URLs for JSON/CSV/XML/XLSX/HTML/RSS exports.
- `detailsPageUrl` - a deep link to the run in Apify Console.
- Strip fields useless to host users (`meta`, `stats`, `options`, `userId`, raw `output`, `standby`).
Downstream steps map fields by name regardless of which action produced the run, because every action yields the same shape. Describe this shape to the host UI with sampled output fields (sample ~10 items, merge their keys).
## 8. Guard against large datasets
Estimate the full download size before inlining: fetch one item, multiply by the requested item count with a safety margin (e.g. 1.2x), compare to a payload cap (e.g. 10 MB). If it exceeds the cap, **do not attempt the inline download** - return a warning item plus the dataset file URLs. When a trigger fetches more items than the cap allows, push a warning pointing users to the "Get Dataset Items" action with `run.defaultDatasetId` for full results. The host platform has payload/time limits the Apify API does not; always offer a file-download fallback.
## 9. Files and binary records
Map the host's file-handling primitives (dehydration / stashing / signed URLs) to Apify's binary KV records. For "Get KV record":
- JSON/text under a size threshold (e.g. 20 MB) -> parsed and returned as fields the user can map directly.
- Binary or larger records -> a dehydrated pointer the host lazily fetches when a downstream step needs it.
- Records above a hard limit (e.g. 120 MB, with margin below the host's 150 MB ceiling) -> rejected outright with an actionable message.
## 10. Run-finished trigger UX
Use the host's hook type backed by Apify webhooks:
- **Subscribe**: create an Apify webhook scoped to an `actorId` or `actorTaskId`, with `eventTypes` from the user-selected terminal statuses. The webhook's `requestUrl` is the host's target URL. Make registration **idempotent** - derive a key from every field that distinguishes one registration from another (`resource:id:sortedEvents:requestUrl`) so re-activating a workflow does not create duplicate webhooks. Persist the created webhook ID so deactivation can delete it.
- **Unsubscribe**: delete the webhook by its stored ID.
- **Perform**: read the webhook payload and enrich it (section 7).
- **Fallback list**: fetch the 3 most recent matching runs so users see realistic test data when configuring the trigger without waiting for a real event.
Offer an event multi-select (SUCCEEDED / FAILED / TIMED-OUT / ABORTED) plus an "any" shortcut that expands to all four. Build the `condition: { actorId }` or `{ actorTaskId }` from the watched resource. Pass the webhook payload through as workflow data - it already carries the run metadata.
## 11. Error UX
Centralize API error mapping in one place, not per action. Match each Apify error to the host's error category:
- 5xx and 429 -> retryable (so backoff applies).
- `token-not-found` / auth errors -> an authentication error the user must fix.
- `full-permission-actor-not-approved` -> a non-retryable error with the **validated** approval URL in the message (approval is a manual Console action; retrying cannot help).
- 404 on KV record GET/HEAD -> return empty rather than throw (a missing key is a valid "no data" result).
- Other 4xx -> a plain error with the API's actual message, or a generic fallback.
Use retryable errors only when retrying can actually succeed. Keep error *codes* (like `EPERM`) out of any message field the host replaces, so the real text survives.
## 12. Auth UX
For consumer-facing automation platforms, prefer **OAuth2 with PKCE** over an API-token field. Users authorize through a browser; no raw token typing. Disable auto-refresh only if the host cannot surface a refresh failure gracefully - otherwise leave it on. Validate the token with a `GET /v2/users/me` test call and populate the connection label with the username/email.
If the host is headless-only, fall back to an API-token credential with the same verify call on a "Verify" button.
## 13. Convenience operations: "Web Fetch"
Beyond generic "run Actor", ship a curated **Web Fetch** action on the Server Actor endpoint described in `SKILL.md`: a URL in, the page content out. It starts no run, so the run machinery in sections 6-8 (sync/async toggle, cost field, polling, enriched run shape) does not apply to it.
- **Fields**: `url`, `formats` (multi-select of `markdown`, `text`, `html`, `links`, `raw`; default `["markdown"]`), and optional `headers` sent to the target site - leave them out of the body when empty. Do not expose `unwrap`.
- **Before the request**: validate the URL (`new URL()` + `http(s)` protocol + hostname) with an actionable error. Set the client timeout above 120 s, the Actor's budget for one fetch, or just under the host's step limit if that is shorter.
- **Output**: return the response as it is - `url`, `fetch`, `metadata`, and one key per requested format, which can be `null` when that format cannot be produced.
- **Errors**: parse Web Fetch errors before the generic mapping in section 11. A 502 or 504 here describes the target site, not an Apify outage.
- **Migration**: if the integration already ships a crawler-based "Scrape single URL" action, add Web Fetch under a new key and deprecate the old one. Never rename or remove the old key - existing workflows reference it.
## Definition-of-done checklist
- [ ] Capability matrix from section 2 is fully surfaced (or trimmed with documented reason).
- [ ] Resource -> operation organization; no flat operation soup.
- [ ] Actor/Task selection offers recently-used + Store sources with URL/ID extraction.
- [ ] Actor input is translated dynamically from the build schema; degradation is graceful.
- [ ] Run actions expose sync/async toggle and a `maxTotalChargeUsd` field (null = unlimited).
- [ ] Polling is bounded; never `while (true)`.
- [ ] All run-producing and run-finding actions return the one enriched run shape.
- [ ] Large-dataset guard with file-URL fallback is in place.
- [ ] Run-finished trigger is webhook-backed, idempotent, and has fallback sample data.
- [ ] Error mapping is centralized; approval URLs are validated; codes don't clobber messages.
- [ ] OAuth2 PKCE is the default consumer auth path; token fallback has a verify call.
- [ ] A "Web Fetch" convenience action calls the Server Actor endpoint with explicit `formats` and pre-request URL validation; any legacy "Scrape single URL" action keeps its key.
- [ ] `x-apify-integration-platform` header is sent on every outbound request; `x-apify-integration-origin: apify-integration-development-skill` included if built from this skill.
- [ ] Two test modes (mocked + live E2E) pass.SKILL.md
---
name: apify-integration-development
description: Design and build an official Apify integration for a company's product - workflow-automation apps (Zapier/n8n-style), AI agent plugins (coding-agent skills+MCP bundles or OpenClaw/Hermes-style harnesses), AI framework packages (LangChain/LlamaIndex-style), or direct application clients via apify-client. Use when planning, creating, or reviewing an integration that exposes Apify Actors, runs, datasets, or key-value stores inside another product.
---
# Apify Integration Development
Design and build an **official Apify integration** for a company's product, with minimal help from Apify. This skill covers every integration shape Apify supports - workflow-automation apps, AI agent plugins (coding agents and harnesses), AI framework packages, and direct application clients - so a partner team can ship a first-class Apify integration end to end. The cross-cutting rules below apply to all of them, and one category-specific reference file carries the rest.
> **Building an official integration?** Once you publish it, contact **integrations@apify.com** so the Apify team can review, test, and validate your integration before it reaches users. We'll check the capability surface, cost controls, error handling, and attribution headers, and help you close any gaps.
## Step 0 - Learn the Apify model first (required)
Before designing anything, fetch and read `https://apify.com/agents.md`. It is the canonical quickstart for AI agents and the single source of truth for vocabulary, the run flow, and the cost rule. If the fetch fails, the mini-glossary below keeps the skill usable.
Apify vocabulary (always written with a capital A on the platform):
- **Actor** - a serverless cloud program that takes JSON input, performs a task, and produces structured output. Not an AI agent.
- **Actor Run** - one execution of an Actor. Each run has its own dataset, key-value store, and request queue, and ends in a terminal status (`SUCCEEDED`, `FAILED`, `TIMED-OUT`, `ABORTED`).
- **Dataset** - append-only structured storage for a run's results. An Actor call returns the dataset ID, not its contents.
- **Key-Value Store** - unstructured/file storage (screenshots, HTML, OUTPUT).
- **Actor Task** - a saved, parameterized configuration for running an Actor.
- **Apify Store** - the marketplace of Actors at `https://apify.com/store.md`.
- **Apify Console** - the web UI at `https://console.apify.com`.
- **Compute Unit (CU)** - billing unit: memory (MB) x duration (hours).
- **Server Actor** - an Actor that runs as an always-ready HTTP server on its own stable URL (may be still called Standby in the API and docs). You send a request and get the result in the response - no run to start, no dataset to read.
Further terms (build, request queue, proxy, pricing models): `https://docs.apify.com/llms.txt`.
## Use Apify MCP for live context while planning
The Apify MCP server is the fastest way to research Actors, schemas, pricing, and docs during integration design. See `https://docs.apify.com/integrations/mcp` (append `.md` for a markdown version).
If Apify MCP tools are already available in this environment, use them:
- `search-actors` - find Actors by platform/product keyword (search by product name, not end goal).
- `fetch-actor-details` - read an Actor's input schema, output format, README, and pricing before you encode its shape into the integration.
- `search-apify-docs` / `fetch-apify-docs` - pull contextual documentation pages.
The anonymous discovery subset (`search-actors`, `fetch-actor-details`, `search-apify-docs`, `fetch-apify-docs`) works without an account, so you can research even before the developer has connected their token.
## Pick your integration shape
Read exactly one reference file based on the product you are integrating into. Each reference carries the category-specific UX design, a canonical capability matrix, and a definition-of-done checklist.
| Product shape | Examples | Read |
|---|---|---|
| Workflow automation platform | Zapier, n8n, Make, Pipedream, Activepieces | `references/workflow-automation.md` |
| AI agent plugin (coding agent or harness) | Cursor, Claude Code, Codex, GitHub Copilot (coding agents); OpenClaw-style runtimes, Hermes-style harnesses (harnesses) | `references/ai-harness-plugin.md` |
| AI framework package (PyPI/npm for LLM frameworks) | LangChain, LlamaIndex, Haystack, Vercel AI SDK | `references/ai-framework-package.md` |
| Application integration (direct client) | A backend service, scheduled job, product feature calling Actors via `apify-client` or REST | `references/sdk-integration.md` |
Paths are relative to this skill folder. If your product spans two shapes (e.g. an AI harness built on top of a framework package), read both - the rules compose. The AI agent plugin reference covers **two approaches with different trade-offs**: a lightweight skills + MCP bundle for skills/MCP-aware coding agents, and a custom tool-registry plugin for OpenClaw/Hermes-style harnesses.
## Cross-cutting design rules (true for every integration type)
These invariants were extracted from every existing Apify integration. Apply them regardless of shape.
### Vocabulary mirroring
Model the integration's resources on Apify's domain (Actor / Run / Dataset / KV Store / Task). Users coming from Apify Console should find the same concepts under the same names.
### Asynchronous run flow with bounded polling
Actors can run for seconds to hours. Use the asynchronous flow, never the 300-second synchronous endpoint for anything but short jobs:
```
POST /v2/actors/{actorId}/runs -> start, return runId
GET /v2/actor-runs/{runId} -> poll until terminal status
GET /v2/datasets/{datasetId}/items -> fetch results on SUCCEEDED
```
Polling must be **bounded**: use the run's own `timeoutSecs` plus a grace buffer, with an absolute ceiling fallback. Never `while (true)`. On a non-terminal status, surface the run ID so the user/agent can poll again or inspect the failure.
### Cost is first-class
Every path that starts a run must expose a cost control. The canonical control is `maxTotalChargeUsd` (caps the run's total charge on most pricing models) and `maxItems` (caps billed items on pay-per-result Actors). Send them as **options / query parameters**, never as Actor input - inside input they are either an Actor-declared field or simply invalid. `0` / empty / null means *no limit*. For LLM-facing integrations, the ceilings are **developer-controlled**; an LLM cannot widen them. A Server Actor request such as Web Fetch (below) starts no run and takes neither option.
### Attribution headers
Stamp an integration header on every outbound request so Apify can attribute traffic: `x-apify-integration-platform: <your-platform>`. When a request is driven by an AI tool (not a human in a UI), also send `x-apify-integration-ai-tool: true`. If the integration was built using this skill, add `x-apify-integration-origin: apify-integration-development-skill` so Apify can distinguish skill-generated integrations from custom ones. One line, big telemetry payoff.
### Authentication
- Browser / consumer-facing (a human completes a sign-in): OAuth2 with PKCE. Do not ask for raw tokens.
- Headless / server / CI (no human present): API token as `Authorization: Bearer <APIFY_TOKEN>`, stored in an env var or secret manager, never hardcoded or logged.
Both paths are real - pick by who is present at auth time, not by which is easier.
### Centralized HTTP layer
One base-URL constant, shared between credentials and the HTTP layer. A Server Actor lives on its own host (`https://web-fetch.apify.actor`) - route it through the same layer so auth and attribution headers still apply. Retries with exponential backoff on 429 and 5xx. **Never retry non-idempotent `POST /runs` on network errors** - a duplicate Actor run is a real, billed, side-effecting operation. This is the single most important correctness invariant in the HTTP layer.
### Error taxonomy
Map Apify errors to the host platform's error categories (retryable vs auth vs permanent). Surface the API's actual error text, not a generic HTTP message. For permission-approval failures (a full-permission Actor needs explicit approval), include the approval URL after validating it is an absolute `http(s)` URL. For LLM consumers, return errors **as data** (JSON error objects), never as raised exceptions - the model needs something to read and reason about.
### Webhooks over polling for run-finished events
When the host supports inbound webhooks, register an Apify webhook scoped to `actorId` or `actorTaskId` with the terminal statuses the user picked. Make registration **idempotent** (a re-activated workflow should not create duplicate webhooks), persist the webhook ID so deactivation can clean it up, and always provide sample/fallback data so users can test the trigger without waiting for a real run.
### Generate from OpenAPI where the host allows it
If the host platform can generate UI fields from an OpenAPI spec, use Apify's spec (`https://apify.com/openapi.json`) and a tag allowlist. Hand-write only what the spec cannot express: convenience wrappers, bill-cap fields, lean AI-tool output contracts.
### High-level convenience operations alongside generic runs
Generic "run Actor" serves power users. Add a few opinionated, high-level actions for the common case so non-power users get a 2-field form instead of a full Actor configuration. The canonical one is **Web Fetch**: one URL in, page content out, built on the `apify/web-fetch` **Server Actor** - `POST https://web-fetch.apify.actor/` with `{"url": "...", "formats": ["markdown"]}` returns the content in the response. No run is started, so the run rules above (asynchronous flow, polling, `maxTotalChargeUsd`) do not apply; the cost is one `fetch` event per successful request. Three things to get right: always send `formats`, check `fetch.httpStatusCode` (HTTP 200 only means Web Fetch itself succeeded), and handle both error shapes - flat `{code, error}` from the Actor, nested `{error: {message}}` from the platform - without auto-retrying its 502/504. Full contract: `https://apify.com/apify/web-fetch.md`. Do not build this on a crawler run with `maxCrawlDepth: 0` - that is the legacy "Scrape single URL" pattern.
### Testing and release
Keep two test modes: mocked (hermetic, no credentials) and live E2E (real API, CI-gated). Automate releases through the host platform's CI on Git tags / GitHub Releases. Never hand-edit versions or changelogs if a release workflow manages them.
## Top anti-patterns to refuse on review
1. Retrying `POST /runs` on a network error - duplicates a billed run.
2. Unbounded `while (true)` polling - ties up the host with no ceiling.
3. Putting `maxTotalChargeUsd` / `maxItems` inside Actor input instead of options - silently not a cap.
4. Dumping a full dataset into an LLM context without size caps or untrusted-content fencing - prompt-injection and context blowout.
5. One monolithic tool list for an LLM agent - routing accuracy degrades past ~8 tools; curate subsets.
6. Surfacing a raw HTTP status/message instead of Apify's actual error text - users can't act on "400".
## Minimal API surface every integration needs
| Purpose | Method + path |
|---|---|
| Start an Actor run | `POST /v2/actors/{actorId}/runs` |
| Start a Task run | `POST /v2/actor-tasks/{taskId}/runs` |
| Poll a run | `GET /v2/actor-runs/{runId}` |
| List runs | `GET /v2/actor-runs` |
| Dataset items | `GET /v2/datasets/{datasetId}/items` |
| KV record | `GET /v2/key-value-stores/{storeId}/records/{key}` |
| Set KV record | `PUT /v2/key-value-stores/{storeId}/records/{key}` |
| Store search | `GET /v2/store` |
| Webhook CRUD | `POST/GET/DELETE /v2/webhooks` |
| Validate token / current user | `GET /v2/users/me` |
| Fetch one URL (Web Fetch, Server Actor) | `POST https://web-fetch.apify.actor/` |
REST reference: `https://docs.apify.com/api/v2`. OpenAPI spec: `https://apify.com/openapi.json`.
## Working workflow
1. Fetch `https://apify.com/agents.md` and internalize the model.
2. Pick the integration shape above and read the matching reference file.
3. Use Apify MCP (if available) to research the concrete Actors, schemas, and pricing the integration will expose.
4. Draft the **capability matrix** for the chosen category (each reference has one) and the UX spec (resource -> operation -> fields -> errors).
5. Scaffold the integration following the category-specific rules in the reference.
6. Verify against the **definition-of-done checklist** at the end of that reference.
## Reference implementations to study
Real, public integrations per category - read their source when in doubt:
- Workflow automation: `@apify/n8n-nodes-apify` (npm), the Apify Zapier app.
- AI agent plugins (coding agents): the Apify plugin bundle (MCP server + skills + router + slash commands) shipped for Cursor, Claude Code, Copilot, and similar tools.
- AI agent plugins (harnesses): `apify-hermes-agent-plugin` (PyPI), `@apify/apify-openclaw-plugin`.
- AI framework packages: `langchain-apify` (PyPI).
- Application integration: see `references/sdk-integration.md` for the canonical `apify-client` usage in JS/TS, Python, and over REST.
Support for integration questions: `integrations@apify.com`. Contact us both for design guidance while you build and for review/testing once you publish - we validate the capability surface, cost controls, error handling, and attribution before the integration reaches users.