Skip to content

About

Tools & lib to test actors on the Apify platform

Resources

Contributing

Stars

8 stars

Watchers

1 watching

Forks

Latest commit

 

History

228 Commits

Folders and files

Repository files navigation

Apify Test Tools

Build, test and release Apify Actors from a Git repository or local source code. The package has three parts:

  1. A vitest library for writing platform tests that run your Actors on Apify and check the results:

    testActor('myteam/web-scraper', 'scrapes the start URL', async ({ expect, run }) => {
        const runResult = await run({ input: { startUrls: [{ url: 'https://apify.com' }] } });
        await expect(runResult).toFinishWith({ datasetItemCount: { min: 1, max: 10 } });
    });
  2. A CLI that finds the Actors a change affects, builds them, and reports the results. It runs anywhere, so you can build your own pipeline on it in any CI:

    npx apify-test-tools get-affected-actors --base origin/master \
      | jq '{ actors: (.functionallyChangedActors + .testChangedActors) }' \
      | npx apify-test-tools build --version-number 0.99
  3. Ready-made GitHub Actions workflows built on the CLI and the library, so a GitHub repo needs only a few lines per workflow:

    jobs:
        buildDevelAndTest:
            uses: apify/apify-test-tools/.github/workflows/public_pr-build-test.yaml@workflows-v0
            secrets:
                # One entry per secret apify-test-tools.config.json names: each `tokenEnvVar` and `testTokenEnvVar`
                CONFIG_SECRETS: |
                    { "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}" }

Most repos only need the library and the workflows: the workflows call the CLI for you. Use the CLI directly only to build your own pipeline, e.g. on a CI platform other than GitHub, or to build and test locally.

Only what changed is retested

Builds and platform tests cost time and money, so the tool skips everything a change can't affect:

  • Only affected Actors. A changed file counts for Actors whose context contains it, subject to ignore rules, or whose actor.json links it. A shared file linked from actor.json affects every Actor that uses it.
  • Cosmetic changes skip tests. Linked README/changelog edits and cosmetic schema changes don't trigger tests on a PR; they're only rebuilt when released.
  • Only new changes on a PR. Each push to a PR retests only the Actors that changed since its last green run, not the whole PR again.
  • Only their tests. Only the tests of the Actors that were built run.

See Change detection for the exact rules.

This README covers what you need to get a repo running. The full reference lives in docs/:

  • Vitest library API — testActor, testStandbyActor, RunTestResult, matchers, running tests locally
  • CLI API — every command, change detection, releasing, building locally
  • Workflow API — every GitHub workflow, its inputs and secrets, versioning

Contributing

Getting started

Your Actors can be written in any programming language, e.g. Python: anything the platform builds from a Dockerfile works. Only the tests and this tooling run on Node.js, so all a repo needs is a package.json at the root with apify-test-tools and vitest in its devDependencies, and tests written in TypeScript or JavaScript. If you use the PR workflow, also define the build-check, lint and test scripts its static checks run, e.g. with your language's own linter and unit tests.

1. Install the package

npm i -D apify-test-tools
  • Requires vitest version 3.2.4 or later (uses annotate)
  • Make sure target and module in your tsconfig.json's compilerOptions are set to ES2022
  • Every Actor must already have at least one build under its default build tag (usually latest), built from a Git repository version whose URL points to this repository. Build it once manually on the platform before CI takes over. See Repository check.

2. Create the config file

Every repo must have an apify-test-tools.config.json file at the root that lists its Actors. See Config file below.

3. Set up Actor folders

Each Actor in the config must have a .actor/actor.json file. The dockerContextDir field in actor.json defines the build context boundary. Resolved documentation, schema, and Dockerfile links in actor.json also count as dependencies even outside that context (see Change detection).

This example has two Actors that share the code at the repo root, and one standalone Actor with its own code:

my-repo
├── apify-test-tools.config.json
├── Dockerfile                      <- shared by web-scraper and email-sender
├── package.json
├── src/                            <- shared by web-scraper and email-sender
├── actors
│   ├── web-scraper
│   │   └── .actor
│   │       └── actor.json          <- points to the repo root
│   └── email-sender
│       └── .actor
│           └── actor.json          <- points to the repo root
├── standalone-actors
│   └── orchestrator
│       ├── .actor
│       │   └── actor.json          <- default context: its own folder
│       ├── Dockerfile
│       ├── package.json
│       └── src/
└── test
    └── platform
        ├── web-scraper.test.ts
        ├── email-sender.test.ts
        └── orchestrator
            └── orchestrator.test.ts

The Actors that share code point their dockerfile and dockerContextDir at the repo root. Both paths are relative to the .actor folder:

{
    "actorSpecification": 1,
    "name": "web-scraper",
    "version": "0.1",
    "dockerfile": "../../../Dockerfile",
    "dockerContextDir": "../../.."
}

A change to the shared src/ then builds and tests both of them, while a change in standalone-actors/orchestrator counts only for the orchestrator, even though it lies inside the others' context. The orchestrator leaves dockerContextDir out, so its context is its own folder.

For a single-Actor repo, set "folder": "." in the config and place .actor/actor.json at the repo root.

4. Write tests

Put platform tests in test/platform. How you organize them inside is up to you. See Writing tests below.

5. Set up CI

On GitHub, add the GitHub workflows below. On another CI platform, call the CLI from your own pipeline; the workflows are a good template for the steps.

Config file

apify-test-tools.config.json lives at the repo root and tells the tool which Actors live in the repo, where they are, and which tokens build and test them. Actors are organized into groups, and each group sets the token once for all of its Actors:

{
    "mode": "grouped",
    "groups": {
        "myteam": {
            "tokenEnvVar": "APIFY_TOKEN_MYTEAM",
            "actors": [
                { "folder": "actors/web-scraper", "actorFullName": "myteam/web-scraper" },
                { "folder": "actors/email-sender", "actorFullName": "myteam/email-sender" },
                {
                    "folder": "standalone-actors/orchestrator",
                    "actorFullName": "myteam/orchestrator",
                    "overrideActorContext": ["standalone-actors/orchestrator", "test/platform/orchestrator"]
                }
            ]
        }
    }
}

The group key (myteam above) is only a label for readability; it doesn't need to match any folder or username. Each group needs a tokenEnvVar and at least one Actor. Add another group for Actors that need a different token, e.g. Actors owned by another account.

The platform tests run each Actor with the same token that builds it, so they run in the account that owns the Actor. Only the owner, or an account with write access to the Actor, can run a standby Actor on a build other than its default one, which is what the tests do on a PR. To test some Actors from another account, name its token in testTokenEnvVar, on the group or on an Actor:

"partner": {
    "tokenEnvVar": "APIFY_TOKEN_PARTNER",
    "testTokenEnvVar": "TESTER_APIFY_TOKEN",
    "actors": [{ "folder": "actors/partner-scraper", "actorFullName": "partner/scraper" }]
}

The orchestrator sets overrideActorContext because its tests live outside its own folder: without it, a change to only test/platform/orchestrator wouldn't test it. The other two already have the whole repo, tests included, as their context.

Field Where Required Description
folder Actor yes Relative path from repo root to the Actor's own project directory — the folder that directly contains .actor/actor.json (i.e. <folder>/.actor/actor.json) and its source. Documentation and schemas may be linked from elsewhere through actor.json. Use "." for a single-Actor repo where .actor/ is at the root.
actorFullName Actor yes Full Actor identifier in owner/name format (e.g. "apify/web-scraper"). This is the source of truth for the Actor name — the name field in actor.json is not used.
tokenEnvVar group or Actor yes Name of the environment variable holding the Apify API token that builds the group's Actors. No fallback — if the env var is not set at build time, the build fails.
testTokenEnvVar group or Actor no Name of the environment variable holding the Apify API token the platform tests run the Actors with. Defaults to the Actor's tokenEnvVar. A group's testTokenEnvVar applies to its Actors even when an Actor sets its own tokenEnvVar. If the env var is not set, the tests of the Actors that use it fall back to TESTER_APIFY_TOKEN for now, or fail without it.
platformTestsGlob top level no Glob from the repo root of the platform tests and their helpers. Default test/platform/**. A changed file that matches it is a test change, also when .dockerignore lists it: the PR workflow builds and tests the Actor whose context holds it, and a release skips it. See Change detection.
overrideActorContext group or Actor no Array of paths (relative to repo root) that define which files are relevant to the Actor. When set, replaces the dockerContextDir from actor.json for change detection. Useful when an Actor depends on shared packages outside its Docker build context. Entries must not be prefixes of one another (e.g. ["", "code"] or ["actors", "actors/foo"] are rejected). The Actor's own folder is always part of its context — if none of the listed entries reach it, it's added automatically.

A value set on an Actor replaces the group's for that Actor only. Values are replaced, not merged: if the myteam group also set overrideActorContext, the orchestrator would still get only the two paths it lists.

No two Actors may share a folder, and no two entries may point at the same actorFullName.

GitHub workflows

The workflows take every change straight to production, with tests in between:

  • Every push to a PR builds the Actors the PR changes into a test version (0.99) that users never run, and runs their platform tests against those builds. Only the Actors whose files changed are built and tested, and a later push only re-tests what changed since the last green run.
  • Every merge to master builds the changed Actors into their default version (latest), which is what users run, and announces the release on Slack. There is no separate release step.
  • Scheduled runs (optional) run the tests against the released builds to catch what breaks without a code change, like a website changing its layout.

These are GitHub Actions workflows. On another CI platform, build the same flow with the CLI.

Add these files to .github/workflows in your repo. They call the reusable workflows in this repo, pinned at the @workflows-v0 major tag — never at @master. Every input, the optional Claude workflows, stale issue closer, and how the tag moves are in the Workflow API.

pr-build-devel-test.yaml

name: PR Test

on:
    pull_request:
        branches: [master]

jobs:
    buildDevelAndTest:
        uses: apify/apify-test-tools/.github/workflows/public_pr-build-test.yaml@workflows-v0
        secrets:
            # One entry per secret apify-test-tools.config.json names: each `tokenEnvVar` and `testTokenEnvVar`
            CONFIG_SECRETS: |
                {
                    "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
                    "APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
                }

release-latest.yaml

name: Release latest

on:
    push:
        branches: [master]

jobs:
    buildLatest:
        uses: apify/apify-test-tools/.github/workflows/public_push-build-latest.yaml@workflows-v0
        secrets:
            SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
            # One entry per secret apify-test-tools.config.json names, here each `tokenEnvVar`
            CONFIG_SECRETS: |
                {
                    "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
                    "APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
                }

Scheduled tests (optional)

How often to run which tests is up to you. test-files-glob is a glob from the repo root, see Splitting the tests. For example, you can run the daily tests once a day:

name: Platform tests - Daily

on:
    schedule:
        # Runs at 00:00 UTC every day
        - cron: '0 0 * * *'
    workflow_dispatch:

jobs:
    platformTestsDaily:
        uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
        with:
            test-files-glob: 'test/platform/{daily,daily-without-pr}/**'
        secrets:
            SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
            # One entry per token the tests run with: each `testTokenEnvVar`, by default each `tokenEnvVar`
            CONFIG_SECRETS: |
                {
                    "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
                    "APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
                }

and a few critical tests every hour:

name: Platform tests - Hourly

on:
    schedule:
        # Runs at the start of every hour
        - cron: '0 * * * *'
    workflow_dispatch:

jobs:
    platformTestsHourly:
        uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
        with:
            test-files-glob: 'test/platform/{hourly,hourly-without-pr}/**'
        secrets:
            SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
            CONFIG_SECRETS: |
                {
                    "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
                    "APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
                }

Secrets

Set these as repository or organization secrets and pass them in each job's secrets: block, as in the examples above. The Actor tokens go packed in one JSON object, CONFIG_SECRETS, because their names differ per repo:

Secret Needed for
the Actor tokens named by tokenEnvVar in apify-test-tools.config.json building Actors and deleting their old builds
the test tokens named by testTokenEnvVar, by default the same Actor tokens running the platform tests
TESTER_APIFY_TOKEN optional for now, the platform tests' fallback for Actors whose own test token isn't passed, and for repos on a lib older than per-Actor tokens
SLACK_BOT_TOKEN announcing releases and reporting scheduled test results to Slack
NPM_TOKEN optional, installing private npm packages. Use a read-only token

Repos in the apify org can pass secrets: inherit instead. Each secret only reaches the step that needs it. See Secrets for both, and for passing CONFIG_SECRETS without listing the tokens.

Writing tests

Platform tests use describe and testActor from apify-test-tools instead of the ones from vitest. testActor registers a test that only runs when that Actor was built for it (or when you ask for all tests), and gives the callback an extended expect and a run function that starts the Actor.

import { describe, testActor } from 'apify-test-tools';

describe('web-scraper', () => {
    testActor('myteam/web-scraper', 'scrapes the start URL', async ({ expect, run }) => {
        const runResult = await run({ input: { startUrls: [{ url: 'https://apify.com' }] } });

        // Status SUCCEEDED, a sane duration, no failed requests, no errors in the log, and the item count
        await expect(runResult).toFinishWith({ datasetItemCount: { min: 1, max: 10 } });

        const { items } = await runResult.getDataset();
        expect(items[0].title, 'Page title').toBeNonEmptyString();
    });
});

Platform tests only run on the Actors test passes to them, each with its testTokenEnvVar token, so your regular unit test run stays unaffected.

To run them locally against the Actors' current default builds:

npx apify-test-tools get-actor-configs | APIFY_TOKEN_MYTEAM=<token> npx apify-test-tools test

The Vitest library API covers everything toFinishWith checks, the other matchers, standby Actors, reading run data, and running tests locally against builds of your own changes.

About

Tools & lib to test actors on the Apify platform

Resources

Contributing

Stars

8 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages