Skip to content

ci: run the suite under a wrapper lifetime mode, with a canary/all pass in the Test job - #590

Draft
edusperoni wants to merge 3 commits into
mainfrom
ci/wrapper-lifetime-lane
Draft

edusperoni wants to merge 3 commits into
mainfrom
ci/wrapper-lifetime-lane

Conversation

@edusperoni

@edusperoni edusperoni commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

The test suite only ever ran under the default resurrecting wrapper lifetime mode. The rooted and canary modes (#559, #561) were only exercised by editing TestRunner/app/package.json by hand. This PR adds a way to run the suite under any mode, and a CI lane for canary/all.

run_tests.sh --wrapper-lifetime <mode>[:<scope>]

  • mode is resurrecting, rooted or canary; scope is extended (default) or all. Accepts --wrapper-lifetime v and --wrapper-lifetime=v, and is listed in --help.
  • It passes the build setting NS_TEST_WRAPPER_LIFETIME=<mode>:<scope> to xcodebuild.
  • A new TestRunner build phase, "Apply Test App Config" (tools/apply-test-app-config.mjs, after Copy Bundle Resources), writes the bundled app/package.json from TestRunner/app/package.json and adds "wrapperLifetime": { mode, scope } when the setting is present.
    • The tracked package.json is never modified, so a failed or interrupted run leaves nothing to restore.
    • The phase rewrites the bundle copy on every build, override or not. Xcode doesn't re-copy an unchanged app folder, so an override would otherwise survive into the next default build.
    • It uses NODE_BINARY from .xcode.env, like the existing js2c and test-runner-deps phases.
  • After the run, scripts/check-wrapper-lifetime-report.sh fails it unless the junit reports contain "reports the configured mode and scope (/)", the spec that names the mode the runtime actually read.
  • Switching modes only relinks; nothing recompiles.

CI

No new runner. The "Test" job builds once and runs the suite twice:

  • xcodebuild build-for-testing, then test-without-building for the default pass, keeping the existing retry policy (the build is no longer inside the retry, so the per-attempt timeout drops from 40 to 25 minutes).
  • A second pass on the same build: the built app's app/package.json is rewritten with tools/apply-test-app-config.mjs --wrapper-lifetime canary:all, and test-without-building runs again with its own result bundle, junit export, report summary and validation, plus check-wrapper-lifetime-report.sh --expect canary/all. It runs whenever the build succeeded, so a default-pass failure still yields the canary results and diagnostics. Both passes' result bundles go into the test-results artifact.
  • test-without-building runs no build phases, so the rewrite stands and nothing compiles or links between the passes. The simulator installs the bundle without re-signing.
  • The ASan job uses the same build/test split, with no second pass.

Cost: the Test job grows by one suite run (about 3.5–4 min); no extra runner, build or cache entry. A rooted:all pass would be one more matrix value and one more suite run.

Testing

Locally on a dedicated simulator: canary:all, rooted:all and default each ran 2157 specs with 0 failures, and each reported the requested mode. A default build right after a canary one bundled a package.json byte-identical to the source. The Catalyst path (--catalyst --wrapper-lifetime) was checked as a build only: the bundle carried the override and codesign --verify --deep --strict passed.

Run the way CI does (build-for-testing, then two test-without-building passes): pass 1 2157/0 under resurrecting/extended in 200 s, pass 2 2157/0 under canary/all in 199 s, with the TestRunner binary unchanged between them. ASan's two-step shape also passed. Earlier, one local canary:all run under heavy machine load timed out in the Workers suite and passed on rerun; worth watching.

…-wrapper-lifetime

run_tests.sh --wrapper-lifetime <mode>[:<scope>] passes the build setting
NS_TEST_WRAPPER_LIFETIME to xcodebuild. A new TestRunner build phase, "Apply
Test App Config" (after Copy Bundle Resources), writes the bundled
app/package.json from TestRunner/app/package.json on every build, adding
"wrapperLifetime": { mode, scope } when the setting is present. The tracked
package.json is never modified, so nothing has to be restored after a failed
or interrupted run, and a later default build drops the override even though
Xcode skips re-copying the unchanged app folder.

After the run, scripts/check-wrapper-lifetime-report.sh fails it unless the
junit reports contain "reports the configured mode and scope (<mode>/<scope>)",
the spec that names the mode the runtime actually read.
A third entry in the PR test matrix, "Test (canary)", builds and runs the
suite like "Test" with NS_TEST_WRAPPER_LIFETIME=canary:all, then checks the
junit reports show the suite ran under canary/all. Its compilation cache
falls back to the default variant's keys, which main seeds, since the
compiled code is the same.
@coderabbitai

coderabbitai Bot commented Nov 21, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Replace the separate "Test (canary)" matrix entry, which rebuilt the app
on its own runner, with a second pass in the "Test" job on the same build.
Both matrix entries now build once with build-for-testing and run the suite
with test-without-building, keeping the retry policy and result bundles.

When a matrix entry sets wrapper-lifetime-pass, tools/apply-test-app-config.mjs
rewrites the built app's app/package.json and the suite runs again with
test-without-building, which runs no build phases, so the rewrite holds. The
simulator installs the rewritten bundle without re-signing. That pass has its
own result folder, retry, junit export, report summary and validation,
including check-wrapper-lifetime-report.sh, and runs even when the default
pass failed. Its xcresults go into the job's test-results artifact.

The canary cache variant and its restore-key fallback are gone; the cache
keys match main again.
@edusperoni edusperoni changed the title ci: run the suite under a wrapper lifetime mode, with a canary/all lane ci: run the suite under a wrapper lifetime mode, with a canary/all pass in the Test job Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant