Repository navigation
ci: run the suite under a wrapper lifetime mode, with a canary/all pass in the Test job - #590
Draft
edusperoni wants to merge 3 commits into
Draft
edusperoni wants to merge 3 commits into
edusperoni wants to merge 3 commits into
Conversation
…-wrapper-lifetime
run_tests.sh --wrapper-lifetime <mode>[:<scope>] passes the build setting
NS_TEST_WRAPPER_LIFETIME to xcodebuild. A new TestRunner build phase, "Apply
Test App Config" (after Copy Bundle Resources), writes the bundled
app/package.json from TestRunner/app/package.json on every build, adding
"wrapperLifetime": { mode, scope } when the setting is present. The tracked
package.json is never modified, so nothing has to be restored after a failed
or interrupted run, and a later default build drops the override even though
Xcode skips re-copying the unchanged app folder.
After the run, scripts/check-wrapper-lifetime-report.sh fails it unless the
junit reports contain "reports the configured mode and scope (<mode>/<scope>)",
the spec that names the mode the runtime actually read.
A third entry in the PR test matrix, "Test (canary)", builds and runs the suite like "Test" with NS_TEST_WRAPPER_LIFETIME=canary:all, then checks the junit reports show the suite ran under canary/all. Its compilation cache falls back to the default variant's keys, which main seeds, since the compiled code is the same.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Replace the separate "Test (canary)" matrix entry, which rebuilt the app on its own runner, with a second pass in the "Test" job on the same build. Both matrix entries now build once with build-for-testing and run the suite with test-without-building, keeping the retry policy and result bundles. When a matrix entry sets wrapper-lifetime-pass, tools/apply-test-app-config.mjs rewrites the built app's app/package.json and the suite runs again with test-without-building, which runs no build phases, so the rewrite holds. The simulator installs the rewritten bundle without re-signing. That pass has its own result folder, retry, junit export, report summary and validation, including check-wrapper-lifetime-report.sh, and runs even when the default pass failed. Its xcresults go into the job's test-results artifact. The canary cache variant and its restore-key fallback are gone; the cache keys match main again.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The test suite only ever ran under the default
resurrectingwrapper lifetime mode. Therootedandcanarymodes (#559, #561) were only exercised by editingTestRunner/app/package.jsonby hand. This PR adds a way to run the suite under any mode, and a CI lane forcanary/all.run_tests.sh --wrapper-lifetime <mode>[:<scope>]modeisresurrecting,rootedorcanary;scopeisextended(default) orall. Accepts--wrapper-lifetime vand--wrapper-lifetime=v, and is listed in--help.NS_TEST_WRAPPER_LIFETIME=<mode>:<scope>to xcodebuild.tools/apply-test-app-config.mjs, after Copy Bundle Resources), writes the bundledapp/package.jsonfromTestRunner/app/package.jsonand adds"wrapperLifetime": { mode, scope }when the setting is present.package.jsonis never modified, so a failed or interrupted run leaves nothing to restore.NODE_BINARYfrom.xcode.env, like the existing js2c and test-runner-deps phases.scripts/check-wrapper-lifetime-report.shfails it unless the junit reports contain "reports the configured mode and scope (/)", the spec that names the mode the runtime actually read.CI
No new runner. The "Test" job builds once and runs the suite twice:
xcodebuild build-for-testing, thentest-without-buildingfor the default pass, keeping the existing retry policy (the build is no longer inside the retry, so the per-attempt timeout drops from 40 to 25 minutes).app/package.jsonis rewritten withtools/apply-test-app-config.mjs --wrapper-lifetime canary:all, andtest-without-buildingruns again with its own result bundle, junit export, report summary and validation, pluscheck-wrapper-lifetime-report.sh --expect canary/all. It runs whenever the build succeeded, so a default-pass failure still yields the canary results and diagnostics. Both passes' result bundles go into thetest-resultsartifact.test-without-buildingruns no build phases, so the rewrite stands and nothing compiles or links between the passes. The simulator installs the bundle without re-signing.Cost: the Test job grows by one suite run (about 3.5–4 min); no extra runner, build or cache entry. A
rooted:allpass would be one more matrix value and one more suite run.Testing
Locally on a dedicated simulator:
canary:all,rooted:alland default each ran 2157 specs with 0 failures, and each reported the requested mode. A default build right after a canary one bundled apackage.jsonbyte-identical to the source. The Catalyst path (--catalyst --wrapper-lifetime) was checked as a build only: the bundle carried the override andcodesign --verify --deep --strictpassed.Run the way CI does (build-for-testing, then two test-without-building passes): pass 1 2157/0 under resurrecting/extended in 200 s, pass 2 2157/0 under canary/all in 199 s, with the
TestRunnerbinary unchanged between them. ASan's two-step shape also passed. Earlier, one localcanary:allrun under heavy machine load timed out in the Workers suite and passed on rerun; worth watching.