Gate iOS 27 in CI: xcode-27 legs for the screenshot and fidelity suites - #5879
Conversation
Three things have to be true before an iOS 27 gate is worth writing, and none of them can be assumed: that the xcode-27 label resolves for this repository at all (it is a public preview image), which Xcode and iOS runtime it carries, and whether an iPhone 16 can be CREATED there -- the image pre-creates iPhone 17 / 17e / 18 Pro / 18 Pro Max / Air and no iPhone 16, and the ios-27-metal goldens were captured on an iPhone 16. Dispatch-only and deleted once answered. Finding out by watching a 60-minute screenshot job fail is the alternative. actionlint ships a static label list that lags GitHub's, so xcode-27 is declared in .github/actionlint.yaml the same way windows-11-arm already is -- rather than muting the rule, so a genuine typo in a runs-on is still caught. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…issing widget FidelityGate refused to write the ios-27-metal baseline at all because two tiles reported no geometry: GlassPanelGrey_normal_dark and GlassPanelRed_normal_dark. Neither is a defect. iOS 27's dark glass over a flat mid-grey backdrop transforms to within a shade of that same grey -- 128 * 0.378 + 79.8 is 128 -- so the panel has no edge to find, and the NATIVE capture agrees: #818181 panel over #808080 backdrop. The same holds for red on red. geometryMetrics already reported this honestly as empty=true with no numbers. The gate could not tell that from geometry being withheld, and its contract -- "a compared pair cannot evade its geometry contract by returning no metrics" -- is right to be strict, so everything stopped. What was actually missing is WHICH SIDE is blank. empty=true was set when either box came back zero, so "the widget is invisible in both renders" and "the widget rendered natively and not in ours" were the same value. The second is a regression, and it was being folded into the first. geometryMetrics now records native_empty and cn1_empty, and the gate exempts a pair from the geometry contract only when BOTH are true. Verified over three cases: both blank updates the baseline; native present with ours blank still fails; and a report carrying empty alone, with no side flags, still fails -- an older report must not acquire an exemption it never measured. Net effect is a stricter gate, not a looser one: the one-sided case used to arrive as "no geometry" and is now named. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he generation A COPY, not a move. build-ios-metal keeps gating macos-15 / Xcode 26, which is the toolchain CN1_XCODE_MAJOR actually pins and what applications are built with today; the new leg runs the identical suite on the Xcode 27 / iOS 27 image with its own baseline directory. The existing job is untouched to the byte -- this commit is a pure insertion -- so its check name, its port-status artifact and anything keyed on either keep working. helloworld does not use the shipped default theme: it installs the modern theme itself, and DualAppearanceBaseTest.pickModernThemeResource named generation 26 outright. Hardcoded, the 27 leg would render the iOS 26 theme and score it against iOS 27 baselines -- a difference in every glass tile with nothing saying why. Both device-side sites now ask the port which generation the build carries (IOSImplementation.getProperty, which answers with the resource that is actually present and falls back to 26 when a bundle lacks the other). The dev-guide screenshot test under src/test is deliberately left alone: it renders on JavaSE for the developer guide and is generation-neutral. The 27 leg deliberately does NOT upload a port status. There is one iOS port and the published report is keyed on `ios-metal`; a second upload under that id would publish whichever leg finished last -- the same clobber the PR-comment marker already had to be split to avoid. scripts/check-ios-metal-legs-match.py holds the two jobs to being the same job. 320 duplicated lines drift, and both legs would stay green while diverging because each passes on its own terms and nothing compares them. It normalises away the differences that are meant to exist and requires the rest to match, so a real difference has to be declared in the script with a reason. Verified to fail on drift introduced from either side, and to exit non-zero when it does. The xcode-27 image is a public preview (actions/runner-images#14404) and its capacity is still being balanced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…th pairs fidelity-ios-metal-27 scores the iOS 27 theme against the ios-27-metal goldens on the Xcode 27 image. A copy rather than a move, for the same reason as the screenshot leg: ios.themeGeneration defaults to 26, so the 26 row keeps gating what applications actually get, and this row is what keeps the 27 theme honest. One variable drives it. CN1SS_FIDELITY_GOLDEN_SET reaches the capture script, the app build (which turns it into ios.themeGeneration=27) and the comparison, so the theme the app ships and the goldens it is scored against cannot disagree -- which they silently did throughout the tuning of gen27.css, and the symptom was two runs whose scores matched to the last decimal. The drift guard is now pair-driven and covers both copies rather than one, and is renamed accordingly. Each pair declares the differences that are meant to exist -- runner, toolchain pin, golden set, baseline and artifact names -- and anything left over is drift. Verified to fail, and to exit non-zero, on drift introduced into the fidelity pair as well as the screenshot pair. This row cannot pass until the ios-27-metal frame goldens are seeded and its baseline recorded; both are deliberate local acts and neither has landed yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ctually are Removing the generated *-ios-source directory does not invalidate enough. Measured: with only that removed, a rebuild after changing a CORE constant returned in 36 SECONDS and the emitted C still carried the previous value -- the core jar on the classpath had the new one, the generated C did not. The whole module target goes now, along with the sources archive build-ios-app.sh writes beside it. The stamp's limit is written down rather than left to be discovered: it keys on the generation, so it does not fire for a core edit at the same generation. Maven does not treat a changed dependency as invalidating generated sources, so iterating on core and the fidelity app together still needs a manual clean. Also documents what cost a wrong plan earlier in this work: `<set>-frames` holds CN1 renders, not platform captures -- run-ios-fidelity-tests.sh calls them "self-goldens (CN1 vs committed CN1)" and seeds them from a single FIDELITY_UPDATE_GOLDENS=1 flag. The neighbouring `-anim` directory IS native and does need record-ios-native-anim.sh with a tap-driven tabs recording, so reading one as the other leads to preparing an xcodegen + XCUITest capture for something a flag produces. The note also records what reviewing a seed looks like, since whatever CN1 rendered becomes the reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…elity row non-gating THE FRAMES. ios-27-metal-frames had nothing in it, so every animation frame failed as a missing golden. These are CN1 self-goldens -- run-ios-fidelity-tests .sh compares CN1 against committed CN1 to catch a stuck frame, non-monotonic travel or a broken overshoot -- so they are seeded, not captured, and reviewed rather than trusted. Seeded from an app built with Xcode 27.0 against the iOS 27 theme, then checked: 24 names and sizes identical to the iOS 26 set, none blank, the ten SwitchMorph frames byte-identical to iOS 26 (Switch uses no glass recipe and iOS 27 did not change it), and TabsMorph moved in dark (~6.5) and barely in light (~0.11) -- exactly where the theme changed, since the pill recipe's light variant measured unchanged. Re-validated against a clean rebuild: 4 groups OK. THE OFFSET STAYS AT THE FIT'S 79.8. 80.3 was tried, on the argument that it puts the mid-grey arithmetic at 128.68 against the native's 128.7 for 0.026/255 of fit error. On device it changed nothing: the renderer emits integers and 128.68 lands on 128, the backdrop exactly. The only other reachable value is 129, which overshoots the native by 0.3 rather than undershooting by 0.7, and choosing it would be picking a constant for what the quantiser does with it. Reverted, with the measurement recorded so it is not retried blind. THE 27 FIDELITY ROW IS NON-GATING, for one reason with one exit. Its baseline cannot be recorded: GlassPanelGrey and GlassPanelRed in dark render with no silhouette where the native has a faint one, so ProcessScreenshots reports cn1_empty and FidelityGate refuses a one-sided empty. That refusal is correct and is NOT being softened to get a green tick -- the same check caught a real defect in these very constants, which a 97% fidelity score could not see. The row runs on every pull request, uploads artifacts and reports scores; it just does not fail the build until the material model can carry sub-level detail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ing on it cn1_select_xcode probes every /Applications/Xcode*.app with `xcodebuild -version` and means to skip any that cannot answer -- the `[ -n "$version" ] || continue` right below says so. Under `set -e`, which every CI step here runs with, it never got that far: the assignment itself carries the failing status and kills the caller first, and since the crash text goes to /dev/null the job dies with no output whatsoever. Measured on the GitHub xcode-27 image, where /Applications holds the real Xcode plus three symlinks to it: selection aborted with exit 134 (SIGABRT) and printed nothing. Reduced to the shell semantics on their own, the old form exits 134 without reaching the emptiness check; with `|| true` it reaches it with an empty version and the candidate is skipped, which is what was intended all along. Latent on macos-15 only because nothing there crashes. It would have taken out both new iOS 27 legs, which pin CN1_XCODE_MAJOR=27 and call this first. The probe now also reports each candidate out loud, so which one misbehaves is on the record rather than inferred from an exit code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Compared 12 screenshots: 12 matched. |
|
Developer Guide build artifacts are available for download from this workflow run:
Developer Guide quality checks: |
|
Compared 166 screenshots: 166 matched. Benchmark ResultsDetailed Performance Metrics
|
|
Compared 166 screenshots: 166 matched. Benchmark ResultsDetailed Performance Metrics
|
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Compared 151 screenshots: 151 matched. Native Android coverage
✅ Native Android screenshot tests passed. Native Android coverage
Benchmark ResultsDetailed Performance Metrics
|
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
Cloudflare Preview
|
|
Compared 166 screenshots: 166 matched. |
|
Compared 166 screenshots: 166 matched. |
|
Compared 166 screenshots: 166 matched. Benchmark ResultsDetailed Performance Metrics
|
…probe The probe answered what it was for, on the real image: Xcode_27_Release_Candidate.app -> prints "Xcode 27.0", exit 134 Xcode_27.0.0.app / Xcode_27.0.app / Xcode.app -> 27.0, exit 0 Two traps in one line, not one. The crashing app is FIRST in glob order, so without `|| true` the failing status propagates out of the assignment and `set -e` kills selection on candidate one -- before any working Xcode is tried, and with no output at all, because the crash text went to /dev/null. That much the previous commit fixed. But it also prints a perfectly good version BEFORE aborting, so testing the string alone accepts an Xcode whose xcodebuild dies. On this image that was survivable only because all four report 27.0 and the tie-break keeps the last one, which is luck: had the crashing app sorted last, it would have been chosen and every build on it would have failed somewhere less obvious. Candidates are now judged on their exit status too. A/B against a fake /Applications holding a crasher that sorts first and prints a version, plus a working Xcode: master exits 134 and selects nothing; this selects the working one and exits 0. The probe workflow goes, as its header said it would. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c92f9fbe2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
All six are real; two are holes in fixes from earlier in this same branch. P1 -- the 27 screenshot job pointed at a directory that does not exist. run-ios-ui-tests.sh exits 3 when SCREENSHOT_REF_DIR is not a directory, so every run would have died before the suite started. The directory now exists with a README, and its screenshot STEP is marked as a seeding run: these images can only come from a run on the 27 image, because capturing them on a maintainer's machine records that machine's toolchain rather than the gate's. The README carries the steps that turn it into a gate. P1 -- continue-on-error was on the JOB. That converts every failure into a green tick, so a broken Xcode selection, build, boot or capture would have reported success while producing none of the scores the comment promises. Both 27 legs now tolerate exactly one step, and everything before it still fails the job. P2 -- FidelityGate let a one-sided empty through on a NEW pair. The parse loop added it to neither collection, and the validation below only iterates pairs that already have baseline geometry, so a widget missing from our render passed with exit 0 -- the exact case the native_empty/cn1_empty split was added to catch, and it did not catch it on the pairs most likely to hit it. It now fails at parse time, naming which side is blank. Re-tested over four cases: missing from our render and missing from the native both fail (the first was exit 0 before), both blank still passes, and a report predating the flags is unchanged. P2 -- the two fidelity jobs shared a PR comment and a preview path. run-ios-fidelity-tests.sh exported a fixed marker and subdirectory for every iOS run, so whichever generation finished last hid the other's scores and could replace its images. Marker, preview path and title now derive from the golden set, with ios-26-metal keeping its historical values so existing comments and links are not orphaned. The screenshot suite already hit this and had to split its markers; this is the same failure one workflow over. P2 -- the screenshot workflow's path filters named only screenshots-metal, so a PR changing an iOS 27 baseline would skip the job that validates it. P2 -- the fidelity PR trigger was CSS-only, which is right for the 26 row and wrong for the 27 one: it is the only check on a generation nothing else exercises. THIS pull request proved it -- it edits the workflow, GlassRecipe, the gate and the frame goldens, and neither fidelity row ran. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Native fidelity (Android, Material 3)54 pairs compared -- median 95.6%, worst 91.3% ( Distribution --
Geometry vs native (bbox offset / size ratio / center offset / corner radius) -- gated separately from the visual score
Side-by-side comparisons (worst first)
|
Native fidelity (ios-27-metal)68 pairs compared -- median 94.5%, worst 68.8% ( Distribution --
Geometry vs native (bbox offset / size ratio / center offset / corner radius) -- gated separately from the visual score
Side-by-side comparisons (worst first)
|
|
Compared 181 screenshots: 181 matched. |
Native fidelity (iOS Modern, Metal)68 pairs compared -- median 95.0%, worst 83.5% ( Distribution --
Geometry vs native (bbox offset / size ratio / center offset / corner radius) -- gated separately from the visual score
Side-by-side comparisons (worst first)
|
|
Compared 160 screenshots: 160 matched. Benchmark Results
Detailed Performance Metrics
|
|
Compared 144 screenshots: 144 matched. |
|
Compared 148 screenshots: 148 matched. Benchmark Results
Detailed Performance Metrics
|
|
Compared 149 screenshots: 149 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
|
Compared 217 screenshots: 217 matched. |
…enshots build-ios-metal-27 reported green on its first run and produced NOT ONE screenshot. It failed the native warning census and exited before the suite started; the tolerated screenshot step then turned that into a successful job. A leg that produces nothing and says nothing is worse than a red one, and only looking in the artifact showed it: zero PNGs. The cause is the same shape as everything else in this branch -- a per-generation thing wearing a per-platform identity. The census is a per-TOOLCHAIN ratchet, and a new warning kind fails it by design. Xcode 27's clang does not emit the same set as Xcode 26's, so inheriting CN1_WARNING_LEG=ios-sim-debug compared a 27 build against a 26 baseline and could never have passed. The leg is declared in LEG_PORT_DIRS rather than accepted as free text -- the tool refuses an unregistered leg, which is how this stayed honest -- and its baseline is seeded from THIS job's own build log, 120 entries against the 26 leg's 128. Verified both directions on that same log: against the new baseline the census passes with "621 gating diagnostic(s), all baselined"; against the iOS 26 baseline it still fails, which is the symptom that stopped the job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds an iOS 27 target as its own gate, running on every pull request, with its own baselines.
What lands
build-ios-metal-27— the helloworld iOS screenshot suite on thexcode-27image, scored againstscripts/ios/screenshots-metal-27/. A copy, not a move:build-ios-metalkeeps gatingmacos-15/Xcode 26, which is the toolchainCN1_XCODE_MAJORpins and what applications are built with today. The existing job is untouched to the byte, so its check name and port-status artifact keep working.fidelity-ios-metal-27— the fidelity suite against theios-27-metalgoldens, driven by a singleCN1SS_FIDELITY_GOLDEN_SETso the theme the app ships and the goldens it is scored against cannot disagree.ios-27-metal-frames. These are CN1 self-goldens (run-ios-fidelity-tests.sh: "self-goldens (CN1 vs committed CN1)"), seeded withFIDELITY_UPDATE_GOLDENS=1, not native captures. Reviewed rather than trusted: names and sizes identical to the iOS 26 set, none blank, the tenSwitchMorphframes byte-identical to iOS 26 (Switch uses no glass recipe and iOS 27 did not change it), andTabsMorphmoved in dark (~6.5) and barely in light (~0.11) — exactly where the theme changed. Re-validated against a clean rebuild:4 group(s) OK.scripts/check-duplicated-ci-legs.pyholds each copy to its original. 320 and 78 duplicated lines drift, and both legs of a pair stay green while diverging because each passes on its own terms. Each pair declares the differences that are meant to exist; anything else fails. Verified to fail, and exit non-zero, on drift introduced from either side of either pair.Two bugs this exposed
cn1_select_xcodedied on the Xcode 27 image with exit 134 and no output. It probes every/Applications/Xcode*.appand means to skip any that cannot report a version —[ -n "$version" ] || continuesays so — but underset -ethe assignment carries the failing status and kills the caller first, while the crash text goes to/dev/null. That image holds the real Xcode plus three symlinks to it, and one invocation aborts. Latent onmacos-15only because nothing there crashes; it would have taken out both new legs, which pinCN1_XCODE_MAJOR=27and call this first.Generation invalidation was not invalidating enough. With only the generated
*-ios-sourcedirectory removed, a rebuild after changing a core constant returned in 36 seconds and the emitted C still carried the previous value. The whole module target goes now, plus the sources archive. The stamp's remaining blind spot — a core edit at the same generation — is written down rather than left to be rediscovered.Geometry: a stricter gate, not a looser one
ProcessScreenshotssetempty=truewhen either side had no widget silhouette, so "invisible in both renders" and "rendered natively but not by us" were the same value. The second is a regression and was being folded into the first. It now recordsnative_empty/cn1_emptyseparately, andFidelityGateexempts a pair only when both are true.That check immediately caught a real defect in the iOS 27 constants from #5876 that a 97% fidelity score could not see — see below.
What is deliberately not green
fidelity-ios-metal-27iscontinue-on-error, for one reason with one exit. Its baseline cannot be recorded: in dark,GlassPanelGreyandGlassPanelRedrender with no silhouette at all. Over a flat mid-grey backdrop the material arithmetic gives128 × 0.378 + 79.8 = 128.18, the renderer emits integers, so the panel lands on 128 — the backdrop exactly. The native capture is not flat there, averaging 128.7 across 101 distinct values where ours has 17.An offset putting the arithmetic at 128.68 against the native's 128.7 was tried, at a cost of 0.026/255 of global fit error. On device it changed nothing — 128.68 still quantises to 128 — and the only other reachable value overshoots the native by 0.3 rather than undershooting by 0.7. Reverted to the fit's optimum, with the measurement recorded so it is not retried blind.
Closing it needs the material model to carry sub-level detail, not a different constant. The one-sided-empty refusal is not being softened to get a green tick, because that same check is what caught the bad constant.
The row still runs on every pull request, uploads artifacts and reports scores; it just does not fail the build.
Runner notes
xcode-27is a public preview image (actions/runner-images#14404). A dispatch probe on this branch confirmed: the label resolves, the image is macOS 27.0 / Xcode 27.0 (27A266a) with the iOS 27.0 simulator runtime, and an iPhone 16 boots there throughboot-ios-simulator.shdespite not being pre-installed — which theios-27-metalgoldens depend on. It waited ~24 minutes behind this repo's own macOS backlog, not on image capacity.actionlintships a static label list that lags GitHub's, soxcode-27is declared in.github/actionlint.yamlthe same waywindows-11-armalready is.The probe workflow is temporary and removed once its re-run confirms the
cn1_select_xcodefix.Not covered
The
xcode-27legs have never run end to end — only the probe has. This PR is the first time they execute.🤖 Generated with Claude Code