Use this guide to understand what runs in CI, when, and why: the trigger fan-out, the per-stack regressions, and the nightly sweep. It is for contributors and reviewers who need to predict which checks a change will start, read the Actions tab without guessing, and find where each job stages its reports. The source of truth is .github/workflows/; this guide mirrors it.
There is no needs:, workflow_run:, workflow_call:, or concurrency: anywhere in the five workflows, so nothing chains, orders, or cancels anything else. Every trigger fans out to its matching workflows in parallel, and inside each workflow every top-level job runs in parallel with the others. Matrix legs are also concurrent, subject to the nightly limit of three legs per matrix. The only sequential execution is the ordered list of steps inside a single job. Read the step lists below top to bottom for order within a job, and treat everything else as concurrent.
flowchart LR
subgraph A["Lane A · on code change"]
EVA["pull_request → master<br/>push → master<br/>workflow_dispatch"]
WPR["PR & Push · Fast Checks<br/>pr.yml"]
WJ["Java · Regression (path-filtered)<br/>java.yml"]
WP["Python · Regression (path-filtered)<br/>python.yml"]
WT["TypeScript · Regression (path-filtered)<br/>ts.yml"]
EVA -->|always| WPR
EVA -. if java paths .-> WJ
EVA -. if python paths .-> WP
EVA -. if ts paths .-> WT
end
subgraph B["Lane B · scheduled"]
EVB["schedule 17 3 * * *<br/>03:17 UTC daily<br/>workflow_dispatch"]
WN["Nightly · Cross-Browser + Grid Regression<br/>nightly.yml"]
EVB --> WN
end
Solid arrows mark always-on or scheduled triggers; dotted arrows mark triggers that are conditional on a path filter.
pr.yml) runs on every pull request and every push to master, with no path filter, so it gates all changes. Its five jobs cover repository hygiene, scenario-catalog drift, and a compile-plus-smoke slice for each stack.scenarios/**, tools/**, and its own workflow file:
java.yml) on stacks/java-selenium-testng/**.python.yml) on stacks/python-playwright/**.ts.yml) on stacks/ts-playwright/**.scenarios/** and tools/**, a change under either shared path triggers all three stack workflows at once, while a change confined to one stack directory triggers only that stack.nightly.yml) runs on cron 17 3 * * * (03:17 UTC daily) or on manual workflow_dispatch. The dispatch exposes an include-flaky-demo boolean input; when it is true, the nightly slices also include the @flaky-demo / flaky_demo tests that are otherwise excluded.| Workflow | Trigger | Intent | Jobs | File |
|---|---|---|---|---|
| PR & Push · Fast Checks (lint + smoke) | pull_request and push to master; workflow_dispatch |
Fast lint and smoke gate on every change | repo-hygiene, scenario-catalog, java-smoke, ts-smoke, py-smoke |
pr.yml |
| Java · Regression (path-filtered) | pull_request and push to master, path-filtered; workflow_dispatch |
Java regression across local browsers | java-regression |
java.yml |
| Python · Regression (path-filtered) | pull_request and push to master, path-filtered; workflow_dispatch |
Python regression across browser and mobile projects | python-regression |
python.yml |
| TypeScript · Regression (path-filtered) | pull_request and push to master, path-filtered; workflow_dispatch |
TypeScript regression across browser and mobile projects | typescript-regression |
ts.yml |
| Nightly · Cross-Browser + Grid Regression | schedule 17 3 * * *; workflow_dispatch |
Broad nightly sweep including a Selenium Grid pass | java-nightly-local, java-nightly-grid, typescript-nightly, python-nightly |
nightly.yml |
Steps run top to bottom within each job. A step marked (always) carries if: always(); a step marked (conditional) carries another if: guard.
Job (id) |
Intent | Steps (top to bottom) | File |
|---|---|---|---|
repo-hygiene |
Lint workflows, commit messages, and Markdown, check documentation links, validate the README fast start, and require CI-guide updates alongside workflow changes | 1. Checkout repository, 2. Test CI container helper, 3. Pull CI utility images, 4. Set up Node 22, 5. Lint GitHub Actions workflows, 6. Lint pull request commits (conditional), 7. Lint latest commit (conditional), 8. Lint Markdown, 9. Check Markdown links, 10. Validate README fast start, 11. Require CI guide updates with workflow changes (conditional) | pr.yml |
scenario-catalog |
Fail if the generated scenario matrix or the README embed of it has drifted | 1. Checkout repository, 2. Check scenario catalog | pr.yml |
java-smoke |
Compile the Java tests and run the Chrome smoke suite | 1. Checkout repository, 2. Start test application, 3. Set up JDK 25, 4. Compile Java tests, 5. Run Java smoke, 6. Stage Java reports (always), 7. Upload Surefire XML (always) | pr.yml |
ts-smoke |
Type-check, lint, and format-check, then run the Chromium smoke suite | 1. Checkout repository, 2. Start test application, 3. Set up Node 22, 4. Install TypeScript stack dependencies, 5. Type-check TypeScript stack, 6. Lint TypeScript stack, 7. Check TypeScript stack formatting, 8. Install Chromium browser, 9. Run TypeScript Chromium smoke, 10. Upload Playwright artifacts (always) | pr.yml |
py-smoke |
Lint, format-check, and type-check, then run the Chromium smoke suite | 1. Checkout repository, 2. Start test application, 3. Set up Python 3.14, 4. Install Python stack dependencies, 5. Lint Python stack, 6. Check Python stack formatting, 7. Type-check Python stack, 8. Install Chromium browser, 9. Run Python Chromium smoke, 10. Upload Playwright artifacts (always) | pr.yml |
java-regression |
Run the Java regression suite per matrix browser | 1. Checkout repository, 2. Start test application, 3. Set up JDK 25, 4. Run Java regression, 5. Stage Java reports (always), 6. Upload Surefire XML (always) | java.yml |
python-regression |
Plan and run each Python Playwright slice per matrix project | 1. Checkout repository, 2. Start test application, 3. Set up Python 3.14, 4. Install Python stack dependencies, 5. Plan Python Playwright slice, 6. Install Playwright browser (conditional), 7. Run Python Playwright slice (conditional), 8. Upload Playwright artifacts (always) | python.yml |
typescript-regression |
Plan and run each TypeScript Playwright slice per matrix project | 1. Checkout repository, 2. Start test application, 3. Set up Node 22, 4. Install TypeScript stack dependencies, 5. Plan TypeScript Playwright slice, 6. Install Playwright browsers (conditional), 7. Run TypeScript Playwright slice (conditional), 8. Upload Playwright artifacts (always) | ts.yml |
java-nightly-local |
Run the Java regression suite with a local driver per matrix browser | 1. Checkout repository, 2. Start test application, 3. Set up JDK 25, 4. Run Java nightly regression, 5. Stage Java reports (always), 6. Upload Surefire XML (always) | nightly.yml |
java-nightly-grid |
Run the Java regression suite against Selenium Grid per matrix browser | 1. Checkout repository, 2. Start test application and Selenium Grid, 3. Set up JDK 25, 4. Run Java Selenium Grid nightly regression, 5. Stage Java Grid reports (always), 6. Upload Java Grid Surefire XML (always) | nightly.yml |
typescript-nightly |
Plan and run each TypeScript nightly slice per matrix project | 1. Checkout repository, 2. Start test application, 3. Set up Node 22, 4. Install TypeScript stack dependencies, 5. Plan TypeScript nightly slice, 6. Install Playwright browser (conditional), 7. Run TypeScript nightly slice (conditional), 8. Upload Playwright artifacts (always) | nightly.yml |
python-nightly |
Plan and run each Python nightly slice per matrix project | 1. Checkout repository, 2. Start test application, 3. Set up Python 3.14, 4. Install Python stack dependencies, 5. Plan Python nightly slice, 6. Install Playwright browser (conditional), 7. Run Python nightly slice (conditional), 8. Upload Playwright artifacts (always) | nightly.yml |
This guide is the one document CI checks against its own source. A pull request
that changes anything under .github/workflows/ without touching this file fails
repo-hygiene, because a step-level mirror that nothing enforces is the highest-
risk documentation in the repository: it stays plausible long after it stops
being true. If a workflow change genuinely has no semantic effect here — a
whitespace or comment edit — put the literal token [ci-guide-exempt] in the
pull request title to skip the check. Dependabot GitHub Actions updates are also
exempt because that ecosystem can only replace action-version references; it
cannot change the documented job topology.
CI starts its pinned Docker images from an explicit Start test application
step instead of the job-level services phase. The helper at
tools/ci_containers.py bounds each pull, retries
only that idempotent external operation with exponential backoff and jitter,
starts containers with --pull=never, and polls application/Grid readiness
before language setup or tests begin. Exhausted retries include container
status and logs. Tests themselves are never retried.
The 20 nightly matrix legs make 26 image-pull calls in total, so each leg starts with a randomized delay of at most 20 seconds and each matrix runs at most three legs concurrently. This reduces synchronized Docker Hub bursts without reducing browser coverage.
Each regression and nightly job fans out over a strategy.matrix, with fail-fast disabled so one failing leg never cancels the rest. Nightly matrices cap parallelism at three legs to avoid synchronized external-registry bursts.
| Stack | Regression projects | Nightly projects |
|---|---|---|
| Java | chrome, firefox |
Local driver chrome, edge, firefox; Selenium Grid chrome, edge, firefox |
| TypeScript | chromium, firefox, webkit, Mobile Chrome, Mobile Safari |
The five regression projects plus branded chrome and msedge |
| Python | chromium, firefox, webkit, Mobile Chrome, Mobile Safari |
The five regression projects plus branded chrome and msedge |
TypeScript and Python select tests by tag. Playwright (TypeScript) filters @-prefixed title tags with --grep and --grep-invert; pytest (Python) filters equivalent markers with -m expressions. The two stacks mirror the same taxonomy:
@http / http — resource-layer checks that hit HTTP endpoints; pinned to the default browser project and inverted out of the others so they run once, not per browser.@desktop / desktop — mouse-only behavior; inverted out of the mobile-emulation projects.@mobile-emulation / mobile_emulation — tests that require a mobile device profile; the Mobile Chrome and Mobile Safari projects grep for them and invert @desktop.@flaky-demo / flaky_demo — deliberately unstable teaching examples; excluded from the PR and nightly gates by default, and added to nightly only when the include-flaky-demo dispatch input is true.@not-ci / not_ci — examples unsuitable for scheduled automation; excluded everywhere in CI and run only locally.See docs/flakiness-guide.md for how these tags keep unstable patterns out of the gates.
Each Playwright slice is planned before it runs. A Plan … slice step collects the matching tests (--list for Playwright, --collect-only for pytest) and writes a count; the install and run steps carry if: steps.slice.outputs.count != '0', so a slice with no matching tests is skipped instead of failing. The Java jobs have no plan step and run their TestNG suite directly.
Every stack stages its reports under artifacts/<stack>/<run-id>/<slice>/ before uploading, so a run’s outputs are grouped by stack and slice:
artifacts/java/...) — Surefire XML, staged by a Stage … reports step and uploaded by an Upload Surefire XML step, both if: always().artifacts/ts/...) — the Playwright HTML report plus retained traces, screenshots, and videos, uploaded by Upload Playwright artifacts if: always().artifacts/py/...) — JUnit XML plus retained traces, screenshots, and videos, uploaded by Upload Playwright artifacts if: always().Because the uploads run on always(), artifacts are available even when the test step fails. See the README.md reports section for the matching per-stack local report paths.
This guide describes what runs. For what to do when one of these jobs goes red — which artifact to open first, and how to reproduce each failure class locally — see runbooks/ci-failure-triage.md.