The repository exists to compare automation stacks against the same application. That comparison is only meaningful if the stacks are answering the same questions. Without a shared definition of “the scenarios”, each stack would drift into testing whatever its examples happened to cover, and any claim that one stack covers more than another would be unfalsifiable — the coverage tables would be prose, maintained by whoever last remembered.
The obvious alternatives each fail differently. Deriving coverage from test names alone makes the tests self-certifying: a stack covers whatever it says it covers, and a missing test is invisible because nothing declares it should exist. Keeping a hand-written coverage table in the README makes the table a second source of truth that rots on the first merge.
Make scenarios/catalog.yml the canonical list of
scenarios. Every scenario has a stable ID matching (?:UI|HTTP)-[A-Z0-9-]+, a
title, a priority, and a coverage: map with one boolean per stack. Tests
reference the ID; the catalog leads and the tests follow.
Enforce it with tools/check-scenarios.py,
which reconciles catalog and tests in both directions and fails on a catalog
row marked covered with no test, a test ID with no catalog row, or a test whose
row is not marked covered. The coverage matrix is generated from the catalog, not
written by hand.
../architecture.md for the caveat in full.