Test coverage¶

make test-coverage runs a selected set of tests and ends with three separate coverage percentages: Python lines, theme JavaScript code lines, and mapped feature checks. These measure different things; they are not combined into a single score. Run make test-all for the complete test suite instead.

Which tests run?¶

The coverage target first builds the sample HTML, then runs pytest once under coverage.py with -m 'not slow or coverage_js or coverage_style'. This selects all fast tests and three slow tests. The coverage_js browser journey (tests/test_features.py::test_features_slow) visits the built site in Chromium and tests cookie consent, navigation, the small-screen drawer, tabs, copying code to the clipboard, dark mode, and search; it also collects JavaScript execution data. Two coverage_style checks run in Chromium: tests/test_scss_propagation.py::test_rendered_color verifies the SCSS fixture’s computed colour, and tests/test_structured_toc.py::test_structured_toc_styling_slow verifies Ulwazi’s structured-TOC styling. The shared built_site fixture builds the sample docs inside pytest, so the theme’s Sphinx hooks contribute to Python coverage. A separate HTML build remains a prerequisite for tests that still read docs/_build.

The other slow tests are not run by make test-coverage:

PDF generation checks that a PDF is produced. Its separate build does not exercise the theme’s Python hooks inside the measured pytest process.
Python-version compatibility installs and builds in isolated interpreters; their subprocesses are not instrumented by this coverage run.
The remote-asset check is network-dependent and advisory. It checks availability, not theme code-line execution.
The other browser styling checks, such as list-marker size and sidebar spacing, verify rendered details without a feature-manifest mapping. The two coverage_style checks are included because manifest entries depend on them.
The code quality linter cases run make lint targets in subprocesses that don’t load the theme. The fast code quality checks do run, but they have no tests/features.yaml entries because they check repository consistency, not theme features.

These tests remain useful and are available via make test-slow and make test-all. The coverage command is not a replacement for either: including every slow test would add PDF/system dependencies, network calls, and per-version builds without making the three percentages more meaningful.

What does each number measure?¶

Python line coverage¶

The final summary reads results/coverage.xml, generated by coverage.py for the ulwazi Python package. The numerator is executed source lines; the denominator is executable source lines in that package. Generated ulwazi/_version.py and test code are excluded. The in-process Sphinx build covers theme setup and page transformations; the standalone ulwazi/product_menu_gen.py command is still in the denominator, but is not run by the docs tests. Not every executed line necessarily has its output asserted by a test.

uv run coverage report -m and htmlcov/index.html provide more detail. They also show branch coverage. When branch measurement is enabled, their headline percentage combines statements and branches and can differ from the Python line percentage in the final three-metric summary. The coverage.py HTML report has its own Files, Functions, and Classes tabs; it does not show the JavaScript or feature percentages.

Theme JavaScript line coverage¶

The browser journey records Chromium V8 execution ranges for scripts under ulwazi/theme/ulwazi/static/js/. The numerator is code lines touched by an executed range; the denominator is code lines in all shipped theme JS files, including scripts not executed by the journey. Blank and comment-only lines and third-party assets are excluded. The final percentage pools line counts across files rather than averaging the per-file percentages. See results/js-coverage.json for per-file lines and percentages.

This is Chromium execution coverage, not proof that each executed branch behaved correctly, a CSS coverage measure, or a cross-browser result. A script loaded by the browser can execute without all its behavior being asserted by the feature test.

Mapped feature checks¶

tests/features.yaml is a curated manifest drawn from the MyST and reST cheatsheets, dedicated fixture pages, existing assertions, and the testing strategy. Each entry has a source to explain what it refers to and checks containing pytest node IDs. An entry counts as checked only when all its mapped tests pass in the selected pytest run. Known behaviors without a qualifying test have checks: [] and stay in the denominator as unverified. The final output names them.

The manifest describes specific checks, not whole Sphinx components. For example, the TOC check verifies Vanilla markup and H4/H5 depth on the two cheat sheets, not every navigation variant; the asset check inspects a small set of pages, not the entire site. A feature with both markup and browser checks lists both pytest IDs so that both must pass. A known behavior without a qualifying assertion (such as tab synchronization across sets) remains unverified.

This is not an automatic count of all cheatsheet sections or all possible theme behavior. In particular, a grouped test such as test_features_fast checks several behaviors in one pytest result: its passing result counts for each feature mapped to it, but does not independently measure the quality or completeness of every assertion. When adding a feature or changing a test, review its manifest mapping; avoid mapping a feature merely because it appears in a fixture page. Results for this run are stored in results/feature-coverage.json.

Map a theme feature to coverage¶

First add the test and its reporting category/fast-or-slow tier using the test-addition checklist. The reporting category does not affect feature coverage; the slow marker does affect whether make test-coverage selects the test. Then:

Add an example in both docs/content/myst-cheat-sheet.md and docs/content/rst-cheat-sheet.rst when equivalent syntax exists. Use a dedicated sample page for site-level features.
Assert the rendered result or interaction in a relevant test. A build or script load alone is not an assertion of behavior.
In tests/features.yaml, add or update one narrowly named markup or site entry. Set source to the fixture and checks to every pytest node ID required (for example, tests/test_features.py::test_features_fast). Use checks: [] if there is no qualifying assertion. Do not duplicate an existing feature entry.
Run make test-coverage; check its final summary and results/feature-coverage.json. The entry counts as checked only when all listed tests are selected and pass. Fast tests are selected by default; slow tests need a coverage marker in pyproject.toml, that marker on the test, and inclusion in the Makefile marker expression.

Reading the final summary¶

The last lines of make test-coverage give the three scoped totals and a list of unverified manifest entries. Its summary combines results/coverage.xml, results/js-coverage.json, and results/feature-coverage.json from the same run; missing data fails the command rather than silently displaying an old percentage. Neither a high code-line percentage nor a high mapped-feature percentage alone guarantees that the theme is bug-free.

Pull request check¶

The Coverage GitHub Actions workflow runs make test-coverage on pull requests. Its comparison check is titled Coverage: Py: …, JS: …, Feat: … and its check summary shows the three PR percentages next to the base revision. It compares unrounded rates and fails if any one decreases; Python and JavaScript have source-line denominators, while the feature rate uses the curated manifest, so adding an untested feature can lower the latter.

The base is rebuilt at the PR’s current base commit only if it has the same three-metric reporting format. If it does not (for example, on the first PR introducing coverage), the check displays the PR numbers and says “baseline unavailable” rather than treating the missing baseline as 0%. The check summary is visible on the PR without posting or updating a comment.

The baseline uses its own implementation, tests, fixtures and test inventory. PR test files are not copied into it: new regression assertions can require fixes absent from the base, and new test cases can invalidate its inventory. Local make test-all tests the current checkout; it does not run this separate baseline comparison.