Add an error_later warning filter action

[RonnyPfannschmidt]

Written by Claude Opus 5.5 via Claude Code for the pytest maintainers; I prompted it, it did the work, I read it.

Draft: the name, exit code 7 and the xdist hook in core are open questions, listed below.

A test-runner version of -W error, as suggested in #14931 (review): track where warnings that are errors come from, let the current test continue, and report the full picture as an error.

error raises at the warnings.warn call site. That aborts the code under test halfway through, so a test that emits several warnings reports only the first. error_later records the warning instead, lets the code run to completion, and fails afterwards.

Behaviour

• error_later is accepted wherever a warning filter is: -W, the filterwarnings ini option and @pytest.mark.filterwarnings.
• The stdlib itself decides whether error_later applies, with its own filter matching: message, category, module and line fields all work, and precedence is the stdlib's, including filters installed by the code under test, fixtures and conftest. See "How matching works" below.
• error_later_report = test (the default) fails the phase that emitted the warning. If that phase already failed, its failure gets a "late warning errors" section instead of a second report.
• error_later_report = session lets tests pass and fails the run at the end with a new ExitCode.LATE_WARNING_ERROR (7).
• Warnings that belong to no test end up in the end-of-run section and set exit code 7 in both modes. That covers collection, conftest and plugin imports, pytest_configure, pytest_sessionfinish, and skipped or xfailed tests.
• The summary line counts them as "N late warning error(s)", in red.
• pytest-xdist: workers pass their late warnings to the controller through workeroutput, merged in pytest_testnodedown, so both modes behave the same with -n.
• Suites without an error_later filter pay nothing: the check returns immediately.

Open questions

• Name. error_later has an underscore, while the stdlib actions are single words. It is also rejected by PYTHONWARNINGS and python -W at interpreter start.

• Exit code. ExitCode.LATE_WARNING_ERROR = 7 is new public API.

• xdist hook. Core now implements an xdist hook, pytest_testnodedown, the same way pytest-cov does. It is only registered when the hookspec exists.

• Filters installed by the test outrank error_later. A test's own simplefilter("always") or catch_warnings now takes precedence over an error_later filter, just as it already does over error. Warnings re-emitted by pytest.warns carry a module name derived from the file path, so module-restricted filters miss them, again as for error.

How matching works

Each error_later filter is installed as a real always filter whose module slot holds a probe object. The stdlib calls .match() on it, so the probe sees the exact module name and records a verdict when the filter matches. The stdlib also calls .match() on filters that go on to lose on category or line, so a never-matching loss probe follows each one and clears the verdict. The probe only leaves the verdict pending. A wrapper around warnings._showwarnmsg, which the stdlib calls for every warning it shows, makes it current for that one call, so pytest's recording picks it up whether the warning arrives directly or through a showwarning hook that forwards it (numpy's suppress_warnings, #15139). When the call returns the verdict is gone, so a warning taken by pytest.warns, by the test's own catch_warnings or by a hook that discards it leaves nothing behind for the next one. Without an error_later filter, recording stays a plain list.append.

Relies on

Behaviour that is not documented, checked against the CPython 3.10–3.14 sources:

• _warnings.c:check_matched (and the pure-Python loop) calls .match() on whatever object sits in a filter tuple's message and module slots; only the 5-tuple shape and a str action are validated.
• With action always, the warning is shown synchronously in the emitting thread, with no other filter search in between.
• _warnings.c:call_show_warning (and the pure-Python warn_explicit) looks up warnings._showwarnmsg on every shown warning, so replacing it takes effect at once. catch_warnings does not restore it, so pytest puts it back when its own warnings context exits.
• Private names: warnings.filters (direct insertion), warnings._filters_mutated(), warnings._showwarnmsg, warnings._showwarnmsg_impl, and on 3.14 warnings._get_filters() and warnings._get_context().log.
• The probes are visible in warnings.filters; they expose .pattern but not the rest of re.Pattern.

Verified on 3.10, 3.14, 3.14 with context-aware warnings and free-threaded 3.14t; 3.11–3.13 only through a standalone check of the mechanism. The _showwarnmsg wrapper is verified on 3.11–3.14 and on 3.14 with context-aware warnings, not on 3.10 or 3.14t.

Related

• Fixes error_later is silently dropped when a test wraps warnings.showwarning #15139: error_later was dropped when a test wraps warnings.showwarning.
• Shared PytestWarning instances in deprecated.py accumulate tracebacks under -W error #14912 / fix: stop reusing shared PytestWarning instances whose tracebacks grow under -W error #14931: independent of this PR. The shared-instance traceback growth is fixed there; this PR does not include a fix for it.
• Fix warning filter precedence across test scopes #15087 changes the order in which mark filters apply across scopes. error_later follows the order pytest applies filters in, so it inherits whatever Fix warning filter precedence across test scopes #15087 settles.

Known gaps

• Warnings from pytest_sessionfinish on xdist workers are still lost.
• On 3.14 with context-aware warnings, threads do not inherit the warnings context, so their warnings never reach pytest. That is stdlib behaviour.

Not verified

The Sphinx build, PyPy, 3.13t, and interaction with --pdb and -x.