Skip to content

logging: capture a record once when a non-propagating logger starts propagating (#15064) - #15138

Open
RonnyPfannschmidt wants to merge 3 commits into
pytest-dev:mainfrom
RonnyPfannschmidt:fix-15064-propagation-stand-in
Open

RonnyPfannschmidt wants to merge 3 commits into
pytest-dev:mainfrom
RonnyPfannschmidt:fix-15064-propagation-stand-in

Conversation

@RonnyPfannschmidt

@RonnyPfannschmidt RonnyPfannschmidt commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Requested by Ronny · project thread

🤖 Written by Claude Opus 5.5 via Claude Code for the pytest maintainers; I prompted it, it did the work, I read it.

Fixes #15064.

Since #14375, catching_logs attaches the capture handler to root and also directly to every logger that is non-propagating at enter. Once such a logger propagates again, a record reaches the handler on the logger and again on root.

Change

  • Each logger that is non-propagating at enter, and each of its ancestors, gets a small stand-in handler instead of the capture handler itself.
  • At emit time the stand-in walks up from its logger while propagate is true. It forwards to the capture handler only if that walk ends below root at a logger that nothing else delivers on. Otherwise root, or a stand-in further up, delivers the record. Because ancestors have stand-ins, a record that stops at an ancestor which became non-propagating mid-test (for example from a sibling logger) is still captured.
  • Loggers that share one handlers list cannot tell their records apart, so such a list gets the capture handler itself, as on main. A stand-in per logger there would capture every record twice.
  • The capture handler's own level and filters apply exactly once; the stand-in has none of its own.
  • The stand-in overrides handle(), not emit(), and takes no lock (lock = None, as logging.NullHandler does). It therefore cannot deadlock against the target handler's lock. It also skips Handler.__init__, so it is not registered in logging's global handler list.
  • There is no cache and no global or class-level state, and nothing is created per record.
  • On exit only what was added is removed. Main used to also remove the capture handler from loggers where the user had added it.

Tests

12 new tests on public behaviour (caplog, report sections, live log), across testing/logging/test_fixture.py and test_reporting.py:

  • 5 fail on main: the duplicates (the issue case, a nested logger, subtests, live log) and a sibling's record stopping at an ancestor that became non-propagating.
  • The raising-handler test also fails on main, which delivers the record anyway; this PR follows stdlib semantics there (see below).
  • 6 pass on main and guard against regressions: a logger made non-propagating by a session fixture or by a previous test, no loggers created for foreign record names, a handler also added directly to the logger, the propagation barrier moving to an ancestor, and loggers sharing one handlers list.

Cost

CPython 3.13, medians of 3 runs (emit in ns per record, enter+exit in µs), end to end median of 5:

scenario main this PR
emit, unrelated propagating logger 6234 5986
emit, non-propagating leaf, depth 3 5761 5996
emit, propagating sibling, depth 3 6251 6568
emit, propagating sibling, depth 25 7674 12521
emit, logger flipped to propagating 7353 6391
enter+exit, 300 loggers / 3 non-propagating 18 51
enter+exit, 3000 loggers / 30 non-propagating 148 198
2000 tests, 300 loggers / 3 non-propagating 2.09 s 2.18 s

At realistic depths the difference is within noise. The visible cost is one stand-in visit, about 190 ns, per ancestor a propagating record passes on its way to root, so it grows with the depth of the logger tree below a non-propagating logger. An experiment that kept stand-ins on every logger, to also cover the remaining gap below, cost 1.4–1.5x on every emitted record, so this PR does not do that.

Behaviour notes and open questions

  • Remaining gap, shared with main. A logger that becomes non-propagating mid-test is missed unless it was non-propagating at enter or is an ancestor of such a logger. A logger below one that was non-propagating at enter is still missed. Session-scoped live log and the log file likewise miss loggers made non-propagating after the session started.
  • Shared handlers lists keep main's behaviour, including its duplicate when one of those loggers starts propagating.
  • A handler that raises between the flipped logger and root. The record is lost, as stdlib logging loses it for any propagating logger. Main delivered it only because the capture handler sat directly on the logger. The new test pins the stdlib behaviour. Keep it, or drop the test?
  • logger.handlers now shows the stand-in, not the capture handler. So caplog.handler in logger.handlers changes from True to False for non-propagating loggers. Filters or a formatter added to the stand-in are ignored. Should it honour filters too?
  • Reuse across phases. Is the enter cost acceptable, or should stand-ins be reused across phases? That needs state on LoggingPlugin.

The first commit was verified on CPython 3.14 and 3.11, the follow-ups on 3.13. At the head commit the full suite passes on 3.14, and so does pre-commit run -a.

@psf-chronographer psf-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Oct 5, 2026
@RonnyPfannschmidt
RonnyPfannschmidt marked this pull request as ready for review October 7, 2026 17:07
@RonnyPfannschmidt
RonnyPfannschmidt requested review from bluetech and nicoddemus and removed request for bluetech October 7, 2026 17:08
RonnyPfannschmidt and others added 3 commits October 9, 2026 06:53
…ropagating (pytest-dev#15064)

catching_logs attached the capture handler itself to every logger that
was non-propagating at enter. Once such a logger propagates again, the
record reaches the handler on the logger and on root.

Attach a stand-in instead, which decides at emit time, from the current
propagate values up the chain, whether the walk still ends below root at
a logger nothing else delivers on.

Co-Authored-By: Claude Opus 5.5 via Claude Code <noreply@anthropic.com>
Ancestors of loggers that are non-propagating at enter now get a
stand-in as well, so a record stopping at an ancestor that becomes
non-propagating mid-test (e.g. from a sibling) is still captured.

Loggers sharing one handlers list cannot tell their records apart, so
such a list gets the pytest handler itself, as before pytest-dev#15064; with a
stand-in per logger every record was captured twice.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0183Jee4Yw5vPqSEfExPfgL9
Registering every short-lived stand-in in logging's global handler list
dominated the cost of entering catching_logs once ancestors get one too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0183Jee4Yw5vPqSEfExPfgL9
@RonnyPfannschmidt
RonnyPfannschmidt force-pushed the fix-15064-propagation-stand-in branch from 328519c to 391300f Compare October 9, 2026 05:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duplicate log capture when an initially non-propagating logger enables propagation

1 participant