Skip to content

protype: Unify _pyio.Buffered* around _pyio.Nibbler - #3

Draft
cmaloney wants to merge 4 commits into
mainfrom
exp/pyio_nibbler_v2
Draft

cmaloney wants to merge 4 commits into
mainfrom
exp/pyio_nibbler_v2

Conversation

@cmaloney

Copy link
Copy Markdown
Owner

No description provided.

cmaloney and others added 4 commits May 14, 2026 14:23
Replaces BufferedWriter's bytearray + _write_lock with a _Nibbler buffer
manager built around _Morsel work units. Adds FileIO._writev so the
nibbler can issue a single writev(2) for multi-buffer flushes when
available, falling back to coalesce+write otherwise.

The Nibbler holds its lock end-to-end across raw I/O, matching the
pre-existing cross-thread write ordering guarantee. A future two-lock
split (short buffer lock + I/O lock) is sketched in the class docstring
as the intended path for releasing the I/O lock once profiling justifies
it.

Also restores f.close() in BufferSizeTest.try_one's nested finally and
skips test_slow_close_from_thread pending a way to validate pythongh-76157 /
pythongh-31976 that does not assume the old lock-across-I/O behavior.

See: https://discuss.python.org/t/reworking-buffered-i-o-in-cpython/90019

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Grow _Nibbler to be the actual buffered I/O implementation that
BufferedReader / BufferedWriter / BufferedRandom subclass directly,
mirroring how the _io C struct `buffered` backs all three Python types.
The class absorbs the responsibilities of the deleted _BufferedIOMixin
plus the previous writer-only _Nibbler, and gains a full read API.

  - One concrete implementation. read/write/seek/tell/close/etc. all
    live on _Nibbler. Subclasses define __init__ validation and override
    readable()/writable() to gate which methods are permitted (the same
    pattern _io uses in C).
  - Mode flag (READING/WRITING) tracked under the lock. The first write
    flips READING -> WRITING; the first read after a write flushes and
    flips back. Mode switches happen atomically inside the nibbler so
    BufferedRandom no longer needs ad-hoc flush/seek-back coordination
    in every override.
  - buffer_size=0 supported on both reader and writer. Writer flushes
    on every append. Reader passes through to raw for read/read1/
    readinto. peek raises UnsupportedOperation since there is no
    buffer to peek into.
  - BufferedRandom keeps the (BufferedWriter, BufferedReader) MRO; the
    cooperative super() chain produces a single _Nibbler.__init__ call.
    Random must override readable()/writable() to True since MRO would
    otherwise inherit False from BufferedWriter.
  - BufferedRWPair is unchanged - it composes two independent streams
    so the unified-buffer model does not apply.

Test changes: PyBufferedReader.test_constructor's buffer_size=0
ValueError check is gated to the C implementation (matches the
pre-existing gating for the writer side).

The follow-on stage routes open(buffering=0) through BufferedIO so
all open() forms retry partial writes uniformly.

See: https://discuss.python.org/t/reworking-buffered-i-o-in-cpython/90019

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Stage 2 of the unified-Nibbler work. With BufferedReader / BufferedWriter
/ BufferedRandom now accepting buffer_size=0 (Stage 1), open(buffering=0)
no longer needs to return a raw FileIO - it can return a BufferedIO with
buffer_size=0 that retries partial writes. This unifies the open() forms
so all of them either consume all data or raise (PEP-475 EINTR plus the
write-retry loop), removing FileIO's POSIX-style "check the return size"
quirk for callers that just used buffering=0 to avoid buffering.

  - open(buffering=0, mode='rb'/'wb'/'r+b'/'ab'/'xb') now returns a
    BufferedReader / BufferedWriter / BufferedRandom with buffer_size=0
    (was: raw FileIO for binary, ValueError for text).
  - The text-mode rejection of buffering=0 stays.
  - Add _Nibbler.readall() so BufferedWriter (now what's returned for
    open('wb', 0)) raises UnsupportedOperation through self.read(-1)
    when callers reach for the FileIO-style readall, matching the prior
    error type.
  - Update test_abc_inheritance / test_abc_inheritance_official to
    reflect that _pyio.open(buffering=0, 'wb') returns a BufferedIOBase
    instance, not a RawIOBase. The C io.open is unchanged so the assertion
    is gated on which implementation the test is exercising.

This is _pyio only. The C _io.open will need the same routing change
when the C nibbler lands.

See: https://discuss.python.org/t/reworking-buffered-i-o-in-cpython/90019

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant