fix(sdk): split chunks >1.8 GiB in forwarding_thread to prevent gRPC 2 GiB crash (#11993) - #12937
Open
Saraswat123 wants to merge 1 commit into
Open
Saraswat123 wants to merge 1 commit into
Saraswat123 wants to merge 1 commit into
Conversation
… limit crash gRPC/protobuf uses a signed 32-bit length prefix, capping single messages at 2 GiB. Chunks larger than this limit cause a panic/disconnect in the write path. Wire Chunk::split_rows (already used in re_chunk_store) into forwarding_thread so that any chunk exceeding 1.8 GiB is split into sub-chunks before encoding. The 1.8 GiB threshold leaves headroom for encoding overhead. Fixes rerun-io#11993
Contributor
There was a problem hiding this comment.
Hi! Thanks for opening this pull request.
Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wire the existing
Chunk::split_rowsutility intoforwarding_threadso that any chunk exceeding 1.8 GiB is split into sub-chunks before encoding and forwarding to sinks.Fixes #11993.
Background
gRPC (and protobuf) use a signed 32-bit length prefix, capping a single message at 2 GiB. When a recording contains very dense data — e.g. a 2 GiB+ point cloud logged in a single
log()call — the resulting chunk hits this limit and the gRPC write path crashes/disconnects.The fix, suggested by @emilk in the issue, is to split oversized chunks before transmitting them.
Chunk::split_rowsalready exists for exactly this purpose (it is used inre_chunk_storefor storage bookkeeping) but was never called in the forwarding path.Solution
Add a
FORWARDING_CHUNK_SPLIT_OPTIONSconstant withchunk_max_bytes = 1_800_000_000(1.8 GiB — leaves ~340 MiB headroom below the 2 GiB limit) and callChunk::split_rowsin both chunk-receive paths insideforwarding_thread:while let Ok(chunk) = chunks.try_recv()drain loop (runs on flush/shutdown).recv(chunks)arm of theselect!loop (the hot path).For chunks under 1.8 GiB
split_rowsreturns the original chunk immediately (no allocation), so there is no overhead on the normal code path.Testing
The repro from the issue (C++ snippet logging a 2 GiB+ point cloud via gRPC) should no longer crash. I do not have the full build environment here to run the full test suite, but the change is confined to
forwarding_threadand the logic is straightforward.🤖 Generated with Claude Code
https://claude.ai/code/session_018Thk1PDVQb2A4yrRmg8vRj