Repository navigation
fix(network): cap the size of replies read from indexers - #805
Open
MoonBoi9001 wants to merge 4 commits into
Open
MoonBoi9001 wants to merge 4 commits into
MoonBoi9001 wants to merge 4 commits into
Conversation
Dipper asks each indexer's own server for its DIPs info and its indexing progress, and used to read whatever came back in full. A server sending a huge or endless reply could run dipper out of memory, so a reply over 1 MiB (DIPs info) or 4 MiB (progress) now counts as no answer.
Reading a reply into memory as structured data can take about 16 times its length, so a limit far above what an honest indexer sends still lets 1 indexer use a lot of memory. DIPs info is now limited to 64 KiB, and a progress reply to 64 KiB plus 1 KiB for each deployment it covers.
A reply is refused either by the length it declares or by counting what arrives, and the tests could not tell the 2 apart; 1 would also have filled memory had the counting been broken. Each now uses a server that stalls, so a missing check shows up as a test that times out.
MoonBoi9001
marked this pull request as ready for review
November 10, 2026 20:52
# Conflicts: # bin/dipper-service/src/network/service/liveness_checker.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dipper now stops reading a reply from an indexer's server once it passes a size limit and treats it as no answer: 64 KiB for DIPs info, and 64 KiB plus 1 KiB for each deployment asked about for indexing progress. Before, the whole reply was held in memory however long it was, so any registered indexer could run dipper out of memory with a huge or endless reply.