Skip to content

Off-by-one buffer overrun in external scanner serialize() with deep indentation and an open string #350

Description

@Pyre909

tree_sitter_python_external_scanner_serialize writes the indent stack in a loop guarded by size < TREE_SITTER_SERIALIZATION_BUFFER_SIZE, then writes two bytes per iteration. When the header before the loop has odd length (2 + delimiter_count, so one open string delimiter), size can reach 1023. The last iteration then writes one byte past the 1024-byte buffer and returns 1025, and the runtime aborts:

Assertion failed: (length <= 1024), function ts_parser__external_scanner_serialize, file parser.c, line 409.

uint32_t iter = 1;
for (; iter < scanner->indents.size && size < TREE_SITTER_SERIALIZATION_BUFFER_SIZE; ++iter) {
uint16_t indent_value = *array_get(&scanner->indents, iter);
buffer[size++] = (char)(indent_value & 0xFF);
buffer[size++] = (char)((indent_value >> 8) & 0xFF);
}

Repro (tree-sitter-python 0.25.0, tree-sitter 0.27.0 runtime, macOS arm64; src/scanner.c is byte-identical on master at 26855ea): 600 lines of if 1:, each indented one space more than the last, then 'a' as the body of the innermost one.

python3 -c "print(''.join(' ' * i + 'if 1:\n' for i in range(600)) + ' ' * 600 + \"'a'\")" > deep.py

The same shape at 400 levels parses.

Suggested fix: test for the bytes about to be written:

for (; iter < scanner->indents.size && size + 2 <= TREE_SITTER_SERIALIZATION_BUFFER_SIZE; ++iter) {

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions