[Bug] [Python] Floating-point primary keys can lose rows during merge #10492

Search before asking

I searched existing issues for NaN and floating-point primary keys. #8532 covers Java generated record equality; this report concerns the Python primary-key writer and merge reader.

Paimon version

PyPaimon from master at 5d902e8e047f4e2e8fb1341647d52e21bb12761b (2.2-SNAPSHOT).

Compute Engine

Python API, Python 3.12.6 and PyArrow 19.0.1. The reproduction uses Parquet, one bucket, and Python planning/reading/writing/committing.

Minimal reproduce step

Write two distinct DOUBLE primary keys, 1.0 and NaN, in separate commits, then read the table:

import math
import tempfile

import pyarrow as pa
from pypaimon import CatalogFactory, Schema

with tempfile.TemporaryDirectory() as warehouse:
    catalog = CatalogFactory.create({'warehouse': warehouse})
    catalog.create_database('default', True)
    arrow_schema = pa.schema([('id', pa.float64()), ('value', pa.string())])
    catalog.create_table('default.nan_key', Schema.from_pyarrow_schema(
        arrow_schema, primary_keys=['id'], options={
            'bucket': '1', 'file.format': 'parquet',
            'scan.native-plan.enabled': 'false', 'read.native.enabled': 'false',
            'write.native.enabled': 'false', 'commit.native.enabled': 'false',
        }), False)
    table = catalog.get_table('default.nan_key')
    for row in [{'id': 1.0, 'value': 'finite'}, {'id': float('nan'), 'value': 'nan'}]:
        builder = table.new_batch_write_builder()
        writer, committer = builder.new_write(), builder.new_commit()
        try:
            writer.write_arrow(pa.Table.from_pylist([row], schema=arrow_schema))
            committer.commit(writer.prepare_commit())
        finally:
            writer.close()
            committer.close()
    rb = table.new_read_builder()
    splits = rb.new_scan().plan().splits()
    actual = rb.new_read().to_arrow(splits).to_pylist()
    print('files:', sum(len(split.files) for split in splits))
    print('raw_convertible:', [split.raw_convertible for split in splits])
    print('actual:', [dict(row, id='NaN' if math.isnan(row['id']) else row['id']) for row in actual])
    assert sorted(row['value'] for row in actual) == ['finite', 'nan'], 'finite primary key was dropped'

On the baseline, both files contain their input rows, but the merged read returns:

files: 2
raw_convertible: [False]
actual: [{'id': 'NaN', 'value': 'nan'}]
AssertionError: finite primary key was dropped

The same reproduction fails with pa.float32().

What doesn't meet your expectations?

The reader should return both rows. Java's floating-point key comparator distinguishes a finite number from NaN, treats NaNs as equal for the same remaining key fields, and orders -0.0 before +0.0 as distinct keys.

The Python primary-key comparator uses < and > to infer equality. Both return false for a finite number versus NaN, so the merge reader treats unrelated keys as equal and keeps only one version. The file-interval comparator has the same NaN problem for composite keys.

The writer also needs the same ordering and equality rules: Arrow's floating sort places NaNs first with the current null-placement option and treats signed zeros as ties, while Python tuple equality folds the two zeros together and can leave multiple NaN versions in one file. Changing only the reader would violate its sorted-run assumptions for these files.

Anything else?

I propose aligning the Python writer's sort/fold and the reader's key/interval comparisons with Java's FLOAT/DOUBLE semantics. Existing Python files may have incompatible ordering or already-folded signed-zero rows; handling those files needs a separate rewrite or recovery path. The proposed fix covers the semantics of new writes.

This report uses one bucket and full scans. NaN predicate statistics and routing different NaN payloads across buckets need separate investigation.

Are you willing to submit a PR?

• I'm willing to submit a PR!