You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg write merges -0.0 and 0.0 rows into one float/double identity partition #6138
The native Iceberg writer puts rows with -0.0 and 0.0 in a float/double identity partition column into the same partition. iceberg-java keeps them apart.
iceberg-rust holds float partition values as OrderedFloat, whose Eq and Hash treat -0.0 and 0.0 as equal. Both places the native writer groups rows by partition compare those values:
the fanout path keys its per-partition state on the partition struct (native/core/src/execution/operators/iceberg_write.rs:643-645), and iceberg-rust's FanoutWriter keys partition_writers: HashMap<Struct, _> the same way
the clustered path splits a batch into runs with *current == value (iceberg_write.rs:937)
So whichever of the two values arrives first names the partition, and the other value's rows are written into it. iceberg-java compares partition keys with Float.compare / Double.compare, which orders -0.0 before 0.0, so it writes two partitions.
This was found by reading the code (iceberg-rust pin 665c64e, ordered-float 4.6), not by a reproduction.
Steps to reproduce
CREATETABLEt (id INT, f DOUBLE) USING iceberg PARTITIONED BY (f);
-- with spark.comet.iceberg.write.enabled=true and the split operator on,-- in a single task:INSERT INTO t VALUES (1, CAST('-0.0'AS DOUBLE)), (2, 0.0D);
SELECT partition, record_count FROMt.partitions;
SELECT*FROM t WHERE f =0.0D;
Expected behavior
Two partitions, f=-0.0 and f=0.0, as iceberg-java writes. With the native writer, one data file holds both rows under a single partition value, so a filter that prunes on the other value's partition can drop rows.
Possible fixes: group rows by the float's bit pattern rather than by OrderedFloat equality, or decline native writes for float/double identity partitions until iceberg-rust distinguishes signed zeros.
Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649.
The bit-exact grouping would fix the partition split differences but it won't fix the writer issue because iceberg-rust would still produce one file because of equality check in value (0.0 vs -0.0).
Activity
sam-1112 commentedon Sep 24, 2026
take
sam-1112 commentedon Oct 1, 2026
untake
zhangfengcdt commentedon Oct 1, 2026
take
zhangfengcdt commentedon Oct 1, 2026
The bit-exact grouping would fix the partition split differences but it won't fix the writer issue because iceberg-rust would still produce one file because of equality check in value (0.0 vs -0.0).
We will need this issue to be fixed upstream - apache/iceberg-rust#3325
For now, we will need to decline these writes as an interim fix.
fix: fall back for Iceberg writes with float or double identity parti…
feat: bump iceberg-rust and write float and double identity partition…