Skip to content

Native Iceberg write merges -0.0 and 0.0 rows into one float/double identity partition #6138

Closed

Description

@andygrove

Describe the bug

The native Iceberg writer puts rows with -0.0 and 0.0 in a float/double identity partition column into the same partition. iceberg-java keeps them apart.

iceberg-rust holds float partition values as OrderedFloat, whose Eq and Hash treat -0.0 and 0.0 as equal. Both places the native writer groups rows by partition compare those values:

  • the fanout path keys its per-partition state on the partition struct (native/core/src/execution/operators/iceberg_write.rs:643-645), and iceberg-rust's FanoutWriter keys partition_writers: HashMap<Struct, _> the same way
  • the clustered path splits a batch into runs with *current == value (iceberg_write.rs:937)

So whichever of the two values arrives first names the partition, and the other value's rows are written into it. iceberg-java compares partition keys with Float.compare / Double.compare, which orders -0.0 before 0.0, so it writes two partitions.

This was found by reading the code (iceberg-rust pin 665c64e, ordered-float 4.6), not by a reproduction.

Steps to reproduce

CREATE TABLE t (id INT, f DOUBLE) USING iceberg PARTITIONED BY (f);
-- with spark.comet.iceberg.write.enabled=true and the split operator on,
-- in a single task:
INSERT INTO t VALUES (1, CAST('-0.0' AS DOUBLE)), (2, 0.0D);
SELECT partition, record_count FROM t.partitions;
SELECT * FROM t WHERE f = 0.0D;

Expected behavior

Two partitions, f=-0.0 and f=0.0, as iceberg-java writes. With the native writer, one data file holds both rows under a single partition value, so a filter that prunes on the other value's partition can drop rows.

Possible fixes: group rows by the float's bit pattern rather than by OrderedFloat equality, or decline native writes for float/double identity partitions until iceberg-rust distinguishes signed zeros.

Additional context

Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649.

Activity

  1. sam-1112 commented on Sep 24, 2026

    @sam-1112
    Contributor

    take

  2. added
    priority:criticalData corruption, silent wrong results, security issues
    and removed on Sep 28, 2026
  3. added this to the 1.2.0 milestone on Sep 28, 2026
  4. sam-1112 commented on Oct 1, 2026

    @sam-1112
    Contributor

    untake

  5. zhangfengcdt commented on Oct 1, 2026

    @zhangfengcdt
    Member

    take

  6. zhangfengcdt commented on Oct 1, 2026

    @zhangfengcdt
    Member

    The bit-exact grouping would fix the partition split differences but it won't fix the writer issue because iceberg-rust would still produce one file because of equality check in value (0.0 vs -0.0).

    We will need this issue to be fixed upstream - apache/iceberg-rust#3325

    For now, we will need to decline these writes as an interim fix.

  7. added a commit that references this issue on Oct 2, 2026
    f57b232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions