[Bug] range-bitmap file index: IS NULL returns only the first and last row when a column is entirely NULL in a file #10332

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

2.0.0 (paimon-bundle-2.0.0)

Compute Engine

None — reproduced with the Java API (ReadBuilder.withFilter), local filesystem catalog, Java 21.

Minimal reproduce step

Create an append table with a range-bitmap file index on c, write 5 rows where c is NULL in every row, then read with c IS NULL:

Identifier id = Identifier.create("db", "t_isnull");
catalog.createTable(
    id,
    Schema.newBuilder()
        .column("id", DataTypes.INT())
        .column("c", DataTypes.INT())
        .option("file-index.range-bitmap.columns", "c")
        .build(),
    false);
Table table = catalog.getTable(id);
List<GenericRow> rows = new ArrayList<>();
for (int i = 0; i < 5; i++) {
  rows.add(GenericRow.of(i, null));
}
write(table, rows);

Predicate isNull = new PredicateBuilder(table.rowType()).isNull(1);
System.out.println(count(table, isNull));   // prints 2
System.out.println(count(table, null));     // prints 5

Helper methods:

static void write(Table table, List<GenericRow> rows) throws Exception {
  BatchWriteBuilder builder = table.newBatchWriteBuilder();
  try (BatchTableWrite write = builder.newWrite();
      BatchTableCommit commit = builder.newCommit()) {
    for (GenericRow row : rows) write.write(row);
    commit.commit(write.prepareCommit());
  }
}

static long count(Table table, Predicate filter) throws Exception {
  ReadBuilder builder = table.newReadBuilder();
  if (filter != null) builder = builder.withFilter(filter);
  long[] count = new long[1];
  try (RecordReader<InternalRow> reader =
      builder.newRead().createReader(builder.newScan().plan())) {
    reader.forEachRemaining(row -> count[0]++);
  }
  return count[0];
}

The same table with file-index.bitmap.columns or file-index.bsi.columns instead of range-bitmap returns 5.

What doesn't meet your expectations?

c IS NULL returns 2 rows; all 5 rows match. The rows returned are the first and the last row of the file; every row in between is dropped.

The same table with file-index.bitmap.columns or file-index.bsi.columns instead of range-bitmap returns 5.

Anything else?

Root cause is in org.apache.paimon.fileindex.rangebitmap.RangeBitmap#isNull. When the column has no non-null value in the file (cardinality <= 0), it returns:

RoaringBitmap32.bitmapOf(0, rid - 1)

bitmapOf(int...) takes a list of positions, so this is the two-element set {0, rid - 1}, not the range [0, rid). Any file with three or more rows where the indexed column is entirely NULL loses its middle rows for IS NULL.

Suggested fix: return the full range, e.g. RoaringBitmap32.bitmapOfRange(0, rid). isNotNull on the same reader is correct (it returns an empty bitmap for such a file), so the complement of isNotNull over [0, rid) gives the right answer too.

Are you willing to submit a PR?

• I'm willing to submit a PR!