Skip to content

[Bug] pmod returns a wider decimal type than Spark #23895

Closed

Description

@andygrove

Describe the bug

SparkPmod::return_type returns arg_types[0] unchanged. For decimal inputs
Spark instead derives the result type with Pmod.resultDecimalType, which
follows the Remainder rule:

val resultScale = max(s1, s2)
val resultPrecision = min(p1 - s1, p2 - s2) + resultScale

Because DataFusion's Signature::numeric first coerces both arguments to a
common decimal type, the two precisions it sees are already equal and the rule
collapses to the input precision. Spark applies the rule to the declared
argument types, so it produces a narrower type.

The computed values agree. Only the reported precision differs.

To Reproduce

Spark 4.2.0 (verified with pyspark==4.2.0):

SELECT typeof(pmod(CAST(2.5 AS DECIMAL(3,1)), CAST(1.2 AS DECIMAL(2,1))));
-- decimal(2,1)

DataFusion:

SELECT arrow_typeof(pmod(2.5::decimal(3,1), 1.2::decimal(2,1)));
-- Decimal128(3, 1)

Expected behavior

The result type follows Spark's resultDecimalType, including the
spark.sql.decimalOperations.allowPrecisionLoss branch that calls
DecimalType.adjustPrecisionScale.

Impact

Low. The values are identical and no test in the tree depends on the narrower
type. It matters for schema-sensitive consumers and for any downstream overflow
check that is driven by the declared precision.

Relevant code

datafusion/spark/src/function/math/modulus.rs, SparkPmod::return_type.
SparkMod::return_type has the same shape and Spark's Remainder uses the same
rule, so a fix should probably cover both.

Surfaced by the audit-datafusion-spark-expression skill.

Activity

  1. added
    bugSomething isn't working
    sparkChanges to datafusion-spark crate
    on Jul 25, 2026
  2. amitvijapur commented on Aug 5, 2026

    @amitvijapur
    Contributor

    take

  3. added a commit that references this issue on Sep 6, 2026
    262936e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingsparkChanges to datafusion-spark crate

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions