You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] pmod returns a wider decimal type than Spark #23895
SparkPmod::return_type returns arg_types[0] unchanged. For decimal inputs
Spark instead derives the result type with Pmod.resultDecimalType, which
follows the Remainder rule:
Because DataFusion's Signature::numeric first coerces both arguments to a
common decimal type, the two precisions it sees are already equal and the rule
collapses to the input precision. Spark applies the rule to the declared
argument types, so it produces a narrower type.
The computed values agree. Only the reported precision differs.
The result type follows Spark's resultDecimalType, including the spark.sql.decimalOperations.allowPrecisionLoss branch that calls DecimalType.adjustPrecisionScale.
Impact
Low. The values are identical and no test in the tree depends on the narrower
type. It matters for schema-sensitive consumers and for any downstream overflow
check that is driven by the declared precision.
Relevant code
datafusion/spark/src/function/math/modulus.rs, SparkPmod::return_type. SparkMod::return_type has the same shape and Spark's Remainder uses the same
rule, so a fix should probably cover both.
Surfaced by the audit-datafusion-spark-expression skill.
Activity
amitvijapur commentedon Aug 5, 2026
take
docs: add an agent skill for auditing datafusion-spark expressions (a…
docs: add an agent skill for auditing datafusion-spark expressions (a…
fix(spark): derive pmod's decimal result type from the declared argum…
fix(spark): derive pmod's decimal result type from the declared argum…