You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug] pmod rejects string arguments that Spark implicitly casts to a numeric type #23896
Spark's Pmod declares inputType = NumericType and relies on the analyzer's
string promotion rule, so a string argument is implicitly cast. DataFusion's SparkPmod uses Signature::numeric(2, Volatility::Immutable), which rejects
strings at planning time.
To Reproduce
Spark 4.2.0 (verified with pyspark==4.2.0), and covered by Spark's own golden
file sql/core/src/test/resources/sql-tests/results/typeCoercion/native/promoteStrings.sql.out:
SELECT pmod('1', CAST(1AS TINYINT)); -- 0, result type bigintSELECT pmod('10', CAST(3ASINT)); -- ansi on: 1 (int); ansi off: 1.0 (double)SELECT pmod('1', '1'); -- DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE
Note that Spark's promotion target differs by mode: with spark.sql.ansi.enabled = true the string is cast to the other argument's type,
and with it disabled the pair is promoted to double. Two strings are rejected
in both modes.
DataFusion:
SELECT pmod('10'::string, 3::int);
-- Error during planning: For function 'pmod' Utf8 and Int32 are not coercible-- to a common numeric type.
Expected behavior
A string argument is accepted and coerced the way Spark coerces it.
Scope
This is not specific to pmod. Every datafusion-spark function that uses Signature::numeric inherits it, and reproducing Spark's mode-dependent
promotion target needs a decision about how far datafusion-spark should go in
emulating Spark's analyzer. Filing it so the gap is visible rather than
proposing a pmod-local workaround.
A commented-out query recording the Spark 4.2.0 result is in datafusion/sqllogictest/test_files/spark/math/pmod.slt.
Activity
amitvijapur commentedon Aug 5, 2026
take
docs: add an agent skill for auditing datafusion-spark expressions (a…
docs: add an agent skill for auditing datafusion-spark expressions (a…