feat(spark): add to_binary and try_to_binary

[arsathS]

Which issue does this PR close?

• Part of [EPIC] Complete datafusion-spark Spark Compatible Functions #15914

Rationale for this change

Spark's to_binary(str[, fmt]) converts a string to binary, choosing how based
on fmt: hex, utf-8/utf8, or base64. try_to_binary is the same
function except that input it cannot convert yields NULL instead of raising an
error.

Neither is available in the datafusion-spark crate, and no core DataFusion
function can stand in for them:

• core decode accepts only base64, base64pad and hex, so there is no
  way to ask it for Spark's utf-8 conversion
• core decode always raises on input it cannot convert, so there is nothing
  to build try_to_binary's NULL-returning behaviour from

What changes are included in this PR?

• New SparkToBinary and SparkTryToBinary UDFs in
  datafusion/spark/src/function/string/to_binary.rs
  • to_binary(str[, fmt]) where fmt is a case-insensitive literal of
    hex, utf-8, utf8 or base64, defaulting to hex
  • hex decodes two characters per byte, left-padding an odd-length input
    with 0, matching Unhex
  • utf-8 / utf8 returns the string's own UTF-8 bytes
  • base64 matches Java's MIME decoder, which Spark uses: the standard
    alphabet, padding optional, and the unused trailing bits of a short final
    group ignored
  • A NULL value or a NULL fmt yields NULL in both functions
  • to_binary raises on a malformed value or an unrecognised fmt;
    try_to_binary returns NULL for both, matching nullOnInvalidFormat
  • fmt must be foldable, as Spark requires
• unhex_scalar is made pub(crate) and reused for the hex path rather
  than duplicating the decoder
• Registered in mod.rs (make_udf_function!, export_functions!,
  functions())
• Filled in the commented-out SLT scaffolds for both functions

Are these changes tested?

Yes.

• 10 unit tests in to_binary.rs (each format, hex as the default, empty
  input, NULL value and NULL format, invalid value, invalid format, column
  input, and a column where one row is invalid)
• 15 SLT assertions across spark/string/to_binary.slt and
  spark/string/try_to_binary.slt

Are there any user-facing changes?

No. These are new functions in the datafusion-spark crate only.

[arsathS]

cc @Jefffrey This is my first time contribution. Can you please take a look?

[Jefffrey]

ci errors are because base64 has a 0.23.1 now, so need to update the lockfile (just rerun cargo check/build and commit again)

[arsathS]

@Jefffrey Sorry for the delay. I have resolved the comments.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.05155% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.23%. Comparing base (ebea069) to head (ac92e5a).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/spark/src/function/string/to_binary.rs	84.81%	27 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #24266    +/-   ##
========================================
  Coverage   81.22%   81.23%            
========================================
  Files        1113     1114     +1     
  Lines      392508   392714   +206     
  Branches   392508   392714   +206     
========================================
+ Hits       318833   319024   +191     
- Misses      54916    54930    +14     
- Partials    18759    18760     +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[arsathS]

Thanks @Jefffrey

[Jefffrey]

thanks @arsathS

[arsathS]

Sorry to disturb @Jefffrey. When can we start using these functions which are getting added? Should we need to wait for the release? If so what will be the frequency of the releases which are made?

[Jefffrey]

itll like come in v56 release targeted for october

• Release DataFusion 56.0.0 (Oct/Nov 2026) #24461