feat(workflow-operator): export the JSON Lines and Arrow sources as Python

[kz930]

What changes were proposed in this PR?

The JSON Lines and Arrow sources implement StandaloneCodeGenerator, so an exported script starts from the same data the workflow did.

Arrow is read into pandas' nullable dtypes rather than its numpy ones. Arrow says of every value whether it is there, and a numpy column has nowhere to put that: a missing double and a stored NaN both land on NaN, and a missing integer costs the column its integer type. The engine reads the file's own answer, so a set operation keeps a NaN row a null row does not match, a holed integer column is still made of integers, and a long past 2^53 is not rounded on its way through a float. The other sources state the types they need one at a time, their formats having no way to say it once.

A file also states the width and the sign of each of its numbers, and Texera has no column narrower than a double or a 32-bit integer. A single-precision or 16-bit column was arriving null, the parse having no case for the Float and Short those vectors hand back, and an unsigned one arrived as the -1 its storage counts down to. Each now reads as the Texera type that holds it, an unsigned 32-bit column as a long and an unsigned 64-bit one refused for want of a wider one, and the script normalizes the same widths so a later sum comes to the same number on both sides. This is the rule the Parquet source already follows. The end of an exported Arrow window is counted in Long, as the CSV readers count theirs.

A timestamp is read as the engine reads one. The engine hands each field to DateParserUtils, so a row states its own format, any year reads, an offset is moved to the local zone, and the reading is cut to the millisecond. pandas' date parsing does none of that. So JSONL reads the column as text and converts it with the helper #8597 adds.

This PR also changes the engine. ArrowUtils handed a TIMESTAMP column to a Python worker as an Arrow millisecond timestamp labelled UTC. A Python operator saw an aware datetime cut to the millisecond, and a year past 2262 overflowed. It is now a zoneless microsecond timestamp. That is what Texera's own Python side declares for a TIMESTAMP (pa.timestamp("us")), and what an exported script holds. Two things a user can see change. A Python UDF now receives a naive datetime instead of one in UTC. A result downloaded as an Arrow file now carries a zoneless microsecond timestamp column instead of a UTC millisecond one. A TIMESTAMP parameter a Python UDF declares is now parsed the same way, as a naive wall clock with any offset moved to the machine's zone, so it compares with a row's instead of raising.

Any related issues, documentation, discussions?

Part of #8325, step 12b; that issue lists the set in order. It needs #8341 for the shared source code and #8597 for the timestamp helpers, so both merge first.

Closes #8702, the task this change is the whole of.

It also finishes #8602, where a scan window of no rows left the type inference nothing to read and JSONL declared a schema of no columns. #8341 fixes the CSV half.

Closes #8602.

How was this PR tested?

Each operator asserts the block it emits in its own spec, and the Arrow spec runs the engine's own executor and the generated script over the same file. The engine change is covered in both ArrowUtilsSpec files, which include a round trip on either side of the nanosecond range, and in the Python tests of the timestamp parser. Once the verification lands, every operator is also run both ways on every configuration its schema offers; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI.

Arrow is the one worth calling out. Its column of timestamps only matches once #7672 is in: before that the engine read the file's numbers through the JVM's zone while pd.read_feather read the wall clock the file states, and the two differed by whatever the machine was set to. That change has since landed.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @roshiiiz, @carloea2, @aglinxinyuan
  You can notify them by mentioning @roshiiiz, @carloea2, @aglinxinyuan in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.60%. Comparing base (535b97a) to head (d50d410).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #8704   +/-   ##
=========================================
  Coverage     92.59%   92.60%           
- Complexity     5040     5041    +1     
=========================================
  Files          1252     1252           
  Lines         53587    53571   -16     
  Branches       6673     6671    -2     
=========================================
- Hits          49619    49608   -11     
+ Misses         2317     2313    -4     
+ Partials       1651     1650    -1     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.38% <ø> (ø)
agent-service	99.16% <ø> (ø)		Carriedforward from 0215bb6
amber	88.06% <ø> (-0.01%)	⬇️	Carriedforward from 0215bb6
computing-unit-managing-service	60.48% <ø> (ø)
config-service	87.37% <ø> (ø)
file-service	82.00% <ø> (+0.47%)	⬆️
frontend	96.58% <ø> (ø)		Carriedforward from 0215bb6
notebook-migration-service	83.73% <ø> (ø)
pyamber	98.58% <100.00%> (ø)
workflow-compiling-service	74.09% <ø> (ø)		Carriedforward from 0215bb6

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

📊 Arrow Flight E2E bench

(no arrow-flight-e2e.csv in artifact)

Full dashboard · Workflow run

[kz930]

@carloea2 Split out of #8341 so each part reads in one sitting; the code is what you reviewed there, plus today's timestamp fixes.

[carloea2]

With the shared API from the declared #8341 dependency supplied locally, 32 source and Arrow tests pass but the added offset regression fails. The 42 core Arrow tests and 73 Linux Python tests also pass.

[kz930]

@carloea2 Fixed the offset-past-EOF case in d50d410: when the window sample is empty, the JSONL schema now samples from the top of the file, with your offset=10 repro as a test. Could you take another look?

[carloea2]

Retested the current head. The offset past EOF regression is fixed. All 75 focused tests and scalafmtCheckAll pass with the shared source API from #8341 supplied locally.

Command: sbt -batch "WorkflowCore/testOnly *ArrowUtilsSpec -- -oDF" "WorkflowOperator/testOnly *ArrowSourceOpDescSpec *JSONLScanSourceOpDescSpec *ArrowUtilsSpec -- -oDF" "scalafmtCheckAll".

Current CI still fails compilation on that missing API. This verifies the fix, not standalone merge readiness. #8341 and #8597 still need to land first, as the description states.