[dominikpeter]

What

arrowsql.spark: a Spark 4 Python data source. spark.read.format("arrowsql") reads SQL Server in parallel on the executors; df.write.format("arrowsql") bulk-copies each task's Arrow batches.

• Reads stream with read_arrow_stream and yield pyarrow.RecordBatches (no rows, no pandas). Data never passes through the driver.
• Partitioning: partitionColumn + numPartitions (+ optional lowerBound/upperBound; without bounds MIN/MAX is one query on the driver) with half-open slices from arrow_parallel's _bounds_plan; the first slice also reads the NULL rows. Or hashColumns + numPartitions with (CHECKSUM(cols) & 2147483647) % N = i. Default is one slice. Never hashes *: a query with a time-varying column (GETUTCDATE()) gave different rows per slice query.
• uint8/16/32 widened to int16/32/64 (Spark has no unsigned types). Empty results work (schema from SELECT TOP 0 * FROM (<query>) AS q).
• timestampType: ntz (default, exact TimestampNTZType) or timestamp (Spark JDBC's TimestampType, wall clock read in timeZone, default UTC; Spark does not hand its session zone to data sources, so set timeZone when it is not UTC). Writes mirror it.
• Write: append = one write_arrow(workers=1) per task, name-based column mapping. overwrite = TRUNCATE TABLE once on the driver, not transactional across tasks. dbtable is parsed and re-quoted, anything else is an error.
• Packaging: optional extra arrowsql[spark]; import arrowsql does not import pyspark. All arrowsql/pyarrow imports are inside read/write bodies.
• Docs: README section, examples/spark_datasource.py (local, ran here), examples/databricks_spark_datasource.py (notebook source), CHANGELOG, ARCHITECTURE.

Tests

tests/spark (37 tests, local SparkSession local[3], spark.driver.memory=1g, SQL Server via scripts/run_tests.py): plain read equals cursor; range partitioning with/without bounds, NULL partition values, numPartitions > distinct values, empty table, date/datetime columns; hash partitioning; time-varying query columns (hash and range); tinyint widening; empty result; timestamp modes; large read (2M rows) on a 1g driver; SQL Server -> Delta -> read back for datetime/datetime2/decimal/uniqueidentifier/varbinary/nvarchar(max)/tinyint/bit in both timestamp modes; write append/overwrite/empty; invalid options. The module skips when pyspark or Java is missing; the Delta test skips without delta-spark.
Also run: tests/port + tests/common_issues (5502 passed, 218 skipped, 1 xfailed).

No CI job added: pyspark + a JVM + delta jars are a heavy install for the runners and the Spark session start dominates the run time.

Not covered: Spark Connect clients (no session on the data source side), streaming, filter pushdown, time columns (Spark 4 has no TimeType in this path).

🤖 Generated with Claude Code