feat(pyamber): read Arrow rows across chunk boundaries

[carloea2]

What changes were proposed in this PR?

Read Arrow fields by their global row index. Using the first column's chunk coordinates for every column crashes or reads the wrong value when columns have different chunk boundaries.

Any related issues, documentation, discussions?

Closes #8209

Replaces #8210, which the stale bot closed. The original PR cannot be reopened by this account.

How was this PR tested?

Retested with Python 3.12 on Windows in a fresh detached worktree from upstream main at 7426431337, which includes the merged loop-completion change #8938. Regression before implementation: 1 failed. After implementation: 354 passed, 2 deselected, 1 expected failure.

The two deselected tests belong to the existing Windows pre-epoch hash fixture; they are covered by #8957. The Arrow chunk regression is included.

Generate protobufs with bash bin/python-proto-gen.sh, then run from amber:

python -m pytest src/test/python/core/models src/test/python/core/architecture/sendsemantics/test_partitioners.py --deselect src/test/python/core/models/test_tuple.py::TestTuple::test_hash -q --tb=line -p no:cacheprovider
python -m ruff check src/main/python src/test/python
python -m ruff format --check src/main/python src/test/python

Ruff lint and formatting pass for all 217 Python files.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-6)

[carloea2]

This replaces stale-closed #8210. The problem still reproduces on current main. The patch passes its regression, 350 broader tests and Ruff checks.

/request-review @eugenegujing

[carloea2]

/request-review @eugenegujing

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @eugenegujing, @aglinxinyuan
  You can notify them by mentioning @eugenegujing, @aglinxinyuan in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.30%. Comparing base (7426431) to head (25e6239).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8956      +/-   ##
============================================
- Coverage     92.30%   92.30%   -0.01%     
  Complexity     5046     5046              
============================================
  Files          1260     1260              
  Lines         53887    53880       -7     
  Branches       6711     6709       -2     
============================================
- Hits          49743    49734       -9     
- Misses         2486     2488       +2     
  Partials       1658     1658              

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.38% <ø> (ø)		Carriedforward from 7426431
agent-service	99.16% <ø> (ø)		Carriedforward from 7426431
amber	87.32% <ø> (ø)		Carriedforward from 7426431
computing-unit-managing-service	60.48% <ø> (ø)		Carriedforward from 7426431
config-service	87.37% <ø> (ø)		Carriedforward from 7426431
file-service	82.00% <ø> (ø)		Carriedforward from 7426431
frontend	96.55% <ø> (ø)		Carriedforward from 7426431
notebook-migration-service	83.73% <ø> (ø)		Carriedforward from 7426431
pyamber	98.49% <100.00%> (-0.05%)	⬇️
workflow-compiling-service	72.43% <ø> (ø)		Carriedforward from 7426431

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 7 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 7426431 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	528	0.322	18,325/28,386/28,386 us	🔴 +41.2% / 🔴 +83.5%
⚪	bs=100 sw=10 sl=64	1,655	1.01	58,991/69,775/69,775 us	⚪ within ±5% / 🟢 +56.5%
🔴	bs=1000 sw=10 sl=64	1,998	1.219	496,945/610,546/610,546 us	🔴 +7.5% / 🟢 +81.8%

Baseline details

Latest main 7426431 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	528 tuples/sec	682 tuples/sec	822.11 tuples/sec	-22.6%	-35.8%
bs=10 sw=10 sl=64	MB/s	0.322 MB/s	0.416 MB/s	0.502 MB/s	-22.6%	-35.8%
bs=10 sw=10 sl=64	p50	18,325 us	12,981 us	12,129 us	+41.2%	+51.1%
bs=10 sw=10 sl=64	p95	28,386 us	24,588 us	15,469 us	+15.4%	+83.5%
bs=10 sw=10 sl=64	p99	28,386 us	24,588 us	18,357 us	+15.4%	+54.6%
bs=100 sw=10 sl=64	throughput	1,655 tuples/sec	1,672 tuples/sec	1,058 tuples/sec	-1.0%	+56.5%
bs=100 sw=10 sl=64	MB/s	1.01 MB/s	1.021 MB/s	0.646 MB/s	-1.1%	+56.4%
bs=100 sw=10 sl=64	p50	58,991 us	58,269 us	97,399 us	+1.2%	-39.4%
bs=100 sw=10 sl=64	p95	69,775 us	69,202 us	104,184 us	+0.8%	-33.0%
bs=100 sw=10 sl=64	p99	69,775 us	69,202 us	117,615 us	+0.8%	-40.7%
bs=1000 sw=10 sl=64	throughput	1,998 tuples/sec	1,976 tuples/sec	1,099 tuples/sec	+1.1%	+81.8%
bs=1000 sw=10 sl=64	MB/s	1.219 MB/s	1.206 MB/s	0.671 MB/s	+1.1%	+81.7%
bs=1000 sw=10 sl=64	p50	496,945 us	500,052 us	946,743 us	-0.6%	-47.5%
bs=1000 sw=10 sl=64	p95	610,546 us	568,011 us	985,437 us	+7.5%	-38.0%
bs=1000 sw=10 sl=64	p99	610,546 us	568,011 us	1,012,622 us	+7.5%	-39.7%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,378.83,200,128000,528,0.322,18325.43,28386.18,28386.18
1,100,10,64,20,1208.42,2000,1280000,1655,1.010,58990.87,69774.78,69774.78
2,1000,10,64,20,10009.99,20000,12800000,1998,1.219,496944.82,610546.10,610546.10

[carloea2]

/request-review @aglinxinyuan

Could you review this Arrow chunk-boundary change? The regression and focused tests pass locally, and CI is green.