Skip to content

feat(pyamber): preserve minus-one shuffle hashes - #8254

Open
carloea2 wants to merge 1 commit into
apache:mainfrom
carloea2:fix/preserve-minus-one-shuffle-hash
Open

carloea2 wants to merge 1 commit into
apache:mainfrom
carloea2:fix/preserve-minus-one-shuffle-hash

Conversation

@carloea2

@carloea2 carloea2 commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Hash shuffle now uses the tuple's raw Java-compatible hash method directly, avoiding Python's reserved -1 to -2 normalization before receiver selection.

Any related issues, documentation, discussions?

Closes #8253

How was this PR tested?

Retested with Python 3.12 on Windows in a fresh detached worktree from upstream main at 7426431337, which includes the merged loop-completion change #8938. Regression before implementation: 1 failed, 37 passed. After implementation: 38 passed.

The raw-hash change now lives in the shared partitioner's routing override.

Generate protobufs with bash bin/python-proto-gen.sh, then run from amber:

python -m pytest src/test/python/core/architecture/sendsemantics/test_partitioners.py -q --tb=line -p no:cacheprovider
python -m ruff check src/main/python src/test/python
python -m ruff format --check src/main/python src/test/python

Ruff lint and formatting pass for all 217 Python files.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Codex

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Ma77Ball
    You can notify them by mentioning @Ma77Ball in a comment.

@Yicong-Huang Yicong-Huang added the release/v1.2 back porting to release/v1.2 label Aug 31, 2026
@github-actions
github-actions Bot requested a review from xuang7 August 31, 2026 05:22
@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. release/* labels drive the post-merge backport, so add or remove one to change where this fix lands.

Release branch Analysis
⚠️ release/v1.3 Not labeled automatically — none of the files this PR modifies exist on this branch (amber/src/main/python/core/architecture/sendsemantics/hash_based_shuffle_partitioner.py, amber/src/test/python/core/architecture/sendsemantics/test_partitioners.py). The fix may target code that isn't on this release, or the files were moved/renamed after the branch was cut. Please check and add release/v1.3 by hand if this fix should be backported here.
✅ release/v1.2 Already labeled — this fix is queued to backport here.

Auto-label run.

@codecov-commenter

codecov-commenter commented Aug 31, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.30%. Comparing base (7426431) to head (99fd075).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8254      +/-   ##
============================================
- Coverage     92.30%   92.30%   -0.01%     
  Complexity     5046     5046              
============================================
  Files          1260     1260              
  Lines         53887    53887              
  Branches       6711     6711              
============================================
- Hits          49743    49740       -3     
- Misses         2486     2488       +2     
- Partials       1658     1659       +1     
Flag Coverage Δ *Carryforward flag
access-control-service 77.38% <ø> (ø) Carriedforward from 7426431
agent-service 99.16% <ø> (ø) Carriedforward from 7426431
amber 87.32% <ø> (ø) Carriedforward from 7426431
computing-unit-managing-service 60.48% <ø> (ø) Carriedforward from 7426431
config-service 87.37% <ø> (ø) Carriedforward from 7426431
file-service 82.00% <ø> (ø) Carriedforward from 7426431
frontend 96.55% <ø> (ø) Carriedforward from 7426431
notebook-migration-service 83.73% <ø> (ø) Carriedforward from 7426431
pyamber 98.47% <100.00%> (-0.07%) ⬇️
workflow-compiling-service 72.43% <ø> (ø) Carriedforward from 7426431

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 4 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main 7426431 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 524 0.32 18,348/25,451/25,451 us 🟢 -19.0% / 🔴 +64.5%
🔴 bs=100 sw=10 sl=64 1,170 0.714 83,268/123,002/123,002 us 🔴 +5.8% / 🔴 +18.1%
🔴 bs=1000 sw=10 sl=64 1,427 0.871 697,672/798,666/798,666 us 🔴 +6.6% / 🟢 +29.9%
Baseline details

Latest main 7426431 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 524 tuples/sec 543 tuples/sec 822.11 tuples/sec -3.5% -36.3%
bs=10 sw=10 sl=64 MB/s 0.32 MB/s 0.331 MB/s 0.502 MB/s -3.3% -36.2%
bs=10 sw=10 sl=64 p50 18,348 us 15,923 us 12,129 us +15.2% +51.3%
bs=10 sw=10 sl=64 p95 25,451 us 31,422 us 15,469 us -19.0% +64.5%
bs=10 sw=10 sl=64 p99 25,451 us 31,422 us 18,357 us -19.0% +38.6%
bs=100 sw=10 sl=64 throughput 1,170 tuples/sec 1,193 tuples/sec 1,058 tuples/sec -1.9% +10.6%
bs=100 sw=10 sl=64 MB/s 0.714 MB/s 0.728 MB/s 0.646 MB/s -1.9% +10.6%
bs=100 sw=10 sl=64 p50 83,268 us 78,730 us 97,399 us +5.8% -14.5%
bs=100 sw=10 sl=64 p95 123,002 us 129,164 us 104,184 us -4.8% +18.1%
bs=100 sw=10 sl=64 p99 123,002 us 129,164 us 117,615 us -4.8% +4.6%
bs=1000 sw=10 sl=64 throughput 1,427 tuples/sec 1,413 tuples/sec 1,099 tuples/sec +1.0% +29.9%
bs=1000 sw=10 sl=64 MB/s 0.871 MB/s 0.862 MB/s 0.671 MB/s +1.0% +29.9%
bs=1000 sw=10 sl=64 p50 697,672 us 703,582 us 946,743 us -0.8% -26.3%
bs=1000 sw=10 sl=64 p95 798,666 us 749,116 us 985,437 us +6.6% -19.0%
bs=1000 sw=10 sl=64 p99 798,666 us 749,116 us 1,012,622 us +6.6% -21.1%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,381.63,200,128000,524,0.320,18347.78,25450.77,25450.77
1,100,10,64,20,1709.94,2000,1280000,1170,0.714,83267.69,123002.19,123002.19
2,1000,10,64,20,14012.23,20000,12800000,1427,0.871,697671.85,798666.35,798666.35

@Yicong-Huang

Copy link
Copy Markdown
Contributor

I am converting this PR to a draft: please follow our PR template and mark it ready for review.

@Yicong-Huang
Yicong-Huang marked this pull request as draft August 31, 2026 20:06
@carloea2
carloea2 marked this pull request as ready for review August 31, 2026 20:18
@carloea2

Copy link
Copy Markdown
Contributor Author

The description now follows the current template, and this is ready for review.

@carloea2 carloea2 changed the title fix(pyamber): preserve minus-one shuffle hashes feat(pyamber): preserve minus-one shuffle hashes Aug 31, 2026
@mengw15 mengw15 removed the release/v1.2 back porting to release/v1.2 label Aug 31, 2026
@xuang7 xuang7 removed the fix label Sep 1, 2026
@xuang7
xuang7 removed their request for review September 1, 2026 01:44
@carloea2
carloea2 force-pushed the fix/preserve-minus-one-shuffle-hash branch from 02bcf80 to 99fd075 Compare October 15, 2026 06:44
@github-actions github-actions Bot added the fix label Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python hash normalization can change shuffle receivers

5 participants