Repository navigation
Conversation
Automated Reviewer SuggestionsBased on the
|
Backport auto-label reportThis
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8254 +/- ##
============================================
- Coverage 92.30% 92.30% -0.01%
Complexity 5046 5046
============================================
Files 1260 1260
Lines 53887 53887
Branches 6711 6711
============================================
- Hits 49743 49740 -3
- Misses 2486 2488 +2
- Partials 1658 1659 +1
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 524 | 0.32 | 18,348/25,451/25,451 us | 🟢 -19.0% / 🔴 +64.5% |
| 🔴 | bs=100 sw=10 sl=64 | 1,170 | 0.714 | 83,268/123,002/123,002 us | 🔴 +5.8% / 🔴 +18.1% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,427 | 0.871 | 697,672/798,666/798,666 us | 🔴 +6.6% / 🟢 +29.9% |
Baseline details
Latest main 7426431 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 524 tuples/sec | 543 tuples/sec | 822.11 tuples/sec | -3.5% | -36.3% |
| bs=10 sw=10 sl=64 | MB/s | 0.32 MB/s | 0.331 MB/s | 0.502 MB/s | -3.3% | -36.2% |
| bs=10 sw=10 sl=64 | p50 | 18,348 us | 15,923 us | 12,129 us | +15.2% | +51.3% |
| bs=10 sw=10 sl=64 | p95 | 25,451 us | 31,422 us | 15,469 us | -19.0% | +64.5% |
| bs=10 sw=10 sl=64 | p99 | 25,451 us | 31,422 us | 18,357 us | -19.0% | +38.6% |
| bs=100 sw=10 sl=64 | throughput | 1,170 tuples/sec | 1,193 tuples/sec | 1,058 tuples/sec | -1.9% | +10.6% |
| bs=100 sw=10 sl=64 | MB/s | 0.714 MB/s | 0.728 MB/s | 0.646 MB/s | -1.9% | +10.6% |
| bs=100 sw=10 sl=64 | p50 | 83,268 us | 78,730 us | 97,399 us | +5.8% | -14.5% |
| bs=100 sw=10 sl=64 | p95 | 123,002 us | 129,164 us | 104,184 us | -4.8% | +18.1% |
| bs=100 sw=10 sl=64 | p99 | 123,002 us | 129,164 us | 117,615 us | -4.8% | +4.6% |
| bs=1000 sw=10 sl=64 | throughput | 1,427 tuples/sec | 1,413 tuples/sec | 1,099 tuples/sec | +1.0% | +29.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.871 MB/s | 0.862 MB/s | 0.671 MB/s | +1.0% | +29.9% |
| bs=1000 sw=10 sl=64 | p50 | 697,672 us | 703,582 us | 946,743 us | -0.8% | -26.3% |
| bs=1000 sw=10 sl=64 | p95 | 798,666 us | 749,116 us | 985,437 us | +6.6% | -19.0% |
| bs=1000 sw=10 sl=64 | p99 | 798,666 us | 749,116 us | 1,012,622 us | +6.6% | -21.1% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,381.63,200,128000,524,0.320,18347.78,25450.77,25450.77
1,100,10,64,20,1709.94,2000,1280000,1170,0.714,83267.69,123002.19,123002.19
2,1000,10,64,20,14012.23,20000,12800000,1427,0.871,697671.85,798666.35,798666.35|
I am converting this PR to a draft: please follow our PR template and mark it ready for review. |
|
The description now follows the current template, and this is ready for review. |
02bcf80 to
99fd075
Compare
What changes were proposed in this PR?
Hash shuffle now uses the tuple's raw Java-compatible hash method directly, avoiding Python's reserved
-1to-2normalization before receiver selection.Any related issues, documentation, discussions?
Closes #8253
How was this PR tested?
Retested with Python 3.12 on Windows in a fresh detached worktree from upstream main at
7426431337, which includes the merged loop-completion change #8938. Regression before implementation: 1 failed, 37 passed. After implementation: 38 passed.The raw-hash change now lives in the shared partitioner's routing override.
Generate protobufs with
bash bin/python-proto-gen.sh, then run fromamber:Ruff lint and formatting pass for all 217 Python files.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Codex