Skip to content

fix: preserve traversal and PD client contracts - #3395

Merged
imbajin merged 2 commits into
apache:masterfrom
hugegraph:fix/java17-traversal-cleanup
Oct 23, 2026
Merged

imbajin merged 2 commits into
apache:masterfrom
hugegraph:fix/java17-traversal-cleanup

Conversation

@imbajin

@imbajin imbajin commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Purpose of the PR

Counterpart: hugegraph#331. Both PRs use the same source branch.

Algorithm results and filtered queries can lose data or leave scans open. PD lock initialization and leader discovery can use the wrong failure or routing path, and one failing watch listener can stop notifications for every listener. This cleanup preserves API signatures, configuration, retry budgets and stored formats.

Main Changes

Trigger Before After
String IDs contain backslashes or control characters Algorithm output can contain invalid JSON Existing JSON encoding preserves the original strings and numeric values
Counting or algorithm execution fails Scans can remain open or close errors hide the original failure Scans close and secondary errors remain suppressed
Query pagination probes a filter repeatedly Records can be skipped or repeated Pending records are consumed once and closing discards them
An intersection reaches its final spilled group The final group can bypass the requested size Every group uses the same predicate and retains its last representative
First lock initialization is interrupted or discovery finds another leader An interrupted caller can issue an RPC; calls can retain the follower channel Interruption stops the call and discovery selects the leader channel
A business watch listener throws a runtime exception Later listeners and future events can lose their notification stream The failure is logged and remaining listeners continue in order

Watch cache updates and fatal errors retain their existing propagation; failed business-listener work is not retried automatically.

Shared counting logic removes duplication. Small switch-expression and instanceof cleanups use stable Java 17 syntax.

Verifying these changes

  • Java 17 formatting and repository clean compile.
  • Focused Server, PD and Store regressions against updated artifacts, including public algorithm calls, interrupted lock ownership, leader routing, watch dispatch, real spill files and single-record RPC batches.
  • Independent code review.

An existing local PD discovery test showed intermittent connection failures before the final selected suite passed. Multi-node cluster validation is not included.

Does this PR potentially affect the following parts?

  • Dependencies
  • Modify configurations
  • The public API
  • Other affects: result correctness, failure reporting and iterator cleanup

Documentation Status

  • Doc - Done: in-repository lifecycle contracts are updated.

See storage lifecycle contracts. No website configuration or API-reference changes are required.

设计图解

以下图片说明本次状态、资源关闭和 PD 通知契约,不代表运行、性能或集群验收结果。

遍历消费与资源收尾

遍历消费与资源收尾:预取、逐项消费及关闭异常传播

PD 路由与监听器隔离

PD 路由与监听器隔离:Leader 路由、通知隔离及中断处理

- retain encoded algorithm values and scan failure causes
- stabilize filtered iteration and spilled group matching
- preserve PD lock interruption and discovered leader routing
- document lifecycle contracts and cover public call paths
- retain callback order after individual listener failures
- preserve cache failures and fatal error propagation
- cover public watch dispatch with deterministic channels
- document failure isolation without automatic replay
@imbajin imbajin changed the title fix: preserve traversal and query results fix: preserve traversal and PD client contracts Oct 22, 2026
@codecov

codecov Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.22034% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.18%. Comparing base (c2ec570) to head (5f70abe).
⚠️ Report is 14 commits behind head on master.

Files with missing lines Patch % Lines
...versal/algorithm/records/record/RecordFactory.java 60.00% 1 Missing and 1 partial ⚠️
.../java/org/apache/hugegraph/pd/client/PDClient.java 94.11% 0 Missing and 1 partial ⚠️
...che/hugegraph/job/algorithm/AbstractAlgorithm.java 96.77% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3395      +/-   ##
============================================
+ Coverage     55.93%   57.18%   +1.25%     
- Complexity     2277     2778     +501     
============================================
  Files           774      776       +2     
  Lines         68534    68907     +373     
  Branches       9333     9425      +92     
============================================
+ Hits          38334    39406    +1072     
+ Misses        26634    25875     -759     
- Partials       3566     3626      +60     
Flag Coverage Δ *Carryforward flag
commons 96.37% <ø> (ø) Carriedforward from c2ec570
hstore 36.19% <7.89%> (+0.38%) ⬆️
pd 51.70% <95.23%> (+2.52%) ⬆️
server_hbase 41.96% <ø> (+0.07%) ⬆️ Carriedforward from c2ec570
server_memory 55.95% <92.10%> (+1.61%) ⬆️
server_rocksdb 42.67% <7.89%> (+0.33%) ⬆️
store 33.04% <4.76%> (+1.11%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 10/10. Covers 5f70abe.

The fixes match the PR description and the new tests cover them. The new store and server tests pass locally. CI is green; an earlier docker-build failure was a Maven Central 502 and passed on rerun.

Two older wrong-result bugs sit in these iterators. Neither is in the changed lines or made worse here:

  • Important: BackendColumn has no hashCode, so in-memory counts never merge equal rows from real scans (see inline).
  • Important: once a union spills (over 10000 candidate rows), NoRepeatValueIterator (UnionFilterIterator.java:182-238) drops every key seen more than once, which is every key the wrapper sends there. The in-memory path and MapUnionIterator (BusinessHandlerImpl.java:624) return each key once. A probe with 20000 keys read twice returns 0 rows. Worth a follow-up issue or a TODO.
  • Note: the RecordFactory switch expression and instanceof patterns are unrelated refactors that make backports a little harder.

BackendColumn second = BackendColumn.of(new byte[]{2}, new byte[]{12});
BackendColumn third = BackendColumn.of(new byte[]{3}, new byte[]{13});
List<BackendColumn> rows = kind == FilterKind.UNION_UNIQUE ? List.of(first, second, third) :
Arrays.asList(first, second, third, first, second, third);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: these fixtures reuse three BackendColumn objects, so the counts here merge rows by identity. BackendColumn overrides equals but not hashCode (RocksDBSession.java:1075), and real scans build a new object per row (RocksDBScanIterator.java:179).

With a new object per row, IntersectionFilterIterator (size 2) over 100 keys read twice returns 0 rows instead of 100, and UnionFilterIterator returns 200 instead of 100. Only the in-memory paths are affected (below 100000 intersection or up to 10000 union candidates); the intersection spill path compares by name and is correct.

Older than this PR and not blocking. Could the fixtures build a new BackendColumn.of(...) per row, with hashCode from name and value added here or in a follow-up issue?

@imbajin
imbajin merged commit 8338402 into apache:master Oct 23, 2026
33 of 34 checks passed
@imbajin
imbajin deleted the fix/java17-traversal-cleanup branch October 23, 2026 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants