[jeffchen006]

Fork CI validation for a proposed upstream PR. Compatibility repairs are on this fork's dev branch; the optimization branch remains focused.

What and why

Avoid fully counting lazy-search candidates whose lengths cannot improve the current length/offset score. Hash-chain and row searches accept a useful-length floor; initial/extDict searches keep the existing floor and the binary-tree wrapper keeps its existing behavior.

A candidate replaces the current solution only if 4*ml2 - highbit(ofb2) > gain1. Searched matches have ofb2 >= ZSTD_REP_NUM+1, hence highbit(ofb2) >= 2: lengths ml2 <= floor((gain1+2)/4) cannot improve the score. The higher floor is used only with a current match length >=8 and is capped below the remaining input length. No public API, format or compression parameter changes.

RFC status: performance trade-offs remain unresolved. The Mac L9 benefit reproduces, but an x86/GCC random-input slowdown reproduces too. This is not a general performance-win claim.

Measurements

Baseline 49cf51799ea051457f57243527aa2058f999f005 (dev); candidate 222606cee24d2fc3cb1c514640903a21b6f40dfc. The original measured commit 09fd6a0 has the identical full source tree d821a0cc6f0fc989bc95675bc8b734102382de53.

Apple M4 Max (14 cores/36 GiB), macOS 26.3, Apple Clang 17/GCC 15.2; AMD EPYC 9654, Ubuntu 22.04, Clang 14/GCC 11.4. Separate binaries, CFLAGS=-O3 DEBUGLEVEL=0 HAVE_ZLIB=0 HAVE_LZMA=0 HAVE_LZ4=0, single thread. Silesia is all 12 files, 211938580 bytes, concatenated alphabetically, not tar. Other inputs include official enwik8 and seeded 16 MiB random data.

Fresh independent checks use five alternating pairs, built-in -q -bLEVEL -i5 DATA. Throughputs are separate medians; changes are medians of paired ratios. Ranges are individual pair extrema, not confidence intervals.

Platform/compiler	Case	Old MB/s	New MB/s	Change	Pair range
mac / apple-clang17	silesia-L5	162.58	162.39	+0.27%	-1.44 to +87.18%
mac / apple-clang17	silesia-L9	75.31	77.99	+3.62%	+3.42 to +4.94%
mac / gcc15	silesia-L9	74.50	76.84	+3.52%	+3.11 to +5.96%
epyc / clang14	silesia-L13	13.15	14.31	-0.69%	-3.73 to +24.43%
epyc / clang14	silesia-L9	73.66	75.04	+1.63%	-0.97 to +2.20%
epyc / gcc11	random16MiB-L9	2485.69	2288.01	-7.84%	-10.74 to -3.66%
epyc / gcc11	silesia-L9	73.35	73.38	+0.61%	-0.52 to +1.45%

The original separate seven-round Mac/Clang experiment found Silesia L9+3.65%, L6+0.55%, L1/L3 near neutral; per-file geometric means were L9+3.16%, L6+0.58%. Fresh longer Mac/Clang L6 file checks remain mixed: x-ray-1.18%, sao-1.02%, osdb-0.54%. These datasets/configurations are not pooled into one score.

The supplemental screen covers levels 5/6/7/9/12/13, random/text/mixed inputs, row/hash-chain, 32 KiB chunks, 1/32 KiB streaming and raw-dictionary compression. Both throughput directions, compressed sizes, all losses, A/A controls, outliers, code-size growth and fixed-work counter/profile diagnostics are retained in the full report.

GCC random-data A/A ranges from -1.78% to +1.37%. A separate fixed-work diagnostic has nearly unchanged GCC instruction/cycle counts, so the cause of the in-memory slowdown remains unresolved; it is not proof of an algorithmic or architecture-specific regression.

Mac desktop apps remained open; CPU placement/frequency/ASLR were uncontrolled. EPYC is shared: benchmark affinity and scheduling/SMT guards do not control shared memory/cache/frequency. Initial EPYC A/A had several-percent variation. Initial Mac A/A overlapped data transfer and is excluded; fresh final and data-matched random A/A controls are separate. No architecture-wide, decoder, dictionary or streaming improvement is claimed. No alignment-perturbation experiment was performed.

Validation and evidence

• make check passed baseline/candidate; bounded ASan/UBSan fuzzer 60s and streaming 30s passed.
• 696 additional boundary configurations passed on each machine, both compilers, row/hash-chain, sizes 0..131073, repetitive/random input: byte-identical frames and round trips. Dictionary full-frame checks passed; measured compressed sizes are unchanged.
• Local make staticAnalyze actually analyzed 41 units including zstd_lazy.c for baseline, candidate and the separate CLI fix. All three exit 2 with the same 12 inherited diagnostics; no new diagnostics. This is not a clean static-analysis pass. Analyzer/tool versions, diagnostics and reproduction commands are retained.
• 106 checks passed on the fork validation PR, including 104 workflow jobs. Its tested merge includes separate package-install and strict-Clang CLI repairs on the fork base; these are excluded from this optimization branch and benchmark builds. The optimization alone does not repair unchanged upstream CI.
• Supplement, raw results and standalone reproducer; original benchmark/CI evidence. Existing evidence blobs were preserved. Smoke timings are excluded from claims. Checks are bounded and do not replace upstream CI or review.

Related work

Searched open/historical issues and PRs. facebook#4841 changes the same matchfinder area differently; this floor has not been benchmarked in combination with it. facebook#2587 concerns another continuation-search idea. Merged facebook#3552/facebook#3295/facebook#4165/facebook#3543/facebook#4418 informed the expanded validation; their evidence does not establish a universal acceptance threshold.

Checklist

• Searched open and historical issues/PRs; related work is disclosed above. No exact floor-based equivalent was found in the recorded searches.
• Bug-fix reproducer (not applicable: optimization).
• Repeated performance runs include CPU, compiler, flags, data, regressions and configuration limits.
• AI/automated tools are disclosed below.

CLA signed (contributor confirmation).

Tools used

An AI-assisted optimization pipeline produced the candidate; Codex assisted with validation, benchmarking, analysis and PR preparation.