[rluvaton]

What and why

Speeds up the row-based lazy matchfinder (ZSTD_greedy / ZSTD_lazy / ZSTD_lazy2 with the row hash, i.e. levels 5-12) by 2-8% on x86-64 and AArch64. Compressed output is byte-identical to dev at every level. All changes were guided by profiling (perf, Arm SPE, Intel PEBS / top-down, llvm-mca).

One commit per change:

1. lazy: drop the row head slot from the match mask once per search: slot 0 of a row holds the head, so it was tested (matchPos == 0) on every tag hit; clear its bit group from the mask once instead.
2. lazy: use 64-bit row offsets in the row matchfinder: U32 row offsets forced zero-extensions and, on AArch64, uxtw addressing (an extra uop per access). Also read hashSalt once (tag-table BYTE stores may alias *ms) and reuse the head byte already loaded for the mask. llvm-mca (Neoverse-N1): insertion loop 33 → 30 uops per position.
3. lazy: count row matchfinder candidates with a branchless 16-byte head: on Sapphire Rapids ~24% of level-6 pipeline slots are lost to branch mispredicts, and ZSTD_count's "first 8 bytes equal?" branch and loop exit are among the worst (57% of candidate lengths are 4-7 bytes, ~85% are < 16). New ZSTD_countShort() resolves the first 16 bytes with selects; it is used only by the row matchfinder: applied to ZSTD_count() itself it slowed dfast / btlazy2 / btopt by 2-6% (levels 3, 13, 16), so ZSTD_count() is unchanged.
4. compress: vectorize ZSTD_seqToCodes with NEON: 8 sequences per iteration, branchless. NEON only: an SSE2 version measured ~1% slower than scalar on x86-64, so x86 keeps the scalar loop. Verified equal to the scalar loop for every 16-bit litLength/mlBase and every offBase bit width.
5. lazy: inline the noDict row search on x86-64: perf attributed ~10% of level-6 cycles on x86-64 to the out-of-line call (callee-saved spills, mls/rowLog dispatch, re-deriving constants). Measured neutral on AArch64, so it is x86-64 only, and only the noDict row search is inlined. Cost: +98 KB .text on x86-64 (gcc 11; inlining all search functions would be ~2x .text for ~1% more, so it is not done).

Results

One-shot ZSTD_compressCCtx() on silesia.tar (211,957,760 bytes), speed in MB/s of input. Median of 5 interleaved runs (binaries run in random order each round), each run = best of repeated compressions over 3 s, pinned to one core of an otherwise idle machine. Same ratio at every level (output is identical).

level	c7i.metal-24xl (x86-64)	vs dev	c6g.metal (AArch64)	vs dev
1	527.5	-0.1%	291.2	+1.3%
3	298.5	-0.4%	136.6	+1.0%
5	160.4	+6.2%	80.3	+2.7%
6	117.3	+5.8%	60.2	+3.6%
7	102.1	+6.2%	51.5	+4.5%
8	81.3	+6.7%	40.9	+5.1%
9	77.8	+5.7%	38.8	+5.1%
12	41.3	+2.2%	18.5	+7.6%
13	15.8	0.0%	10.5	0.0%
16	7.1	0.0%	5.0	0.0%

Code-layout noise: x86-64 results move by up to ~1.4% with function/loop alignment alone, so each kept change was also re-measured with -falign-functions=64 -falign-loops=32 (and a second alignment) on both sides; only changes that held at every alignment were kept.

Setup

	x86-64	AArch64
AWS instance	c7i.metal-24xl	c6g.metal
CPU	Intel Xeon Platinum 8488C (Sapphire Rapids)	AWS Graviton2 (Neoverse-N1)
Region	il-central-1	il-central-1
OS / kernel	Amazon Linux 2023, 6.18.54	Amazon Linux 2023, 6.18.54
Compiler	gcc 11.5.0	gcc 11.5.0
Flags	default make -C lib libzstd.a (-O3), no -march	same

Not measured: other compilers (clang), other CPUs, data other than Silesia, streaming / dictionary modes (the extDict / dictMatchState paths share commits 1-3, but their speed was not benchmarked).

Verification

• Compressed output of silesia.tar byte-identical to dev at levels 1-19, on both machines.
• tests/fuzzer -T60s and tests/zstreamtest -T60s pass on both machines.
• Each commit builds with -Wall -Wextra -Werror for arm64 and x86_64 (clang).

Checklist

• I searched open pull requests and issues: this problem is not already being addressed.
• Bug fix: it comes with a reproducer or a test that fails without the fix. (not a bug fix)
• Performance change: numbers from repeated runs are included, with CPU, compiler, flags and data.
  A gain in one configuration is not a general gain: see the Performance section of CONTRIBUTING.md.
• I stated below whether AI or other automated tools were used.

Tools used

Claude Code (AI) was used to profile (perf, Arm SPE, Intel PEBS / top-down, llvm-mca), to write the code changes, and to run the benchmarks and tests. Every change was measured on the machines above, and changes that did not measure as faster were dropped.