Skip to content

fix: asymmetric zero-point packing, format-scoped 8-bit asym, W5/W6/W7 llm_compressor presets - #2273

Merged
wenhuach21 merged 3 commits into
intel:mainfrom
avtc:feature/fixes-to-asym-and-packing
Sep 3, 2026
Merged

wenhuach21 merged 3 commits into
intel:mainfrom
avtc:feature/fixes-to-asym-and-packing

Conversation

@avtc

@avtc avtc commented Sep 1, 2026 •

Copy link
Copy Markdown
Collaborator

Description

This PR fixes silent-corruption bugs in asymmetric weight packing and scopes the 8-bit-asym policy by servability:

  • llm_compressor export: two asymmetric-packing bugs, found while quantizing a tencent/hy3 MoE model with asymmetric group search.
    (1) Group zero points arrive as float32 tensors but compressed-tensors' packer requires int8, crashing with Tensor must be quantized to torch.int8 before packing on the first asym layer.
    (2) More subtle: compressed-tensors re-quantizes with a signed integer range and clamps after adding the zero point, while auto_round uses the unsigned level convention - the unsigned zp therefore clamped every level above 2^(b-1)-1, silently collapsing the upper half of each group's range (such exports load fine but produce garbage). Zero points are now validated against the unsigned level range [0, 2^bits-1] (plus finiteness/integrality - NaN passes every min/max comparison), shifted into the signed convention, and range-checked; a round-trip test verifies (q - zp) * s is bit-exact through the compressed-tensors pack path.
  • Native/AWQ packers: rounding overflow could shift bits into the neighboring packed value. Integer weights are clamped to the valid range in qlinear_torch, qlinear_torch_zp and the AWQ packing path; the classic (zp-1) packing clamps zp = 0 to a decodable stored 0 instead of corrupting the whole packed word (tensor paths, 2/4/8-bit and 3-bit scalar paths, and the model-free GPTQ packer; scalars are normalized to int - a float scalar zp crashed the bit-shift packing).
  • 8-bit asym is scoped by what can actually serve it: native auto_round / auto_gptq / auto_awq / marlin refuse it at configuration time (vLLM's GPTQ-format W8 kernels are symmetric-only and Marlin's zero points are 4-bit only), with uniform --asym pinning 8-bit options back to symmetric and explicit sym: false on 8-bit entries raising instead of failing late at pack time. The llm_compressor format keeps 8-bit asym: compressed-tensors represents it and vLLM serves it (Machete on SM90+ at group sizes 64/128/-1, Conch on SM80+ at group size 128 or channelwise, Humming as the fallback where enabled)), with a one-time warning for group sizes only SM90 serves. AutoScheme option pins and fixed layer pins (lm_head etc.) follow the same rule, with the effective allowance derived from the export format at layer-config generation time. A new allow_w8_asym flag (CLI, AutoRound, and ModelFreeCompressor) lifts every refusal and pin for serving stacks beyond stock vLLM.
  • New presets: W7A16, G64/G32 group-size variants (W2-W8), and W5/W6/W7 unlocked for the llm_compressor format (compressed-tensors is the vLLM serving route for these widths).

Type of Change

Bug fix / Feature (opt-in flag + presets)

Related Issues

Not found

Checklist Before Submitting

  • My code has been tested locally.
  • Documentation has been updated as needed.
  • New or updated tests are included where applicable.
  • The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.

@avtc

avtc commented Sep 1, 2026 •

Copy link
Copy Markdown
Collaborator Author

This is a prerequisite before NeUQI addition.

Manually verified with phi-2 (full-vocab fp32 KL vs the base model, openwebtext 8x8192, streaming dequant student - bit-exact against compressed-tensors' own decompressor modulo the intended bf16 cast):

1. SignRound asym + quant_lm_head + llm_compressor - the tensor-zp int8 cast + signed-convention shift on an outside-block layer. Branch: clean export. Main: crashes at export after tuning completes (Tensor must be quantized to torch.int8 before packing).

autoround --model microsoft/phi-2 --asym --iters 1 --quant_lm_head --nsamples 16 \
  --format auto_round:llm_compressor --output_dir /tmp/t1_asym_lmhead

2. Uniform W8A16 --asym - the silent-corruption comparison. A correct asym quantization is never worse than sym, so the W8-sym run is the control:

same protocol mean median p99 top1 logit bias
W8 sym (control) 0.0230 0.0185 0.0980 94.2% -0.004
W8 asym, main 0.0356 0.0273 0.1624 92.3% -0.084
W8 asym, this branch + llm_compressor 0.0234 0.0189 0.0986 93.8% -0.012

On main the export succeeds but zero points above 127 wrap negative in the int8 cast: 1.55x the control with a 20x larger logit bias, and vLLM refuses the artifact at load (Unsupported packing_format: auto_round). On this branch the native-format route is refused at startup with a message pointing at the servable alternatives, and the llm_compressor route lands at parity with the control and loads/serves in vLLM (compressed-tensors WNA16 kernel path).

autoround --model microsoft/phi-2 --iters 0 --scheme W8A16 --asym \
  --format auto_round:llm_compressor --output_dir /tmp/w8asym_llmc   # branch

3. AutoScheme mixed-bits with --asym + llm_compressor - W7A16G32 + W8A16 at avg 7.75 bits. Branch: completes, 0.0239 mean / 0.0192 median / p99 0.1009 / 93.9% top1 - parity with the same run where 8-bit entries were forced symmetric (0.0242 / 0.0195 / 0.1028 / 93.6%); at 8 bits both are near-lossless, the point is the path. Main cannot run it: no G32 preset, and with plain widths it dies at export with the same int8 error as test 1, after the whole scoring pass.

autoround --model microsoft/phi-2 --avg_bits 7.75 --options "W7A16G32,W8A16" \
  --asym --iters 0 --format auto_round:llm_compressor --output_dir /tmp/t3_as_llmc

@avtc
avtc marked this pull request as draft September 1, 2026 14:08
@avtc

avtc commented Sep 1, 2026 •

Copy link
Copy Markdown
Collaborator Author

I have found that LLM compressor supports int8 asym pack and load in vllm and serve, so will do changes to this PR to block only unsupported paths, with explicit --allow_w8_asym to bypass checks/overrides.

Update: PR description/comment/logic updated with improvements and self-review fixes

@avtc
avtc marked this pull request as ready for review September 1, 2026 23:23
@avtc avtc changed the title fix: asymmetric zero-point packing, reject 8-bit asym, W5/W6/W7 llm_compressor presets fix: asymmetric zero-point packing, format-scoped 8-bit asym, W5/W6/W7 llm_compressor presets Sep 1, 2026
…7 llm_compressor presets

- llm_compressor export: validate asymmetric zero points against the unsigned
  level range (finiteness/integrality included), shift into the signed
  convention compressed-tensors packs with, and range-check before the int8
  cast; fixes a crash on float32 tensor zero points and a silent corruption
  where the unsigned convention clamped the upper half of each group.
- Native/AWQ packers: clamp integer weights into the valid level range before
  nibble packing (searched grids can leave round(W/s+zp) outside it); the
  classic (zp-1) GPTQ packing clamps zp=0 to a decodable value instead of
  corrupting the packed word; scalars normalized to int for bit-shift packing.
- 8-bit asym scoped by servability: native auto_round/auto_gptq/auto_awq/
  marlin refuse it at configuration time; llm_compressor keeps it
  (compressed-tensors represents it and vLLM serves it); allow_w8_asym lifts
  every refusal; AutoScheme options and fixed layer pins follow the same rule.
- Model-free fallback: only an explicitly requested construction format is
  forwarded, so save-time formats (gguf/llm_compressor/gptq/awq) are honored.
- Presets: W7A16, W2-W8 G64/G32 group-size variants, W5/W6/W7 unlocked for
  llm_compressor export; docs EN/CN updated.

Signed-off-by: avtc <tarasenkov@gmail.com>
@avtc
avtc force-pushed the feature/fixes-to-asym-and-packing branch from 047fffb to 9ce12e3 Compare September 2, 2026 01:08
Comment thread docs/step_by_step.md Resolved
@wenhuach21
wenhuach21 requested review from WeiweiZhang1, n1ck-guo and xin3he and a lite review from Copilot September 2, 2026 01:41
@wenhuach21

Copy link
Copy Markdown
Contributor

Really nice work, thanks!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two code paths have internal inconsistencies that can re-break W8-asym correctness/allowance (CT type detection in pack_layer, and save-time W8-asym allowance check ignoring scheme.allow_w8_asym).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR tightens correctness and policy around asymmetric integer weight quantization (especially W8 asym), fixes bit-packing corruption edge cases across multiple packers/export paths, and expands llm_compressor preset coverage (W5/W6/W7 + G32/G64 variants) with updated docs and tests.

Changes:

  • Fix asymmetric zero-point handling for llm_compressor export (dtype + signed/unsigned convention shift) and add round-trip validation tests.
  • Clamp quantized integer levels before bit-packing (native qlinear_torch, qlinear_torch_zp, AWQ packer, and GPTQ “missing_tensors” packer) to prevent neighbor-field corruption.
  • Enforce format-scoped W8-asym policy with an explicit --allow_w8_asym escape hatch; add new presets and document availability (EN + CN).
File summaries
File Description
test/unit/test_cuda/quantization/test_act_quantization.py Updates CUDA test to expect W8-asym refusal at construction time.
test/unit/test_cuda/export/test_autoround_int_export.py Adds CUDA test for allow_w8_asym permitting native export + reload.
test/unit/test_cpu/quantization/test_model_free_parity.py Adds CPU test ensuring W8 asym is refused on both model-free and regular routes by default.
test/unit/test_cpu/export/test_qlinear_pack_clamp.py New CPU tests covering clamping behavior and zero-point edge cases for packers.
test/unit/test_cpu/export/test_llmc_format.py Adds llm_compressor pack_layer tests (int8 zp casting, unsigned-domain validation, convention shift round-trip, mixed-width scheme export).
test/unit/common/utils/test_layer_config_resolver.py Tests W8-asym layer_config refusal/pinning and format/flag allowance handling.
test/unit/common/schemes/test_w8_asym_policy.py New unit tests for format gating + allow_w8_asym override behavior.
test/unit/common/schemes/test_auto_scheme.py Updates AutoScheme tests for new W8-asym pinning/refusal behavior and generation-time allowance derivation.
test/unit/common/quantization/test_model_free.py Tests that only explicitly requested construction formats are forwarded to fallback.
test/unit/common/quantization/test_asym.py Adjusts asym test matrix and adds W8-asym policy/format/flag coverage.
test/unit/common/export/test_format_helpers.py Verifies new presets/variants and llm_compressor scheme gating.
docs/step_by_step.md Documents new presets and the W8-asym servability policy + flag.
docs/step_by_step_CN.md Chinese doc parity update for new presets/policy.
auto_round/utils/missing_tensors.py Fixes GPTQ-style zp packing for zp=0 via clamping of stored (zp-1).
auto_round/schemes.py Adds W8-asym format policy helpers, new presets (W7A16 + G32/G64 variants), and parse-time enforcement/pinning.
auto_round/export/formats/backends/llm_compressor.py Extends supported schemes to W5/W6/W7 and G32/G64 variants.
auto_round/export/export_to_llmcompressor/export.py Implements int8 zp casting, unsigned-domain validation, and signed-convention shift for CT packing.
auto_round/export/export_to_awq/utils.py Clamps integer levels before AWQ packing to prevent packed-word corruption.
auto_round/compressors/model_free.py Tracks explicit format forwarding, adds allow_w8_asym, and re-checks W8-asym policy at save time.
auto_round/compressors/layer_config_resolver.py Enforces W8-asym pin/refusal during layer-config resolution with format/flag allowance.
auto_round/compressors/base.py Threads W8-asym policy through scheme resolution/config generation and adds save-time exportability re-check.
auto_round/cli/parser.py Adds --allow_w8_asym CLI flag.
auto_round/cli/main.py Wires allow_w8_asym into compressor construction/tuning.
auto_round/autoround.py Adds eager validation for format-scoped W8-asym behavior at construction time.
auto_round/auto_scheme/gen_auto_scheme.py Adds policy allowance plumbing (w8_asym_allowed) without mutating caller-owned AutoScheme.
auto_round/auto_scheme/delta_loss.py Enforces W8-symmetric fixed pins unless allowed; adds helper for refusal/pinning.
auto_round_extension/torch/qlinear_torch.py Clamps quantized levels before packing (2/4/8-bit + 3-bit paths).
auto_round_extension/torch/qlinear_torch_zp.py Clamps levels before packing; fixes zp=0 and float-scalar zp handling in classic (zp-1) paths.
Review details
  • Files reviewed: 28/28 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread auto_round/compressors/base.py Resolved
Comment thread auto_round/auto_scheme/delta_loss.py Resolved
Comment thread auto_round/compressors/model_free.py Resolved
Comment thread auto_round/compressors/model_free.py Resolved
Comment thread auto_round/auto_scheme/delta_loss.py Resolved
…M env

Review follow-up: the 8-bit-asym opt-in is a non-standard escape hatch, so it
is now an environment variable instead of a new argument on every layer
(parse_scheme, AutoRound, ModelFreeCompressor, AutoScheme, the layer-config
resolver, and the AutoScheme generation path all drop their flag plumbing and
the CLI flag is removed). The opt-in is read at the policy gates themselves,
which also removes the model-free fallback forwarding and the save-time
re-check argument. The format-scoped policy is unchanged: llm_compressor
keeps W8 asym, native formats refuse or pin it.

Also hardens the compressed-tensors zp shift: coerce the scheme type through
QuantizationType so the check holds whether the field carries the enum member
or its plain string value.
@avtc
avtc marked this pull request as draft September 2, 2026 08:23
Thread the output format into the entry scheme preview and the
model-free route/fallback so format-scoped policies (the 8-bit
asym rule) resolve identically at routing, construction and
export; log when the preview degrades to config overrides
instead of swallowing parse failures silently.

Model-free asymmetric int artifacts declared packing_format
'auto_round' but packed zero points with the gptq convention
(zp - 1, unpacked with +1), shifting every weight by +scale:
W4A16-asym on phi-2 measured full-vocab KL 5.15 vs ~0.12 after
the fix (pre-existing on main; regular-flow exports are
unaffected since asym native routes to the AWQ packer there).
The zero-point convention is now derived from the declared
packing_format via the backend registry constants, shared by
the packer and the config builder so bytes and declaration
cannot drift, and the declared packing is a required keyword
on the packing call.

Signed-off-by: avtc <tarasenkov@gmail.com>
@avtc
avtc marked this pull request as ready for review September 2, 2026 09:31
@avtc

avtc commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 2273 in repo intel/auto-round

@wenhuach21

Copy link
Copy Markdown
Contributor

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@avtc

avtc commented Sep 2, 2026 •

Copy link
Copy Markdown
Collaborator Author

@wenhuach21 something wrong with pipeline ( https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=74046&view=logs&j=abf08a52-6fca-57e2-8281-2cd20bcb00b9&t=47e98654-b9d3-59c3-85c9-e73592dce793&l=135 ):

Using Python 3.12.13 environment at: /root/.venv
   Updating https://github.com/ggml-org/llama.cpp.git (HEAD)
error: Git operation failed
  Caused by: failed to clone into: /root/.cache/uv/git-v0/db/d69dd14cf260707a
  Caused by: failed to fetch commit `0f3a71be15af836d277c9f918adfafb45732677e`
  Caused by: process didn't exit successfully: `/usr/bin/git fetch --force --update-head-ok 'https://github.com/ggml-org/llama.cpp.git' '+0f3a71be15af836d277c9f918adfafb45732677e:refs/commit/0f3a71be15af836d277c9f918adfafb45732677e'` (exit status: 128)
    --- stderr
    fatal: could not read Username for 'https://github.com'/: terminal prompts disabled
    fatal: expected flush after ref listing

@avtc

avtc commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

Looks like it was an issue with github login, encountered it as well locally, please retry run:
/azp run Unit-Test-CUDA-AutoRound

@wenhuach21

Copy link
Copy Markdown
Contributor

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@wenhuach21

Copy link
Copy Markdown
Contributor

@chensuyue Could we also grant the collaborator privilege to run CUDA UT?

@avtc

avtc commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

c37c0bc lost its sign-off during a history squash; all other commits are signed and the tree is unchanged - can you dismiss the DCO check, or shall I rebase with signoff?

@wenhuach21

Copy link
Copy Markdown
Contributor

c37c0bc lost its sign-off during a history squash; all other commits are signed and the tree is unchanged - can you dismiss the DCO check, or shall I rebase with signoff?

done

@wenhuach21
wenhuach21 self-requested a review September 3, 2026 00:49
@wenhuach21
wenhuach21 merged commit 5d2f884 into intel:main Sep 3, 2026
47 of 49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants