Repository navigation
fix: asymmetric zero-point packing, format-scoped 8-bit asym, W5/W6/W7 llm_compressor presets - #2273
Conversation
|
This is a prerequisite before NeUQI addition. Manually verified with phi-2 (full-vocab fp32 KL vs the base model, openwebtext 8x8192, streaming dequant student - bit-exact against compressed-tensors' own decompressor modulo the intended bf16 cast): 1. SignRound asym + quant_lm_head + llm_compressor - the tensor-zp int8 cast + signed-convention shift on an outside-block layer. Branch: clean export. Main: crashes at export after tuning completes ( autoround --model microsoft/phi-2 --asym --iters 1 --quant_lm_head --nsamples 16 \
--format auto_round:llm_compressor --output_dir /tmp/t1_asym_lmhead2. Uniform W8A16 --asym - the silent-corruption comparison. A correct asym quantization is never worse than sym, so the W8-sym run is the control:
On main the export succeeds but zero points above 127 wrap negative in the int8 cast: 1.55x the control with a 20x larger logit bias, and vLLM refuses the artifact at load ( autoround --model microsoft/phi-2 --iters 0 --scheme W8A16 --asym \
--format auto_round:llm_compressor --output_dir /tmp/w8asym_llmc # branch3. AutoScheme mixed-bits with --asym + llm_compressor - W7A16G32 + W8A16 at avg 7.75 bits. Branch: completes, 0.0239 mean / 0.0192 median / p99 0.1009 / 93.9% top1 - parity with the same run where 8-bit entries were forced symmetric (0.0242 / 0.0195 / 0.1028 / 93.6%); at 8 bits both are near-lossless, the point is the path. Main cannot run it: no G32 preset, and with plain widths it dies at export with the same int8 error as test 1, after the whole scoring pass. autoround --model microsoft/phi-2 --avg_bits 7.75 --options "W7A16G32,W8A16" \
--asym --iters 0 --format auto_round:llm_compressor --output_dir /tmp/t3_as_llmc |
|
I have found that LLM compressor supports int8 asym pack and load in vllm and serve, so will do changes to this PR to block only unsupported paths, with explicit --allow_w8_asym to bypass checks/overrides. Update: PR description/comment/logic updated with improvements and self-review fixes |
…7 llm_compressor presets - llm_compressor export: validate asymmetric zero points against the unsigned level range (finiteness/integrality included), shift into the signed convention compressed-tensors packs with, and range-check before the int8 cast; fixes a crash on float32 tensor zero points and a silent corruption where the unsigned convention clamped the upper half of each group. - Native/AWQ packers: clamp integer weights into the valid level range before nibble packing (searched grids can leave round(W/s+zp) outside it); the classic (zp-1) GPTQ packing clamps zp=0 to a decodable value instead of corrupting the packed word; scalars normalized to int for bit-shift packing. - 8-bit asym scoped by servability: native auto_round/auto_gptq/auto_awq/ marlin refuse it at configuration time; llm_compressor keeps it (compressed-tensors represents it and vLLM serves it); allow_w8_asym lifts every refusal; AutoScheme options and fixed layer pins follow the same rule. - Model-free fallback: only an explicitly requested construction format is forwarded, so save-time formats (gguf/llm_compressor/gptq/awq) are honored. - Presets: W7A16, W2-W8 G64/G32 group-size variants, W5/W6/W7 unlocked for llm_compressor export; docs EN/CN updated. Signed-off-by: avtc <tarasenkov@gmail.com>
047fffb to
9ce12e3
Compare
|
Really nice work, thanks! |
There was a problem hiding this comment.
🟡 Changes recommended
Two code paths have internal inconsistencies that can re-break W8-asym correctness/allowance (CT type detection in pack_layer, and save-time W8-asym allowance check ignoring scheme.allow_w8_asym).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR tightens correctness and policy around asymmetric integer weight quantization (especially W8 asym), fixes bit-packing corruption edge cases across multiple packers/export paths, and expands llm_compressor preset coverage (W5/W6/W7 + G32/G64 variants) with updated docs and tests.
Changes:
- Fix asymmetric zero-point handling for
llm_compressorexport (dtype + signed/unsigned convention shift) and add round-trip validation tests. - Clamp quantized integer levels before bit-packing (native
qlinear_torch,qlinear_torch_zp, AWQ packer, and GPTQ “missing_tensors” packer) to prevent neighbor-field corruption. - Enforce format-scoped W8-asym policy with an explicit
--allow_w8_asymescape hatch; add new presets and document availability (EN + CN).
File summaries
| File | Description |
|---|---|
| test/unit/test_cuda/quantization/test_act_quantization.py | Updates CUDA test to expect W8-asym refusal at construction time. |
| test/unit/test_cuda/export/test_autoround_int_export.py | Adds CUDA test for allow_w8_asym permitting native export + reload. |
| test/unit/test_cpu/quantization/test_model_free_parity.py | Adds CPU test ensuring W8 asym is refused on both model-free and regular routes by default. |
| test/unit/test_cpu/export/test_qlinear_pack_clamp.py | New CPU tests covering clamping behavior and zero-point edge cases for packers. |
| test/unit/test_cpu/export/test_llmc_format.py | Adds llm_compressor pack_layer tests (int8 zp casting, unsigned-domain validation, convention shift round-trip, mixed-width scheme export). |
| test/unit/common/utils/test_layer_config_resolver.py | Tests W8-asym layer_config refusal/pinning and format/flag allowance handling. |
| test/unit/common/schemes/test_w8_asym_policy.py | New unit tests for format gating + allow_w8_asym override behavior. |
| test/unit/common/schemes/test_auto_scheme.py | Updates AutoScheme tests for new W8-asym pinning/refusal behavior and generation-time allowance derivation. |
| test/unit/common/quantization/test_model_free.py | Tests that only explicitly requested construction formats are forwarded to fallback. |
| test/unit/common/quantization/test_asym.py | Adjusts asym test matrix and adds W8-asym policy/format/flag coverage. |
| test/unit/common/export/test_format_helpers.py | Verifies new presets/variants and llm_compressor scheme gating. |
| docs/step_by_step.md | Documents new presets and the W8-asym servability policy + flag. |
| docs/step_by_step_CN.md | Chinese doc parity update for new presets/policy. |
| auto_round/utils/missing_tensors.py | Fixes GPTQ-style zp packing for zp=0 via clamping of stored (zp-1). |
| auto_round/schemes.py | Adds W8-asym format policy helpers, new presets (W7A16 + G32/G64 variants), and parse-time enforcement/pinning. |
| auto_round/export/formats/backends/llm_compressor.py | Extends supported schemes to W5/W6/W7 and G32/G64 variants. |
| auto_round/export/export_to_llmcompressor/export.py | Implements int8 zp casting, unsigned-domain validation, and signed-convention shift for CT packing. |
| auto_round/export/export_to_awq/utils.py | Clamps integer levels before AWQ packing to prevent packed-word corruption. |
| auto_round/compressors/model_free.py | Tracks explicit format forwarding, adds allow_w8_asym, and re-checks W8-asym policy at save time. |
| auto_round/compressors/layer_config_resolver.py | Enforces W8-asym pin/refusal during layer-config resolution with format/flag allowance. |
| auto_round/compressors/base.py | Threads W8-asym policy through scheme resolution/config generation and adds save-time exportability re-check. |
| auto_round/cli/parser.py | Adds --allow_w8_asym CLI flag. |
| auto_round/cli/main.py | Wires allow_w8_asym into compressor construction/tuning. |
| auto_round/autoround.py | Adds eager validation for format-scoped W8-asym behavior at construction time. |
| auto_round/auto_scheme/gen_auto_scheme.py | Adds policy allowance plumbing (w8_asym_allowed) without mutating caller-owned AutoScheme. |
| auto_round/auto_scheme/delta_loss.py | Enforces W8-symmetric fixed pins unless allowed; adds helper for refusal/pinning. |
| auto_round_extension/torch/qlinear_torch.py | Clamps quantized levels before packing (2/4/8-bit + 3-bit paths). |
| auto_round_extension/torch/qlinear_torch_zp.py | Clamps levels before packing; fixes zp=0 and float-scalar zp handling in classic (zp-1) paths. |
Review details
- Files reviewed: 28/28 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…M env Review follow-up: the 8-bit-asym opt-in is a non-standard escape hatch, so it is now an environment variable instead of a new argument on every layer (parse_scheme, AutoRound, ModelFreeCompressor, AutoScheme, the layer-config resolver, and the AutoScheme generation path all drop their flag plumbing and the CLI flag is removed). The opt-in is read at the policy gates themselves, which also removes the model-free fallback forwarding and the save-time re-check argument. The format-scoped policy is unchanged: llm_compressor keeps W8 asym, native formats refuse or pin it. Also hardens the compressed-tensors zp shift: coerce the scheme type through QuantizationType so the check holds whether the field carries the enum member or its plain string value.
Thread the output format into the entry scheme preview and the model-free route/fallback so format-scoped policies (the 8-bit asym rule) resolve identically at routing, construction and export; log when the preview degrades to config overrides instead of swallowing parse failures silently. Model-free asymmetric int artifacts declared packing_format 'auto_round' but packed zero points with the gptq convention (zp - 1, unpacked with +1), shifting every weight by +scale: W4A16-asym on phi-2 measured full-vocab KL 5.15 vs ~0.12 after the fix (pre-existing on main; regular-flow exports are unaffected since asym native routes to the AWQ packer there). The zero-point convention is now derived from the declared packing_format via the backend registry constants, shared by the packer and the config builder so bytes and declaration cannot drift, and the declared packing is a required keyword on the packing call. Signed-off-by: avtc <tarasenkov@gmail.com>
|
/azp run Unit-Test-CUDA-AutoRound |
|
Commenter does not have sufficient privileges for PR 2273 in repo intel/auto-round |
|
/azp run Unit-Test-CUDA-AutoRound |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@wenhuach21 something wrong with pipeline ( https://dev.azure.com/lpot-inc/neural-compressor/_build/results?buildId=74046&view=logs&j=abf08a52-6fca-57e2-8281-2cd20bcb00b9&t=47e98654-b9d3-59c3-85c9-e73592dce793&l=135 ): |
|
Looks like it was an issue with github login, encountered it as well locally, please retry run: |
|
/azp run Unit-Test-CUDA-AutoRound |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@chensuyue Could we also grant the collaborator privilege to run CUDA UT? |
|
c37c0bc lost its sign-off during a history squash; all other commits are signed and the tree is unchanged - can you dismiss the DCO check, or shall I rebase with signoff? |
done |
Description
This PR fixes silent-corruption bugs in asymmetric weight packing and scopes the 8-bit-asym policy by servability:
(1) Group zero points arrive as float32 tensors but compressed-tensors' packer requires int8, crashing with
Tensor must be quantized to torch.int8 before packingon the first asym layer.(2) More subtle: compressed-tensors re-quantizes with a signed integer range and clamps after adding the zero point, while auto_round uses the unsigned level convention - the unsigned zp therefore clamped every level above
2^(b-1)-1, silently collapsing the upper half of each group's range (such exports load fine but produce garbage). Zero points are now validated against the unsigned level range[0, 2^bits-1](plus finiteness/integrality - NaN passes every min/max comparison), shifted into the signed convention, and range-checked; a round-trip test verifies(q - zp) * sis bit-exact through the compressed-tensors pack path.qlinear_torch,qlinear_torch_zpand the AWQ packing path; the classic(zp-1)packing clampszp = 0to a decodable stored 0 instead of corrupting the whole packed word (tensor paths, 2/4/8-bit and 3-bit scalar paths, and the model-free GPTQ packer; scalars are normalized to int - a float scalar zp crashed the bit-shift packing).auto_round/auto_gptq/auto_awq/ marlin refuse it at configuration time (vLLM's GPTQ-format W8 kernels are symmetric-only and Marlin's zero points are 4-bit only), with uniform--asympinning 8-bit options back to symmetric and explicitsym: falseon 8-bit entries raising instead of failing late at pack time. Thellm_compressorformat keeps 8-bit asym: compressed-tensors represents it and vLLM serves it (Machete on SM90+ at group sizes 64/128/-1, Conch on SM80+ at group size 128 or channelwise, Humming as the fallback where enabled)), with a one-time warning for group sizes only SM90 serves. AutoScheme option pins and fixed layer pins (lm_head etc.) follow the same rule, with the effective allowance derived from the export format at layer-config generation time. A newallow_w8_asymflag (CLI,AutoRound, andModelFreeCompressor) lifts every refusal and pin for serving stacks beyond stock vLLM.W7A16, G64/G32 group-size variants (W2-W8), and W5/W6/W7 unlocked for thellm_compressorformat (compressed-tensors is the vLLM serving route for these widths).Type of Change
Bug fix / Feature (opt-in flag + presets)
Related Issues
Not found
Checklist Before Submitting
/azp run Unit-Test-CUDA-AutoRound.