Skip to content

fix: layer_config for lm_head takes effect in model-free mode - #2182

Merged
xin3he merged 11 commits into
mainfrom
copilot/fix-layer-config-priority
Aug 21, 2026
Merged

xin3he merged 11 commits into
mainfrom
copilot/fix-layer-config-priority

Conversation

Copilot AI commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

In model-free quantization, explicitly setting lm_head in layer_config (e.g. --layer_config "{lm_head:{bits:4}}" or --layer_config "{lm_head:{scheme:W4A16}}") was silently ignored — lm_head was always skipped unless --quant_lm_head was also set.

Root cause: _build_ignore_patterns() unconditionally appended both "lm_head" and "head" (DeepSeek v4 compatibility) to ignore_patterns whenever quant_lm_head=False. Since "head" is a substring pattern that matches "lm_head", fixing only the "lm_head" entry was insufficient.

Fix:

  • In _build_ignore_patterns(), check whether any key in self.layer_config contains "lm_head" before appending either "lm_head" or "head" to the ignore list — user-specified layer_config now takes highest priority over defaults
# before
if not self.quant_lm_head and "lm_head" not in ignore_patterns:
    ignore_patterns.append("lm_head")
    ignore_patterns.append("head")  # for deepseek v4

# after
lm_head_in_layer_config = any("lm_head" in key for key in (self.layer_config or {}))
if not self.quant_lm_head and not lm_head_in_layer_config:
    if "lm_head" not in ignore_patterns:
        ignore_patterns.append("lm_head")
    if "head" not in ignore_patterns:
        ignore_patterns.append("head")  # for deepseek v4
  • Added two regression tests: test_layer_config_lm_head_bits_takes_effect and test_layer_config_lm_head_scheme_takes_effect

Type of Change

Bug fix

Related Issues

Checklist Before Submitting

  • My code has been tested locally.
  • Documentation has been updated as needed.
  • New or updated tests are included where applicable.
  • The CUDA CI has passed. You can trigger it by commenting /azp run Unit-Test-CUDA-AutoRound.

Co-authored-by: xin3he <83260933+xin3he@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix layer_config to take effect in model free fix: layer_config for lm_head takes effect in model-free mode Aug 18, 2026
Copilot AI requested a review from xin3he August 18, 2026 08:57
@xin3he

xin3he commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

@wenhuach21

@xin3he
xin3he marked this pull request as ready for review August 18, 2026 09:00
@AutoRoundBot

Copy link
Copy Markdown
Collaborator

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@chensuyue chensuyue added this to the 0.15.0 milestone Aug 19, 2026
Comment thread auto_round/compressors/model_free.py Resolved
Copilot AI and others added 2 commits August 19, 2026 05:19
…nfig keys

Co-authored-by: xin3he <83260933+xin3he@users.noreply.github.com>
Co-authored-by: xin3he <83260933+xin3he@users.noreply.github.com>
Copilot AI requested a review from xin3he August 19, 2026 05:23
Co-authored-by: xin3he <83260933+xin3he@users.noreply.github.com>

Copilot AI commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

@wenhuach21

Extended the lm_head name variants in _LM_HEAD_PATTERNS to include embed_out (Pythia/Dolly) and output (some InternLM variants), in addition to lm_head and head (DeepSeek v4). The _build_ignore_patterns method now iterates over this constant. Added unit tests for both new patterns. Commit: fix: extend lm_head ignore patterns to include embed_out and output.

Copilot AI requested a review from xin3he August 19, 2026 06:16
@xin3he

xin3he commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

@copilot model free应该还有其他地方涉及到lm_head的处理,都考虑全面各种已知的名称,优化代码。

Co-authored-by: xin3he <83260933+xin3he@users.noreply.github.com>

Copilot AI commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

@copilot model free应该还有其他地方涉及到lm_head的处理,都考虑全面各种已知的名称,优化代码。

Updated in commit fix: use _LM_HEAD_PATTERNS for all lm_head handling in model_free.py.

All remaining hardcoded "lm_head" checks in the MXFP and INT export config builders now iterate over _LM_HEAD_PATTERNS ("lm_head", "head", "embed_out", "output"), so variants like DeepSeek v4's head, Pythia's embed_out, and InternLM's output are handled consistently throughout model-free mode.

Comment thread auto_round/compressors/model_free.py Resolved
@xin3he

xin3he commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

…fig-priority

# Conflicts:
#	auto_round/compressors/model_free.py

Co-authored-by: xin3he <83260933+xin3he@users.noreply.github.com>

Copilot AI commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Merge conflicts resolved. The main branch had refactored several functions out of model_free.py into the new auto_round/utils/model_free_utils.py. I've merged main, applied all our _LM_HEAD_PATTERNS fixes to the new location in model_free_utils.py, and updated the import in model_free.py.

@xin3he

xin3he commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

@copilot

  • issue 1: (fix it)
    "module": "auto_round.compressors.model_free",
    "obj": "",
    "line": 618,
    "column": 0,
    "endLine": null,
    "endColumn": null,
    "path": "/auto-round/auto_round/compressors/model_free.py",
    "symbol": "line-too-long",
    "message": "Line too long (133/120)",
    "message-id": "C0301"
  • issue 2: (please check whether the UT is wrong or the code change is wrong, fix it)
    FAILED unit/test_cpu/quantization/test_model_free.py::TestModelFreeQuantize::test_layer_config_lm_head_bits_takes_effect - AssertionError: assert 'lm_head.qweight' in {'lm_head.weight', 'model.decoder.layers.0.fc1.qweight', 'model.decoder.layers.0.self_attn.k_proj.qzeros', 'model.decoder.layers.0.self_attn.q_proj.scales', 'model.decoder.layers.0.fc1.bias', 'model.decoder.layers.0.self_attn.k_proj.scales', 'model.decoder.layers.0.self_attn.q_proj.qweight', 'model.decoder.layers.0.fc1.scales', 'model.decoder.layers.0.fc1.qzeros', 'model.decoder.embed_tokens.weight', 'model.decoder.layers.0.fc2.qzeros', 'model.decoder.layers.0.self_attn.k_proj.qweight', 'model.decoder.layers.0.fc2.scales', 'model.decoder.layers.0.self_attn.q_proj.qzeros', 'model.decoder.layers.0.fc2.qweight'}
  • where {'lm_head.weight', 'model.decoder.layers.0.fc1.qweight', 'model.decoder.layers.0.self_attn.k_proj.qzeros', 'model.decoder.layers.0.self_attn.q_proj.scales', 'model.decoder.layers.0.fc1.bias', 'model.decoder.layers.0.self_attn.k_proj.scales', 'model.decoder.layers.0.self_attn.q_proj.qweight', 'model.decoder.layers.0.fc1.scales', 'model.decoder.layers.0.fc1.qzeros', 'model.decoder.embed_tokens.weight', 'model.decoder.layers.0.fc2.qzeros', 'model.decoder.layers.0.self_attn.k_proj.qweight', 'model.decoder.layers.0.fc2.scales', 'model.decoder.layers.0.self_attn.q_proj.qzeros', 'model.decoder.layers.0.fc2.qweight'} = _read_output_keys('/tmp/pytest-of-hostuser/pytest-7/test_layer_config_lm_head_bits0/output')
    FAILED unit/test_cpu/quantization/test_model_free.py::TestModelFreeQuantize::test_layer_config_lm_head_scheme_takes_effect - AssertionError: assert 'lm_head.qweight' in {'lm_head.weight', 'model.decoder.layers.0.fc1.qweight', 'model.decoder.layers.0.self_attn.k_proj.qzeros', 'model.decoder.layers.0.self_attn.q_proj.scales', 'model.decoder.layers.0.fc1.bias', 'model.decoder.layers.0.self_attn.k_proj.scales', 'model.decoder.layers.0.self_attn.q_proj.qweight', 'model.decoder.layers.0.fc1.scales', 'model.decoder.layers.0.fc1.qzeros', 'model.decoder.embed_tokens.weight', 'model.decoder.layers.0.fc2.qzeros', 'model.decoder.layers.0.self_attn.k_proj.qweight', 'model.decoder.layers.0.fc2.scales', 'model.decoder.layers.0.self_attn.q_proj.qzeros', 'model.decoder.layers.0.fc2.qweight'}
  • where {'lm_head.weight', 'model.decoder.layers.0.fc1.qweight', 'model.decoder.layers.0.self_attn.k_proj.qzeros', 'model.decoder.layers.0.self_attn.q_proj.scales', 'model.decoder.layers.0.fc1.bias', 'model.decoder.layers.0.self_attn.k_proj.scales', 'model.decoder.layers.0.self_attn.q_proj.qweight', 'model.decoder.layers.0.fc1.scales', 'model.decoder.layers.0.fc1.qzeros', 'model.decoder.embed_tokens.weight', 'model.decoder.layers.0.fc2.qzeros', 'model.decoder.layers.0.self_attn.k_proj.qweight', 'model.decoder.layers.0.fc2.scales', 'model.decoder.layers.0.self_attn.q_proj.qzeros', 'model.decoder.layers.0.fc2.qweight'} = _read_output_keys('/tmp/pytest-of-hostuser/pytest-7/test_layer_config_lm_head_sche0/output')

Co-authored-by: xin3he <83260933+xin3he@users.noreply.github.com>

Copilot AI commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

@copilot

  • issue 1: (fix it)
    "module": "auto_round.compressors.model_free",
    ...

Both issues addressed in the latest commit:

  1. Line too long (133/120): Split the any(...) expression across multiple lines — all lines now within 120 chars.

  2. Test failures (test_layer_config_lm_head_bits_takes_effect / test_layer_config_lm_head_scheme_takes_effect): The root cause was that to_standard_regex("head") produces ".*head.*" (substring match). So even though "lm_head" was correctly excluded from ignore_patterns when present in layer_config, the "head" pattern was still added — and .*head.* matched "lm_head" as a substring, causing should_ignore("lm_head.weight") to return True.

    Fix: added lm_head_pattern in key to the pattern_in_config check, mirroring how _build_ignore_regex uses .*pattern.* substring matching. Now if a layer_config key contains the pattern as a substring (e.g. "lm_head" contains "head"), the pattern is also excluded from ignore_patterns.

@xin3he

xin3he commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

@copilot
修复下面的问题。
FAILED unit/test_cpu/quantization/test_model_free.py::TestBuildIgnorePatterns::test_head_still_ignored_when_lm_head_in_layer_config_but_not_head - AssertionError: assert 'head' in ['embed_out', 'output', 'vision', 'visual', 'image', 'img', 'audio', 'speech', 'wav', 'waveform']

  • where ['embed_out', 'output', 'vision', 'visual', 'image', 'img', 'audio', 'speech', 'wav', 'waveform'] = <auto_round.compressors.model_free._ModelFreeCompressorCore object at 0x7a014a2bd4f0>.ignore_patterns

…er.should_ignore

Co-authored-by: xin3he <83260933+xin3he@users.noreply.github.com>

Copilot AI commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

@copilot
修复下面的问题。
FAILED unit/test_cpu/quantization/test_model_free.py::TestBuildIgnorePatterns::test_head_still_ignored_when_lm_head_in_layer_config_but_not_head - AssertionError...

Fixed. The previous approach (lm_head_pattern in key) incorrectly treated "head" in "lm_head" as True, removing "head" from ignore_patterns when only "lm_head" was in layer_config.

The fix is at the _PatternMatcher.should_ignore level: if a tensor's layer name has an explicit entry in layer_config, always return False — explicit layer_config takes priority over all ignore patterns, including the .*head.* substring regex that would otherwise match lm_head. _build_ignore_patterns reverts to exact/prefix match only (key == lm_head_pattern or key.startswith(lm_head_pattern + ".")), so "head" stays in ignore_patterns when only "lm_head" is in layer_config.

@AutoRoundBot

Copy link
Copy Markdown
Collaborator

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@AutoRoundBot

Copy link
Copy Markdown
Collaborator

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@xin3he
xin3he merged commit 1d05046 into main Aug 21, 2026
49 checks passed
@xin3he
xin3he deleted the copilot/fix-layer-config-priority branch August 21, 2026 05:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: layer_config does not take effect in model free

5 participants