Skip to content

fix: skip exporting fp8 attention q_max - #2103

Merged
chensuyue merged 3 commits into
mainfrom
fix-fp8attn
Jul 31, 2026
Merged

chensuyue merged 3 commits into
mainfrom
fix-fp8attn

Conversation

@yiliu30

@yiliu30 yiliu30 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • skip serializing the transient FP8 attention q_max calibration tensor during immediate shard writes
  • keep q_scale, k_scale, and v_scale in exported checkpoints
  • add export regression checks for both llm_compressor and auto_round formats

Root cause

clean_up_hooked_attention() removes q_max from the in-memory model, but immediate saving writes shard files before that cleanup runs. As a result, q_max leaked into exported safetensors even though it is calibration-only.

Validation

  • manual repro with nm-testing/tinysmokeqwen3 + static_attention_dtype=fp8 + format=llm_compressor
    • qmax_count = 0
    • q_scale_count = 6
    • k_scale_count = 6
    • v_scale_count = 6
  • manual repro with the same model + format=auto_round
    • qmax_count = 0
    • q_scale_count = 6
    • k_scale_count = 6
    • v_scale_count = 6

@AutoRoundBot

Copy link
Copy Markdown
Collaborator

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

@AutoRoundBot

Copy link
Copy Markdown
Collaborator

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@AutoRoundBot

Copy link
Copy Markdown
Collaborator

/azp run Unit-Test-CUDA-AutoRound

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@chensuyue chensuyue left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XPU test issue not related to this PR.

@chensuyue
chensuyue merged commit 2a267e7 into main Jul 31, 2026
46 of 48 checks passed
@chensuyue
chensuyue deleted the fix-fp8attn branch July 31, 2026 06:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants