[CI] Allow single file tests to run as fast tests by DN6 · Pull Request #14324 · huggingface/diffusers

DN6 · 2026-07-29T16:41:29Z

What does this PR do?

We've had trouble running single file testing with fast tests for a while because creating a dummy checkpoint for models in their original format is non-trivial. In this PR, we leverage the fact that the hub allows downloading safetensors metadata via
HfApi().parse_safetensors_file_metadata. This includes information about the checkpoint keys, which we can use to build a dummy state dict with empty tensors on meta device. The primary thing we want to test with single file loading is the loading and conversion step, and we can use these dummy checkpoints for that purpose.

To run a check on whether converted parameters match 1:1 would still require downloading a real checkpoints, so we move just that test to nightly.

Also added coverage for models that support single file loading that were missed during the first test refactor pass.

Fixes # (issue)

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2026-07-29T16:49:07Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul

Really nice!

sayakpaul · 2026-07-30T03:21:13Z

 import torch
-from huggingface_hub import hf_hub_download, snapshot_download
+from accelerate import init_empty_weights


Maybe we don't need init_empty_weights() here and we could just use with torch.device("meta")?

sayakpaul · 2026-07-30T03:24:40Z

        )
+        assert model_single_file.dtype == dtype, f"Expected dtype {dtype}, got {model_single_file.dtype}"


Should it resemble

diffusers/tests/models/testing_utils/common.py

Lines 519 to 523 in 3c46892

for name, param in model.named_parameters():

if any(module_to_keep_in_fp32 in name.split(".") for module_to_keep_in_fp32 in fp32_modules):

assert param.dtype == torch.float32, f"Parameter {name} should be float32 but got {param.dtype}"

else:

assert param.dtype == torch.float16, f"Parameter {name} should be float16 but got {param.dtype}"

?

… FlashAttention (drop `aiter` dependency) (#14436) Use Hub kernel Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

sayakpaul

Thanks! Just a few nits. Additionally, it would helpful to briefly summarize what's being tested (fast) in the single file test suite to get a quick idea. Optional though.

sayakpaul · 2026-08-18T11:57:29Z

+    env:
+      HF_TOKEN: ""


What is going here? IMO, it's better to scope the token to the steps where it's needed.

sayakpaul · 2026-08-18T11:57:49Z

 | `flash_varlen` | [FlashAttention](https://github.com/Dao-AILab/flash-attention) | Variable length FlashAttention |
 | `flash_varlen_hub` | [FlashAttention](https://github.com/Dao-AILab/flash-attention) | Variable length FlashAttention from kernels |
-| `aiter` | [AI Tensor Engine for ROCm](https://github.com/ROCm/aiter) | FlashAttention for AMD ROCm |
+| `aiter_fa2_hub` | [AI Tensor Engine for ROCm](https://github.com/ROCm/aiter) | FlashAttention-2 for AMD ROCm from kernels |


Seems unrelated but okay.

sayakpaul · 2026-08-18T11:58:26Z

-            target_key = "model.diffusion_model.patch_embedding.weight"
+            prefix = "model.diffusion_model."
        else:
-            target_key = "patch_embedding.weight"
+            prefix = ""
+
+        target_key = f"{prefix}patch_embedding.weight"

-        if CHECKPOINT_KEY_NAMES["wan_vace"] in checkpoint:
+        if f"{prefix}{CHECKPOINT_KEY_NAMES['wan_vace']}" in checkpoint:


Did these come about during writing the tests being added in this PR?

Yup. VACE auto config loading wasn't working for models with keys prefixed with model.diffusion_model.

sayakpaul · 2026-08-18T11:58:48Z

 if torch.distributed.is_available():
    import torch.distributed._functional_collectives as funcol

 from ..utils import (


Seems completely unrelated.

sayakpaul · 2026-08-18T12:02:08Z

+            model = self.model_class.from_config(self.pretrained_model_name_or_path, **self.pretrained_model_kwargs)
+
+        config = model.config
+        model_single_file = self.model_class.from_single_file(self.get_dummy_single_file_state_dict(), device="meta")


Curious. We're doing torch.empty with a meta device in get_dummy_single_file_state_dict(). Do we still need to pass "meta" here?

torch.empty still allocates CPU memory. Setting to meta device avoids that.

sayakpaul · 2026-08-18T12:09:51Z

 # See the License for the specific language governing permissions and
 # limitations under the License.

+import functools


Also from the PR description:

To run a check on whether converted parameters match 1:1 would still require downloading a real checkpoints, so we move just that test to nightly.

Here I am guessing we want to test if the from_single_file() path and the respective from_pretrained() path lead to the same results? But I didn't see any such tests. Maybe I am missing something. Which test did you refer to here? from_pretrained() params and from_single_file() params match with one another?

Edit: nevermind. I think you referred to test_single_file_model_parameters().

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

# Conflicts: # tests/models/transformers/test_models_transformer_sd3.py

DN6 added 3 commits July 29, 2026 16:33

update

f9ca252

update

c2b11a0

update

5948054

DN6 requested a review from sayakpaul July 29, 2026 16:41

github-actions Bot added size/L PR with diff > 200 LOC tests single-file labels Jul 29, 2026

sayakpaul reviewed Jul 30, 2026

View reviewed changes

sayakpaul and others added 5 commits July 30, 2026 09:02

Merge branch 'main' into single-file-test-update

3f52546

update

7c407e9

Merge branch 'main' into single-file-test-update

93b1ec4

[FA2] Use kernels-community/aiter-flash-attn-ck Hub kernel for ROCm…

189b086

… FlashAttention (drop `aiter` dependency) (#14436) Use Hub kernel Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

add HF_TOKEN to pr workflow

8bd92b2

github-actions Bot added documentation Improvements or additions to documentation models utils CI labels Aug 14, 2026

fix token

8242d40

sayakpaul reviewed Aug 18, 2026

View reviewed changes

DN6 added 2 commits August 18, 2026 22:39

Merge remote-tracking branch 'origin/main' into single-file-test-update

01fcdd4

clean up

d0d98fe

github-actions Bot removed documentation Improvements or additions to documentation models utils labels Aug 18, 2026

sayakpaul approved these changes Aug 18, 2026

View reviewed changes

sayakpaul and others added 3 commits August 18, 2026 19:23

Merge branch 'main' into single-file-test-update

fa1b648

Apply suggestions from code review

daf7865

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

Merge branch 'main' into single-file-test-update

f3dce3c

Merge remote-tracking branch 'origin/main' into single-file-test-update

29a1734

# Conflicts: # tests/models/transformers/test_models_transformer_sd3.py

DN6 merged commit aeda394 into main Aug 19, 2026
18 of 20 checks passed

DN6 deleted the single-file-test-update branch August 19, 2026 05:02

iwr-redmond mentioned this pull request Aug 23, 2026

The Diffusers MVP 🚀 — Ideas Track 💡 #14529

Open

		)
		assert model_single_file.dtype == dtype, f"Expected dtype {dtype}, got {model_single_file.dtype}"

	for name, param in model.named_parameters():
	if any(module_to_keep_in_fp32 in name.split(".") for module_to_keep_in_fp32 in fp32_modules):
	assert param.dtype == torch.float32, f"Parameter {name} should be float32 but got {param.dtype}"
	else:
	assert param.dtype == torch.float16, f"Parameter {name} should be float16 but got {param.dtype}"

Conversation

DN6 commented Jul 29, 2026

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Jul 29, 2026

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sayakpaul Aug 18, 2026 •

edited

Loading