Activity

1. github-actions

   added

   bugSomething isn't workingSomething isn't working

   pipelines

   on Aug 10, 2026
2. RudraMantri123 commented on Aug 12, 2026

   

   RudraMantri123

   on Aug 12, 2026

   More actions

   Hi @pupa3066 reproduced, and I can answer your unconfirmed-hardware question: this is not limited to 8 GB devices.

   Apple M5 Pro, 24 GB unified memory, macOS 26.4.1, Python 3.14, torch 2.13.0 (MPS available), diffusers 0.39.0 — running your script verbatim: pixel mean 0.0, pixel max 0, all-black PNG, and the same RuntimeWarning: invalid value encountered in cast at image_processor.py:142. All 20 denoising steps complete normally; only the VAE decode is affected, matching your description.

   Since it reproduces with 3× your RAM, memory pressure may not be the (only) trigger — this looks more like a deterministic issue in the fp16 VAE decode on MPS under component-level offload.

   If nobody has picked this up, I'd like to work on it. Plan:

   Localize where the first NaN appears in the VAE decode under enable_model_cpu_offload() (forward hooks on the decoder blocks), comparing against the working enable_sequential_cpu_offload() path.
   Test whether upcasting the VAE decode to fp32 under model offload on MPS resolves it.
   Turn the finding into a fix + regression test (or a documented guard if it's an upstream PyTorch/MPS issue).
   @yiyixuxu @sayakpaul @DN6 — happy to open a PR if this scope looks right.

   AI disclosure per the contribution policy: I use AI assistance for code navigation and drafting; all debugging and verification are run and reviewed by me on real hardware.
3. RudraMantri123

   added a commit that references this issue on Aug 12, 2026

   Fix NaN attention scores on MPS from uninitialized baddbmm buffer

   ...

   This issue will close once commit 061b461 is merged into the 'main' branch.

   061b461
4. pupa3066 commented on Aug 12, 2026

   

   pupa3066

   on Aug 12, 2026

   Author

   More actions

   Update: Corrected Root Cause

   After further isolation testing, the root cause is more specific than initially reported:

   The bug is triggered by enable_attention_slicing() combined with enable_model_cpu_offload() on MPS. The VAE is not the failure point — the NaN originates in the UNet's SlicedAttnProcessor during denoising.

   Isolation results (Apple M1 8GB, same script otherwise):

   Configuration	Result
   model_cpu_offload() without enable_attention_slicing()	✅ Valid image (pixel mean=140.73)
   model_cpu_offload() with enable_attention_slicing()	❌ All black (pixel mean=0.0)
   sequential_cpu_offload() with enable_attention_slicing()	✅ Valid image (pixel mean=140.72)

   Why:

   SlicedAttnProcessor splits attention into chunks. Under model_cpu_offload on MPS, attention slices appear to operate on uninitialized/stale memory regions left by previous component offloads via accelerate's hooks. The resulting NaN propagates through the UNet latents and the VAE faithfully decodes the corrupted latents to zeros (black).

   With sequential_cpu_offload, each layer is freshly loaded before computation, so slices always see valid data. Without attention slicing, the full attention path avoids the issue entirely.

   Additional evidence:

   • @RudraMantri123's reproduction on M5 Pro 24GB confirms this is not memory-pressure-related — it is deterministic regardless of available RAM
   • Bug is prompt-agnostic, seed-agnostic, scheduler-agnostic, and VAE-agnostic (tested with both madebyollin/sdxl-vae-fp16-fix and default SDXL VAE)

   Immediate workaround:

   pipe.enable_model_cpu_offload()
   # pipe.enable_attention_slicing()  # Do not use with model_cpu_offload on MPS

   Note on my original reproduction script:

   The reproduction code in the issue body includes pipe.enable_attention_slicing() — which is indeed the trigger. Removing that single line produces valid output. Apologies for the initial misdirection toward the VAE — the controlled isolation came later.

   I am working on a proper fix (either auto-disabling attention slicing on MPS + model_cpu_offload, or adding sync barriers in SlicedAttnProcessor) and will open a PR once the edge cases are covered.
5. pupa3066

   changed the title [-]SDXL + enable_model_cpu_offload() produces all-black images on MPS (Apple Silicon)[/-] [+]SDXL: enable_attention_slicing() + enable_model_cpu_offload() produces all-black images on MPS (Apple Silicon)[/+] on Aug 12, 2026
6. RudraMantri123 commented on Aug 12, 2026

   

   RudraMantri123

   on Aug 12, 2026

   More actions

   Great isolation, @pupa3066 — your table matches my config matrix exactly (seeds 42/7/2024 across five configurations). I can add the missing piece: the precise mechanism, traced to a specific PyTorch MPS defect — which affects which fix will work.

   Root cause: Attention.get_attention_scores computes scores as torch.baddbmm(input=torch.empty(...), beta=0, ...). Per the PyTorch docs, beta=0 means the input is ignored and NaN/Inf in it must not propagate. On MPS, that contract is violated — garbage in the uninitialized buffer leaks into the output. Only the sliced processors call this code (the default path uses SDPA), which is why slicing is the trigger. Minimal repro, no diffusers involved:

   import torch
   B, T, D = 10, 4096, 64
   junk = torch.full((B, T, T), float("nan"), device="mps", dtype=torch.float16)
   del junk  # allocator will recycle these NaN pages
   q = torch.randn(B, T, D, device="mps", dtype=torch.float16)
   k = torch.randn(B, T, D, device="mps", dtype=torch.float16)
   buf = torch.empty(B, T, T, device="mps", dtype=torch.float16)
   print(torch.isnan(torch.baddbmm(buf, q, k.mT, beta=0, alpha=0.125)).any())  # True — bug
   print(torch.isnan(torch.bmm(q, k.mT) * 0.125).any())                        # False — control

   This refines two conclusions from your update:

   1. It isn't offload-specific — plain pipe.to("mps") + enable_attention_slicing("max") produces the same NaNs with no offload involved; offload just churns the allocator so torch.empty recycles dirty pages far more often.
   2. Sync barriers won't fix it — I tested single-sync-point variants (before UNet call 0, per step-end) and they still NaN; it's recycled uninitialized memory, not a timing race. And auto-disabling slicing would remove the memory-saving feature from exactly the 8GB configurations that need it — it can be made safe instead.

   Fix: opened #14459 — on MPS with no mask, compute scores with a buffer-free scaled bmm. No reliance on the broken beta=0 contract, one less scores-sized allocation (lower peak memory), and ~35% faster than the current path at SDXL slice dimensions on Apple Silicon. fp32 matches the CPU reference exactly. Includes a regression test that fails deterministically on current main, plus a CPU-parity test. Your original script with slicing enabled now renders correctly across seeds on my M5 Pro.

   @pupa3066 — since you were heading toward a PR yourself: would you co-verify #14459 on your M1 8GB? That's the memory-constrained case I can't cover, and your isolation work already shaped the fix. I'll also file the underlying baddbmm contract violation upstream against PyTorch.

   AI disclosure per the contribution policy: AI-assisted debugging and drafting; all experiments run and verified by me on real hardware.
7. RudraMantri123

   added a commit that references this issue on Aug 12, 2026

   Fix NaN attention scores on MPS from uninitialized baddbmm buffer

   ...

   This issue will close once commit 92bd5b5 is merged into the 'main' branch.

   92bd5b5
8. RudraMantri123 commented on Aug 16, 2026

   

   RudraMantri123

   on Aug 16, 2026

   More actions

   @pupa3066 — whenever you get a chance, #14459 is ready for a test on your M1 8GB: pip install git+https://github.com/RudraMantri123/diffusers.git@fix-mps-sliced-attention-nan — would love to confirm the fix holds on the memory-constrained config before review.