MiniMax-H3: segmented offload (max_vram) deterministically produces an unrelated image (SD1.5 unaffected) #1865

Activity

1. dkcompactica commented on Aug 8, 2026

   

   dkcompactica

   on Aug 8, 2026

   Author

   More actions

   Root cause found, and it's a one-line fix. Verified byte-identical output on the exact repro above.

   Root cause

   is_graph_cut_tensor (src/core/ggml_graph_cut.cpp) classifies cut tensors by name prefix:

   return std::strncmp(tensor->name, GGML_RUNNER_CUT_PREFIX, std::strlen(GGML_RUNNER_CUT_PREFIX)) == 0;

   But ggml's auto-naming gives derived nodes the parent's name plus a suffix — a view of a cut tensor is named "<cut-name> (view)", a cont of that "<cut-name> (view) (cont)". So every view/cont derived from a cut tensor is itself classified as a cut tensor. Two things then go wrong at once:

   1. each derived node becomes a segment output, cached by name, and
   2. sibling views of the same parent share one identical auto-name, so they collide in the cache — and when the next segment reads them back, bind_segment_cached_inputs binds every one of them to the same tensor at offset zero.

   MiniMax-H3 hits this on every budgeted run because each transformer block slices hidden_states into per-modality rows (audio 17 / video 16 / text 810 at 864×480×56). When the block just past a segment boundary slices the boundary tensor, all three slices read the same rows, and every downstream block processes the wrong modality. That explains the exact failure signature: a deterministic, coherent, unrelated clip with correct frame count, duration and audio — nothing ever errors.

   It also explains why flux and SD1.5 measure byte-identical under segmentation: they never slice a cut tensor across a segment boundary, so the misclassification has nothing to collide.

   How it was isolated: a leaf-content probe (log every graph leaf's name/shape/content signature after external data lands) diffed a budgeted vs unbudgeted run of the same seed. All 539 shared leafs were byte-identical; the budgeted run's segment-2 graph had four extra leafs — the base ggml_runner_cut:minimax_h3.blocks.39|hidden_states [5376,843] plus three ... (view) (cont) leafs [5376,17], [5376,16], [5376,810], all three with the same content signature — i.e. all bound to the base tensor at offset 0.

   Fix

   Require the exact cut name — a derived node's " (view)"/" (cont)" suffix disqualifies it:

   bool is_graph_cut_tensor(const ggml_tensor* tensor) {
       if (tensor == nullptr || tensor->name[0] == '\0') {
           return false;
       }
       if (std::strncmp(tensor->name, GGML_RUNNER_CUT_PREFIX, std::strlen(GGML_RUNNER_CUT_PREFIX)) != 0) {
           return false;
       }
       // ggml auto-names derived nodes "<parent> (view)" / "<parent> (view) (cont)", so a prefix
       // match alone also classifies views/conts OF a cut tensor as cut tensors. Sibling views share
       // one identical auto-name, collide in the by-name segment cache, and all bind to the same
       // data at offset zero in the next segment.
       return std::strstr(tensor->name, " (") == nullptr;
   }

   The derived nodes then become ordinary internal nodes again, recomputed inside the consuming segment from the properly bound base tensor.

   Verification (RTX 5070 Ti 16 GB, CUDA 13, commit ea7f0c8 + this fix)

   Repro config from the issue: minimax_h3_fl2va_pruned-Q4_K_M + Q4 encoder + both VAEs, 864×480×56, 8 steps, seed 7, te=cpu.

   Configuration	Before	After
   max_vram cuda=9	unrelated clip (deterministic)	byte-identical to unsegmented (7.3 vs 6.8 min)
   max_vram cuda=4.5 + stream_layers	not tested (assumed worse)	byte-identical to unsegmented (7.2 min, 13.6 s/step vs 10.4)

   The probe confirms the budgeted graph still splits into the same two segments (10077 + 2564 nodes, boundary at blocks.39) — the fix doesn't suppress segmentation; the colliding derived leafs are simply gone, recomputed in segment 2.

   The 4.5 GiB result is worth a note: deep oversubscription is nearly free on this model, which makes 8 GB cards a realistic target for H3 through max_vram once this lands.

   Happy to open a PR with the fix if useful.

   👍React with 👍2Reacted by Lexi Hale and farawayso
2. CAHbKA-IV commented on Aug 16, 2026

   

   CAHbKA-IV

   on Aug 16, 2026

   Contributor

   More actions

   For ROCm backend fix also works, thank you! Tested on targets gfx1030, gfx1100.
3. farawayso commented on Aug 16, 2026

   

   farawayso

   on Aug 16, 2026

   More actions

   我也是这个问题，求pr，但是我不知道怎么提？
4. farawayso commented on Aug 16, 2026

   

   farawayso

   on Aug 16, 2026

   More actions

   #1865 issue comment draft — copy and paste to #1865

   ---

   Additional reproduction: AMD Radeon RX 7900 XTX (gfx1100), Vulkan & ROCm backends

   We can confirm the same bug on a 7900 XTX 24 GB with both Vulkan and ROCm backends (commit de298c2).

   Setup:

   • GPU: AMD Radeon RX 7900 XTX (gfx1100), 24 GB VRAM, Vulkan & ROCm 7.14
   • Model: minimax_h3_fl2va_pruned-Q4_K_M.gguf + qwen3vl_32b_minimax_h3-Q4_K_M.gguf
   • Backend: Vulkan (eager-load, single GPU) and ROCm (stream-layers, params-backend cpu)

   Observations:

   --max-vram	Segments	Content correct?
   Vulkan1=22	51→1 (merged single segment)	✅ Yes
   Vulkan1=7	51→17 (budgeted)	❌ No — unrelated but coherent clip
   Vulkan1=6	51→8 (budgeted)	❌ No
   Streaming (ROCm, fully segmented)	49~51 segments	❌ No

   The symptom matches the report exactly: deterministic, coherent, unrelated output with correct frame count, duration and audio — nothing ever errors, the content is just wrong.

   Workaround confirmed: a budget large enough to merge the entire forward pass into a single segment (e.g., Vulkan1=22 / ROCm0=22) completely avoids the bug, producing prompt-correct output. This is consistent with the root cause analysis (no segment boundary → no view/cont collision).

   The one-line fix from the comment (return std::strstr(tensor->name, " (") == nullptr;) is expected to resolve this on both Vulkan and ROCm backends — we have not tested with a patched build but the symptom is identical across all three backends (CUDA, Vulkan, ROCm).

   Thanks for the excellent root cause analysis — this completely explains our testing results.
5. leejet

   closed this as completedin #1883on Aug 19, 2026