Upload README.md with huggingface_hub

README.md

@@ -7,6 +7,7 @@ tags:
   - audio-video
   - lora
   - minimax-h3
+  - comfyui
 pipeline_tag: text-to-video
 ---
 
@@ -16,63 +17,63 @@ A LoRA for [MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3) that render
 joint **video + synchronized stereo audio** in **4 sampling steps** instead of
 the usual ~20 — roughly a 5× speedup in sampling wall-clock.
 
-> ⚠️ **This is a demo / preview checkpoint, not a finished model.** It is an
-> early snapshot from an in-progress run — under-trained, the time-averaged
-> (EMA) variant hasn't matured, and overall quality is **nowhere near what a
-> full run will produce**. What it *does* already show is a clear improvement
-> over the base model at 4 steps: sharper detail and cleaner, better-synced
-> audio than the base gives you in the same 4 steps. Treat it as a taste of the
-> direction, not the destination. No comfyui support yet, will add later.
-
-> If you get completely broken audio after loading this in ComfyUI, it's almost
-> certainly a scheduler mismatch. With the correct sampler the audio here is a bit
-> rough but coherent (this is an early preview checkpoint); *completely* broken —
-> blown-out, distorted, or noise-like audio — is the signature of the wrong sampler.
-> MiniMax-H3 runs video and audio on two different flow schedules (shift 12 for
-> video, 3 for audio). The model returns the audio velocity pre-scaled so that a
-> plain single-schedule ("flat") sampler — what ComfyUI's stock samplers do — can
-> step both streams with one schedule. That approximation holds at the usual ~20
-> steps, but at ultra-low step counts like 4 the flat step massively overshoots the
-> audio schedule on the last steps, and the audio energy explodes.
-> To make audio work at 4 steps you have to replace the flat single-schedule
-> sampler with a dual-clock one that integrates the audio stream on its own shifted
-> schedule (recovering the raw audio velocity by dividing the schedule derivative
-> back out). That is exactly what generate.py in this repo does, and it matches
-> MiniMax-H3's official dual-scheduler behavior.
-
-
-Two weight files are included:
-
-| file | notes |
-|---|---|
-| `minimax_h3_turbo_4step.safetensors` | trained weights — crisper, holds up better on fast motion |
-| `minimax_h3_turbo_4step_ema.safetensors` | time-averaged weights — smoother, but **immature at this checkpoint** (EMA hasn't warmed up), so it can look soft |
-
-Both are bf16, ~744 MB, apply as a standard low-rank update
-(`W_eff = W + lora_B @ lora_A`, alpha = rank so no extra scaling).
-
-## Quick start
-
-The base model, VAEs and text encoder come from the official MiniMax-H3 release,
-and the model definitions live in ComfyUI, so you need a ComfyUI checkout:
+> ⚠️ **Early prototype — the weights *and* the tooling.** These weights are an
+> early, under-trained checkpoint (not production quality), and the ComfyUI nodes
+> below are prototype code: **functionality and compatibility are not
+> guaranteed.** It is already a clear step up from the base model at 4 steps —
+> sharper detail, cleaner and better-synced audio — but treat everything here as
+> a work-in-progress taste, not a finished product. **If something breaks, please
+> open an issue** on the [node repo](https://github.com/Larryvrh/ComfyUI-MiniMax-H3-Turbo/issues).
+
+## Use it in ComfyUI (recommended)
+
+Custom nodes: **[Larryvrh/ComfyUI-MiniMax-H3-Turbo](https://github.com/Larryvrh/ComfyUI-MiniMax-H3-Turbo)**
+— or search **"MiniMax-H3 Turbo"** in ComfyUI-Manager.
+
+1. Install the nodes (Manager, or `git clone` into `ComfyUI/custom_nodes`).
+2. Download a `.safetensors` from this repo into `ComfyUI/models/loras/`.
+3. Start from the official [MiniMax-H3 workflow](https://docs.comfy.org/tutorials/video/minimax/minimax-h3)
+   (text-to-video or image-to-video) and make two changes:
+   - insert **MiniMax-H3 Turbo LoRA** between the model loader and the sampler;
+   - replace the sampler feeding `SamplerCustomAdvanced` with **MiniMax-H3 Turbo
+     Sampler (4-step)**, and set the scheduler to **4 steps** (`simple`).
+
+Everything else stays as in the official workflow, so both t2v and i2v work.
+The custom sampler is required: MiniMax-H3 runs video and audio on two different
+flow schedules, and a stock sampler over-steps the audio at 4 steps and it breaks.
+
+## Weights
+
+Three checkpoints, all bf16, ~744 MB, applied as a standard low-rank update
+(`W_eff = W + lora_B @ lora_A`, alpha = rank so no extra scaling):
+
+| file | ~steps | notes |
+|---|---|---|
+| `minimax_h3_turbo_4step_ema_ckpt500.safetensors` | ~500 | **newest, most-trained — recommended default** (time-averaged) |
+| `minimax_h3_turbo_4step.safetensors` | ~200 | initial release, trained weights — a bit sharper on very fast motion |
+| `minimax_h3_turbo_4step_ema.safetensors` | ~200 | initial release, time-averaged — softer; superseded by the ckpt500 file |
+
+Training is ongoing; newer checkpoints will land here over time.
+
+## Standalone (no ComfyUI graph)
+
+`generate.py` is a single self-contained file — loads the base DiT + a LoRA,
+encodes the prompt, runs the 4-step dual-schedule sampler, decodes and muxes an
+mp4. It still needs a ComfyUI checkout for the H3 model / VAE / text-encoder
+definitions:
 
 ```bash
-# 1. ComfyUI (pinned to the commit these weights were validated against)
+# ComfyUI (pinned to the commit these weights were validated against)
 git clone https://github.com/comfyanonymous/ComfyUI
 cd ComfyUI && git checkout 14b05228cef127ce529bc0c08660770d4af3e9a8
 pip install -r requirements.txt && cd ..
+pip install -r requirements.txt   # this repo: torch, safetensors, imageio-ffmpeg
 
-# 2. this release
-pip install -r requirements.txt        # torch, safetensors, imageio-ffmpeg, ...
-
-# 3. base weights (from Comfy-Org/MiniMax-H3): the bf16 DiT, the int8 Qwen3-VL
-#    text encoder, and the video + audio VAEs, into a models/ tree.
-
-# 4. generate
+# base weights from Comfy-Org/MiniMax-H3 into a models/ tree, then:
 python generate.py \
   --comfyui ./ComfyUI \
   --base   models/diffusion_models/minimax_h3_fl2va_bf16.safetensors \
-  --lora   minimax_h3_turbo_4step.safetensors \
+  --lora   minimax_h3_turbo_4step_ema_ckpt500.safetensors \
   --te     models/text_encoders/qwen3vl_32b_minimax_h3_int8_convrot.safetensors \
   --video-vae models/vae/minimax_h3_video_vae_fp16.safetensors \
   --audio-vae models/vae/minimax_h3_audio_vae_fp32.safetensors \
@@ -80,20 +81,13 @@ python generate.py \
   --width 1344 --height 768 --frames 124 --out corgi.mp4
 ```
 
-`generate.py` is a single self-contained file: it loads the base DiT and merges
-the LoRA, encodes the prompt with Qwen3-VL, runs 4-step sampling on the model's
-native dual video/audio schedule, decodes both streams and muxes an mp4.
-
 ## Notes
 
-- **Resolution / duration**: width and height are multiples of 16 (the canvas is
-  32-based); the short edge is typically 768. Frame count is at 24 fps and snaps
-  up to the model's 17·k+5 grid (124 ≈ 5 s). The validated range is ~124–362
-  frames (~5–15 s).
-- **VRAM**: the base model is large (~33 B). `--offload-adaln` keeps the biggest
-  timestep-conditioned weights in CPU fp32, saving ~13 GB; an 80 GB GPU runs
-  comfortably, smaller cards can try with offload on.
-- **Steps**: 4 is the design point. `--steps 8` is a little cleaner if you have
-  the budget; the schedule is the same either way.
-- **Audio**: the model emits 32 kHz stereo aligned to the video; the two streams
-  ride different flow schedules and `generate.py` integrates each on its own.
+- **Resolution / duration**: width/height are multiples of 32; the short edge is
+  typically 768. Frame count is at 24 fps and snaps to the model's 17·k+5 grid
+  (124 ≈ 5 s). Validated range ~124–362 frames (~5–15 s).
+- **VRAM**: the base model is large (~33 B); an 80 GB GPU is comfortable
+  (`--offload-adaln` in the standalone script trades ~13 GB of VRAM for CPU RAM).
+- **Steps**: 4 is the design point, scheduler `simple`.
+- **Audio**: 32 kHz stereo, aligned to the video; the two streams ride different
+  flow schedules and are integrated each on its own clock.