Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request optimizes memory usage and performance for the Wan model by replacing eager nnx.vmap with a jitted, scanned, and batch-sharded implementation (_scanned_compute_kv) for cross-attention KV cache computation. It also introduces dynamic vae_spatial mesh extent selection in the VAE pipeline to prevent OOMs and improve decoding speed. A critical issue was identified in the JIT compilation of _scanned_compute_kv, where blocks_graphdef must be marked as a static argument to avoid a JAX TypeError since it is not a valid JAX PyTree.
2a78845 to
182357d
Compare
182357d to
98ea89f
Compare
98ea89f to
2b7e2bb
Compare
6a91b43 to
4ac4bbf
Compare
mbohlool
left a comment
There was a problem hiding this comment.
Nice memory win. Main ask: fail fast when the decode batch doesn't divide the batch-parallel VAE mesh.
06cd3dc to
947eb99
Compare
mbohlool
left a comment
There was a problem hiding this comment.
just one nit. LGTM otherwise.
|
|
||
| def _decode_latents_to_video(self, latents: jax.Array, trace: Optional[dict] = None) -> np.ndarray: | ||
| """Decodes latents to video frames and postprocesses.""" | ||
| if self.vae_mesh is not None and hasattr(self.vae_mesh, "shape"): |
There was a problem hiding this comment.
nit: self.vae_mesh is always a Mesh (L520) and L1001 enters it right after, so the is not None and hasattr(self.vae_mesh, "shape") guard can go.
There was a problem hiding this comment.
Done in fe4aa3d4 — dropped the guard; self.vae_mesh is always a Mesh there.
04f3eb7 to
fe4aa3d
Compare
entrpn
left a comment
There was a problem hiding this comment.
Nice HBM and VAE decode speedup! Found two issues that need a quick fix before merging (plus one fail-fast suggestion):
_scanned_compute_kvreadsnn.logical_to_mesh_axes((BATCH,))inside@jax.jit, where Flax's thread-localaxis_rulescontext is not part of JAX'sjitcache key.WanPipeline2_1setsbatch_parallel_vae=True, which also affectsWanCheckpointer2_1callers that runvae.encode(wan_txt2vid_data_preprocessing.pyanddata_processing_test.py).
|
|
||
|
|
||
| @functools.partial(jax.jit, static_argnames=("num_layers", "mesh")) | ||
| def _scanned_compute_kv(blocks_graphdef, blocks_state, encoder_hidden_states, encoder_attention_mask, num_layers, mesh): |
There was a problem hiding this comment.
_scanned_compute_kv is decorated with @jax.jit, and _batch_sharding (L59) reads nn.logical_to_mesh_axes((BATCH,))[0] inside the traced function. Because nn_partitioning.axis_rules(...) is thread-local Python state and not part of JAX's jit cache key, calling compute_kv_cache once outside axis_rules (e.g., under bare with mesh:) or under different axis_rules caches an executable whose sharding is silently reused on subsequent calls with the same (blocks_graphdef, blocks_state, encoder_hidden_states, num_layers, mesh), leaving the KV cache replicated (P()) or sharded on the wrong mesh axes even when axis_rules is active.
Could we resolve batch_mesh_axes = nn.logical_to_mesh_axes((BATCH,))[0] (canonicalized to None or tuple[str, ...]) in compute_kv_cache before entering _scanned_compute_kv, and pass it via static_argnames=("num_layers", "mesh", "batch_mesh_axes")?
| load_text_encoder=load_text_encoder, | ||
| load_scheduler=load_scheduler, | ||
| # T2V only runs the VAE on the final full-batch decode, so the | ||
| # batch-aware (N, 1) mesh is safe here; see _auto_vae_spatial. |
There was a problem hiding this comment.
WanCheckpointer2_1 instantiates WanPipeline2_1, which is also used for vae.encode in src/maxdiffusion/data_preprocessing/wan_txt2vid_data_preprocessing.py:147 and src/maxdiffusion/tests/data_processing_test.py:74 under default base_wan_14b.yml (vae_spatial: -1, per_device_batch_size: 1.0 -> global_batch_size_to_train_on == N -> vae_mesh = (N, 1)):
wan_txt2vid_data_preprocessing.py:118-127encodes chunks ofbatch_size = 10(plus any trailing remainder), which raises XLAIndivisibleErrorinsideWanCausalConv3dwhenredundantis4or8.data_processing_test.py:74-89encodes a single video (batch = 1), which degradesP('redundant', ...)toP()and runs the full 81-frame video encode replicated on every chip instead of spatially sharded on(1, N).
Those encode/checkpoint paths should keep a full-spatial (1, N) VAE mesh (and AutoencoderKLWan.encode / vae_encode_pass may also want the same redundant divisibility guard as _decode_latents_to_video so batch=1 on a (N, 1) mesh fails fast rather than encoding replicated).
| redundant = self.vae_mesh.shape.get("redundant", 1) | ||
| if redundant > 1 and latents.shape[0] % redundant != 0: | ||
| raise ValueError( | ||
| f"VAE decode batch size ({latents.shape[0]}) must be divisible by the 'redundant' mesh axis " |
There was a problem hiding this comment.
Optional (fail-fast UX): Since base_wan_*.yml defaults to per_device_batch_size: 1.0 and vae_spatial: -1 (redundant == total_devices), a direct Python call like pipeline(prompt="...") (batch_size = 1, output_type="np") will currently run all 40-50 denoising steps before hitting this ValueError at decode time. Checking redundant divisibility up front in check_inputs / _prepare_model_inputs when output_type != "latent" (in addition to keeping the guard here) would fail fast before denoising.
Two memory fixes for Wan 2.2 T2V at per-device batch >= 1, motivated by a
batch-scaling sweep (B=1..16) at 720p/81f/40 steps on v6e-8 and tpu7x-8.
1. Cross-attention KV cache (transformer_wan.py)
compute_kv_cache ran an eager nnx.vmap over all 40 layers on replicated
prompt embeds, so every device materialised the full CFG-doubled cache
plus the FP32 RMSNorm temporaries for every layer at once. It is now one
jitted executable that pins the batch axis to ('data', 'fsdp') on input
and output and scans the layers one at a time. The cross-attn consumer
already re-constrains K/V to that layout, so nothing is resharded at use
time; each device just holds its own B_local slice.
If the batch is not divisible by data * fsdp (e.g. data_parallelism > 1
with per_device_batch_size = 1 and CFG off, the case the pipelines
already tolerate for the CFG cache) the batch constraint is skipped and
the cache stays replicated as before. with_sharding_constraint does not
raise on a non-divisible axis (JAX 0.11); skipping keeps that case
explicit instead of depending on how the constraint treats it.
WanModel.compute_kv_cache is shared, so the i2v pipelines get the same
cache layout. VACE has its own compute_kv_cache and is unchanged.
Sharding the 512-token text axis on 'context' as well was tried and
rejected: XLA hoists the per-layer all-gather out of the layer scan, so
the forward program's reservation grew by 5.8 GiB (net loss on v6e-8).
2. VAE mesh (wan_pipeline.py), T2V pipelines only
With vae_spatial=-1 (auto) the VAE mesh is now ('redundant', 'vae_spatial')
= (N, 1) when the global batch is a multiple of the device count N, i.e.
one video per device with no cross-chip collectives, and (1, N) otherwise
(unchanged). Mixed meshes were measured and are not a win: at B=2 on
v6e-8 a (2, 4) mesh needed a 21.2 GiB decode program vs 15.5 GiB for
(1, 8). An explicit vae_spatial still overrides.
This is on by default for T2V: the base configs ship vae_spatial: -1 and
per_device_batch_size: 1.0, and the global batch is device count *
per_device_batch_size, so any integer per-device batch selects (N, 1).
The batch-aware choice is gated by a new batch_parallel_vae argument on
_create_common_components, enabled only by wan_pipeline_2_1 / 2_2. The
VAE constrains activations to P('redundant', ..., 'vae_spatial', ...); on
a (N, 1) mesh the batch axis is the only sharded one, so every tensor
through the VAE needs a batch divisible by N (_decode_latents_to_video
raises ValueError otherwise, rather than silently decoding replicated on
every chip). That holds for T2V (only the final full-batch decode)
but not for VACE (reference images encoded one at a time,
[1, 1, H, W, C]) or the i2v / animate conditioning encodes; those keep
the (1, N) mesh.
Tests: new tests/wan/wan_vae_mesh_test.py (4 host devices on CPU):
- the _auto_vae_spatial table for B in {0..16}, gated and ungated;
- a guard that only the T2V pipelines set the gate;
- _batch_sharding / _shard_kv_cache_on_batch layouts including the
non-divisible fallback;
- compute_kv_cache on a tiny scan_layers WanModel under the production
logical axis rules against the eager nnx.vmap it replaced (f32 and bf16,
batch divisible and not divisible by data * fsdp), including that the
batch axis stays sharded and that the cached forward matches the
uncached one.
- _decode_latents_to_video rejecting a batch that does not divide the
'redundant' axis.
generate_wan_smoke_test.py and wan_kv_cache_test.py called the T2V pipeline
with a batch of 1 under the default per_device_batch_size; they now pass
global_batch_size_to_train_on videos, as generate_wan.py does.
Measured before rebasing onto current main (DP=2, CP=4 unless noted,
use_kv_cache=true):
tpu7x-8 B=2 peak HBM 27.49 -> 16.65 GiB, denoise 270.5 -> 269.8 s
tpu7x-8 B=8 peak HBM 68.12 -> 24.32 GiB, VAE decode 41.6 -> 6.3 s,
denoise 897.8 -> 893.4 s
v6e-8 B=8 (DP=1, FSDP=2, CP=4, vae_decode_chunk=1) main OOMs in
compute_kv_cache (6.25 GiB alloc, 4.99 GiB free); with this
change it completes: peak 17.41 GiB, denoise 1302.6 s,
VAE decode 4.8 s.
fe4aa3d to
6347d1a
Compare
Two memory fixes for Wan 2.2 T2V at per-device batch >= 1, from a B=1..16 batch-scaling sweep at 720p / 81 frames / 40 steps on v6e-8 and tpu7x-8.
1. Cross-attention KV cache is batch-sharded and scanned (
transformer_wan.py)compute_kv_cacheran an eagernnx.vmapover all 40 layers on replicated prompt embeds, so every device materialised the full CFG-doubled cache plus the FP32 RMSNorm temporaries for every layer at once (~6.8 GiB per video per device at 720p). It is now a single jitted executable that pins the batch axis to('data', 'fsdp')on input and output andnnx.scans the layers one at a time. The cross-attention consumer already re-constrains K/V to(BATCH, CROSS_ATTN_HEAD, None, D_KV), so nothing is resharded at use time — each device simply holds its ownB_localslice.If the batch is not divisible by
data * fsdp(e.g.data_parallelism > 1withper_device_batch_size = 1and CFG off — the same case the pipelines already tolerate for the CFG cache), the batch constraint is skipped and the cache stays replicated, as before.with_sharding_constraintdoes not raise on a non-divisible axis (JAX 0.11); skipping keeps that case explicit instead of depending on how the constraint treats it.WanModel.compute_kv_cacheis shared, so the i2v pipelines get the same cache layout. VACE has its owncompute_kv_cacheand is unchanged.2.
vae_spatial=-1picks a batch-aware VAE mesh for T2V (wan_pipeline.py)The VAE mesh is
('redundant', 'vae_spatial'); batch is sharded onredundant, and the longer spatial axis (height for portrait videos, width otherwise) / conv channels onvae_spatial. With the old auto value (vae_spatial = N) the batch axis is fully replicated, so atB = Nevery chip holds all N videos' activations and the causal conv stack pays ~10x (and OOMs on 32 GiB parts). Auto now resolves to(N, 1)— one video per device, no cross-chip collectives — when the global batch is a multiple of the device count, and to the unchanged(1, N)otherwise. Mixed meshes were measured and are not a win (at B=2 on v6e-8,(2, 4)needed a 21.2 GiB decode program vs 15.5 GiB for(1, 8)). An explicitvae_spatialstill overrides.This is on by default for T2V. The base configs ship
vae_spatial: -1andper_device_batch_size: 1.0, and the global batch is device count ×per_device_batch_size, so any integer per-device batch selects(N, 1).Scope: the batch-aware choice is gated by a new
batch_parallel_vaeargument on_create_common_components, enabled only by the T2V pipelines (wan_pipeline_2_1,wan_pipeline_2_2). The VAE constrains every activation toP('redundant', …, 'vae_spatial', …); on a(N, 1)mesh the batch axis is the only sharded one, so every tensor that goes through the VAE needs a batch divisible by N;_decode_latents_to_videoraisesValueErrorotherwise, rather than silently decoding replicated on every chip. That holds for T2V (the only VAE call is the final full-batch decode) but not for VACE (reference images are encoded one at a time,[1, 1, H, W, C]) or the i2v / animate conditioning encodes. Those pipelines keep the previous(1, N)mesh, so nothing changes for them.The mesh is picked from the config batch at load time, so calling a T2V pipeline directly with a batch that is not a multiple of the device count now raises on such a mesh (pass a full batch, or set
vae_spatialexplicitly).generate_wan.pyalways pads to the full batch.Measurements (before rebasing onto current
main;DP=2, CP=4,use_kv_cache=trueunless noted)DP=1, FSDP=2, CP=4,vae_decode_chunk=1)compute_kv_cache(6.25 GiB alloc, 4.99 GiB free)Denoise time is unchanged within noise; the wins are −64% peak HBM and 6.6x faster VAE decode at B=8 on tpu7x-8, and B=8 with the KV cache becoming possible at all on v6e-8.
Validation
New
tests/wan/wan_vae_mesh_test.py(4 host devices on CPU):_auto_vae_spatialtable for B in {0,1,2,3,4,8,12,16} with the gate on and off.batch_parallel_vae=True._batch_sharding/_shard_kv_cache_on_batchlayout checks, including the non-divisible-batch fallback.compute_kv_cacheon a tinyscan_layersWanModelunder the production logical axis rules, against the eagernnx.vmapit replaced: f32 and bf16, batch divisible and not divisible bydata * fsdp. It also checks that the batch axis stays sharded and that the cached forward pass matches the uncached one._decode_latents_to_videorejecting a batch that does not divide theredundantaxis.Other notes:
generate_wan_smoke_test.pyandwan_kv_cache_test.pycalled the T2V pipeline with a batch of 1 under the defaultper_device_batch_size; they now passglobal_batch_size_to_train_onvideos, asgenerate_wan.pydoes.wan_kv_cache_test.pyis TPU-only (attention=flash); not runnable on CPU before or after this change.pyinkclean. The branch is rebased onto currentmain, which fixed the two unrelated files the Linter workflow was failing on.