An interactive autoregressive world model with real-time camera control, prompt switching, and long-horizon memory consistency.
- [2026-07-16] Inference code released and pretrained weights available on 🤗 Hugging Face. See Quick Start.
- [2026-07-08] Project page and technical report released.
- Inference code
- Pretrained weights — 🤗 AlayaLab/AlayaWorld
- Training code
- Training data (partial)
AlayaWorld is built around four core properties — interaction, consistency, stability, and runtime.
Two control channels: a rendered 3D cache with lightweight AdaLN camera modulation for grounded, trajectory-aware navigation, and chunk-level prompt switching to introduce new events mid-generation.
Two forms of complementary memory: an explicit 3D cache reprojected to the queried view for spatial recall, plus a compressed frame-history embedding for temporal continuity, so revisited places stay recognizable.
Long-horizon stability from training on drifted histories and an error bank that re-injects accumulated artifacts into both memory and target, preventing errors from compounding over minute-long rollouts.
Real-time interaction via few-step DMD distillation and short temporal chunks, with prompt switching at chunk boundaries to minimize both visual and semantic latency.
Inference is image-to-video: give the model a first-frame image, a camera trajectory, and a text prompt, and it rolls out a video chunk by chunk along the camera path (1 chunk ≈ 1.33s @ 24fps; ~45 chunks ≈ 1 minute).
1. Environment — a CUDA GPU and PyTorch ≥ 2.6 (the DiT uses flex_attention).
2. Weights — the model runs on four pieces. Only merged_infer.safetensors
(AlayaWorld's own weights) is hosted on our 🤗 repo; the text encoder and the
depth model are third-party — get them from their original sources:
Path under checkpoints/ |
Source |
|---|---|
merged_infer.safetensors — DiT + VAE + text-encoder + history-encoder bundle |
🤗 AlayaLab/AlayaWorld |
gemma-3-12b-it-qat-q4_0-unquantized/ — Gemma text encoder |
🤗 google/gemma-3-12b-it-qat-q4_0-unquantized (gated — accept Google's license first) |
Depth-Anything-3/ — DA3 code repo |
GitHub ByteDance-Seed/Depth-Anything-3 (see step 3) |
hf_cache/ — DA3 weights, in HF-cache layout |
🤗 depth-anything/DA3NESTED-GIANT-LARGE-1.1 (see step 3) |
taeltx2_3_wide.pth — optional TAEHV bank decoder (--bank-taehv) |
GitHub madebyollin/taehv |
From the repo root:
# AlayaWorld weights (this repo)
hf download AlayaLab/AlayaWorld merged_infer.safetensors --local-dir checkpoints
# Gemma text encoder (gated: log in + accept the license on its HF page first)
hf download google/gemma-3-12b-it-qat-q4_0-unquantized \
--local-dir checkpoints/gemma-3-12b-it-qat-q4_0-unquantizedTarget layout (paths: in configs/infer.yaml points here; repoint it if your
weights live elsewhere):
checkpoints/
├── merged_infer.safetensors
├── gemma-3-12b-it-qat-q4_0-unquantized/
├── Depth-Anything-3/ # step 3
├── hf_cache/ # step 3
└── taeltx2_3_wide.pth # optional
3. Depth-Anything-3 (required) — the spatial-memory branch depends on DA3, an external repo (inference errors out without it):
git clone https://github.com/ByteDance-Seed/Depth-Anything-3 checkpoints/Depth-Anything-3
pip install -e checkpoints/Depth-Anything-3 # run AFTER installing the torch stack (see requirements.txt)Its weights (depth-anything/DA3NESTED-GIANT-LARGE-1.1) load from
checkpoints/hf_cache/ — either let them download there on first run, or fetch
them ahead of time:
HF_HOME=checkpoints/hf_cache hf download depth-anything/DA3NESTED-GIANT-LARGE-1.14. Run a ready-made case (cases live under playground/).
The one-command launcher renders the bundled case1 (~1 min):
# single GPU
bash inference/run.sh
# multi-GPU (Ulysses context parallel; e.g. 2 or 4 GPUs)
GPUS=4 bash inference/run.shrun.sh just forwards to python -m inference.run (defaulting to
--input playground/case1/case1); call the module directly to run any case or
pass extra flags:
PYTORCH_ALLOC_CONF=expandable_segments:True \
python -m inference.run --input playground/case1/case1 --seed 1234The result mp4 is written under outputs/. See inference/README.md
for the full option list.
5. Use your own input — a "case" is three files sharing a prefix:
<prefix>_image.png first frame (seeds the history)
<prefix>_camera.pt camera trajectory: cam_c2w [F,4,4] + intrinsics
<prefix>_prompt.txt text prompt
Point --input at the prefix. For long (~1 min) rollouts, add --ttc to curb appearance drift. See inference/README.md for the full option list.
- Core Lead: Kaipeng Zhang
- Lead: Chuanhao Li
- Core Contributors: Chuanhao Li, Kaipeng Zhang, Yifan Zhan, Yongtao Ge, Yuanyang Yin
- Contributors: Jiaming Tan, Kang He, Liaoyuan Fan, Ruicong Liu, Xiaojie Xu, Xuangeng Chu, Zhen Li, Zhengyuan Lin, Zhixiang Wang, Zian Meng, Zihui Gao
For collaboration or business inquiries, contact kaipeng.zhang@shanda.com.
If you find AlayaWorld useful for your research, please cite:
@article{team2026alayaworld,
title={AlayaWorld: Long-Horizon and Playable Video World Generation},
author={Team, AlayaWorld and Zhang, Kaipeng and Li, Chuanhao and Zhan, Yifan and Ge, Yongtao and Yin, Yuanyang and Tan, Jiaming and He, Kang and Fan, Liaoyuan and Liu, Ruicong and others},
journal={arXiv preprint arXiv:2607.06291},
year={2026}
}This project is based on LTX-2 by Lightricks Ltd. Portions of the original LTX-2
codebase (flash_alaya/ltx2/) have been modified by Alaya Lab (Shanda AI Research
Tokyo) for academic and research purposes only, and the released weights
(merged_infer.safetensors) are fine-tuned from LTX-2.3. Accordingly, this
project — code and weights — is released under the
LTX-2 Community License Agreement. All original copyright, license,
patent, trademark, and attribution notices from LTX-2 are retained.
For academic research and non-commercial use only. For commercial use of LTX-2 or its derivatives, contact Lightricks Ltd. (entities with ≥ $10M annual revenue require a commercial license).
See NOTICE and THIRD_PARTY_LICENSES.md for full attribution. Third-party weights (Gemma-3 text encoder; Depth-Anything-3) are not redistributed here — obtain them from their original sources under their own licenses.