Dual-State RaStream

Official code skeleton for Dual-State RaStream: Edge-Deployable Streaming Human Mesh Recovery from mmWave Radar.

RaStream recovers SMPL-X human mesh parameters from short-window mmWave radar tensors. The public code is organized around the method described in the paper:

1. RaSS spatial encoder: localizes the body in a radar tensor window, extracts a subject-centered region, and outputs a compact radar token.
2. Dual-State RaStream: consumes causal RaSS token sequences and maintains a slow morphology state for shape/gender and a fast motion state for pose/root/translation.

This branch intentionally removes private experiment outputs, checkpoints, compiled paper artifacts, deployment binaries, and unrelated baselines.

Repository Layout

RaSS/
  model.py                          # RaSS spatial encoder
  train.py                          # single-window RaSS trainer
  extract_features.py               # RaSS token extraction
  dataset/                          # M4Human radar tensor dataset loader
  losses.py                         # SMPL-X, rotation, and RaSS losses
  evaluation.py                     # RaSS evaluation utilities
RaStream/
  model.py                          # DualStateRaStream
  data.py                           # precomputed token sequence dataset
  losses.py                         # dual-state regularizers
  temporal_losses.py                # temporal parameter losses
  metrics.py                        # temporal/mesh metrics
  train.py                          # temporal training entry point
configs/
  rass_tiny.yaml                    # RaSS tiny training config
  rass_small.yaml                   # RaSS small training config
  rass_base.yaml                    # RaSS base training config
  extract_rass_small.yaml           # RaSS token extraction config
  rastream.yaml                     # Dual-State RaStream training config
scripts/
  train_rass.py                     # train RaSS tiny/small/base
  extract_rass_features.py          # extract RaSS tokens
  train_rastream.py                 # train Dual-State RaStream

Setup

Tested development environment:

• Python 3.9+
• PyTorch 2.x
• CUDA 11.8 or newer for GPU training

Install Python dependencies:

pip install -r requirements.txt

Download SMPL-X model files from the official SMPL-X website and place them at:

models/smplx/
  SMPLX_MALE.npz
  SMPLX_FEMALE.npz
  SMPLX_NEUTRAL.npz

Model files, datasets, checkpoints, extracted features, and experiment outputs are intentionally ignored by git.

Data

The code expects the processed M4Human radar dataset layout used by the original benchmark:

<M4HUMAN_DATA_ROOT>/rf3dpose_all/
  radar_comp.lmdb
  radar_pc.lmdb
  params.lmdb
  calib.lmdb
  indicator.lmdb
  indeces.pkl.gz

Set paths.cached_root in configs/rass_small.yaml to the parent directory containing rf3dpose_all/.

For temporal training, RaStream expects precomputed RaSS token files:

data/features/rass_small/
  train.pt
  val.pt
  test.pt
  metadata.json

Each split .pt file contains features, seq_ids, frame_ids, targets, and optionally base_pred. See RaStream/data.py for the exact schema.

Train RaSS

For more detail on the spatial encoder, see RaSS/README.md.

Train the single-window RaSS spatial encoder:

python -m RaSS.train \
  --config-name rass_small \
  paths.cached_root=/path/to/MR-Mesh \
  run.name=rass_small

Multi-GPU training:

torchrun --nproc_per_node=4 -m RaSS.train \
  --config-name rass_small \
  paths.cached_root=/path/to/MR-Mesh

Equivalent script:

python scripts/train_rass.py --config-name rass_small \
  paths.cached_root=/path/to/MR-Mesh

Extract RaSS Tokens

After training RaSS, extract dense causal tokens for the temporal model:

python -m RaSS.extract_features \
  --config configs/extract_rass_small.yaml \
  --splits train val test

Equivalent script:

python scripts/extract_rass_features.py \
  --config configs/extract_rass_small.yaml \
  --splits train val test

Override paths as needed in the YAML:

• paths.cached_root: processed M4Human root
• paths.feature_root: output feature directory
• pretrained.backbone_checkpoint: trained RaSS checkpoint

Train Dual-State RaStream

Train the temporal module from precomputed RaSS tokens:

python -m RaStream.train \
  --config configs/rastream.yaml \
  paths.feature_root=data/features/rass_small \
  paths.smplx_model_path=models/smplx

Equivalent script:

python scripts/train_rastream.py \
  --config configs/rastream.yaml

Citation

If this project helps your research, please cite your paper here:

@article{dong2026rastream,
  title={RaStream: Edge-Deployable Streaming Human Mesh Recovery from mmWave Radar},
  author={Dong, Jiazhen and Liu, Lei},
  journal={arXiv preprint arXiv:2608.11791},
  year={2026}
}