Abstract

Acquiring physically plausible motor skills across diverse and unconventional morphologies—from humanoids to ants—is crucial for robotics and simulation. We introduce No-data Imitation Learning (NIL), which:

1. Generates a reference video with a pretrained video diffusion model from a single simulation frame + text prompt.
2. Learns a policy in simulation by comparing rendered agent videos against the generated reference via video-encoder embeddings and segmentation-mask IoU.

NIL matches or outperforms baselines trained on real motion-capture data, effectively replacing expensive data collection with generative video priors.

NIL Overview

Experiments

We validate NIL on locomotion tasks for multiple morphologies (humanoids, quadrupeds, animals).

• Reward components: ablation of video vs. mask vs. reg.
• Policy performance: matches or exceeds motion-capture-trained baselines.
• Generalization: works zero-shot on unseen morphologies.

BibTeX

@inproceedings{albaba2026nil,
  title     = {{NIL}: No-data Imitation Learning by Leveraging Pre-trained Video Diffusion Models},
  author    = {Albaba, Mert and Li, Chenhao and Diomataris, Markos and Taheri, Omid and Krause, Andreas and Black, Michael},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026},
}