Anymove

---
name: anymove
description: Motion capture for any creature from one AI video, with the anymove repo. A video model films a rigged 3D creature from two cameras in one frame, a DINOv2 joint reader trained on that creature finds every joint in both views, and the rig is fitted to them and exported as a GLB. Use when someone wants to animate a non-human creature (dragon, spider, wolf, monster) from a text prompt or an AI video, asks how anymove works, or needs to know which step comes next; it routes to anymove-rig, anymove-film, anymove-reader, anymove-capture, anymove-pack and anymove-lora.
license: Apache-2.0
compatibility: Needs an NVIDIA GPU with CUDA (12 GB to train a reader, 24 GB is safer), Blender 5.x, Python 3.11, and a fal.ai key for video generation. Runs on Linux and on Windows in Git Bash.
---

# anymove

Animate any rigged creature from one AI video. The trick is the frame: one image holds two orthographic views of the
creature at the same scale, front on the left and side on the right, sharing the vertical axis. A joint sits on the
same pixel row in both halves, so the front view gives left-right and height, the side view gives front-back and
height, and together they pin the joint in 3D. The method is @CanItCode's; the LoRA step is @Graalitoo's idea.

## The pipeline

| step | once per | what happens | skill |
|---|---|---|---|
| 1. mesh | creature | a textured 3D model (Tripo H3.1 from one image, or your own GLB), facing -Y, real size | anymove-rig |
| 2. skeleton | creature | joints placed in `assets/<c>/rig.json`, rig built and skinned in Blender | anymove-rig |
| 3. limb colours | creature | each limb painted its own colour so left and right never swap | anymove-rig |
| 4. two cameras | move | the painted reference frame plus a strict prompt go to MiniMax H3 Max on fal | anymove-film |
| 5. teach a reader | creature | 2,200 random poses rendered with known joints; DINOv2 learns this creature | anymove-reader |
| 6. read every joint | clip | the reader finds every joint in both views, every frame | anymove-capture |
| 7. two views, 3D | clip | shared heights tie the views; the pair gives each joint in 3D | anymove-capture |
| 8. the rig follows | clip | forward kinematics in torch fitted to both views; GLB out | anymove-capture |
| 9. the pack | creature | score the takes, keep the best, cut loops and root motion; one GLB with every move | anymove-pack |
| optional | creature | a MiniMax H3 LoRA that keeps the two-camera layout and colours more reliably | anymove-lora |

## Before anything

0. **Check it's the right tool.** If the person only wants a creature, look in Patch's Assets first: nine are ready.
   anymove suits big single gestures, and doesn't do eight-legged walking, lying down, bird wings or tail swipes (see the
   README). Check there's an NVIDIA card before anything else; without one, stop and say so.
1. Work in an anymove checkout. If the working directory has no `tools/capture.sh`, clone it:
   `git clone https://github.com/Patch-gg/anymove && cd anymove`.
2. Set up once: `python3.11 -m venv .venv`, then
   `.venv/bin/pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu128` and
   `.venv/bin/pip install -r requirements.txt`. Blender 5.x must be on PATH, or set `BLENDER=/path/to/blender`.
   **On Windows,** run everything in Git Bash, use `py -3.11 -m venv .venv` and `.venv/Scripts/` where these say
   `.venv/bin/`, and set `BLENDER`. `tools/*.sh` handle the rest. If you call Blender yourself rather than through them,
   give it absolute paths: on Windows it reads a relative output path from the drive root.
3. For video generation and LoRA training, put `FAL_KEY=...` in `.env` (git ignores it) or export it. Never print it.
4. Every command takes a creature name; its config is `assets/<c>/creature.json` and every output lands in
   `runs/<c>/` (reference frame, dataset, `reader.pt`, videos, clips). The scripts set `MOCAP_CREATURE` for you; when
   you run a `python -m mocap...` module by hand, set `MOCAP_CREATURE=assets/<c>` first, or the wolf's layout applies.

## Which step next

- No `assets/<c>/model.glb`: make or fetch the mesh. Examples: `tools/fetch_examples.sh <c>` (wolf, dragon, spider,
  demon). New creature: anymove-rig.
- No `assets/<c>/rigged/rigged.blend` or `runs/<c>/ref/ref.png`: `tools/prep.sh <c>` then `tools/rig.sh <c>`
  (anymove-rig).
- No `runs/<c>/reader.pt`: anymove-reader (about 45 min on one RTX 4090), or `tools/fetch_examples.sh <c> --readers`.
- No clip for the move: anymove-film (about $0.26 a 5 s clip, and about 7 clips for a usable move).
- A clip: anymove-capture, `tools/capture.sh <c> <video.mp4> <name>`.
- Several good takes: anymove-pack, `tools/pack.sh <c>`.
- Clips keep swapping limb colours or the halves drift apart: anymove-lora.

## Try it without a GPU

The repo carries the dragon's roar as read by its reader. Fit it on the CPU in about a minute and a half:

```bash
mkdir -p runs/dragon/clips/roar
MOCAP_CREATURE=assets/dragon .venv/bin/python -m mocap.solve examples/dragon_roar/det.npz \
  assets/dragon/rigged/rig_info.json runs/dragon/clips/roar/motion.json
```

It prints the paw agreement (about 0.93) and the reprojection error per view; `motion.json` holds the keyframes.

## Costs and time (October 2026, one RTX 4090)

**Quote the all-in numbers, not the price of one clip:** about $3 for a usable move, and about $17 and a couple of hours
for a new creature with about 5 moves (about $11 of AI services on the person's own card). A move took about 7 clips on
average ($1.80), and 58% of captured clips passed the checks.

- Tripo H3.1 mesh with HD texture: $0.40. MiniMax H3 Max clip: about $0.26 for 5 s (H3: $0.30).
- Dataset and reader training: about 75 min on one RTX 4090. Reading a clip: about 12 s. Fitting: about 45 s. GLB and
  renders: about 3 min.
- A LoRA: $0.01 a training step, so 1,200 steps cost about $12 and take about 2 h on fal.

Say what a step will cost before calling fal, and confirm with the person before spending on a batch.