From One Phone-Shot Dance Video to a 3D Character Dancing the Whole Song
Oct 16, 2026 · Seen on YouTube · 0 views
Hina · live-action video to VRMA
くじら
Original choreography by Mitsuru. In the Short above, the top half is her original video and the bottom half is Hina, with the camera matched to the original. From a dance video shot on a phone with a fixed camera, AI reads a 3D skeleton for every frame (SAM 3D Body), which is converted into a motion file (VRMA) and applied to Hina. The whole song, about 60 seconds. The motion is an AI estimate, so details differ from the original choreography.
Idol stage version · 31 camera moves
The same motion, now in an idol outfit with a half-up princess hairstyle, danced on a concert stage. The shots were planned automatically from a catalog of camera moves, and her mouth opens and closes with the vocals.
- 31 shots
- 15 camera-move types
- Lowest camera 1.01 m
- 59.8 s
From raw capture to round 5 · the same first 20 seconds
Each clip has three panels: the original on the left, Hina in the middle (camera matched to the original), and Hina from the side on the right. Ordered oldest to newest.
Raw capture · no fixes yet
The estimated skeleton applied as is: hunched back, head down, toes lifted, pelvis twisted, and the skirt keeps getting flipped up.
- Full-song pose estimation 806 s (GPU)
Rounds 1–2 · deformation and joint limits
The estimation model and Hina have different rest poses, so instead of copying where each bone points, we copy how far each bone has rotated away from its rest pose. Her body straightens and her soles land flat. Then the motion is kept within the joint angles Hina's mesh can hold.
- Soles through the floor 651 → 0 frames
- Over joint limits 0 frames
Round 3 · foot drift
Hina used to be pinned in place, so whenever the dancer moved, her planted foot slid across the floor. Now the planted foot is locked where it lands and her body position is worked back from the feet, so she travels across the floor along with the dancer.
- Planted foot sliding 0.82 → 0 cm/frame
- Planted foot floating 13 → 0 frames
- Foot plants: left 40 times · right 39 times
Round 4 · removing jitter
The AI estimates each frame's pose independently, so the estimated feet jump a little every frame. Each bone gets a bit of smoothing over time, and the foot lifts between two plants are joined with interpolation.
- Foot acceleration p99 27.8 → 6.1 cm/frame²
Round 5 · leg shape, skipping the fade-in
Hina's stance was narrower than the dancer's, with her knees close together, so the thighs and shins now align directly with the dancer's direction. The original fades in at the start and out at the end; the darkest frames can't be estimated reliably, so they are skipped. Both finished videos above use this version's motion file.
- Skipped: first 28 frames · last 10 frames
Credit
How it was made
From a dance video shot on a phone, SAM 3D Body estimates a 3D skeleton frame by frame, which is converted to VRMA and applied to Hina. Applied directly, she hunches, her toes lift, her feet slide on the floor, and the motion jerks. It took five rounds of fixes before it looked like dancing, and each round was measured to confirm the fix worked before moving on.
For testing we rendered only the first 20 seconds, but the motion file covered the whole song from the start, so extending to 60 seconds only took a re-render. In both finished videos, her mouth opens and closes with the vocals separated out of the song, and blinks are scheduled on video time: rendering a frame takes longer than playing one, so blinks timed by the computer clock make her blink nonstop in the video, as in the five process clips above. The stage version also got 31 automatically planned shots; the camera never goes below 1 m and only full-body shots are taken from behind. The rules live in the shot-planning code and block bad shots before rendering.
A Short whose song is claimed through Content ID is blocked worldwide if it runs over one minute, so both were cut to 59.8 seconds.
Not solved yet: when a thigh lifts forward and swings outward at the same time (the moment at 55.9 s in the side-by-side version, arms raised and one leg up), holes open in Hina's skirt mesh. The limit of each single joint has been measured; the combined limit has not.
Looking back
- The same pipeline also worked on videos of two other dancers performing “Kujira”; the motion came through just as well. These are other people's dances, so before publishing I asked all three dancers: one agreed, one preferred not to have it published, and one did not reply, so only the one who agreed is shown.
Pose estimation: SAM 3D Body | Vocal separation: demucs | Made with Claude Code (Opus 5.5)
The previous post, “When Great Dance Moves Turn into a Dance of Despair, Can AI Fix It?”, concluded “to fix it, fix the motion data.” This note is the result of doing exactly that. Camera work and MMD-to-VRM are covered in the two earlier notes | October 16, 2026
Videos
