We present BiomechGPT, the first motion-language model built to answer detailed, clinically meaningful questions about how a person moves.
How someone moves carries rich information about their health, and recent advances in markerless motion capture have made it easier to record biomechanics in accessible clinical settings. But each clinical question, from measuring a gait parameter to suggesting a diagnosis, usually needs its own custom pipeline. Multi-modal language models offer a natural way to unify these tasks. We build motion tokenizers that convert motion into discrete tokens for a language model to process, and we extend the motion-language model framework to clinical motion analysis, a domain with diverse, well-defined questions that have precise ground truth and direct clinical meaning.
We use 71 hours of biomechanical data from 750 participants, many with movement impairments from causes such as stroke or lower-limb prosthesis use, performing tasks commonly used in clinical mobility assessment. To expand our training data, we designed a cross-format tokenizer that encodes motion from SMPL body models into a single shared representation. This let us pool our clinical dataset with the public HumanML3D dataset and share annotations between them. On top of this, we built a large set of motion question–answer pairs to fine-tune the Gemma 3 language model.
A single BiomechGPT model handled ten clinical tasks at once, spanning both classification (activity, impairment, diagnosis, and assistive-device use) and regression (walking speed, cadence, and timed test scores). It worked well on data from both multi-camera systems and a single smartphone, showing the approach doesn't depend on how the movement was recorded. Performance rose steadily as we scaled up both model size and training data, including data pooled across different clinical recording setups and the general-purpose HumanML3D dataset. We report per-task performance and adidtionally, provided analysis on the effect of tokenization method.
These findings show that motion-language models can serve as a flexible, natural-language interface to biomechanical data, and that clinical motion understanding is a precise, measurement-grounded way to evaluate such models. We see this as a promising direction for rehabilitation and movement science. To support further research and clinical translation, we release our full code and trained model checkpoints.
