OSSO: Obtaining Skeletal Shape from Outside
Marilyn Keller, Silvia Zuffi, Michael J. Black and Sergi Pujades
CVPR 2022
[Paper] [Supplementary] [Video] [Poster]
[Code] [Dataset]

From DXA scans we learn the correlation between the body surface and the skeleton (left).
Then, given a body shape, we can predict the underlying skeleton shape (right).

Abstract

We address the problem of inferring the anatomic skeleton of a person, in an arbitrary pose, from the 3D surface of the body; i.e. we predict the inside (bones) from the outside (skin). This has many applications in medicine and biomechanics. Existing state-of-the-art biomechanical skeletons are detailed but do not easily generalize to new subjects. Additionally, computer vision and graphics methods that predict skeletons are typically heuristic, not learned from data, do not leverage the full 3D body surface, and are not validated against ground truth. To our knowledge, our system, called OSSO (Obtaining Skeletal Shape from Outside), is the first to learn the mapping from the 3D body surface to the internal skeleton from real data. We do so using 1000 male and 1000 female dual-energy X-ray absorptiometry (DXA) scans. To these, we fit a parametric 3D body shape model (STAR) to capture the body surface and a novel part-based 3D skeleton model to capture the bones. This provides inside/outside training pairs. We model the statistical variation of full skeletons using PCA in a pose-normalized space. We then train a regressor from body shape parameters to skeleton shape parameters and refine the skeleton to satisfy constraints on physical plausibility. Given an arbitrary 3D body shape and pose, OSSO predicts a realistic skeleton inside. In contrast to previous work, we evaluate the accuracy of the skeleton shape quantitatively on held out DXA scans, outperforming the state-of-the art. We also show 3D skeleton prediction from varied and challenging 3D bodies. The code to infer a skeleton from a body shape is available for research, and the dataset of paired outer surface (skin) and skeleton (bone) meshes are available as a Biobank Returned Dataset. This research has been conducted using the UK Biobank Resource.

Dataset release

Acknowledgements

This research has been conducted using the UK Biobank Resource under the Approved Project ID 51951. The authors thank the International Max Planck Research School for Intelligent Systems for supporting Marilyn Keller.  Sergi Pujades’ work was funded by the ANR SEMBA project. We thank Anatoscope (www.anatoscope.com) for the initial skeleton mesh and useful discussions.

@inproceedings{Keller:CVPR:2022,
  title = {{OSSO}: Obtaining Skeletal Shape from Outside},
  author = {Keller, Marilyn and Zuffi, Silvia and Black, Michael J. and Pujades, Sergi},
  booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
  month = jun,
  year = {2022},
  month_numeric = {6},
  pages = {20492-20501}}