Markerless Reconstruction of Dynamic Scenes

The goal of our research is thus the development of new performance capture algorithms. These algorithms reconstruct detailed models of geometry, motion, and appearance of dynamic scenes from multi-view video data. The reconstruction is to be performed without optical interference with the scene, for instance, fiducial markers, as are commonly used in motion capture. In the past, we have pioneered the development of performance capture algorithms that are able to reconstruct detailed models of persons in general apparel or of arbitrary moving objects from multi-view video. Unfortunately, the application of these methods is constrained to indoor studios with many cameras, controlled lighting, and controlled backgrounds. In the reporting period. we therefore began to redefine the algorithmic foundations of dynamic scene reconstruction to enable, in the long run, performance capture of arbitrarily complex outdoor scenes under uncontrolled lighting and with only a few cameras.

In the reporting period, we developed some of the first methods for the detailed marker-less skeletal motion capture of humans and animals in outdoor scenes with few cameras (2 – 3). Another milestone was the development of new inverse rendering algorithms that allow us to measure models of appearance, illumination, and detailed geometry from video footage recorded under uncontrolled conditions. This new methodology is the basis for other important results: the first approach to capture relightable 3D videos in uncontrolled scenes, the first methods to capture highly detailed dynamic face models from stereo or monocular video, and the first approach for detailed full body performance capture with a single stereo camera [ see figure 1 ]. An important new strand of research is also the development of algorithms for marker-less hand motion capture from a few cameras, or even from a  single camera view [ see figure 2 ].