Human Motion from Video and IMUs

Real-time Human Pose from 6 IMU: While less intrusive, SIP is inherently offline, which limits a lot of applications. In recent work, Figure 1-right, we present a Deep Learning based real time algorithm for full body reconstruction from 6 IMUs alone. We found that propagation of information forward and backward in time is crucial for reconstructing natural human motion, for which we use a bi-directional Recursive Neural Network. We learn from synthetic IMU data and generalize to real data with transfer learning.

Visual-Inertial Human Pose: In contrast to visual measurements, IMU cannot provide absolute joint position information. This makes pure IMU based methods inaccurate for certain types of motions. Hence, in recent work, we introduce VIP, which combines IMUs and single a moving camera, to robustly recover human pose in challenging outdoor scenes. The moving camera, sensor heading drift, cluttered background, occlusions and many people visible in the video make the problem very hard. We associate 2D pose detections in each image to the corresponding IMU equipped persons by solving a novel graph based optimization problem that forces 3D to 2D coherency within a frame and across long range frames. Given these associations, we jointly optimize the pose of the SMPL body model, the camera pose and heading drift using a continuous optimization.

Using VIP, we collected the 3DPW dataset, which includes videos of humans in challenging scenes with accurate 3D parameters that provide, for the first time, the means to quantitatively evaluate monocular methods in difficult scenes and stimulate new research in this area, see Figure2.