Movie heroes are soon to be transferred to virtual worlds more easily and realistically

Hollywood devotes great effort to chasing monsters through realistic-looking environments. Researchers at the Max Planck Institute for Informatics in Saarbrücken have now developed a technology that greatly simplifies the production of such scenes. Actors' movements are captured with a few cameras in a real scene and then transferred extremely realistically to virtual characters. This will not only simplify the work of cartoon makers, but also assist doctors and sportsmen with motion analysis. The new technology will soon be marketed by a newly-established business.

Whenever computer-animated characters roam through wild landscapes, such a Gollum in Lord of the Rings, there were real actors at work. Film studios usually use a procedure called ‘motion capture’. The actors wear skin-tight suits with markers attached to them reflecting beams of infrared light that are sent out and received by a special camera system. In this way, the movements of a real actor are recorded and can later be transferred to a virtual character, using animation software. “However, the suits are very uncomfortable for the actors, and the markers interfere with their movements”, says Nils Hasler from the Max Planck Institute for Informatics in Saarbrücken. For this reason, the Computer Graphics researchers there have developed a method that eliminates the need for markers but captures the movements quickly and realistically.
This method allows actors in their normal clothing to be filmed with ordinary cameras. The movements are then analysed with special computer software and transferred to a virtual character in the form of a skeleton. “We require only a few cameras instead of the several dozen cameras needed for the special effects in Hollywood. The movements are computed so quickly that we can transfer them directly to the animated character without time delay,” Hasler explains. The meanwhile patented computation method has been further refined in the past months. It can now deal with scenes in which several participants are simultaneously active and body parts overlap. “The system even detects a person’s movements when they are covered up by other objects or when there are disturbances in the background. This will allow us to shoot visual effects outside of the studio in the future, for example, out in open nature,” the Saarbrücken-based researcher reckons.
The computer scientists in Christian Theobalt’s “Graphics, Vision & Video” team were able to solve yet another problem in the past few months, as Hasler explains proudly: “It was difficult for our software to reconstruct the body movements of actors wearing big coats or women entering a scene in long ballroom dresses. Our new computation method enables us to capture surfaces in such precise detail that, e.g., the draping folds of clothing can be reproduced realistically.” The new technology is also useful in areas outside the film and game industry. Athletes could use it to analyse specific, individual body movements without bothersome markers. Sports journalists would be able to comment on motion sequences, like in pole vault and discus competitions, in live television broadcasts.
“The field of medicine would also profit. It would be easier for doctors to depict and track the degree of recovery after operations on joints,” Hasler explains. The researcher from the Max Planck Institute for Informatics wants to establish a company together with Professor Christian Theobalt and his research colleague, Carsten Stoll, in order to offer their software as a commercial product. “We have already had quite a few inquiries from companies in the film and sports marketing industries,” Hasler reveals.

Technical Background

The technology used in this method is quite affordable. Anywhere from five to twelve ordinary video cameras are needed. The computer scientists use their software to produce a 3-D model of the depicted person from a skeleton with 58 joints. In order to capture the movements, the computation method continuously works on overlaying the two-dimensional image from the video camera and the 3-D model as exactly as possible. The researchers can solve the necessary equations for this task efficiently and quickly. With this method, they capture filmed movements and visualize them in the virtual characters within just a few milliseconds.

Computer Science on Saarland University Campus

The Max Planck Institute for Informatics is just one of several institutes on Saarland University’s campus devoted to research in computer science. In direct vicinity are located: the Department of Computer Science of Saarland University, the Max Planck Institute for Software Systems, the Center for Bioinformatics, the Center for IT-Security, Privacy and Accountability, the recently renewed Cluster of Excellence “Multimodal Computing and Interaction”, the German Research Center for Artificial Intelligence (DFKI), and the Intel Visual Computing Institute.

More Information:

A. Elhayek, C. Stoll, N. Hasler, K. I. Kim, H.-P. Seidel, C. Theobalt
Spatio-temporal Motion Tracking with Unsynchronized Cameras
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, USA, 2012,
http://gvv.mpi-inf.mpg.de/files/old_site_files/ElhStoHasKimSeiThe12Spacetime.pdf

Online-Video:
www.mpi-inf.mpg.de/~theobalt/sog.mp4

Press pictures:

www.uni-saarland.de/pressefotos

Questions are answering:

Dr. Nils Hasler

Max Planck Institute for Informatics

Tel.: +49 681 9325 4055

E-Mail: hasler@mpi-inf.mpg.de

Professor Christian Theobalt

Max Planck Institute for Informatics

E-Mail: theobalt@mpii.de

Tel.: +49 681 9325-428