Addressing scalability in object recognition

Figure 1.Re-use of model components. The expected variation of the spatial layout of horse parts is combined with a single giraffe example, resulting in a new giraffe object class model (left) that can be used for recognition in an image (right).

Re-using object class model components. In [1], we design a shape-based object class model that allows to re-use previously learned components of one model in order to facilitate the learning of another model, hence increasing scalability. The model represents object classes as assemblies of parts, and captures the local shape of each part as well as their global spatial layout and pairwise symmetry relations (see Fig. 1 for visualizations of horse and giraffe models). Since the corresponding probabilistic formulation includes a distinct factor for each of these constituent model aspects, individual components can be re-used and combined to form new object class models. In particular, given two topologically similar object classes, such as horses and giraffes, the expected variation of the spatial part layout of one class (horse) can be re-used in order to build a model of a second class (giraffe) from just a single training image, as shown in the figure. The part-based nature of our object class representation further allows to re-use model components even for relatively dissimilar classes, such as horses and swans.

According to our probabilistic formulation, object recognition amounts to finding the MAP hypothesis over all possible assignments between model and image features. Since the exact solution to this problem is intractable, we resort to efficient approximate inference techniques instead. Specifically, we use data-driven Markov chain Monte Carlo (DDMCMC) sampling, which allows to guide MAP search by bottom-up part proposals.

In our experimental evaluation, we demonstrate the effectiveness of re-using object class model components for scarce training data, and, at the time of publication of [1], achieve state-of-the-art recognition performance on a standard benchmark data set for shape-based recognition approaches.

Figure 2.Automatically discovered parts of motorbikes and horses. For each object class, corresponding colours denote corresponding part labels.

Automatic discovery of meaningful object parts: While the work presented in [1] relies on the manual selection of re-usable object class components and object parts, a second line of our research focuses on the automatic discovery of meaningful object parts, thereby reducing manual annotation effort. The key technical difference to [1] consists in the treatment of object parts as latent rather than observed variables in a probabilistic formulation.

In [2] , we phrase the problem of finding meaningful parts as inferring latent part labels in a conditional random field (CRF) model, that subsumes discriminative local part classifiers and their spatial layout in a coherent probabilistic framework. In contrast to prior work, we consider parts with flexible shapes and sizes, and marginalize over part occurrences instead of maximization. Further, we allow for and model arbitrary part topologies rather than committing to a single fixed topology. Fig. 2 illustrates automatically discovered parts for horses and motorbikes. Training our object class models involves inferring the values of latent variables, and hence uses the expectation maximization (EM) algorithm. Object recognition in test images proceeds in a sliding window fashion, marginalizing over latent part labels for each potential window using sum-product belief propagation.

In our experiments, we demonstrate superior recognition performance compared to previous work on a variety of different object classes of the challenging Pascal VOC 2007 data set. In particular, we show the benefit of marginalizing over part labels, and learning part shape and topology rather than committing to a fixed layout.

In [3], we explore the decomposition of object class templates based on histograms of oriented gradients (HOG) features into collections of latent topics, using latent Dirichlet allocation (LDA). The decompositions are obtained in a fully data-driven fashion, allowing for the automatic discovery of different viewpoints, object sub-classes, and object parts from training images. The resulting decomposition can further serve as an intermediate representation for object class recognition, in combination with discriminative classifiers. In our experiments, we demonstrate the benefit of this representation over the original HOG template on the Pascal VOC 2006 data set, achieving state-of-the-art performance on several object classes at the time of publication.