Pose estimation for augmented reality: a hands-on survey

Table of Contents

• Introduction
• Installation
  • Prerequisities
  • How to build
• Pose estimation relying on a 3D model
  • Pose estimation from a known model
  • Extension to markerless model-based tracking
• Pose estimation relying on an image model
  • Homography estimation
  • Direct motion estimation through template matching

Introduction

Augmented Reality (AR) has now progressed to the point where real-time applications are being considered and needed. At the same time it is important that synthetic elements are rendered and aligned in the scene in an accurate and visually acceptable way. In order to address these issues, real-time, robust and efficient tracking algorithms have to be considered. The tracking of objects in the scene amounts to calculating the location (or pose) between the camera and the scene.

In the paper [pdf]:

E. Marchand, H. Uchiyama and F. Spindler. Pose estimation for augmented reality:
a hands-on survey. IEEE Trans. on Visualization and Computer Graphics, 22(12):2633-2651, December 2016.

a brief but almost self contented introduction to the most important approaches dedicated to camera localization along with a survey of the extension that have been proposed in the recent years. We also try to link these methodological concepts to the main libraries and SDK available on the market.

The aim of this paper is then to provide researchers and practitioners with an almost comprehensive and consolidate introduction to effective tools to facilitate research in augmented reality. It is also dedicated to academics involved in teaching augmented reality at the undergraduate and graduate level.

For most of the presented approaches, we also provide links to code of short examples. This should allow readers to easily bridge the gap between theoretical aspects and practice. These examples have been written using OpenCV but also ViSP developed at Inria. This page contains the documentation of these documented source code proposed as a supplementary material of the paper.

We hope this article and source code will be accessible and interesting to experts and students alike.

Installation

Prerequisities

The source code available from http://github.com/lagadic/camera_localization was designed to work with OpenCV or with ViSP. During CMake configuration the user can choose which of these two 3rd parties are used. Prior to build this project, the user has to download and install OpenCV and/or ViSP. For example under Ubuntu OpenCV installation is done using:

sudo apt-get libopencv-dev

while ViSP installation is performed using:

sudo apt-get libvisp-dev

How to build

Once ViSP is installed, download the lastest source code release from github https://github.com/lagadic/camera_localization/releases.

Unzip the archive:

$ unzip camera_localization-2.0.1.zip

or extract the code from tarball:

$ tar xvzf camera_localization-2.0.1.tar.gz

Using cmake run:

$ cd camera_localization
$ cmake .
$ make

To generate the documentation you can run:

$ make doc

Pose estimation relying on a 3D model

Pose estimation from a known model

In this section we give base algorithm for camera localization.

• Pose from Direct Linear Transform method using OpenCV or using ViSP
  In this first tutorial a simple solution known as Direct Linear Transform (DLT) [4] [7] based on the resolution of a linear system is considered to estimate the pose of the camera from at least 6 non coplanar points.
• Pose from Dementhon's POSIT method using OpenCV or using ViSP
  In this second tutorial we give Dementhon's POSIT method [3] [6] used to estimate the pose based on the resolution of a linear system introducing additional constraints on the rotation matrix. The pose is estimated from at least 4 non coplanar points.
• Pose from homography estimation using OpenCV or using ViSP
  In this tutorial we explain how to decompose the homography to estimate the pose from at least 4 coplanar points.
• Pose from a non-linear minimization method using OpenCV or using ViSP
  In this other tutorial we give a non-linear minimization method to estimate the pose from at least 4 points. This method requires an initialization of the pose to estimate. Depending on the points planarity, this initialization could be performed using one of the previous pose algorithms.

Extension to markerless model-based tracking

• Pose from markerless model-based tracking using ViSP
  This tutorial focuses on markerless model-based tracking that allows to estimate the pose of the camera.

Pose estimation relying on an image model

Homography estimation

• Homography estimation using OpenCV or using ViSP
  In this tutorial we describe the estimation of an homography using Direct Linear Transform (DLT) algorithm. At least 4 coplanar points are requested to achieve the estimation.

Direct motion estimation through template matching

• Direct motion estimation through template matching using ViSP
  In this other tutorial we propose a direct motion estimation through template matching approach. Here the reference template should be planar.