ViSP  2.9.0
Tutorial: Model-based tracking

With ViSP it is possible to track an object using its cad model. Considered objects should be modeled by lines or cylinders. The model of the object could be defined in vrml format, or in cao format.

The model-based tracker can consider moving-edges behind the lines of the model (see section Model-based edges tracker). It can also consider keypoints that are detected and tracked on each visible face of the model (see section Model-based KLT tracker). The tracker can also handle moving-edges and keypoints in an hybrid scheme (see section Model-based hybrid tracker).

While the Model-based edges tracker is appropriate to track untextured object, the Model-based KLT tracker is more designed to exploit textured objects with edges that are not really visible. The Model-based hybrid tracker is appropriate to track textured objects with visible edges.

In the following sections, we consider the tracking of a tea box modeled in cao format. To simplify the source code, the tracking is performed on a single image. The extension to a sequence of images or to images acquired from a camera is easy.

Model-based edges tracker

The following example that comes from tutorial-mb-edge-tracker.cpp allows to track the tea box using vpMbEdgeTracker class.

#include <visp/vpDisplayGDI.h>
#include <visp/vpDisplayX.h>
#include <visp/vpImageIo.h>
#include <visp/vpMbEdgeTracker.h>
int main()
{
try {
vpImageIo::read(I, "teabox.pgm");
#if defined(VISP_HAVE_X11)
vpDisplayX display(I,100,100,"Model-based edge tracker");;
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI display(I,100,100,"Model-based edge tracker");;
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
vpMbEdgeTracker tracker;
#ifdef VISP_HAVE_XML2
tracker.loadConfigFile("teabox.xml");
#else
vpMe me;
me.setMaskSize(5);
me.setMaskNumber(180);
me.setRange(8);
me.setThreshold(10000);
me.setMu1(0.5);
me.setMu2(0.5);
tracker.setMovingEdge(me);
cam.initPersProjWithoutDistortion(839, 839, 325, 243);
tracker.setCameraParameters(cam);
tracker.setAngleAppear( vpMath::rad(70) );
tracker.setFarClippingDistance(100.0);
#endif
tracker.setOgreVisibilityTest(false);
tracker.loadModel("teabox.cao");
//tracker.loadModel("teabox.wrl");
tracker.setDisplayFeatures(true);
tracker.initClick(I, "teabox.init");
while(1){
tracker.track(I);
tracker.getPose(cMo);
tracker.getCameraParameters(cam);
tracker.display(I, cMo, cam, vpColor::red, 2);
vpDisplay::displayFrame(I, cMo, cam, 0.025, vpColor::none, 3);
if (vpDisplay::getClick(I, false))
break;
}
#ifdef VISP_HAVE_XML2
#endif
#ifdef VISP_HAVE_COIN
SoDB::finish();
#endif
}
catch(vpException e) {
std::cout << "Catch an exception: " << e << std::endl;
}
}

The video below shows the result of the tea box model-based edges tracking.

Hereafter is the description of the new lines introduced in this example.

#include <visp/vpMbEdgeTracker.h>

Here we include the header of the vpMbEdgeTracker class that allows to track an object from its cad model by using the moving-edges. The tracker will use image I and the intrinsic camera parameters cam as input.

As output, it will estimate cMo, the pose of the object in the camera frame.

Once the input image teabox.pgm is loaded in I, a window is created and initialized with image I. Then we create an instance of the tracker.

There are then two different ways to initialize the tracker.

An important setting concerns the visibility test that is used to determine if a face is visible. Note that moving-edges are tracked only on visible faces. Two different visibility tests are implemented; with or without Ogre ray tracing. The default test is the one without Ogre. The function vpMbEdgeTracker::setOgreVisibilityTest() allow to select which test is to use.

Let us now highlight how the visibility test works. As illustrated in the following figure, the angle $ \alpha $ between the normal of the face and the line going from the camera to the center of gravity of the face is used to determine if the face is visible. If we consider two parameters; the angle to determine if a face is appearing $ \alpha_{appear} $, and the angle to determine if the face is disappearing $ \alpha_{disappear} $, a face will be considered as visible if $ \alpha \leq \alpha_{disappear} $. We consider also that a new face is appearing if $ \alpha \geq \alpha_{appear} $. These two parameters can be set either in the xml file:

<conf>
...
<face>
<angle_appear>70</angle_appear>
<angle_disappear>80</angle_disappear>
</face>

or in the code:

tracker.setAngleAppear( vpMath::rad(70) );
Note
When these two angle parameters are not set, their default values set to 89 degrees are used.
img-tracker-mb-visibility.jpg
Principle of the visibility test used to determine if a face is visible.
  • When Ogre visibility test is disabled (we recall that this is the default behavior), the algorithm that computes the normal of the face is very simple. It makes the assumption that faces are convex and oriented counter clockwise. Here the face is considered as appearing if $ \alpha < 70$ degrees, and disappearing if $ \alpha > 80$ degrees. When only moving-edges are used (nor keypoints) and when the object to track is simple like a single box, we suggest as here to disable Ogre visibility test.
    tracker.setOgreVisibilityTest(false); // Default behavior
  • When Ogre visibility test is enabled, the algorithm used to determine the visibility of a face is the same than previously except that once visible faces are detected thanks to their normal, we add an other test to reject faces that are partially occluded by an other one. This additional test is performed using Ogre ray-tracing capability.
    tracker.setOgreVisibilityTest(true);

Additionally to the visibility test described above, it is also possible to use clipping. Firstly, the algorithm removes the faces that are not visibles, according to the visibility test used, then it will also remove the faces or parts of the faces that are out of the clipping planes. As illustrated in the following figure, different clipping planes can be enabled.

img-fov.png
Camera field of view and clipping planes.

Let's consider two plane categories: the ones belonging to the field of view or FOV (Left, Right, Up and Down), and the Near and Far clipping planes. The FOV planes can be enabled by:

which is equivalent to:

Of course, if the user just wants to activate Right and Up clipping, he will use:

For the Near and Far clipping it is quite different. Indeed, thoses planes require clipping distances. Here there are two choices, either the user uses default values and activate them with:

or the user can specify the distances in meters, which will automatically activate the clipping for thoses planes:

tracker.setFarClippingDistance(100.0);

It is also possible to enable them in the xml file. This is done with the following lines:

<conf>
...
<face>
...
<near_clipping>0.1</near_clipping>
<far_clipping>100.0</far_clipping>
<fov_clipping>0</fov_clipping>
</face>

Here for simplicity, the user just has the possibility to either activate all the FOV clipping planes or none of them (fov_clipping requires a boolean).

Note
When clipping parameters are not set in the xml file, nor in the code, clipping is not used. Usually clipping is not helpfull when the oject to track is simple.

Now we are ready to load the cad model of the object. ViSP supports cad model in cao format or in vrml format. The cao format is a particular format only supported by ViSP. It doesn't require an additional 3rd party rather then vrml format that require Coin 3rd party. We load the cad model in cao format with:

tracker.loadModel("teabox.cao");

The file teabox.cao describes first the vertices of the box, then the edges that corresponds to the faces. A more complete description of this file is provided in CAD model in cao format (teabox.cao example). The next figure gives the index of the vertices that are defined in teabox.cao.

To load the cad model in vrml the user has to replace the previous line by the following:

tracker.loadModel("teabox.wrl");

As for the cao format, teabox.wrl describes first the vertices of the box, then the edges that corresponds to the faces. A more complete description of this file is provided in CAD model in vrml format.

img-teabox-cao.jpg
Index of the vertices used to model the tea box in cao format.

Once the model of the object to track is loaded, with the next line the display in the image window of additional drawings in overlay such as the moving edges positions, is then enabled by:

tracker.setDisplayFeatures(true);

Now we have to initialize the tracker. With the next line we choose to use a user interaction. As explained later, the user has to match in the image four vertices with their 3D coordinates. Matched 2D and 3D coordinates are then used to compute an initial pose used to initialize the tracker:

tracker.initClick(I, "teabox.init");

The content of teabox.init file that defines 3D coordinates of some points of the model used during user intialization is provided hereafter:

1 4
2 0.000 0 0
3 0.165 0 0
4 0.165 0 -0.08
5 0.165 0.068 -0.08
6 

We give now the signification of each line of this file:

  • line 1: Number of 3D points that should be defined in this file. At least 4 points are required. Four is the minimal number of points requested to compute a pose.
  • line 2: Each point is defined by its 3D coordinates. Here we define the first point with coordinates (0,0,0). In the previous figure it corresponds to vertex 0 of the tea box. This point is also the origin of the frame in which all the points are defined.
  • line 3: 3D coordinates of vertex 3.
  • line 4: 3D coordinates of vertex 2.
  • line 5: 3D coordinates of vertex 5.

Here the user has to click on vertex 0, 3, 2 and 5 in the window that displays image I. From the 3D coordinates defined in teabox.init and the corresponding 2D coordinates of the vertices obtained by user interaction a pose is computed that is than used to initialize the tracker.

Next, in the infinite while loop, after displaying the next image, we track the object on a new image I.

tracker.track(I);

The result of the tracking is a pose cMo that could be obtained by:

tracker.getPose(cMo);

Next lines are used first to retrieve the camera parameters used by the tracker, then to display the visible part of the cad model using red lines with 2 as thickness, and finally to display the object frame at the estimated position cMo. Each axis of the frame are 0.025 meters long. Using vpColor::none indicates that x-axis is displayed in red, y-axis in green, while z-axis in blue. The thickness of the axis is 3.

tracker.getCameraParameters(cam);
tracker.display(I, cMo, cam, vpColor::red, 2);
vpDisplay::displayFrame(I, cMo, cam, 0.025, vpColor::none, 3);

The last lines are here to free the memory allocated by libxml2 or Coin 3rd party libraries:

#ifdef VISP_HAVE_XML2
#endif
#ifdef VISP_HAVE_COIN
SoDB::finish();
#endif

Model-based KLT tracker

The following example that comes from tutorial-mb-klt-tracker.cpp allows to track the tea box using vpMbKltTracker class.

#include <visp/vpDisplayGDI.h>
#include <visp/vpDisplayX.h>
#include <visp/vpImageIo.h>
#include <visp/vpMbKltTracker.h>
int main()
{
#ifdef VISP_HAVE_OPENCV
try {
vpImageIo::read(I, "teabox.pgm");
#if defined(VISP_HAVE_X11)
vpDisplayX display(I,100,100,"Model-based keypoints tracker");;
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI display(I,100,100,"Model-based keypoints tracker");;
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
vpMbKltTracker tracker;
#ifdef VISP_HAVE_XML2
tracker.loadConfigFile("teabox.xml");
#else
tracker.setMaskBorder(5);
vpKltOpencv klt_settings;
klt_settings.setMaxFeatures(300);
klt_settings.setWindowSize(5);
klt_settings.setQuality(0.015);
klt_settings.setMinDistance(8);
klt_settings.setHarrisFreeParameter(0.01);
klt_settings.setBlockSize(3);
klt_settings.setPyramidLevels(3);
tracker.setKltOpencv(klt_settings);
cam.initPersProjWithoutDistortion(839, 839, 325, 243);
tracker.setCameraParameters(cam);
tracker.setAngleAppear( vpMath::rad(70) );
tracker.setFarClippingDistance(100.0);
#endif
tracker.setOgreVisibilityTest(true);
tracker.loadModel("teabox-triangle.cao");
tracker.setDisplayFeatures(true);
tracker.initClick(I, "teabox.init");
while(1){
tracker.track(I);
tracker.getPose(cMo);
tracker.getCameraParameters(cam);
tracker.display(I, cMo, cam, vpColor::red, 2, true);
vpDisplay::displayFrame(I, cMo, cam, 0.025, vpColor::none, 3);
if (vpDisplay::getClick(I, false))
break;
}
#ifdef VISP_HAVE_XML2
#endif
#ifdef VISP_HAVE_COIN
SoDB::finish();
#endif
}
catch(vpException e) {
std::cout << "Catch an exception: " << e << std::endl;
}
#endif
}

The video below shows the result of the tea box model-based KLT tracking where keypoints are used as input features.

This example is very similar to the one presented in Model-based edges tracker except that here we use vpMbKltTracker class to track the tea box.

As previously, there are two different ways to initialize the tracker.

  • The first one, if libxml2 is available, consists in reading the settings from an xml file.
    tracker.loadConfigFile("teabox.xml");
    The teabox.xml file used here contains the following:
    <?xml version="1.0"?>
    <conf>
    <klt>
    <mask_border>5</mask_border>
    <max_features>300</max_features>
    <window_size>5</window_size>
    <quality>0.015</quality>
    <min_distance>8</min_distance>
    <harris>0.01</harris>
    <size_block>3</size_block>
    <pyramid_lvl>3</pyramid_lvl>
    </klt>
    <camera>
    <u0>325.66776</u0>
    <v0>243.69727</v0>
    <px>839.21470</px>
    <py>839.44555</py>
    </camera>
    <face>
    <angle_appear>70</angle_appear>
    <angle_disappear>80</angle_disappear>
    <near_clipping>0.1</near_clipping>
    <far_clipping>100</far_clipping>
    <fov_clipping>1</fov_clipping>
    </face>
    </conf>
  • The second one consists in initializing the parameters directly in the source code:
    tracker.setAngleAppear(70);
    tracker.setAngleDisappear(80);
    tracker.setMaskBorder(5);
    vpKltOpencv klt_settings;
    klt_settings.setMaxFeatures(300);
    klt_settings.setWindowSize(5);
    klt_settings.setQuality(0.015);
    klt_settings.setMinDistance(8);
    klt_settings.setHarrisFreeParameter(0.01);
    klt_settings.setBlockSize(3);
    klt_settings.setPyramidLevels(3);
    tracker.setKltOpencv(klt_settings);
    cam.initPersProjWithoutDistortion(839, 839, 325, 243);
    tracker.setCameraParameters(cam);

Note also that in this example we model the tea box with triangles:

tracker.loadModel("teabox-triangle.cao");

The file teabox-triangle.cao describes first the vertices of the box, then the triangular faces. A more complete description of this file is provided in CAD model in cao format (teabox-triangle.cao example).

Note that this is the only tracker for which lines of the model are not necessary edges of the object.

Model-based hybrid tracker

The following example that comes from tutorial-mb-hybrid-tracker.cpp allows to track the tea box using vpMbEdgeKltTracker class.

#include <visp/vpDisplayGDI.h>
#include <visp/vpDisplayX.h>
#include <visp/vpImageIo.h>
#include <visp/vpMbEdgeKltTracker.h>
int main()
{
#ifdef VISP_HAVE_OPENCV
try {
vpImageIo::read(I, "teabox.pgm");
#if defined(VISP_HAVE_X11)
vpDisplayX display(I,100,100,"Model-based hybrid tracker");;
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI display(I,100,100,"Model-based hybrid tracker");;
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
#ifdef VISP_HAVE_XML2
tracker.loadConfigFile("teabox.xml");
#else
vpMe me;
me.setMaskSize(5);
me.setMaskNumber(180);
me.setRange(8);
me.setThreshold(10000);
me.setMu1(0.5);
me.setMu2(0.5);
tracker.setMovingEdge(me);
tracker.setMaskBorder(5);
vpKltOpencv klt_settings;
klt_settings.setMaxFeatures(300);
klt_settings.setWindowSize(5);
klt_settings.setQuality(0.015);
klt_settings.setMinDistance(8);
klt_settings.setHarrisFreeParameter(0.01);
klt_settings.setBlockSize(3);
klt_settings.setPyramidLevels(3);
tracker.setKltOpencv(klt_settings);
cam.initPersProjWithoutDistortion(839, 839, 325, 243);
tracker.setCameraParameters(cam);
tracker.setAngleAppear( vpMath::rad(70) );
tracker.setFarClippingDistance(100.0);
#endif
tracker.setOgreVisibilityTest(true);
tracker.loadModel("teabox.cao");
tracker.setDisplayFeatures(true);
tracker.initClick(I, "teabox.init");
while(1){
tracker.track(I);
tracker.getPose(cMo);
tracker.getCameraParameters(cam);
tracker.display(I, cMo, cam, vpColor::red, 2, true);
vpDisplay::displayFrame(I, cMo, cam, 0.025, vpColor::none, 3);
if (vpDisplay::getClick(I, false))
break;
}
#ifdef VISP_HAVE_XML2
#endif
#ifdef VISP_HAVE_COIN
SoDB::finish();
#endif
}
catch(vpException e) {
std::cout << "Catch an exception: " << e << std::endl;
}
#endif
}

The video below shows the result of the tea box model-based hybrid tracking where moving-edges and keypoints are used as input features.

The source code is very similar to the one described in Model-based edges tracker and Model-based KLT tracker. It doesn't require additional line by line explanation. We provide just hereafter the content of the teabox.xml file:

<?xml version="1.0"?>
<conf>
<ecm>
<mask>
<size>5</size>
<nb_mask>180</nb_mask>
</mask>
<range>
<tracking>8</tracking>
</range>
<contrast>
<edge_threshold>10000</edge_threshold>
<mu1>0.5</mu1>
<mu2>0.5</mu2>
</contrast>
</ecm>
<sample>
<step>4</step>
<nb_sample>250</nb_sample>
</sample>
<klt>
<mask_border>5</mask_border>
<max_features>300</max_features>
<window_size>5</window_size>
<quality>0.015</quality>
<min_distance>8</min_distance>
<harris>0.01</harris>
<size_block>3</size_block>
<pyramid_lvl>3</pyramid_lvl>
</klt>
<camera>
<u0>325.66776</u0>
<v0>243.69727</v0>
<px>839.21470</px>
<py>839.44555</py>
</camera>
<face>
<angle_appear>70</angle_appear>
<angle_disappear>80</angle_disappear>
<near_clipping>0.1</near_clipping>
<far_clipping>100</far_clipping>
<fov_clipping>1</fov_clipping>
</face>
</conf>

CAD model in cao format

cao format is specific to ViSP. It allows to describe the CAD model of an object using a text file with extension .cao.

teabox.cao example

The content of the file teabox.cao used in the Model-based edges tracker and Model-based hybrid tracker examples is given here:

1 V1
2 8
3 0 0 0
4 0 0 -0.08
5 0.165 0 -0.08
6 0.165 0 0
7 0.165 0.068 0
8 0.165 0.068 -0.08
9 0 0.068 -0.08
10 0 0.068 0
11 0
12 0
13 6
14 4 0 1 2 3
15 4 1 6 5 2
16 4 4 5 6 7
17 4 0 3 4 7
18 4 5 4 3 2
19 4 0 7 6 1
20 0

This file describes the model of the tea box corresponding to the next image:

img-teabox-cao.jpg
Index of the vertices used to model the tea box in cao format.

We make the choice to describe the faces of the box from the 3D points that correspond to the vertices. We provide now a line by line description of the file:

  • line 1: Header of the .cao file
  • line 2: The model is defined by 8 3D points. Here the 8 points correspond to the 8 vertices of the tea box presented in the previous figure. Thus, next 8 lines define the 3D points coordinates.
  • line 3: 3D point with coordinate (0,0,0) corresponding to vertex 0 of the tea box. This point is also the origin of the frame in which all the 3D points are defined.
  • line 4: 3D point with coordinate (0,0,-0.08) corresponding to vertex 1.
  • line 5 to 10: The other 3D points corresponding to vertices 2 to 7 respectively.
  • line 11: Number of single lines of the model. Sometimes, it could be useful to track a single line that doesn't lead to a face.
  • line 12: Deprecated parameter that should always be set to 0.
  • line 13: The number of faces defined by a set of points. Here our tea box has 6 faces. Thus, next 6 lines describe each face from the 3D points defined previously line 3 to 10.
  • line 14: First face defined by 4 points, respectively vertices 0,1,2,3. The orientation of the face is counter clockwise by going from vertex 0 to vertex 1, then 2 and 3. This fixes the orientation of the normal of the face going outside the object.
  • line 15: Second face also defined by 4 points, respectively vertices 1,6,5,2 to have a counter clockwise orientation.
  • line 16 to 19: The four other faces of the box.
  • line 20: Number of cylinders describing the model. Since we model a simple box, the number of cylinders is 0.

teabox-triangle.cao example

The content of the file teabox-triangle.cao used in the Model-based KLT tracker example is given here:

1 V1
2 8
3 0 0 0
4 0 0 -0.08
5 0.165 0 -0.08
6 0.165 0 0
7 0.165 0.068 0
8 0.165 0.068 -0.08
9 0 0.068 -0.08
10 0 0.068 0
11 0
12 0
13 12
14 3 0 1 2
15 3 0 2 3
16 3 0 3 7
17 3 3 4 7
18 3 4 5 6
19 3 4 6 7
20 3 1 6 5
21 3 1 5 2
22 3 5 3 2
23 3 5 4 3
24 3 7 6 1
25 3 7 1 0
26 0

This file describes the model of the tea box corresponding to the next image:

img-teabox-cao-triangle.jpg
Index of the vertices used to model the tea box in cao format with triangles.

Until line 12, the content of this file is similar to the one described in teabox.cao example. Line 13 we specify that the model contains 12 faces. Each face is then described as a triangle.

CAD model in vrml format

ViSP support vrml format only if Coin 3rd party is installed. This format allows to describe the CAD model of an object using a text file with extension .wrl.

teabox.wrl example

The content of the teabox.wrl file used in the Model-based edges tracker is given hereafter. This content is to make into relation with teabox.cao described in teabox.cao example.

1 #VRML V2.0 utf8
2 
3 DEF fst_0 Group {
4 children [
5 
6 # Object "teabox"
7 Shape {
8 
9 geometry DEF cube IndexedFaceSet {
10 
11 coord Coordinate {
12 point [
13 0 0 0 ,
14 0 0 -0.08,
15 0.165 0 -0.08,
16 0.165 0 0 ,
17 0.165 0.068 0 ,
18 0.165 0.068 -0.08,
19 0 0.068 -0.08,
20 0 0.068 0 ]
21 }
22 
23 coordIndex [
24  0,1,2,3,-1,
25  1,6,5,2,-1,
26  4,5,6,7,-1,
27  0,3,4,7,-1,
28  5,4,3,2,-1,
29  0,7,6,1,-1]}
30 }
31 
32 ]
33 }

This file describes the model of the tea box corresponding to the next image:

img-teabox-cao.jpg
Index of the vertices used to model the tea box in vrml format.

We provide now a line by line description of the file where the faces of the box are defined from the vertices:

  • line 1 to 10: Header of the .wrl file.
  • line 13 to 20: 3D coordinates of the 8 tea box vertices.
  • line 34 to 29: Each line describe a face. In this example, a face is defined by 4 vertices. For example, the first face join vertices 0,1,2,3. The orientation of the face is counter clockwise by going from vertex 0 to vertex 1, then 2 and 3. This fixes the orientation of the normal of the face going outside the object.

You are now ready to see the next Tutorial: Template tracking.