Visual Servoing Platform  version 3.5.1 under development (2023-09-22)
Tutorial: Moving-edges tracking

Introduction

ViSP moving-edges tracker provide real-time tracking capabilities of points normal to the object contours. Such a tracker allow to track a line, an ellipse or more complex objects using model-based approaches.

All the material (source code and video) described in this tutorial is part of ViSP source code and could be downloaded using the following command:

$ svn export https://github.com/lagadic/visp.git/trunk/tutorial/tracking/moving-edges

Line tracking

With ViSP you can track a line using moving edges. The following example code available in tutorial-me-line-tracker.cpp shows how to use ViSP vpMeLine class to this end.

#include <visp3/core/vpConfig.h>
#ifdef VISP_HAVE_MODULE_SENSOR
#include <visp3/sensor/vp1394CMUGrabber.h>
#include <visp3/sensor/vp1394TwoGrabber.h>
#include <visp3/sensor/vpV4l2Grabber.h>
#endif
#include <visp3/gui/vpDisplayGDI.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/gui/vpDisplayX.h>
#include <visp3/me/vpMeLine.h>
#if defined(HAVE_OPENCV_VIDEOIO)
#include <opencv2/videoio.hpp>
#endif
int main()
{
#if (defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_CMU1394) || defined(VISP_HAVE_V4L2) || defined(HAVE_OPENCV_VIDEOIO))
try {
#if defined(VISP_HAVE_DC1394)
vp1394TwoGrabber g(false);
#elif defined(VISP_HAVE_CMU1394)
#elif defined(VISP_HAVE_V4L2)
#elif defined(HAVE_OPENCV_VIDEOIO)
cv::VideoCapture g(0); // open the default camera
if (!g.isOpened()) { // check if we succeeded
std::cout << "Failed to open the camera" << std::endl;
return EXIT_FAILURE;
}
cv::Mat frame;
#endif
#if defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_V4L2) || defined(VISP_HAVE_CMU1394)
g.acquire(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
g >> frame; // get a new frame from camera
#endif
#if defined(VISP_HAVE_X11)
vpDisplayX d(I, 0, 0, "Camera view");
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI d(I, 0, 0, "Camera view");
#elif defined(HAVE_OPENCV_HIGHGUI)
vpDisplayOpenCV d(I, 0, 0, "Camera view");
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
vpMe me;
me.setRange(25);
me.setThreshold(20);
me.setSampleStep(10);
vpMeLine line;
line.setMe(&me);
line.initTracking(I);
while (1) {
#if defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_V4L2) || defined(VISP_HAVE_CMU1394)
g.acquire(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
g >> frame;
#endif
line.track(I);
}
}
catch (const vpException &e) {
std::cout << "Catch an exception: " << e << std::endl;
}
#endif
}
Firewire cameras video capture based on CMU 1394 Digital Camera SDK.
void acquire(vpImage< unsigned char > &I)
Class for firewire ieee1394 video devices using libdc1394-2.x api.
static const vpColor red
Definition: vpColor.h:211
Display for windows using GDI (available on any windows 32 platform).
Definition: vpDisplayGDI.h:128
The vpDisplayOpenCV allows to display image using the OpenCV library. Thus to enable this class OpenC...
Use the X11 console to display images on unix-like OS. Thus to enable this class X11 should be instal...
Definition: vpDisplayX.h:132
static void display(const vpImage< unsigned char > &I)
static void flush(const vpImage< unsigned char > &I)
error that can be emitted by ViSP classes.
Definition: vpException.h:59
static void convert(const vpImage< unsigned char > &src, vpImage< vpRGBa > &dest)
Class that tracks in an image a line moving edges.
Definition: vpMeLine.h:148
void display(const vpImage< unsigned char > &I, const vpColor &color, unsigned int thickness=1)
Definition: vpMeLine.cpp:181
void track(const vpImage< unsigned char > &I)
Definition: vpMeLine.cpp:649
void initTracking(const vpImage< unsigned char > &I)
Definition: vpMeLine.cpp:186
@ RANGE_RESULT
Definition: vpMeSite.h:75
void setDisplay(vpMeSite::vpMeSiteDisplayType select)
Definition: vpMeTracker.h:238
void setMe(vpMe *p_me)
Definition: vpMeTracker.h:259
Definition: vpMe.h:122
void setSampleStep(const double &s)
Definition: vpMe.h:390
void setRange(const unsigned int &r)
Definition: vpMe.h:383
void setLikelihoodThresholdType(const vpLikelihoodThresholdType likelihood_threshold_type)
Definition: vpMe.h:445
@ NORMALIZED_THRESHOLD
Easy-to-use normalized likelihood threshold corresponding to the minimal luminance contrast to consid...
Definition: vpMe.h:132
void setThreshold(const double &t)
Definition: vpMe.h:435
Class that is a wrapper over the Video4Linux2 (V4L2) driver.

The video shows the result of the tracking:

Here after we explain line by line the program.

Images that are processed could be acquired from a firewire camera on Unix or Windows, of from an usb camera under Unix. That is allowed by including the grabber headers.

#include <visp3/sensor/vp1394CMUGrabber.h>
#include <visp3/sensor/vp1394TwoGrabber.h>
#include <visp3/sensor/vpV4l2Grabber.h>

To display these images we then include the headers of the viewers.

#include <visp3/gui/vpDisplayGDI.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/gui/vpDisplayX.h>

The Graphical Display Interface (GDI) allows to display images under Windows, while X11 allows this feature under unix-like systems.

Finally, to track a line with the moving edges, we include the header of the vpMeLine class.

#include <visp3/me/vpMeLine.h>

In the main() function, The source code is build only if one of the grabbers is available.

#include <visp3/me/vpMeLine.h>

Here we create a gray level image container I that will contain the images acquired by our camera.

Then, we create a grabber instance, first for a firewire camera under Unix if libdc1394 3rd party is installed, secondly for a firewire camera under Windows if CMU1394 3rd party is installed, next for an usb camera under Unix if none of the previous 3rd party are installed and if libv4l is installed, and finally with OpenCV. The Tutorial: Image frame grabbing gives more details concerning the framegrabbing.

#if defined(VISP_HAVE_DC1394)
vp1394TwoGrabber g(false);
#elif defined(VISP_HAVE_CMU1394)
#elif defined(VISP_HAVE_V4L2)
#elif defined(HAVE_OPENCV_VIDEOIO)
cv::VideoCapture g(0); // open the default camera
if (!g.isOpened()) { // check if we succeeded
std::cout << "Failed to open the camera" << std::endl;
return EXIT_FAILURE;
}
cv::Mat frame;
#endif

We then open the connection with the grabber and acquire an image in I.

#if defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_V4L2) || defined(VISP_HAVE_CMU1394)
g.acquire(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
g >> frame; // get a new frame from camera
#endif

To be able to display image I and the tracking results in overlay in a window, we create a display instance.

#if defined(VISP_HAVE_X11)
vpDisplayX d(I, 0, 0, "Camera view");
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI d(I, 0, 0, "Camera view");
#elif defined(HAVE_OPENCV_HIGHGUI)
vpDisplayOpenCV d(I, 0, 0, "Camera view");
#else
std::cout << "No image viewer is available..." << std::endl;
#endif

Then we display the image in the window created previously.

We then initialize the moving edges parameters used later by the tracker. From the previous position of a moving edge, we are tracking its new position along the normal of the contour with a range of 25 pixels. For each pixel along the normal we will compute the oriented convolution. The pixel that will be selected by the moving edges algorithm will be the one that has a convolution higher than 15000. Between two consecutive moving edges on the contour we keep a space of 10 pixels.

We then, create an instance of the vpMeTracker class that will track our line. We initialize the tracker with the previous moving-egdes parameters. We allow also the tracker to display additional information on the viewer overlay. The user has than to initialize the tracker on image I by clicking on two points located on the line to track.

vpMeLine line;
line.setMe(&me);
line.initTracking(I);

Once the tracker is initialized, we enter in a while loop where we successively acquire a new image, display it, track the line, display the tracking results and finally flush the overlay drawings in the viewer.

while (1) {
#if defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_V4L2) || defined(VISP_HAVE_CMU1394)
g.acquire(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
g >> frame;
#endif
line.track(I);
}

Ellipse tracking

With ViSP you can also track an ellipse using moving edges. The following example code available in tutorial-me-ellipse-tracker.cpp shows how to use ViSP vpMeEllipse class to this end.

#include <visp3/core/vpConfig.h>
#ifdef VISP_HAVE_MODULE_SENSOR
#include <visp3/sensor/vp1394CMUGrabber.h>
#include <visp3/sensor/vp1394TwoGrabber.h>
#include <visp3/sensor/vpV4l2Grabber.h>
#endif
#include <visp3/gui/vpDisplayGDI.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/gui/vpDisplayX.h>
#include <visp3/me/vpMeEllipse.h>
#if defined(HAVE_OPENCV_VIDEOIO)
#include <opencv2/videoio.hpp>
#endif
int main()
{
#if (defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_CMU1394) || defined(VISP_HAVE_V4L2) || defined(HAVE_OPENCV_VIDEOIO))
try {
#if defined(VISP_HAVE_DC1394)
vp1394TwoGrabber g(false);
#elif defined(VISP_HAVE_CMU1394)
#elif defined(VISP_HAVE_V4L2)
#elif defined(HAVE_OPENCV_VIDEOIO)
cv::VideoCapture g(0); // open the default camera
if (!g.isOpened()) { // check if we succeeded
std::cout << "Failed to open the camera" << std::endl;
return EXIT_FAILURE;
}
cv::Mat frame;
#endif
#if defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_V4L2) || defined(VISP_HAVE_CMU1394)
g.acquire(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
g >> frame; // get a new frame from camera
#endif
#if defined(VISP_HAVE_X11)
vpDisplayX d(I, 0, 0, "Camera view");
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI d(I, 0, 0, "Camera view");
#elif defined(HAVE_OPENCV_HIGHGUI)
vpDisplayOpenCV d(I, 0, 0, "Camera view");
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
vpMe me;
me.setRange(25);
me.setThreshold(20);
me.setSampleStep(10);
vpMeEllipse ellipse;
ellipse.setMe(&me);
ellipse.initTracking(I);
while (1) {
#if defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_V4L2) || defined(VISP_HAVE_CMU1394)
g.acquire(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
g >> frame;
#endif
ellipse.track(I);
ellipse.display(I, vpColor::red);
}
}
catch (const vpException &e) {
std::cout << "Catch an exception: " << e << std::endl;
}
#endif
}
Class that tracks an ellipse using moving edges.
Definition: vpMeEllipse.h:90
void display(const vpImage< unsigned char > &I, const vpColor &col, unsigned int thickness=1)
void initTracking(const vpImage< unsigned char > &I, bool trackCircle=false, bool trackArc=false)
void track(const vpImage< unsigned char > &I)

The video shows the result of the tracking:

This example is very similar to the one presented in Line tracking. It differs only in the name of the class and its header that is used to allow ellipse tracking:

vpMeEllipse ellipse;
ellipse.setMe(&me);
ellipse.initTracking(I);

Note here that the user has to initialize the tracker on image I by clicking on five points located on the ellipse to track.

Next tutorial

You are now ready to see the next Tutorial: Markerless generic model-based tracking using a color camera.