Visual Servoing Platform  version 3.6.1 under development (2024-12-13)
Tutorial: Blob tracking

Introduction

With ViSP you can track a blob using either vpDot or vpDot2 classes. By blob we mean a region of the image that has the same gray level. The blob can be white on a black background, or black on a white background.

In this tutorial we focus on vpDot2 class that provides more functionalities than vpDot class. As presented in section Blob auto detection and tracking, it allows especially to automize the detection of blobs that have the same characteristics than a reference blob.

The next videos show the result of ViSP blob tracker on two different objects:

Note that all the material (source code and images) described in this tutorial is part of ViSP source code (in tutorial/tracking/blob folder) and could be found in https://github.com/lagadic/visp/tree/master/tutorial/tracking/blob.

Blob tracking

In the next subsections we explain how to achieve this kind of tracking, first using a firewire live camera, then using a v4l2 live camera that can be an usb camera, or a Raspberry Pi camera module.

From a firewire live camera

The following code also available in tutorial-blob-tracker-live-firewire.cpp file provided in ViSP source code tree allows to grab images from a firewire camera and track a blob. The initialisation is done with a user mouse click on a pixel that belongs to the blob.

To acquire images from a firewire camera we use vp1394TwoGrabber class on unix-like systems or vp1394CMUGrabber class under Windows. These classes are described in the Tutorial: Image frame grabbing.

#include <visp3/core/vpConfig.h>
#ifdef VISP_HAVE_MODULE_SENSOR
#include <visp3/sensor/vp1394CMUGrabber.h>
#include <visp3/sensor/vp1394TwoGrabber.h>
#endif
#include <visp3/blob/vpDot2.h>
#include <visp3/gui/vpDisplayGDI.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/gui/vpDisplayX.h>
#if defined(HAVE_OPENCV_VIDEOIO)
#include <opencv2/videoio.hpp>
#endif
int main()
{
#if (defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_CMU1394) || defined(HAVE_OPENCV_VIDEOIO)) && \
(defined(VISP_HAVE_X11) || defined(VISP_HAVE_GDI) || defined(VISP_HAVE_OPENCV))
#ifdef ENABLE_VISP_NAMESPACE
using namespace VISP_NAMESPACE_NAME;
#endif
vpImage<unsigned char> I; // Create a gray level image container
#if defined(VISP_HAVE_DC1394)
vp1394TwoGrabber g(false);
g.open(I);
#elif defined(VISP_HAVE_CMU1394)
g.open(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
cv::VideoCapture g(0); // open the default camera
if (!g.isOpened()) { // check if we succeeded
std::cout << "Failed to open the camera" << std::endl;
return EXIT_FAILURE;
}
cv::Mat frame;
g >> frame; // get a new frame from camera
#endif
#if defined(VISP_HAVE_X11)
vpDisplayX d(I, 0, 0, "Camera view");
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI d(I, 0, 0, "Camera view");
#elif defined(HAVE_OPENCV_HIGHGUI)
vpDisplayOpenCV d(I, 0, 0, "Camera view");
#endif
vpDot2 blob;
blob.setGraphics(true);
bool init_done = false;
while (1) {
try {
#if defined(VISP_HAVE_DC1394) || defined(VISP_HAVE_CMU1394)
g.acquire(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
g >> frame;
#endif
if (!init_done) {
vpDisplay::displayText(I, vpImagePoint(10, 10), "Click in the blob to initialize the tracker", vpColor::red);
if (vpDisplay::getClick(I, germ, false)) {
blob.initTracking(I, germ);
init_done = true;
}
}
else {
blob.track(I);
}
}
catch (...) {
init_done = false;
}
}
#endif
}
Firewire cameras video capture based on CMU 1394 Digital Camera SDK.
void acquire(vpImage< unsigned char > &I)
void open(vpImage< unsigned char > &I)
Class for firewire ieee1394 video devices using libdc1394-2.x api.
static const vpColor red
Definition: vpColor.h:217
Display for windows using GDI (available on any windows 32 platform).
Definition: vpDisplayGDI.h:130
The vpDisplayOpenCV allows to display image using the OpenCV library. Thus to enable this class OpenC...
static bool getClick(const vpImage< unsigned char > &I, bool blocking=true)
static void display(const vpImage< unsigned char > &I)
static void flush(const vpImage< unsigned char > &I)
static void displayText(const vpImage< unsigned char > &I, const vpImagePoint &ip, const std::string &s, const vpColor &color)
This tracker is meant to track a blob (connex pixels with same gray level) on a vpImage.
Definition: vpDot2.h:125
void track(const vpImage< unsigned char > &I, bool canMakeTheWindowGrow=true)
Definition: vpDot2.cpp:452
void setGraphics(bool activate)
Definition: vpDot2.h:318
void setGraphicsThickness(unsigned int thickness)
Definition: vpDot2.h:326
void initTracking(const vpImage< unsigned char > &I, unsigned int size=0)
Definition: vpDot2.cpp:269
static void convert(const vpImage< unsigned char > &src, vpImage< vpRGBa > &dest)
Class that defines a 2D point in an image. This class is useful for image processing and stores only ...
Definition: vpImagePoint.h:82

From now, we assume that you have successfully followed the Tutorial: How to create and build a project that uses ViSP and CMake on Unix or Windows and the Tutorial: Image frame grabbing. Here after we explain the new lines that are introduced.

vpDot2 blob;

Then we are modifying some default settings to allow drawings in overlay the contours pixels and the position of the center of gravity with a thickness of 2 pixels.

blob.setGraphics(true);

Then we are waiting for a user initialization throw a mouse click event in the blob to track.

blob.initTracking(I, germ);

The tracker is now initialized. The tracking can be performed on new images:

blob.track(I);

From a v4l2 live camera

The following code also available in tutorial-blob-tracker-live-v4l2.cpp file provided in ViSP source code tree allows to grab images from a camera compatible with video for linux two driver (v4l2) and track a blob. Webcams or more generally USB cameras, but also the Raspberry Pi Camera Module can be considered.

To acquire images from a v4l2 camera we use vpV4l2Grabber class on unix-like systems. This class is described in the Tutorial: Image frame grabbing.

#include <visp3/core/vpConfig.h>
#ifdef VISP_HAVE_MODULE_SENSOR
#include <visp3/sensor/vpV4l2Grabber.h>
#endif
#include <visp3/blob/vpDot2.h>
#include <visp3/gui/vpDisplayGDI.h>
#include <visp3/gui/vpDisplayGTK.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/gui/vpDisplayX.h>
#if defined(HAVE_OPENCV_VIDEOIO)
#include <opencv2/videoio.hpp>
#endif
int main(int argc, char **argv)
{
#if ((defined(VISP_HAVE_V4L2) || defined(HAVE_OPENCV_VIDEOIO)) && \
(defined(VISP_HAVE_X11) || defined(VISP_HAVE_GDI) || defined(VISP_HAVE_OPENCV) || defined(VISP_HAVE_GTK)))
#ifdef ENABLE_VISP_NAMESPACE
using namespace VISP_NAMESPACE_NAME;
#endif
int device = 0;
if (argc == 2) {
device = std::atoi(argv[1]);
}
vpImage<unsigned char> I; // Create a gray level image container
std::stringstream ss;
ss << "/dev/video" << device;
std::cout << "Connect to: " << ss.str() << std::endl;
#if defined(VISP_HAVE_V4L2)
g.setDevice(ss.str());
g.open(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
cv::VideoCapture g(device); // open the default camera
if (!g.isOpened()) { // check if we succeeded
std::cout << "Failed to open the camera" << std::endl;
return EXIT_FAILURE;
}
cv::Mat frame;
g >> frame; // get a new frame from camera
#endif
#if defined(VISP_HAVE_X11)
vpDisplayX d(I, 0, 0, "Camera view");
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI d(I, 0, 0, "Camera view");
#elif defined(HAVE_OPENCV_HIGHGUI)
vpDisplayOpenCV d(I, 0, 0, "Camera view");
#elif defined(VISP_HAVE_GTK)
vpDisplayGTK d(I, 0, 0, "Camera view");
#endif
vpDot2 blob;
blob.setGraphics(true);
bool init_done = false;
std::cout << "Click!!!" << std::endl;
while (1) {
try {
#if defined(VISP_HAVE_V4L2)
g.acquire(I);
#elif defined(HAVE_OPENCV_VIDEOIO)
g >> frame;
#endif
if (!init_done) {
vpDisplay::displayText(I, vpImagePoint(10, 10), "Click in the blob to initialize the tracker", vpColor::red);
if (vpDisplay::getClick(I, germ, false)) {
blob.initTracking(I, germ);
init_done = true;
}
}
else {
blob.track(I);
}
}
catch (...) {
init_done = false;
}
}
#else
(void)argc;
(void)argv;
#endif
}
The vpDisplayGTK allows to display image using the GTK 3rd party library. Thus to enable this class G...
Definition: vpDisplayGTK.h:133
Class that is a wrapper over the Video4Linux2 (V4L2) driver.
void open(vpImage< unsigned char > &I)
void setDevice(const std::string &devname)
void acquire(vpImage< unsigned char > &I)

The code is the same than the one presented in the previous subsection, except that here we use the vpV4l2Grabber class to grab images from usb cameras. Here we have also modified the while loop in order to catch an exception when the tracker fail:

try { blob.track(I); }
catch(...) { }

If possible, it allows the tracker to overcome a previous tracking failure (due to blur, blob outside the image,...) on the next available images.

Blob auto detection and tracking

The following example also available in tutorial-blob-auto-tracker.cpp file provided in ViSP source code tree shows how to detect blobs in the first image and then track all the detected blobs. This functionality is only available with vpDot2 class. Here we consider an image that is provided in ViSP source tree.

#include <visp3/core/vpConfig.h>
#include <visp3/blob/vpDot2.h>
#include <visp3/gui/vpDisplayGDI.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/gui/vpDisplayX.h>
#include <visp3/io/vpImageIo.h>
int main()
{
#ifdef ENABLE_VISP_NAMESPACE
using namespace VISP_NAMESPACE_NAME;
#endif
try {
bool learn = false;
vpImage<unsigned char> I; // Create a gray level image container
vpImageIo::read(I, "./target.pgm");
#if defined(VISP_HAVE_X11)
vpDisplayX d(I, 0, 0, "Camera view");
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI d(I, 0, 0, "Camera view");
#elif defined(HAVE_OPENCV_HIGHGUI)
vpDisplayOpenCV d(I, 0, 0, "Camera view");
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
vpDot2 blob;
if (learn) {
// Learn the characteristics of the blob to auto detect
blob.setGraphics(true);
blob.initTracking(I);
blob.track(I);
std::cout << "Blob characteristics: " << std::endl;
std::cout << " width : " << blob.getWidth() << std::endl;
std::cout << " height: " << blob.getHeight() << std::endl;
#if VISP_VERSION_INT > VP_VERSION_INT(2, 7, 0)
std::cout << " area: " << blob.getArea() << std::endl;
#endif
std::cout << " gray level min: " << blob.getGrayLevelMin() << std::endl;
std::cout << " gray level max: " << blob.getGrayLevelMax() << std::endl;
std::cout << " grayLevelPrecision: " << blob.getGrayLevelPrecision() << std::endl;
std::cout << " sizePrecision: " << blob.getSizePrecision() << std::endl;
std::cout << " ellipsoidShapePrecision: " << blob.getEllipsoidShapePrecision() << std::endl;
}
else {
// Set blob characteristics for the auto detection
blob.setWidth(50);
blob.setHeight(50);
#if VISP_VERSION_INT > VP_VERSION_INT(2, 7, 0)
blob.setArea(1700);
#endif
blob.setGrayLevelMin(0);
blob.setGrayLevelMax(30);
blob.setSizePrecision(0.65);
}
std::list<vpDot2> blob_list;
blob.searchDotsInArea(I, 0, 0, I.getWidth(), I.getHeight(), blob_list);
if (learn) {
// The blob that is tracked by initTracking() is not in the list of auto
// detected blobs We add it:
blob_list.push_back(blob);
}
std::cout << "Number of auto detected blob: " << blob_list.size() << std::endl;
std::cout << "A click to exit..." << std::endl;
while (1) {
for (std::list<vpDot2>::iterator it = blob_list.begin(); it != blob_list.end(); ++it) {
(*it).setGraphics(true);
(*it).setGraphicsThickness(3);
(*it).track(I);
}
if (vpDisplay::getClick(I, false))
break;
}
}
catch (const vpException &e) {
std::cout << "Catch an exception: " << e << std::endl;
}
}
unsigned int getGrayLevelMin() const
Definition: vpDot2.h:219
unsigned int getGrayLevelMax() const
Definition: vpDot2.h:225
double getEllipsoidShapePrecision() const
Definition: vpDot2.cpp:651
void searchDotsInArea(const vpImage< unsigned char > &I, int area_u, int area_v, unsigned int area_w, unsigned int area_h, std::list< vpDot2 > &niceDots)
void setGrayLevelMax(const unsigned int &max)
Definition: vpDot2.h:357
double getArea() const
Definition: vpDot2.cpp:628
void setSizePrecision(const double &sizePrecision)
Definition: vpDot2.cpp:756
void setGrayLevelPrecision(const double &grayLevelPrecision)
Definition: vpDot2.cpp:726
void setGrayLevelMin(const unsigned int &min)
Definition: vpDot2.h:338
void setHeight(const double &height)
Definition: vpDot2.cpp:695
double getSizePrecision() const
Definition: vpDot2.cpp:642
double getGrayLevelPrecision() const
Definition: vpDot2.cpp:635
void setWidth(const double &width)
Definition: vpDot2.cpp:683
double getWidth() const
Definition: vpDot2.cpp:614
void setEllipsoidShapePrecision(const double &ellipsoidShapePrecision)
Definition: vpDot2.cpp:801
void setArea(const double &area)
Definition: vpDot2.cpp:707
double getHeight() const
Definition: vpDot2.cpp:621
error that can be emitted by ViSP classes.
Definition: vpException.h:60
static void read(vpImage< unsigned char > &I, const std::string &filename, int backend=IO_DEFAULT_BACKEND)
Definition: vpImageIo.cpp:147
unsigned int getWidth() const
Definition: vpImage.h:242
unsigned int getHeight() const
Definition: vpImage.h:181
VISP_EXPORT int wait(double t0, double t)

Here is a screen shot of the resulting program :

And here is the detailed explanation of the source :

First we create an instance of the tracker.

vpDot2 blob;

Then, two cases are handled. The first case, when learn is set to true, consists in learning the blob characteristics. The user has to click in a blob that serves as reference blob. The size, area, gray level min and max, and some precision parameters will than be used to search similar blobs in the whole image.

if (learn) {
// Learn the characteristics of the blob to auto detect
blob.setGraphics(true);
blob.initTracking(I);
blob.track(I);
std::cout << "Blob characteristics: " << std::endl;
std::cout << " width : " << blob.getWidth() << std::endl;
std::cout << " height: " << blob.getHeight() << std::endl;
#if VISP_VERSION_INT > VP_VERSION_INT(2, 7, 0)
std::cout << " area: " << blob.getArea() << std::endl;
#endif
std::cout << " gray level min: " << blob.getGrayLevelMin() << std::endl;
std::cout << " gray level max: " << blob.getGrayLevelMax() << std::endl;
std::cout << " grayLevelPrecision: " << blob.getGrayLevelPrecision() << std::endl;
std::cout << " sizePrecision: " << blob.getSizePrecision() << std::endl;
std::cout << " ellipsoidShapePrecision: " << blob.getEllipsoidShapePrecision() << std::endl;
}

If you have an precise idea of the dimensions of the blob to search, the second case consists is settings the reference characteristics directly.

else {
// Set blob characteristics for the auto detection
blob.setWidth(50);
blob.setHeight(50);
#if VISP_VERSION_INT > VP_VERSION_INT(2, 7, 0)
blob.setArea(1700);
#endif
blob.setGrayLevelMin(0);
blob.setGrayLevelMax(30);
blob.setSizePrecision(0.65);
}

Once the blob characteristics are known, to search similar blobs in the image is simply done by:

std::list<vpDot2> blob_list;
blob.searchDotsInArea(I, 0, 0, I.getWidth(), I.getHeight(), blob_list);

Here blob_list contains the list of the blobs that are detected in the image I. When learning is enabled, the blob that is tracked is not in the list of auto detected blobs. We add it to the end of the list:

if (learn) {
// The blob that is tracked by initTracking() is not in the list of auto
// detected blobs We add it:
blob_list.push_back(blob);
}

Finally, when a new image is available we do the tracking of all the blobs:

for (std::list<vpDot2>::iterator it = blob_list.begin(); it != blob_list.end(); ++it) {
(*it).setGraphics(true);
(*it).setGraphicsThickness(3);
(*it).track(I);
}

Next tutorial

You are now ready to see the next Tutorial: Keypoint tracking.