Visual Servoing Platform  version 3.3.0 under development (2020-02-17)
Tutorial: Image frame grabbing

Table of Contents

Introduction

In this tutorial you will learn how to grab images with ViSP, either from cameras or from a video stream.

Grabbing images from a real camera is only possible if you have installed the corresponding 3rd party. The complete list of 3rd parties supported by ViSP and dedicated to framegrabbing is given here. From this page you will also found useful information to install these 3rd parties.

All the material (source code and videos) described in this tutorial is part of ViSP source code and could be downloaded using the following command:

$ svn export https://github.com/lagadic/visp.git/trunk/tutorial/grabber

Frame grabbing using FlyCapture SDK

After ViSP 3.0.0, we introduce vpFlyCaptureGrabber class, a wrapper over PointGrey FlyCapture SDK that allows to grab images from any PointGrey camera. This grabber was tested under Ubuntu and Windows with the following cameras:

  • Flea3 USB 3.0 cameras (FL3-U3-32S2M-CS, FL3-U3-13E4C-C)
  • Flea2 firewire camera (FL2-03S2C)
  • Dragonfly2 firewire camera (DR2-COL)

It should also work with GigE PGR cameras.

The following example also available in tutorial-grabber-flycapture.cpp shows how to use vpFlyCaptureGrabber to capture grey level images from a PointGrey camera under Ubuntu or Windows. The following example suppose that a window renderer (libX11 on Ubuntu or GDI on Windows) and FlyCapture SDK 3rd party are available throw VISP.

Source code explained

#include <visp3/core/vpImage.h>
#include <visp3/gui/vpDisplayGDI.h>
#include <visp3/gui/vpDisplayX.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/sensor/vpFlyCaptureGrabber.h>
#include "record_helper.h"
int main(int argc, char **argv)
{
#ifdef VISP_HAVE_FLYCAPTURE
try {
std::string opt_seqname;
int opt_record_mode = 0;
bool opt_change_settings = false;
for (int i = 0; i < argc; i++) {
if (std::string(argv[i]) == "--seqname")
opt_seqname = std::string(argv[i + 1]);
else if (std::string(argv[i]) == "--record")
opt_record_mode = std::atoi(argv[i + 1]);
else if (std::string(argv[i]) == "--change_settings")
opt_change_settings = true;
else if (std::string(argv[i]) == "--help" || std::string(argv[i]) == "-h") {
std::cout << "\nUsage: " << argv[0]
<< " [--seqname <sequence name (default: empty>] [--record <0: continuous | 1: single shot (default: 0)>]"
" [--change_settings] [--help] [-h]\n"
<< "\nExample to visualize images:\n"
<< " " << argv[0] << "\n"
<< "\nExamples to record a sequence:\n"
<< " " << argv[0] << " --seqname I%04d.png \n"
<< " " << argv[0] << " --seqname folder/I%04d.png --record 0\n"
<< "\nExamples to record single shot images:\n"
<< " " << argv[0] << " --seqname I%04d.png --record 1\n"
<< " " << argv[0] << " --seqname folder/I%04d.png --record 1\n"
<< std::endl;
return 0;
}
}
std::cout << "Settings : " << (opt_change_settings ? "modified" : "current") << std::endl;
std::cout << "Recording : " << (opt_seqname.empty() ? "disabled" : "enabled") << std::endl;
std::string text_record_mode = std::string("Record mode: ") + (opt_record_mode ? std::string("single") : std::string("continuous"));
if (! opt_seqname.empty()) {
std::cout << text_record_mode << std::endl;
std::cout << "Record name: " << opt_seqname << std::endl;
}
vpImage<unsigned char> I; // Create a gray level image container
vpFlyCaptureGrabber g; // Create a grabber based on FlyCapture SDK third party lib
if (opt_change_settings) {
try {
g.setShutter(true); // Turn auto shutter on
g.setGain(true); // Turn auto gain on
g.setVideoModeAndFrameRate(FlyCapture2::VIDEOMODE_640x480Y8, FlyCapture2::FRAMERATE_60);
} catch (...) { // If settings are not available just catch execption to
// continue with default settings
std::cout << "Warning: cannot modify camera settings" << std::endl;
}
}
g.open(I);
std::cout << "Image size : " << I.getWidth() << " " << I.getHeight() << std::endl;
#if defined(VISP_HAVE_X11)
vpDisplayX d(I);
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI d(I);
#elif defined(VISP_HAVE_OPENCV)
vpDisplayOpenCV d(I);
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
bool quit = false;
while (! quit) {
double t = vpTime::measureTimeMs();
g.acquire(I);
vpDisplay::display(I);
quit = record_helper(opt_seqname, opt_record_mode, I);
std::stringstream ss;
ss << "Acquisition time: " << std::setprecision(3) << vpTime::measureTimeMs() - t << " ms";
vpDisplay::displayText(I, I.getHeight() - 20, 10, ss.str(), vpColor::red);
vpDisplay::flush(I);
}
} catch (const vpException &e) {
std::cout << "Catch an exception: " << e.getStringMessage() << std::endl;
}
#else
(void) argc;
(void) argv;
std::cout << "Install Flycapture SDK, configure and build ViSP again to use this example" << std::endl;
#endif
}

Here after we explain the source code.

First an instance of the frame grabber is created.

vpFlyCaptureGrabber g; // Create a grabber based on FlyCapture SDK third party lib

Once the grabber is created, we turn auto shutter and auto gain on and set the camera image size, color coding, and framerate. These settings are enclosed in a try/catch to be able to continue if one of these settings are not supported by the camera.

if (opt_change_settings) {
try {
g.setShutter(true); // Turn auto shutter on
g.setGain(true); // Turn auto gain on
g.setVideoModeAndFrameRate(FlyCapture2::VIDEOMODE_640x480Y8, FlyCapture2::FRAMERATE_60);
} catch (...) { // If settings are not available just catch execption to
// continue with default settings
std::cout << "Warning: cannot modify camera settings" << std::endl;
}
}

Then the grabber is initialized using:

g.open(I);

From now the grey level image I is also initialized with the size corresponding to the grabber settings.

Then we enter in a while loop where image acquisition is simply done by:

g.acquire(I);

This image is then displayed using libX11 or GDI renderer:

vpDisplay::display(I);

Depending on the command line options we are recording a sequence of images, or single shot images. We are also waiting for a non blocking mouse event to quit the while loop.

quit = record_helper(opt_seqname, opt_record_mode, I);

How to acquire images

Once tutorial-grabber-flycapture.cpp is built, the binary usage is given by:

./tutorial-grabber-flycapture --help

To just view images, run:

./tutorial-grabber-flycapture

To grab single shot images (for camera calibration for example), you may use:

./tutorial-grabber-flycapture --seqname I%04d.pgm --record 1

To grab a sequence of images, you may rather use:

./tutorial-grabber-flycapture --seqname I%04d.pgm --record 0

Frame grabbing using libdc1394 SDK

The next example also available in tutorial-grabber-1394.cpp shows how to use a framegrabber to acquire color images from a firewire or USB3 camera under Unix. The following example suppose that libX11 and libdc1394-2 3rd party are available.

Source code explained

The source code is the following:

#include <visp3/core/vpImage.h>
#include <visp3/gui/vpDisplayX.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/sensor/vp1394TwoGrabber.h>
#include "record_helper.h"
int main(int argc, char **argv)
{
#ifdef VISP_HAVE_DC1394
try {
std::string opt_seqname;
int opt_record_mode = 0;
bool opt_change_settings = false;
for (int i = 0; i < argc; i++) {
if (std::string(argv[i]) == "--seqname")
opt_seqname = std::string(argv[i + 1]);
else if (std::string(argv[i]) == "--record")
opt_record_mode = std::atoi(argv[i + 1]);
else if (std::string(argv[i]) == "--change_settings")
opt_change_settings = true;
else if (std::string(argv[i]) == "--help" || std::string(argv[i]) == "-h") {
std::cout << "\nUsage: " << argv[0]
<< " [--seqname <sequence name (default: empty>] [--record <0: continuous | 1: single shot (default: 0)>]"
" [--change_settings] [--help] [-h]\n"
<< "\nExample to visualize images:\n"
<< " " << argv[0] << "\n"
<< "\nExamples to record a sequence:\n"
<< " " << argv[0] << " --seqname I%04d.png \n"
<< " " << argv[0] << " --seqname folder/I%04d.png --record 0\n"
<< "\nExamples to record single shot images:\n"
<< " " << argv[0] << " --seqname I%04d.png --record 1\n"
<< " " << argv[0] << " --seqname folder/I%04d.png --record 1\n"
<< std::endl;
return 0;
}
}
std::cout << "Settings : " << (opt_change_settings ? "modified" : "current") << std::endl;
std::cout << "Recording : " << (opt_seqname.empty() ? "disabled" : "enabled") << std::endl;
std::string text_record_mode = std::string("Record mode: ") + (opt_record_mode ? std::string("single") : std::string("continuous"));
if (! opt_seqname.empty()) {
std::cout << text_record_mode << std::endl;
std::cout << "Record name: " << opt_seqname << std::endl;
}
vpImage<unsigned char> I; // Create a gray level image container
bool reset = false; // Disable bus reset during construction (default)
vp1394TwoGrabber g(reset); // Create a grabber based on libdc1394-2.x third party lib
if (opt_change_settings) {
try {
g.setVideoMode(vp1394TwoGrabber::vpVIDEO_MODE_640x480_MONO8);
g.setFramerate(vp1394TwoGrabber::vpFRAMERATE_60);
}
catch(...) { // If settings are not available just catch execption to
// continue with default settings
std::cout << "Warning: cannot modify camera settings" << std::endl;
}
}
g.open(I);
std::cout << "Image size : " << I.getWidth() << " " << I.getHeight() << std::endl;
#ifdef VISP_HAVE_X11
vpDisplayX d(I);
#elif defined(VISP_HAVE_OPENCV)
vpDisplayOpenCV d(I);
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
bool quit = false;
while (! quit) {
double t = vpTime::measureTimeMs();
g.acquire(I);
vpDisplay::display(I);
quit = record_helper(opt_seqname, opt_record_mode, I);
std::stringstream ss;
ss << "Acquisition time: " << std::setprecision(3) << vpTime::measureTimeMs() - t << " ms";
vpDisplay::displayText(I, I.getHeight() - 20, 10, ss.str(), vpColor::red);
vpDisplay::flush(I);
}
} catch (const vpException &e) {
std::cout << "Catch an exception: " << e << std::endl;
}
#else
(void) argc;
(void) argv;
std::cout << "Install libdc1394, configure and build ViSP again to use this example" << std::endl;
#endif
}

Here after we explain the new lines that are introduced.

First an instance of the frame grabber is created. During the creating a bus reset is send. If you don't want to reset the firewire bus, just turn reset to false.

vp1394TwoGrabber g(reset); // Create a grabber based on libdc1394-2.x third party lib

Once the grabber is created, we set the camera image size, color coding, and framerate.

if (opt_change_settings) {
try {
g.setVideoMode(vp1394TwoGrabber::vpVIDEO_MODE_640x480_MONO8);
g.setFramerate(vp1394TwoGrabber::vpFRAMERATE_60);
}
catch(...) { // If settings are not available just catch execption to
// continue with default settings
std::cout << "Warning: cannot modify camera settings" << std::endl;
}
}

Note that here you can specify some other settings such as the firewire transmission speed. For a more complete list of settings see vp1394TwoGrabber class.

g.setIsoTransmissionSpeed(vp1394TwoGrabber::vpISO_SPEED_800);

Then the grabber is initialized using:

g.open(I);

From now the color image I is also initialized with the size corresponding to the grabber settings.

Then we enter in a while loop where image acquisition is simply done by:

g.acquire(I);

As in the previous example, depending on the command line options we are recording a sequence of images, or single shot images. We are also waiting for a non blocking mouse event to quit the while loop.

quit = record_helper(opt_seqname, opt_record_mode, I);

In the previous example we use vp1394TwoGrabber class that works for firewire cameras under Unix. If you are under Windows, you may use vp1394CMUGrabber class. A similar example is provided in tutorial-grabber-CMU1394.cpp.

How to acquire images

Once tutorial-grabber-1394.cpp is built, the binary usage is given by:

./tutorial-grabber-1394 --help

To just view images, run:

./tutorial-grabber-1394

To grab single shot images (for camera calibration for example), you may use:

./tutorial-grabber-1394 --seqname I%04d.pgm --record 1

To grab a sequence of images, you may rather use:

./tutorial-grabber-1394 --seqname I%04d.pgm --record 0

Frame grabbing using libv4l2 SDK

If you want to grab images from an usb camera under Unix, you may use vpV4l2Grabber class that is a wrapper over Video For Linux SDK. To this end libv4l should be installed. An example is provided in tutorial-grabber-v4l2.cpp.

How to acquire images

Once tutorial-grabber-v4l2.cpp is built, the binary usage is given by:

./tutorial-grabber-v4l2 --help

To just view images, run:

./tutorial-grabber-v4l2

To grab single shot images (for camera calibration for example), you may use:

./tutorial-grabber-v4l2 --seqname I%04d.pgm --record 1

To grab a sequence of images, you may rather use:

./tutorial-grabber-v4l2 --seqname I%04d.pgm --record 0

Frame grabbing using Pylon SDK

It is also possible to grab images using Pylon, the SDK for Basler cameras. You may find an example in tutorial-grabber-basler-pylon.cpp.

How to acquire images

Once tutorial-grabber-basler-pylon.cpp is built, the binary usage is given by:

./tutorial-grabber-basler-pylon --help

To just view images, run:

./tutorial-grabber-basler-pylon

To grab single shot images (for camera calibration for example), you may use:

./tutorial-grabber-basler-pylon --seqname I%04d.pgm --record 1

To grab a sequence of images, you may rather use:

./tutorial-grabber-basler-pylon --seqname I%04d.pgm --record 0

Frame grabbing using Realsense SDK

It is also possible to grab images using librealsense, the SDK provided for Intel Realsense RDB-D cameras. You may find an example in tutorial-grabber-realsense.cpp.

How to acquire images

Once tutorial-grabber-realsense.cpp is built, the binary usage is given by:

./tutorial-grabber-realsense --help

To just view images, run:

./tutorial-grabber-realsense

To grab single shot images (for camera calibration for example), you may use:

./tutorial-grabber-realsense --seqname I%04d.pgm --record 1

To grab a sequence of images, you may rather use:

./tutorial-grabber-realsense --seqname I%04d.pgm --record 0

Frame grabbing using OpenCV

It is also possible to grab images using OpenCV. You may find an example in tutorial-grabber-opencv.cpp.

How to acquire images

Once tutorial-grabber-opencv.cpp is built, the binary usage is given by:

./tutorial-grabber-opencv --help

To just view images, run:

./tutorial-grabber-opencv

To grab single shot images (for camera calibration for example), you may use:

./tutorial-grabber-opencv --seqname I%04d.pgm --record 1

To grab a sequence of images, you may rather use:

./tutorial-grabber-opencv --seqname I%04d.pgm --record 0

Frame grabbing using CMU1394 SDK

It is also possible to grab images using CMU1394 SDK if you want to use a firewire camera under Windows. You may find an example in tutorial-grabber-CMU1394.cpp.

Frame grabbing using Parrot Bebop 2 drone

It is also possible to grab images using a Parrot Bebop 2 drone. You may find an example in tutorial-grabber-bebop2.cpp.

Warning
Image grabbing with Bebop 2 drone only works if ViSP was built with Parrot ARSDK3 and OpenCV support.

How to acquire images

Once tutorial-grabber-bebop2.cpp is built, the binary usage is given by:

./tutorial-grabber-bebop2 --help

To just view images, run:

./tutorial-grabber-bebop2

To grab single shot images (for camera calibration for example), you may use:

./tutorial-grabber-bebop2 --seqname I%04d.pgm --record 1

To grab a sequence of images, you may rather use:

./tutorial-grabber-bebop2 --seqname I%04d.pgm --record 0

You can chose to record HD 720p pictures from the drone (instead of default 480p) using –hd_resolution option :

./tutorial-grabber-bebop2 --seqname I%04d.pgm --record 0 --hd_resolution

Images from a video stream

With ViSP it also possible to get images from an input video stream. Supported formats are *.avi, *.mp4, *.mov, *.ogv, *.flv and many others... To this end we exploit OpenCV 3rd party.

The example below available in tutorial-video-reader.cpp shows how to consider an mpeg video stream.

Warning
We recall that this example only works if ViSP was built with OpenCV support.

Source code explained

The source code is the following:

#include <visp3/gui/vpDisplayGDI.h>
#include <visp3/gui/vpDisplayOpenCV.h>
#include <visp3/gui/vpDisplayX.h>
#include <visp3/core/vpTime.h>
#include <visp3/io/vpVideoReader.h>
int main(int argc, char **argv)
{
#if (VISP_HAVE_OPENCV_VERSION >= 0x020100)
try {
std::string videoname = "video.mpg";
for (int i = 0; i < argc; i++) {
if (std::string(argv[i]) == "--name")
videoname = std::string(argv[i + 1]);
else if (std::string(argv[i]) == "--help" || std::string(argv[i]) == "-h") {
std::cout << "\nUsage: " << argv[0] << " [--name <video name> (default: " << videoname << ")] [--help] [-h]\n" << std::endl;
return 0;
}
}
vpImage<unsigned char> I;
vpVideoReader g;
g.setFileName(videoname);
g.open(I);
std::cout << "Video name : " << videoname << std::endl;
std::cout << "Video framerate: " << g.getFramerate() << "Hz" << std::endl;
std::cout << "Video dimension: " << I.getWidth() << " " << I.getHeight() << std::endl;
#ifdef VISP_HAVE_X11
vpDisplayX d(I);
#elif defined(VISP_HAVE_GDI)
vpDisplayGDI d(I);
#elif defined(VISP_HAVE_OPENCV)
vpDisplayOpenCV d(I);
#else
std::cout << "No image viewer is available..." << std::endl;
#endif
vpDisplay::setTitle(I, "Video reader");
unsigned cpt = 1;
while (!g.end()) {
double t = vpTime::measureTimeMs();
g.acquire(I);
vpDisplay::display(I);
vpDisplay::displayText(I, 20, 20, "Click to quit", vpColor::red);
std::stringstream ss;
ss << "Frame: " << cpt ++;
vpDisplay::displayText(I, 40, 20, ss.str(), vpColor::red);
vpDisplay::flush(I);
if (vpDisplay::getClick(I, false))
break;
vpTime::wait(t, 1000. / g.getFramerate());
}
std::cout << "End of video was reached" << std::endl;
} catch (const vpException &e) {
std::cout << e.getMessage() << std::endl;
}
#else
(void)argc;
(void)argv;
std::cout << "Install OpenCV and rebuild ViSP to use this example." << std::endl;
#endif
}

We explain now the new lines that were introduced.

#include <visp3/core/vpTime.h>
#include <visp3/io/vpVideoReader.h>

Include the header of the vpTime class that allows to measure time, and of the vpVideoReader class that allows to read a video stream.

vpVideoReader g;

Create an instance of a video reader.

g.setFileName(videoname);

Set the name of the video stream. Here videoname corresponds to a video file name location. For example we provide the file video.mpg located in the same folder than the executable.

The vpVideoReader class can also handle a sequence of images. For example, to read the following images:

% ls *.png
image0000.png
image0001.png
image0002.png
image0003.png
image0004.png
...

you may use the following:

g.setFileName("./image%04d.png");

where you specify that each image number is coded with 4 digits. Here, we will use libpng or OpenCV to read PNG images. Supported image formats are PPM, PGM, PNG and JPEG.

Then as for any other grabber, you have to initialize the frame grabber using:

g.open(I);

Then we enter in the while loop until the last image was not reached:

while (!g.end()) {

To get the next image in the stream, we just use:

g.acquire(I);

To synchronize the video decoding with the video framerate, we measure the beginning time of each loop iteration:

double t = vpTime::measureTimeMs();

The synchronization is done by waiting from the beginning of the iteration the corresponding time expressed in milliseconds by using:

vpTime::wait(t, 1000. / g.getFramerate());

How to acquire images

Once tutorial-video-reader.cpp is built, the binary usage is given by:

./tutorial-video-reader --help

To read the default video video.mpg, run:

./tutorial-video-reader

To read an other video, let say my-video.mpg, you may use:

./tutorial-grabber-realsense --name my-video.mpg

Next tutorial

You are now ready to see how to continue with: