Tutorial: Keypoint matching (deprecated)

Table of Contents

• Introduction
• SURF key points detection and matching

Introduction

Note

This tutorial is deprecated if your OpenCV version is equal to 3.0.0 or more recent. If so you should rather follow Tutorial: Keypoints matching.

This tutorial focuses on SURF key points manipulation. You will learn how to detect SURF key points on a reference image considered here as the first image of an mpeg video. Then in the next images of the video, key points that match those detected in the reference image using SURF descriptor are displayed.

Note

We assume that you are familiar with video framegrabbing described in Tutorial: Image frame grabbing and with the way to display an image in a window described in Tutorial: How to create and build a CMake project that uses ViSP on Unix or Windows.

SURF key points detection and matching

Let us consider the following source code also available in tutorial-matching-surf-deprecated.cpp.

#include <visp/vpDisplayOpenCV.h>
#include <visp/vpVideoReader.h>
#include <visp/vpKeyPointSurf.h>

int main()
{  
#if defined(VISP_HAVE_OPENCV_NONFREE) && (VISP_HAVE_OPENCV_VERSION < 0x030000)
  vpImage<unsigned char> I;

  vpVideoReader reader;
  reader.setFileName("video-postcard.mpeg");
  reader.acquire(I);

  vpKeyPointSurf surf;
  surf.buildReference(I);

  vpImage<unsigned char> Idisp;
  Idisp.resize(I.getHeight(), 2*I.getWidth());
  Idisp.insert(I, vpImagePoint(0, 0));
  Idisp.insert(I, vpImagePoint(0, I.getWidth()));
  vpDisplayOpenCV d(Idisp, 0, 0, "Matching surf keypoints") ;
  vpDisplay::display(Idisp);
  vpDisplay::flush(Idisp);

  while ( ! reader.end() )
  {
    reader.acquire(I);
    Idisp.insert(I, vpImagePoint(0, I.getWidth()));

    vpDisplay::display(Idisp);
    vpDisplay::displayLine(Idisp, vpImagePoint(0, I.getWidth()), vpImagePoint(I.getHeight(), I.getWidth()), vpColor::white, 2);

    int nbMatch = surf.matchPoint(I);

    vpImagePoint iPref, iPcur;
    for (int i = 0; i < nbMatch; i++)
    {
      surf.getMatchedPoints(i, iPref, iPcur);
      vpDisplay::displayLine(Idisp, iPref, iPcur + vpImagePoint(0, I.getWidth()), vpColor::green);
    }
    vpDisplay::flush(Idisp);

    if (vpDisplay::getClick(Idisp, false))
      break;
  }

  vpDisplay::getClick(Idisp);
#endif

  return 0;
}

Here after is the resulting video. The left image represents the reference image. The right images correspond to the successive images of the input video. All the green lines extremities represent the points that are matched.

Now, let us explain the lines dedicated to the SURF keypoint usage.

First we have to include the header of the vpKeyPointSurf class that is a wrapper over OpenCV classes.

#include <visp/vpKeyPointSurf.h>

Note that this class is only available if ViSP was build with OpenCV non free module. This is ensured by the check of VISP_HAVE_OPENCV_NONFREE macro. To grab the images from the mpeg video stream we need also that ViSP was build with ffmpeg 3rd party. That's why we check VISP_HAVE_FFMPEG macro definition:

#if defined(VISP_HAVE_OPENCV_NONFREE) && (VISP_HAVE_OPENCV_VERSION < 0x030000)

Then we open the mpeg video stream and grab the first image of the video that is stored in I container. A Surf keypoint class is instantiated and keypoints are detected on the first image which is considered as the reference image:

  vpKeyPointSurf surf;
  surf.buildReference(I);

The next lines are used to create image Idisp to render the matching results; left image for the reference image, right image for the current image that is processed:

  vpImage<unsigned char> Idisp;
  Idisp.resize(I.getHeight(), 2*I.getWidth());
  Idisp.insert(I, vpImagePoint(0, 0));
  Idisp.insert(I, vpImagePoint(0, I.getWidth()));

Then a display using OpenCV is created and image Idisp is rendered:

  vpDisplayOpenCV d(Idisp, 0, 0, "Matching surf keypoints") ;
  vpDisplay::display(Idisp);
  vpDisplay::flush(Idisp);

We enter then in the while() loop where a new image is acquired from the video stream and inserted in the right part of image Idisp dedicated to rendering of the matching results.

    reader.acquire(I);
    Idisp.insert(I, vpImagePoint(0, I.getWidth()));

We start the rendering by displaying the rendered image and by drawing a white vertical line to separate the reference image from the current one:

    vpDisplay::display(Idisp);
    vpDisplay::displayLine(Idisp, vpImagePoint(0, I.getWidth()), vpImagePoint(I.getHeight(), I.getWidth()), vpColor::white, 2);

Keypoint matches between the reference image and the current image I are detected using:

    int nbMatch = surf.matchPoint(I);

Then we parse all the matches to retrieve the coordinates of the points in the reference image (in iPref variable) and in the current image (in iPcur variable):

    vpImagePoint iPref, iPcur;
    for (int i = 0; i < nbMatch; i++)
    {
      surf.getMatchedPoints(i, iPref, iPcur);

Next we draw green lines between the matched points:

      vpDisplay::displayLine(Idisp, iPref, iPcur + vpImagePoint(0, I.getWidth()), vpColor::green);

At the end of the iteration, we flush all the previous display to the render window:

    vpDisplay::flush(Idisp);

You can now follow Tutorial: Homography estimation from points (deprecated) to see how to exploit couple of matched points in order to estimate an homography that allows to track the position of an object.