Visual Servoing Platform  version 3.5.0 under development (2022-02-15)
Tutorial: Bridge over OpenCV

Table of Contents

ViSP is interfaced with OpenCV third party. In this tutorial we explain how to convert data such as camera parameters or images from ViSP to OpenCV or vice versa.

Camera parameters conversions

ViSP camera parameters are implemented in vpCameraParameters class. If you want to calibrate a camera with ViSP tools follow Tutorial: Camera intrinsic calibration.

Let us recall the pinhole camera model implemented in ViSP. In this model, a scene view is formed by projecting 3D points into the image plane using a perspective transformation.

\[ \left[ \begin{array}{c} u \\ v \\ 1 \end{array}\right] = \left[ \begin{array}{ccc} p_x & 0 & u_0 \\ 0 & p_y & v_0 \\ 0 & 0 & 1 \end{array}\right] \left[ \begin{array}{c} X_c \\ Y_c \\ Z_c \end{array}\right] \]

where:

  • $(X_c,Y_c,Z_c)$ are the coordinates of a 3D point in the camera frame
  • $(u,v)$ are the coordinates in pixels of the projected 3D point
  • $(u_0,v_0)$ is a principal point that is usually near the image center
  • $(p_x,p_y)$ are the focal lengths expressed in pixel units.

When $Z_c \neq 0$, the previous equation is equivalent to the following:

\[ \begin{array}{lcl} x &=& X_c / Z_c \\ y &=& Y_c / Z_c \\ u &=& u_0 + x \; p_x \\ v &=& v_0 + y \; p_y \end{array} \]

Real lenses usually have some radial distortion. So, the above model is extended as:

\[ \begin{array}{lcl} x &=& X_c / Z_c \\ y &=& Y_c / Z_c \\ x^{'} &=& x (1 + k_{ud} r^2) \\ y^{'} &=& y (1 + k_{ud} r^2) \\ r^2 &=& x^2 + y^2 \\ u &=& u_0 + x^{'} \; p_x \\ v &=& v_0 + y^{'} \; p_y \end{array} \]

where $k_{ud}$ is the first order radial distortion. Higher order distortion coefficients are not considered in ViSP.

Note
In ViSP we introduce an extra parameter named $k_{du}$ which is the radial first order distortion that allows to transform pixels in meters. If this parameter is unknown as in OpenCV a good approximation is to consider $k_{du} = - k_{ud}$

Even if OpenCV notations are different, this model is exactly the same then the one used in OpenCV and described here where higher order OpenCV distortion parameters are turned to 0.

The following table gives the correspondences between ViSP and OpenCV parameters:

\[ \begin{array}{l|l} ViSP & OpenCV \\ \hline u_0 & c_x\\ v_0 & c_y \\ p_x & f_x \\ p_y & f_y \\ k_{ud} & k_1 \\ k_{du} & -k_1 \\ & k_2 = 0 \\ & k_3 = 0 \\ & k_4 = 0 \\ & k_5 = 0 \\ & k_6 = 0 \\ & p_1 = 0 \\ & p_2 = 0 \end{array} \]

From a coding point of view, let us consider the following code also available in tutorial-bridge-opencv-camera-param.cpp where we initialize camera parameters using ViSP:

double u0 = 326.6;
double v0 = 215.0;
double px = 582.7;
double py = 580.6;
double kud = -0.3372;
double kdu = 0.4021;
vpCameraParameters cam(px, py, u0, v0, kud, kdu);

These parameters could be used to initialize OpenCV camera parameters:

cv::Mat K = (cv::Mat_<double>(3, 3) << cam.get_px(), 0, cam.get_u0(), 0, cam.get_py(), cam.get_v0(), 0, 0, 1);
cv::Mat D = (cv::Mat_<double>(4, 1) << cam.get_kud(), 0, 0, 0);

Image conversions

ViSP image is implemented in vpImage class, while OpenCV images in cv::Mat class. All the functions that allow image conversion from ViSP to OpenCV or vice versa are implemented in vpImageConvert.

Some image conversion examples could be found in tutorial-bridge-opencv-camera-param.cpp. There is also tutorial-bridge-opencv-image.cpp from where the following snippets are extracted.

From ViSP to OpenCV image conversion

  • Color image conversion

    The following code allows to read a color image with ViSP:

    vpImage<vpRGBa> Irgba;
    vpImageIo::read(Irgba, "monkey.jpeg");

    and then convert the image in OpenCV using:

    cv::Mat cv_img_color;
    vpImageConvert::convert(Irgba, cv_img_color);

    before saving the converted color image

    cv::imwrite("monkey-cv-color.jpeg", cv_img_color);

  • Grey scale image conversion

    The following code allows to read a grey scale image with ViSP. Note here that if the input image (ie. monkey.jpeg) is a color image, vpImageIo::read() converts implicitly the color image in a grey scale image:

    vpImage<unsigned char> Igrey;
    vpImageIo::read(Igrey, "monkey.jpeg");

    and then convert the image in OpenCV using:

    cv::Mat cv_img_grey;
    vpImageConvert::convert(Igrey, cv_img_grey);

    before saving the converted grey image

    cv::imwrite("monkey-cv-grey.jpeg", cv_img_grey);

From OpenCV to ViSP image conversion

  • Color image conversion

    The following code allows to read a color image with OpenCV:

    #if VISP_HAVE_OPENCV_VERSION >= 0x030000
    cv::Mat cv_img_color = cv::imread("monkey.jpeg", cv::IMREAD_COLOR);
    #else
    cv::Mat cv_img_color = cv::imread("monkey.jpeg", CV_LOAD_IMAGE_COLOR);
    #endif

    and then convert the image in ViSP using:

    vpImage<vpRGBa> Irgba;
    vpImageConvert::convert(cv_img_color, Irgba);

    before saving the converted color image

    vpImageIo::write(Irgba, "monkey-vp-color.jpeg");

  • Grey scale image conversion

    The following code allows to read a grey scale image with OpenCV:

    #if VISP_HAVE_OPENCV_VERSION >= 0x030000
    cv::Mat cv_img_grey = cv::imread("monkey.jpeg", cv::IMREAD_GRAYSCALE);
    #else
    cv::Mat cv_img_grey = cv::imread("monkey.jpeg", CV_LOAD_IMAGE_GRAYSCALE);
    #endif

    and then convert the image in ViSP using:

    vpImage<unsigned char> Igrey;
    vpImageConvert::convert(cv_img_grey, Igrey);

    before saving the converted grey image

    vpImageIo::write(Igrey, "monkey-vp-grey.jpeg");

Matrix conversions

ViSP matrices are implemented in different classes:

while OpenCV matrices are always implemented as a cv::Mat.

Up to now, there is no specific function that does the conversion between ViSP and OpenCv matrices.

Note that each element of a ViSP matrix is a double and that ViSP matrices are row-major.

In the following subsections we illustrate how to convert a vpMatrix to a cv::Mat, but the code could be easily adapted to other ViSP specific matrices as long as the corresponding OpenCV matrix contains double elements and that it has the same size as the ViSP specific matrix.

From OpenCV to ViSP matrix conversion

  • The following code also available in tutorial-bridge-opencv-matrix.cpp shows how to create an OpenCV row-major matrix with size 3 rows by 4 cols that contains double elements.
    cv::Mat M_cv = (cv::Mat_<double>(3, 4) << 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12);
    std::cout << "M_cv: \n" << M_cv << std::endl;

  • Deep copy conversion

    To convert the previous matrix in a vpMatrix using a deep copy, you may use:

    vpMatrix M(static_cast<unsigned int>(M_cv.rows), static_cast<unsigned int>(M_cv.cols));
    memcpy(M.data, M_cv.data, sizeof(double) * static_cast<size_t>(M_cv.rows * M_cv.cols));
    std::cout << "M: \n" << M << std::endl;

    The content of the matrices is now the following: 

    M_cv: 
[1, 2, 3, 4;
 5, 6, 7, 8;
 9, 10, 11, 12]
M: 
1  2  3  4
5  6  7  8
9  10  11  12

From ViSP to OpenCV conversion

  • The following code also available in tutorial-bridge-opencv-matrix.cpp shows how to create a ViSP row-major matrix with size 3 rows by 4 cols that contains double elements.
    vpMatrix M(3, 4, {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12} );
    std::cout << "M: \n" << M << std::endl;

  • Deep copy conversion

    To convert the previous matrix in a cv::Mat using a deep copy, you may use:

    cv::Mat tmp(static_cast<int>(M.getRows()), static_cast<int>(M.getCols()), CV_64F, static_cast<void *>(M.data));
    cv::Mat M_cv_deep = tmp.clone();
    std::cout << "M_cv_deep: \n" << M_cv_deep << std::endl;

    The content of the matrices is now the following: 

    M: 
1  2  3  4
5  6  7  8
9  10  11  12
M_cv_deep: 
[1, 2, 3, 4;
 5, 6, 7, 8;
 9, 10, 11, 12]

  • Pointer assignment, without deep copy conversion

    For performance issues, there is also the possibility to just copy the data pointer. To this end, you may use:

    cv::Mat M_cv(static_cast<int>(M.getRows()), static_cast<int>(M.getCols()), CV_64F, static_cast<void *>(M.data));
    std::cout << "M_cv: \n" << M_cv << std::endl;

    The content of the matrices is now the following: 

    M: 
1  2  3  4
5  6  7  8
9  10  11  12
M_cv: 
[1, 2, 3, 4;
 5, 6, 7, 8;
 9, 10, 11, 12]

    As a side effect, you may know that modifying one of the matrices (ie. M like below or M_cv) will affect the content of both matrices. Let us illustrate this behavior, where the diagonal of M is set to 1:

    std::cout << "Set M = eye" << std::endl;
    M.eye();
    std::cout << "M: \n" << M << std::endl;
    std::cout << "M_cv: \n" << M_cv << std::endl;

    As expected, the content of M_cv matrix is also changed: 

    Set M = eye
M: 
1  0  0  0
0  1  0  0
0  0  1  0
M_cv: 
[1, 0, 0, 0;
 0, 1, 0, 0;
 0, 0, 1, 0]