Visual Servoing Platform  version 3.6.1 under development (2024-05-27)
Tutorial: Visual servo simulation on a pioneer-like unicycle robot

This tutorial focuses on visual servoing simulation on a unicycle robot. The study case is a Pioneer P3-DX mobile robot equipped with a camera.

We suppose here that you have at least followed the Tutorial: Image-based visual servo that may help to understand this tutorial.

Note that all the material (source code) described in this tutorial is part of ViSP source code (in tutorial/robot/pioneer folder) and could be found in

Unicycle with a fixed camera

In this section we consider the following unicycle:

This robot has 2 dof: $(v_x, w_z)$, the translational and rotational velocities that are applied at point E, considered as the end-effector. A camera is rigidly attached to the robot at point C. The homogeneous transformation between C and E is given by cMe. This transformation is constant.

The robot position evolves with respect to a world frame; wMe. When a new joint velocity is applied to the robot using setVelocity(), the position of the camera wrt the world frame is also updated; wMc.

To control the robot by visual servoing we need to introduce two visual features. If we consider a 3D point at position O as the target, to position the robot relative to the target we can consider the coordinate $x$ of the point in the image plane and $log(Z/Z^*)$, with $Z$ the distance of point in the camera frame, as visual features. The first feature implemented in vpFeaturePoint allows to control $w_z$, while the second one implemented in vpFeatureDepth $v_x$. The position of the target in the world frame is given by wMo transformation. Thus the current visual feature ${\bf s} = (x, log(Z/Z^*))^\top$ and the desired feature ${\bf s}^* = (0, 0)^\top$.

The code that does the simulation is provided in tutorial-simu-pioneer.cpp and given hereafter.

#include <iostream>
#include <visp3/core/vpHomogeneousMatrix.h>
#include <visp3/core/vpVelocityTwistMatrix.h>
#include <visp3/gui/vpPlot.h>
#include <visp3/robot/vpSimulatorPioneer.h>
#include <visp3/visual_features/vpFeatureBuilder.h>
#include <visp3/visual_features/vpFeatureDepth.h>
#include <visp3/visual_features/vpFeaturePoint.h>
#include <visp3/vs/vpServo.h>
int main()
try {
cdMo[1][3] = 1.2;
cdMo[2][3] = 0.5;
cMo[0][3] = 0.3;
cMo[1][3] = cdMo[1][3];
cMo[2][3] = 1.;
vpRotationMatrix cRo(0, atan2(cMo[0][3], cMo[1][3]), 0);
wMo = wMc * cMo;
vpPoint point(0, 0, 0);
vpServo task;
cVe = robot.get_cVe();
vpMatrix eJe;
vpFeaturePoint s_x, s_xd;, 0, cdMo[2][3]);
task.addFeature(s_x, s_xd, vpFeaturePoint::selectX());
vpFeatureDepth s_Z, s_Zd;
double Z = point.get_Z();
double Zd = cdMo[2][3];, s_x.get_y(), Z, log(Z / Zd));, 0, Zd, 0);
task.addFeature(s_Z, s_Zd);
// Create a window (800 by 500) at position (400, 10) with 3 graphics
vpPlot graph(3, 800, 500, 400, 10, "Curves...");
// Init the curve plotter
graph.initGraph(0, 2);
graph.initGraph(1, 2);
graph.initGraph(2, 1);
graph.setTitle(0, "Velocities");
graph.setTitle(1, "Error s-s*");
graph.setTitle(2, "Depth");
graph.setLegend(0, 0, "vx");
graph.setLegend(0, 1, "wz");
graph.setLegend(1, 0, "x");
graph.setLegend(1, 1, "log(Z/Z*)");
graph.setLegend(2, 0, "Z");
int iter = 0;
for (;;) {
cMo = wMc.inverse() * wMo;
Z = point.get_Z();, s_x.get_y(), Z, log(Z / Zd));
graph.plot(0, iter, v); // plot velocities applied to the robot
graph.plot(1, iter, task.getError()); // plot error vector
graph.plot(2, 0, iter, Z); // plot the depth
if (task.getError().sumSquare() < 0.0001) {
std::cout << "Reached a small error. We stop the loop... " << std::endl;
graph.saveData(0, "./v2.dat");
graph.saveData(1, "./error2.dat");
const char *legend = "Click to quit...";
vpDisplay::displayText(graph.I, (int)graph.I.getHeight() - 60, (int)graph.I.getWidth() - 150, legend, vpColor::red);
// Kill the servo task
} catch (const vpException &e) {
std::cout << "Catch an exception: " << e << std::endl;
Implementation of column vector and the associated operations.
Definition: vpColVector.h:163
double sumSquare() const
static const vpColor red
Definition: vpColor.h:211
static bool getClick(const vpImage< unsigned char > &I, bool blocking=true)
static void flush(const vpImage< unsigned char > &I)
static void displayText(const vpImage< unsigned char > &I, const vpImagePoint &ip, const std::string &s, const vpColor &color)
error that can be emitted by ViSP classes.
Definition: vpException.h:59
static void create(vpFeaturePoint &s, const vpCameraParameters &cam, const vpDot &d)
Class that defines a 3D point visual feature which is composed by one parameters that is that defin...
vpFeatureDepth & build(const double &x, const double &y, const double &Z, const double &LogZoverZstar)
Class that defines a 2D point visual feature which is composed by two parameters that are the cartes...
static unsigned int selectX()
double get_y() const
double get_x() const
vpFeaturePoint & build(const double &x, const double &y, const double &Z)
Implementation of an homogeneous matrix and operations on such kind of matrices.
vpHomogeneousMatrix inverse() const
void insert(const vpRotationMatrix &R)
Implementation of a matrix and operations on matrices.
Definition: vpMatrix.h:146
This class enables real time drawing of 2D or 3D graphics. An instance of the class open a window whi...
Definition: vpPlot.h:109
Class that defines a 3D point in the object frame and allows forward projection of a 3D point in the ...
Definition: vpPoint.h:77
void get_eJe(vpMatrix &eJe) vp_override
void setVelocity(const vpRobot::vpControlFrameType frame, const vpColVector &vel) vp_override
Definition: vpRobot.h:78
Implementation of a rotation matrix and operations on such kind of matrices.
void setInteractionMatrixType(const vpServoIteractionMatrixType &interactionMatrixType, const vpServoInversionType &interactionMatrixInversion=PSEUDO_INVERSE)
Definition: vpServo.cpp:378
Definition: vpServo.h:162
void addFeature(vpBasicFeature &s_cur, vpBasicFeature &s_star, unsigned int select=vpBasicFeature::FEATURE_ALL)
Definition: vpServo.cpp:329
void set_cVe(const vpVelocityTwistMatrix &cVe_)
Definition: vpServo.h:1028
void print(const vpServo::vpServoPrintType display_level=ALL, std::ostream &os=std::cout)
Definition: vpServo.cpp:169
void setLambda(double c)
Definition: vpServo.h:976
void set_eJe(const vpMatrix &eJe_)
Definition: vpServo.h:1091
void setServo(const vpServoType &servo_type)
Definition: vpServo.cpp:132
vpColVector getError() const
Definition: vpServo.h:504
Definition: vpServo.h:229
vpColVector computeControlLaw()
Definition: vpServo.cpp:703
Definition: vpServo.h:202
Class that defines the Pioneer mobile robot simulator equipped with a static camera.
vpVelocityTwistMatrix get_cVe() const
Definition: vpUnicycle.h:70

We provide now a line by line explanation of the code.

Firstly we define cdMo the desired position the camera has to reach wrt the target. $t_y=1.2$ should be different from zero to be non singular. The camera has to keep a distance of 0.5 meter from the target.

cdMo[1][3] = 1.2; // ty
cdMo[2][3] = 0.5; // tz

Secondly we specify cMo the initial position of the camera wrt the target.

cMo[0][3] = 0.3; // tx
cMo[1][3] = cdMo[1][3]; // ty
cMo[2][3] = 1.; // tz
vpRotationMatrix cRo(0, atan2( cMo[0][3], cMo[1][3]), 0);

Thirdly by introducing our simulated robot we can compute the position of the target wMo and of the camera wMc wrt the world frame.

wMo = wMc * cMo;

Once all the frames are defined, we define a 3D point and its coordinates (0,0,0) in the object frame as the target.

vpPoint point;
void setWorldCoordinates(double oX, double oY, double oZ)
Definition: vpPoint.cpp:110

We compute then its coordinates in the camera frame.

void track(const vpHomogeneousMatrix &cMo)

A visual servo task is then instantiated.

vpServo task;

With the next line, we specify the king of visual servoing control law that will be used to control our mobile robot. Since the camera is mounted on the robot, we consider the case of an eye-in-hand visual servo. The robot controller provided in vpSimulatorPioneer allows to send $(v_x, w_z)$ velocities. This controller implements also the robot jacobian $\bf ^e J_e$ that links the end-effector velocity skew vector $\bf v_e$ to the control velocities $(v_x, w_z)$. The also provided velocity twist matrix $\bf ^c V_e$ allows to transform a velocity skew vector expressed in the end-effector frame in the camera frame.

We specify then that the interaction matrix $\bf L$ is computed from the visual features at the desired position. The constant gain that allows an exponential decrease of the features error is set to 0.2.

To resume, with the previous line, the following control law will be used:

\[ \left[\begin{array}{c} v_x \\ w_z \end{array}\right] = -0.2 \left( {\bf L_{s^*} {^c}V_e {^e}J_e}\right)^{+} ({\bf s} - {\bf s}^*) \]

From the robot position we retrieve the velocity twist transformation $\bf ^c V_e$ that is then re-injected to the task.

cVe = robot.get_cVe();

We do the same with the robot jacobian $\bf ^e J_e$.


Let us now consider the visual features. We first instantiate the current and desired position of the 3D target point as a visual feature point.

vpFeaturePoint s_x, s_xd;

The current visual feature is directly computed from the perspective projection of the point position in the camera frame.

The desired position of the feature is set to (0,0). The depth of the point cdMo[2][3] is required to compute the feature position., 0, cdMo[2][3]);

Finally only the position of the feature along x is added to the task.

We consider now the second visual feature $log(Z/Z^*)$ that corresponds to the depth of the point. The current and desired features are instantiated with:

vpFeatureDepth s_Z, s_Zd;

Then, we get the current Z and desired Zd depth of the target.

double Z = point.get_Z();
double Zd = cdMo[2][3];
double get_Z() const
Get the point cZ coordinate in the camera frame.
Definition: vpPoint.cpp:449

From these values, we are able to initialize the current depth feature:, s_x.get_y(), Z, log(Z/Zd));

and also the desired one:, 0, Zd, 0);

Finally, we add the feature to the task:

task.addFeature(s_Z, s_Zd);

Then comes the material used to plot in real-time the curves that shows the evolution of the velocities, the visual error and the estimation of the depth. The corresponding lines are not explained in this tutorial, but should be easily understand by reading Tutorial: Real-time curves plotter tool.

In the visual servo loop we retrieve the robot position and compute the new position of the camera wrt the target:

robot.getPosition(wMc) ;
cMo = wMc.inverse() * wMo;

We compute the coordinates of the point in the new camera frame:


Based on these new coordinates, we update the point visual feature s_x:

and also the depth visual feature:

Z = point.get_Z() ;, s_x.get_y(), Z, log(Z/Zd)) ;

We also update the task with the values of the velocity twist matrix cVe and the robot jacobian eJe:


After all these updates, we are able to compute the control law:

Computed velocities are send to the robot:

At the end, we stop the infinite loop when the visual error reaches a value that is considered as small enough:

if (task.getError().sumSquare() < 0.0001) {
std::cout << "Reached a small error. We stop the loop... " << std::endl;

Unicycle with a moving camera

In this section we consider the following unicycle:

This robot has 3 dof: $(v_x, w_z, \dot q_{1})$, as previously the translational and rotational velocities that are applied here at point M, and $\dot q_{1}$ the pan of the head. The position of the end-effector E depends on $ q_{1}$ position. The camera at point C is attached to the robot at point E. The homogeneous transformation between C and E is given by cMe. This transformation is constant.

If we consider the same visual features than previously ${\bf s} = (x, log(Z/Z^*))^\top$ and the desired feature ${\bf s}^* = (0, 0)^\top$, we are able to simulate this new robot simply by replacing vpSimulatorPioneer by vpSimulatorPioneerPan. The code is available in tutorial-simu-pioneer-pan.cpp.

You can just notice here that we compute the control law using the current interaction matrix; the one computed with the current visual feature values.

The following control law is used:

\[ \left[\begin{array}{c} v_x \\ w_z \\ \dot q_{1} \end{array}\right] = -0.2 \left( {\bf L_{s} {^c}V_e {^e}J_e}\right)^{+} ({\bf s} - {\bf s}^*) \]

Next tutorial

You are now ready to see the next Tutorial: How to boost your visual servo control law.