Introduction

Since ViSP 3.6.0, a 3rd party library was introduced to allow the seamless use of JSON (JavaScript Object Notation) in ViSP. The library that is used is JSON for modern C++. To install it on your system, look at JSON for modern C++ installation instructions for your system.

The main use case for using JSON in your programs is to save your experimental results. The main advantage of using JSON is that it is widely supported by default in other languages.

Thanks to this 3rd party, JSON serialization was introduced for vpCameraParameters, vpArray2D, vpColVector, vpPoseVector, vpHomogeneousMatrix, vpMe and vpMbGenericTracker.

In this tutorial, we will use JSON to:

Read program arguments from a configuration file. This makes it easier to run your program many times and makes storing different configurations easier
Save the data generated by running a visual servoing loop
Generate plots in Python

JSON overview

First of all, it is necessary to understand the structure of JSON documents to better use them. For a detailed description, you may be interested in the Mozilla Developer Network documentation.

Much of this section is a repeat of the library's documentation, available here.

To use JSON in your code you should first include the relevant header:

#include VISP_NLOHMANN_JSON(json.hpp)

using json = nlohmann::json; // For convenience

This introduces a new type named json which supports many types out of the box:

json j = {
  {"bool", false},
  {"double", 3.14156},
  {"string", "Hello World!"},
  {"aVector", {1.0, 2.0, 3.0}}
};
std::cout << j.dump(4) << std::endl;

This snippet will print

{
  "aVector": [1.0, 2.0, 3.0],
  "bool": false,
  "double": 3.14156,
  "string": "Hello World!"
}

Defining a JSON conversion for your types

The library also allows to define explicit conversion between a custom type and JSON. This can be achieved by defining the two following functions:

void from_json(const json& j, YourType& t);

void to_json(json& j, const YourType& t);

These functions must be defined in the same scope namespace as YourType and must be accessible everywhere YourType is used. It is thus common to define it in the header where YourType is defined.

#include VISP_NLOHMANN_JSON(json.hpp)
 
class YourType {
public:
  YourType(double x): x(x) {}
 
private:
  double x;
  friend from_json(const json& j, YourType& t);    // Declare friends if you wish to access private members
  friend void to_json(json& j, const YourType& t);
};
 
inline void from_json(const json& j, YourType& t)
{
  t.x = j.at("x");
}
 
inline void to_json(json& j, const YourType& t)
{
  j["x"] = t.x;
}

Once this conversion is defined you can convert to and from json with:

YourType t(5.0);
json j = t;                         // j == {"x": 5.0}, calls to_json
YourType t2 = j;                    // t2.x == 5.0, calls from_json
j = {YourType(1.0), YourType(2.0)}; // j == [{"x": 1.0}, {"x": 2.0}]
                                    // JSON conversion for std vector is already known and call
                                    // to_json for each of the elements of the vector

To better understand the requirements and caveats when writing custom conversions, we encourage you to read the documentation on arbitrary type conversion.

Example: Using JSON for a visual servoing experiment

This example will demonstrate some basic usage of JSON to save the data generated by running an IBVS and exporting it to python to plot some figures. This example is a modification tutorial-ibvs-4pts.cpp and the full code is available at tutorial-ibvs-4pts-json.cpp.

This program can be run with:

$ cd $VISP_WS/visp-build/tutorial/visual-servo/ibvs
$ ./tutorial-ibvs-4pts-json --settings ibvs_settings.json --output results.json

Using JSON for program arguments

We first start by defining the program arguments:

class Arguments
{
public:
  // Default values
  Arguments() :
    lambda(0.5), cdMo(0, 0, 0.75, 0, 0, 0),
    cMo(0.15, -0.1, 1., vpMath::rad(10), vpMath::rad(-10), vpMath::rad(50)),
    samplingTime(0.04), errorThreshold(0.0001), interactionMatrixType(CURRENT)
  { }
 
  vpServo::vpServoIteractionMatrixType getInteractionMatrixType() const
  {
    switch (interactionMatrixType) {
    case CURRENT:
      return vpServo::CURRENT;
    case DESIRED:
      return vpServo::DESIRED;
    case MEAN:
      return vpServo::MEAN;
    default:
      throw vpException(vpException::badValue, "Unexpected value");
    }
    return vpServo::CURRENT;
  }
 
  double lambda;            // Control law gain
  vpHomogeneousMatrix cdMo; // Target (desired) camera pose
  vpHomogeneousMatrix cMo;  // Initial camera pose
  double samplingTime;      // Robot sampling time
  double errorThreshold;    // Error threshold. Once error is below, consider servoing as successful
  vpInteractionMatrixTypeSubset interactionMatrixType;
};

This struct contains the control law gain, the desired and starting pose, as well as the visual error threshold and robot sampling time. Finally it contains a value of type vpInteractionMatrixTypeSubset which is defined as

enum vpInteractionMatrixTypeSubset
{
  UNKNOWN = -1,
  CURRENT,
  DESIRED,
  MEAN
};

This value will be converted to a vpServo::vpServoIteractionMatrixType when defining the control law, by calling Arguments::getInteractionMatrixType().

Now that our arguments are defined we must now know to parse JSON to get them. We start by defining how to serialize our enumeration vpInteractionMatrixTypeSubset:

NLOHMANN_JSON_SERIALIZE_ENUM(vpInteractionMatrixTypeSubset, {
  {UNKNOWN, nullptr}, // Default value if the json string is not in "current", "desired" or "mean"
  {CURRENT, "current"},
  {DESIRED, "desired"},
  {MEAN, "mean"} }
  );

Note that if the value is not in CURRENT, DESIRED or MEAN, then the parsing result will default to the value vpInteractionMatrixTypeSubset::UNKNOWN.

Next we define how to parse the full list of arguments:

// Read script arguments from JSON. All values are optional and if an argument is not present,
// the default value defined in the constructor is kept
void from_json(const json &j, Arguments &a)
{
#ifdef ENABLE_VISP_NAMESPACE
  using VISP_NAMESPACE_ADDRESSING from_json;
#endif
  a.lambda = j.value("lambda", a.lambda);
  if (a.lambda <= 0) {
    throw vpException(vpException::badValue, "Lambda should be > 0");
  }
 
  a.cMo = j.value("cMo", a.cMo);
  a.cdMo = j.value("cdMo", a.cdMo);
 
  a.samplingTime = j.value("samplingTime", a.samplingTime);
  if (a.samplingTime <= 0) {
    throw vpException(vpException::badValue, "Sampling time should be > 0");
  }
 
  a.errorThreshold = j.value("errorThreshold", a.errorThreshold);
  if (a.errorThreshold <= 0) {
    throw vpException(vpException::badValue, "Error threshold should be > 0");
  }
 
  a.interactionMatrixType = j.value("interactionMatrix", a.interactionMatrixType);
  if (a.interactionMatrixType == UNKNOWN) {
    throw vpException(vpException::badValue, "Unknown interaction matrix type defined in JSON");
  }
}
 
void to_json(json &j, const Arguments &a)
{
#ifdef ENABLE_VISP_NAMESPACE
  using VISP_NAMESPACE_ADDRESSING to_json;
#endif
  j = json {
    {"lambda", a.lambda},
    {"cMo", a.cMo},
    {"cdMo", a.cdMo},
    {"errorThreshold", a.errorThreshold},
    {"samplingTime", a.samplingTime} ,
    {"interactionMatrix", a.interactionMatrixType}
  };
}

Where the method from_json() is used to parse JSON and, inversely, to_json() is used to convert the arguments to JSON, which we will use afterwards.

The syntax

x = j.value("key", x);

is used to fill in optional values from the JSON document. If the key is not found then x will be left unchanged This syntax can also be used to set a default value, e.g.:

a.lambda = j.value("lambda", 0.5);

In the from_json() method, we ensure that all the arguments are correctly set:

Lambda should be > 0
Sampling time should be > 0
Error threshold should be > 0
The type of interaction matrix should be correct

With this conversion defined, we can read the program settings from a file (ibvs_settings.json), containing

{
    "lambda": 0.2,
    "samplingTime": 0.04,
    "errorThreshold": 0.0001,
    "interactionMatrix": "current",
    "cdMo": {
        "cols": 1,
        "rows": 6,
        "type": "vpPoseVector",
        "data": [0.0, 0.0, 0.75, 0.0, 0.0, 0.0]
    },
    "cMo": {
        "cols": 1,
        "rows": 6,
        "type": "vpPoseVector",
        "data": [0.3, 0.3, 1.2, 0.349066, 0.349066, 1.5708]
    }
}

Note that while the Arguments class represents the desired and starting poses as vpHomogeneousMatrix elements, they are defined in the JSON as vpPoseVector representations. ViSP will automatically convert one representation to the other when parsing JSON.

To read from the ibvs_settings.json file, we will define the following method:

Arguments readArguments(const std::string &path)
{
  Arguments a;
 
  if (!path.empty()) {
    std::ifstream file(path);
    if (!file.good()) {
      std::stringstream ss;
      ss << "Problem opening file " << path << ". Make sure it exists and is readable" << std::endl;
      throw vpException(vpException::badValue, ss.str());
    }
    json j;
    try {
      j = json::parse(file);
    }
    catch (json::parse_error &e) {
      std::stringstream msg;
      msg << "Could not parse JSON file : \n";
 
      msg << e.what() << std::endl;
      msg << "Byte position of error: " << e.byte;
      throw vpException(vpException::ioError, msg.str());
    }
    a = j; // Call from_json(const json& j, Argument& a) to read json into arguments a
    file.close();
  }
  else {
    std::cout << "Using default arguments. Try using a JSON file to set the arguments of the visual servoing." << std::endl;
  }
  return a;
}

Which we will call in the main(), to finally obtain our program arguments:

  std::string arguments_path = "";
  std::string output_path = "";
  for (int i = 1; i < argc; ++i) {
    if (std::string(argv[i]) == "--settings" && i + 1 < argc) {
      arguments_path = std::string(argv[i + 1]);
    }
    else if (std::string(argv[i]) == "--output" && i + 1 < argc) {
      output_path = std::string(argv[i + 1]);
    }
  }
 
  if (output_path.empty()) {
    std::cerr << "JSON output path must be specified" << std::endl;
    return EXIT_FAILURE;
  }
  const Arguments args = readArguments(arguments_path);

We can now start servoing and acquiring data, that will be saved to another JSON file.

Collecting and saving visual servoing data

To save the data to a JSON file, we first define what we'll save

class ServoingExperimentData
{
public:
  ServoingExperimentData(const Arguments &arguments, const std::vector<vpFeaturePoint> &desiredFeatures) :
    m_arguments(arguments), m_desiredFeatures(desiredFeatures)
  { }
 
  void onIter(const vpHomogeneousMatrix &cMo, const double errorNorm, const std::vector<vpFeaturePoint> &points,
    const vpColVector &velocity, const vpMatrix &interactionMatrix)
  {
    vpPoseVector r(cMo);
    m_trajectory.push_back(r);
    m_errorNorms.push_back(errorNorm);
    m_points3D.push_back(points);
    m_velocities.push_back(velocity);
    m_interactionMatrices.push_back(interactionMatrix);
  }
 
private:
  Arguments m_arguments;
  std::vector<vpFeaturePoint> m_desiredFeatures;
  std::vector<vpPoseVector> m_trajectory;
  std::vector<double> m_errorNorms;
  std::vector<std::vector<vpFeaturePoint> > m_points3D;
  std::vector<vpColVector> m_velocities;
  std::vector<vpMatrix> m_interactionMatrices;
  friend void to_json(json &j, const ServoingExperimentData &res);
};
 
void to_json(json &j, const ServoingExperimentData &res)
{
#ifdef ENABLE_VISP_NAMESPACE
  using VISP_NAMESPACE_ADDRESSING to_json;
#endif
  j = json {
    {"parameters", res.m_arguments},
    {"trajectory", res.m_trajectory},
    {"errorNorm", res.m_errorNorms},
    {"features", res.m_points3D},
    {"desiredFeatures", res.m_desiredFeatures},
    {"velocities", res.m_velocities},
    {"interactionMatrices", res.m_interactionMatrices}
  };
}

This includes:

The arguments used to launch the program
The desired visual features
For each iteration
- The visual features
- The camera pose
- The squared norm of the visual error
- The velocity
- The interaction matrix

Before starting the servoing loop we create a results object:

ServoingExperimentData results(args, features);

We then start the visual servoing loop:

    while (!end) {
      robot.getPosition(wMc);
      cMo = wMc.inverse() * wMo;
      for (unsigned int i = 0; i < 4; i++) {
        point[i].track(cMo);
        vpFeatureBuilder::create(p[i], point[i]);
        features[i] = p[i];
      }
      const vpColVector v = task.computeControlLaw();
      robot.setVelocity(vpRobot::CAMERA_FRAME, v);
      const double errorNorm = task.getError().sumSquare();
 
      results.onIter(cMo, errorNorm, features, v, task.getInteractionMatrix());
 
      if (errorNorm < args.errorThreshold) {
        end = true;
      }
      vpTime::wait(10);
      iter++;
    }

where at each iteration, we update our results object with

results.onIter(cMo, errorNorm, features, v, task.getInteractionMatrix());

Finally, to save the results to a file, we define a function to write to a file

void saveResults(const ServoingExperimentData &results, const std::string &path)
{
  std::ofstream file(path);
  const json j = results;
  file << j.dump(4);
  file.close();
}

and call it at the end of the program

saveResults(results, output_path);

Our data is now saved in a JSON file, and we can reuse it in other programs, as we demonstrate further below.

Reusing the data in Python to generate plots

We will now reparse the JSON file saved in Collecting and saving visual servoing data section to generate high quality plots in Python with Matplotlib. The script, plot-ibvs-control-law.py can be found in the script folder of the ViSP git repository. It has been tested with Python 3.7.9, matplotlib 3.3.2 and numpy 1.17.3.

If not already done, to use the script install the following:

Install the latest Python3 release
Check if pip3 and python3 are correctly installed
```
$ python3 --version
$ pip3 --version
```
Upgrade your pip to avoid errors during installation.
```
$ pip3 install --upgrade pip
```
Enter the following command to install Numpy using pip3
```
$ pip3 install numpy
```
Enter the following command to install Matplotlib using pip3
```
$ pip3 install matplotlib
```

The script gives plot-ibvs-control-law.py a reference on how to plot with JSON data coming from ViSP. You may wish to modify and extend it to fit your use case or control law.

To run the script, replace the json path that contains the data to plot and where you wish to save the plots in the following command:

$ cd VISP_WS/visp/script
$ python3 plot-ibvs-control-law.py --data $VISP_WS/visp-build/tutorial/visual-servo/ibvs/results.json \
                                   --plot_folder ./plots

The script should produce similar output:

Generating error norm plot...
Generating velocity plot...
Generating 2D and 3D trajectory plots...
Generating features plot...
Plots were saved to $VISP_WS/visp/script/plots

You can then view the generated plots in the folder where you saved them (in our case in ./plots/ corresponding to $VISP_WS/visp/script/plots folder).

$ ls $VISP_WS/visp/script/plots
cMo.pdf           error.pdf               velocity.pdf
cMo_3d.pdf      trajectories_2d.pdf

You should obtain plots similar to those below, but with a higher quality (generated in pdf):

Next tutorial

You are now ready to see how to continue with Tutorial: Loading a model-based generic tracker from JSON.

Table of Contents