Visual Servoing Platform  version 3.3.0 under development (2020-02-17)
Tutorial: Visual-servoing with Parrot Bebop 2 drone

Table of Contents

Introduction

This tutorial explains how to do an image-based servoing with a Parrot Bebop 2 drone on Ubuntu or OSX.

img-bebop2.jpg

Prerequisites

The following material is necessary :

  • Parrot Bebop 2 drone
  • An AprilTag from 36h11 family that will serve as target for the visual servoing. Print an Apriltag target.

ViSP must be built with OpenCV support if you want to get the video streamed by the drone, which needs to be decoded.

Note
Before continuing, we recommend that you succeed to complete Tutorial: AprilTag marker detection.

Build Parrot ARSDK3

In order to use Parrot Bebop 2 drone with ViSP, you first need to build Parrot's SDK ARDroneSDK3 (as explained here) :

On Ubuntu

The following steps allow to build ARSDK3 on Ubuntu (tested on 18.04).

1. Get the SDK source code

Create a workspace.

$ cd ${VISP_WS}
$ mkdir ARDroneSDK3 && cd ARDroneSDK3

Initialize the repo.

$ sudo apt install repo
$ repo init -u https://github.com/Parrot-Developers/arsdk_manifests.git -m release.xml

You can then download all the repositories automatically, by executing the following command.

$ repo sync

2. Build the SDK

Install required 3rd parties:

$ sudo apt-get install git build-essential autoconf libtool libavahi-client-dev libavcodec-dev libavformat-dev libswscale-dev libncurses5-dev mplayer

Build the SDK:

$ ./build.sh -p arsdk-native -t build-sdk -j

The output will be located in ${VISP_WS}/ARDroneSDK3/out/arsdk-native/staging/usr

3. Set ARSDK_DIR environment variable

In order for ViSP to find ARDroneSDK3, set ARSDK_DIR environment variable:

$ export ARSDK_DIR=${VISP_WS}/ARDroneSDK3

4. Modify LD_LIBRARY_PATH environment variable to detect ARDroneSDK3 libraries

In order that ViSP binaries are able to find ARDroneSDK3 libraries, set LD_LIBRARY_PATH with:

$ export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${ARSDK_DIR}/out/arsdk-native/staging/usr/lib

On Mac OSX

The following steps allow to build ARSDK3 on macOS Mojave 10.14.5.

1. Get the SDK source code

Create a workspace.

$ cd ${VISP_WS}
$ mkdir ARDroneSDK3 && cd ARDroneSDK3

Initialize the repo.

$ brew install repo
$ repo init -u https://github.com/Parrot-Developers/arsdk_manifests.git -m release.xml

You can then download all the repositories automatically, by executing the following command.

$ repo sync

2. Build the SDK

Install required 3rd parties:

$ brew install ffmpeg

Build the SDK:

$ ./build.sh -p arsdk-native -t build-sdk -j

The output will be located in ${VISP_WS}/ARDroneSDK3/out/arsdk-native/staging/usr

3. Set ARSDK_DIR environment variable

In order for ViSP to find ARDroneSDK3, set ARSDK_DIR environment variable:

$ export ARSDK_DIR=${VISP_WS}/ARDroneSDK3

4. Modify DYLD_LIBRARY_PATH environment variable to detect ARDroneSDK3 libraries

In order that ViSP binaries are able to find ARDroneSDK3 libraries, set DYLD_LIBRARY_PATH with:

$ export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:${ARSDK_DIR}/out/arsdk-native/staging/usr/lib

Build ViSP to detect ARSDK3

In order that ViSP takes into account ARSDK3 fresh installation you need to configure and build ViSP again.

  • Enter ViSP build folder:
    $ cd $VISP_WS/visp-build
  • Configure ViSP to detect ARSDK3 and ffmpeg used to decode images from Bebop2 camera stream:
    $ cmake ../visp
    At this point you should see in Real robots section that ARSDK and ffmpeg are enabled
    Real robots:
    ...
    Use Parrot ARSDK: yes
    \-Use ffmpeg: yes
    ...
  • Now build ViSP using:
    $ make -j4

Image-based visual-servoing

Source code access

An example of image-based visual servoing is implemented in servoBebop2.cpp.

The corresponding source code and CMakeLists.txt file can be downloaded using:

$ svn export https://github.com/lagadic/visp.git/trunk/example/servo-bebop2

Understanding the controller

First, to get the basics of image-based visual servoing, you can read Tutorial: Image-based visual servo.

The following image shows the frames attached to the drone:

img-bebop2-coord-system.png
  • There is the drone control frame, also called end-effector frame, in which we can control the drone in velocity applying $\dot{\bf q}_e = (v_x, v_y, v_z, w_z)$ corresponding respectively to the 3 translational velocities along $X_e, Y_e, Z_e$ axis, and the rotational velocity along $Z_e$ axis. The vpRobotBebop2 class allows to send these velocities. Note that the 6-dim velocity skew vector is named ${\bf v_e}$
  • There is the also the camera frame with $X_c, Y_c, Z_c$ axis in which we define the velocities skew vector ${\bf v_c} = (v_x, v_y, v_z, w_x, w_y, w_z)$.
  • The homogeneous transformation between the camera frame and the end-effector frame is named ${^c}{\bf M}_e$. This transformation is implemented as a vpHomogeneousMatrix.

In servoBebop2.cpp example, we use four visual features ${\bf s}=(n_{g_x}, n_{g_y}, n_a, \arctan(1/\rho))$ for the servoing in order to control the four drone dof $\dot{\bf q}_e$. These visual features are:

  • Centered and normalized gravity center moment ${\bf n_g}=(n_{g_x}, n_{g_y})$ of the tag along camera $X_c$ and $Y_c$ axis. This feature is implemented in vpFeatureMomentGravityCenterNormalized and used to center the tag in the image.
  • Normalized area moment of the tag $n_a$. This feature implemented in vpFeatureMomentAreaNormalized is used to control the distance between the drone and the tag.
  • Horizontal vanishing point position corresponding to the intersection of the two lines passing through top and bottom tag edges. From the polar coordinates $(\rho, \theta)$ of this point, we use $\arctan(1/\rho)$ visual feature. This feature implemented in vpFeatureVanishingPoint is used to control the orientation of the drone along its vertical axis based on the tag orientation.

The corresponding controller is given by:

\[ \dot{\bf q}_e = -\lambda {\left({\bf L_s} {^c}{\bf V}_e {^e}{\bf J}_e\right)}^{+}({\bf s} - {\bf s}^*) \]

where:

  • $\lambda$ is the controller gain implemented in vpAdaptiveGain
  • ${\bf L_s}$ is the interaction matrix corresponding to the visual features ${\bf s}$. This matrix is updated in vpServo
  • ${^c}{\bf V}_e$ is the velocity twist matrix build using ${^c}{\bf M}_e$. Implemented in vpVelocityTwistMatrix it allows to transform a velocity skew from end-effector frame into the camera frame: ${\bf v_c} = {^c}{\bf V}_e \; {\bf v_e}$
  • ${^e}{\bf J}_e$ the robot Jacobian that makes the link between the velocity skew ${\bf v_e}$ and the control dof ${\bf q_e}$ in the end-effector frame: ${\bf v_e} = {^e}{\bf J}_e \; \dot{\bf q}_e$
  • ${\bf s}$ and ${\bf s}^*$ are respectively current and desired visual feature vectors.

To make the relation between this controller description and the code, check the comments in servoBebop2.cpp.

Running the program

The next step is now to run the image-based visual servoing example implemented in servoBebop2.cpp.

Note
Before starting the program, the drone should be turned on and the computer connected to the drone WiFi, as shown in the following pictures :
On Ubuntu:
drone_connexion.png

On Mac OSX :
img-bebop2-osx-wifi.jpg
Warning
CAUTION : It's is strongly recommended to use this program outside or in a large room with non-uniform flooring, as the drone uses a downward-facing camera to estimate its motion from optical flow. If the surface under the drone is uniform, its movements will be inaccurate and dangerous.

If you built ViSP with ffmpeg and Parrot ARSDK3 support, the corresponding binary is available in ${VISP_WS}/visp-build/example/servo-bebop2 folder.

$ cd ${VISP_WS}/visp-build/example/servo-bebop2
$ ./servoBebop2 --tag_size 0.14
Note
Passing the tag size (in meters) as a parameter is required.

On Mac OSX, you may need to allow servoBebop2 to accept incoming network connections :

img-bebop2-osx-accept-connection.jpg

Running the previous command should allow to get same results as the one presented in the video:

Run ./servoBebop2 --help to see which are the command line options available.

  • Adding option --ip allows you to specify the ip of the drone on the network (default is 192.168.42.1). This is useful if you changed your drone ip (see Changing Bebop 2 IP address), if you want to fly multiple drones at once, for instance.
  • Adding option --distance_to_tag 1.5 allows to specify the desired distance (in meters) to the tag for the drone servoing. Values between 0.5 and 2 are recommended (default is 1 meter).
  • Adding option --intrinsic ~/path-to-calibration-file/camera.xml allows you to specify the intrinsic camera calibration parameters. This file can be obtained by completing Tutorial: Camera intrinsic calibration. Without this option, default parameters that are enough for a trial will be used..
  • Adding option --hd_stream enables HD 720p stream resolution instead of default 480p. Increase range and accuracy of the tag detection, but increases latency and computation time.
    Note
    Camera calibration settings are different for the two resolutions.
    Make sure that if you pass custom intrinsic camera parameters, they were obtained with the correct resolution.
  • Adding option --verbose or -v enables the display of information messages from the drone, and the velocity commands sent to the drone.

The program will first connect to the drone, start the video streaming and decoding, and then the drone will take off and hover until it detects one (and one only) 36h11 AprilTag in the image.

img-drone_tag.jpg

We then display the drone video stream with the visible features, as well as the error for each feature :

drone_view.png

servoing_task.png

In this graph :

  • Xn corresponds to the error $(n_{g_x} - {n_{g_x}}^*)$ that allows to control the tag center of gravity along $X_c$ axis,
  • Yn corresponds to the error $(n_{g_y} - {n_{g_y}}^*)$ that allows to control the tag center of gravity along $Y_c$ axis,
  • an corresponds to the error $(a_{n} - {a_{n}}^*)$, used to regulate the distance between the drone and the tag along $Z_c$ axis,
  • atan(1/rho) corresponds to the error $(atan(1/rho) - atan(1/rho^*))$ related to vanishing point. This feature will make the drone move its orientation along $Y_c$ axis to ensure that the two horizontal lines remain parallel.

Clicking on the drone view display will make the drone land, safely disconnect everything and quit the program.

Tips & Tricks

Changing Bebop 2 IP address

If you need to change the drone IP address, for flying multiple drones for instance, you can follow these steps :

  • Turn on your drone and connect to its WiFi network.
  • Press the drone on/off button 4 times.
  • Connect to the drone file system using telnet (if you haven't changed the drone IP yet, the default IP should be 192.168.42.1):
    $ telnet 192.168.42.1
    Note
    If you get the message "Connection refused", you haven't properly pressed the on/off button 4 times.
    If you get the message "Connection timed out", you haven't used the right IP. You can try with 192.168.43.1 .
  • Once you're connected to the drone file system, you need to get write access to the files. You can do so with :
    mount –o remount,rw /
    Warning
    You now have permissions to move, edit or delete any file. Proceed at your own discretion, as you could irreversibly make your drone unusable !
  • Edit /sbin/broadcom_setup.sh :
    cd sbin
    vi broadcom_setup.sh
Note
If you don't know how to use VI text editor :
  • move the cursor using arrow keys,
  • edit the text using i and escape to cancel,
  • press : and enter wq to save and quit, or q! to quit without saving.
  • Edit line IFACE IP AP=”192.168.42.1” to IFACE IP AP=”192.168.x.1”, where x represents any number that you have not assigned to any other drone yet.
  • Save and exit the text editor.
  • Exit Bebop 2 file system by entering exit.
  • Restart your drone. It's IP should now be changed. You will have to adapt your programs accordingly.

Connecting multiple Bebop 2 drones to a single computer

If you want to control multiple drones using one single computer, you're going to need to change the drones ip, by following Changing Bebop 2 IP address.

Once every drone you want to use has a unique IP address, you need to connect your PC to each drone WiFi network. You can use multiple WiFi dongles and you PC WiFi card, if it has one.

For two drones, it should look like this (on Ubuntu) :

img_multiple_drones_connection.png

In ViSP programs that use the drone, you can then use option --ip to specify the IP of the drone to which you want to connect :

$ cd ${VISP_WS}/visp-build/example/servo-bebop2
$ ./keyboardControlBebop2.cpp --ip 192.168.42.1

and in another terminal :

$ cd ${VISP_WS}/visp-build/example/servo-bebop2
$ ./keyboardControlBebop2.cpp --ip 192.168.43.1

In your own programs, you can specify the IP in the constructor of vpRobotBebop2 class :

vpRobotBebop2 drone(false, true, "192.168.43.1"); // This creates the drone with low verbose level, settings reset and corresponding IP



Next tutorial

If needed, you can see Tutorial: Image frame grabbing corresponding section dedicated to Parrot Bebop 2 to get images of the calibration grid.
You can also calibrate your drone camera and generate an XML file usable in the servoing program (see Tutorial: Camera intrinsic calibration).
If you need more details about this program, check the comments in servoBebop2.cpp.
You can check example program keyboardControlBebop2.cpp if you want to see how to control a Bebop 2 drone with the keyboard.
You can also check vpRobotBebop2 to see the full documentation of the Bebop 2 ViSP class.

Finally, if you are more interested to do the same experiment with ROS framework, you can follow How to do visual servoing with Parrot Bebop 2 drone using visp_ros tutorial.