In the context of providing a render-based tracker, this tutorial introduces a new, easy to use renderer based on Panda3D.
This renderer can output:
A color image, with support for textures and lighting
Depth image
Normal maps
In world space
In camera space
It only supports camera models with no distortion.
It is also possible to compute camera clipping values, depending on the pose of an object in the camera frame. This ensures that the depth buffer is as accurate as possible when considering this object.
Below is a set of renders for a textured cube object.
Multi-output rendering is performed via the vpPanda3DRendererSet class, which duplicates the scene across multiple renders and synchronizes changes to objects and the camera. Each Sub renderer implements a specific type of render: geometric (vpPanda3DGeometryRenderer) or color-based (vpPanda3DRGBRenderer) etc. They all inherit from vpPanda3DBaseRenderer, which implements basic functions for a panda renderer.
Panda3D installation
Installation on Ubuntu
It is recommended to install the assimp package before installing Panda3D. This will allow you to load .obj files, which are a common and easily exportable format. Without it, you will be restricted to .bam files, which is a panda specific format. To convert to a .bam file, see Converting a mesh to Panda's file format.
To install assimp, run:
$ sudo apt install libassimp-dev
Installer are available for Ubuntu browsing the download page.
Hereafter you will find the instructions to build and install Panda3D from source on Ubuntu 22.04
install the produced Debian package (recommended) with
$ sudo dpkg -i panda3d1.11_1.11.0_amd64.deb
use the Panda3D libraries located in the built folder without installing the Debian package panda3d1.11_1.11.0_amd64.deb, but in that case you need to set LD_LIBRARY_PATH environment var:
Without setting LD_LIBRARY_PATH you may experience the following error when running a binary that uses Panda3D capabilities:
$ ./tutorial-panda3d-renderer
./tutorial-panda3d-renderer: error while loading shared libraries: libp3dtoolconfig.so.1.11: cannot open shared object file: No such file or directory
Now to build ViSP with Panda3D support when Debian package panda3d1.11_1.11.0_amd64.deb is installed as described in option (1), you may notice that there is nothing specific to do, just run cmake as usual:
$ cd $VISP_WS/visp-build
$ cmake ../visp
$ make -j$(nproc)
There is also the possibility to build ViSP with Panda3D support without installing Debian package panda3d1.11_1.11.0_amd64.deb as described in option (2):
By setting Panda3D_DIR cmake var to the Panda3D cloned folder
Installer are available for macOS browsing the download page.
Note
For the latest Panda3D 1.10.15 SDK there is an Installer for macOS X 10.9+ that is only compatible with architecture x86_64. If you are using a Mac M1 or M2, there is no Panda3D SDK available yet for arm64 architecture. The solution is to build Panda3D from source.
Hereafter you will find the instructions to build Panda3D from source on macOS.
It is recommended to install the assimp package before installing Panda3D. This will allow you to load .obj files, which are a common and easily exportable format. Without it, you will be restricted to .bam files, which is a panda specific format. To convert to a .bam file, see Converting a mesh to Panda's file format. To install assimp, run:
$ brew install assimp
On macOS, you will need to download a set of precompiled third-party packages in order to compile Panda3D. Navigate to Panda3D download page, select the lastest SDK (in our case SDK 1.10.15), and under </> Source Code section, download Thirdparty tools for macOS (in our case panda3d-1.10.15-tools-mac.tar.gz).
Extract third-party tools for macOS from downloaded archive
$ cd ~/Downloads
$ tar xvzf panda3d-1.10.15-tools-mac.tar.gz
Once done clone Panda3D:
$ mkdir -p $VISP_WS/3rdparty/panda3d
$ cd $VISP_WS/3rdparty/panda3d
$ git clone https://github.com/panda3d/panda3d
$ cd panda3d
Move the downloaded third-party tools in Panda3D source code folder
install the produced Panda3D-1.11.0.dmg file (recommended) just by double clicking on it. In the installer window, don't forget to enable the C++ Header Files check box before pressing the installation button. After that you have to set DYLIB_LIBRARY_PATH environment var:
or use the Panda3D libraries located in the built folder without installing .dmg file, but in that case you need to set DYLIB_LIBRARY_PATH environment var:
Without setting DYLD_LIBRARY_PATH you may experience the following error when running a binary that uses Panda3D capabilities:
$ ./tutorial-panda3d-renderer
dyld[257]: Library not loaded: @loader_path/../lib/libpanda.1.11.dylib
Now to build ViSP with Panda3D support when .dmg file Panda3D-1.11.0.dmg is installed, you can just run cmake as usual. Note that PCL is not compatible with Panda3D, that's why we disable here PCL usage (see Segfault: :framework(error): Unable to create window).
$ cd $VISP_WS/visp-build
$ cmake ../visp -DUSE_PCL=OFF
$ make -j$(sysctl -n hw.logicalcpu)
There is also the possibility to build ViSP with Panda3D support without installing the .dmg file
By setting Panda3D_DIR cmake var to the Panda3D cloned folder
Installer are available for Windows browsing the download page.
Converting a mesh to Panda's file format
If you did not install assimp before installing Panda, you are fairly restricted in what mesh format you can load with Panda.
Panda's recommended format is .bam, a binary format specific to the engine itself. If you have Python, you can easily convert your model to a .bam file by doing the following:
Installing the panda3D-gltf package: pip install panda3d-gltf
From your favorite 3D modelling software export your model to GLTF (available in blender)
Running the converter tool:
$ gltf2bam source.gltf output.bam
If you encounter issues, check the tool's settings with gltf2bam -h
Using Panda3D for rendering
An example that shows how to exploit Panda3D in ViSP to render a color image with support for textures and lighting, a depth image, normals in object space and in camera space is given in tutorial-panda3d-renderer.cpp.
To start rendering, we first instanciate a vpPanda3DRendererSet. This object allows to render multiple modalities (color, depth, etc.) in a single pass. To add different rendering modalities, we will use subclasses that will be registered to the renderer set. Internally, each sub renderer has its own scene: the renderer set synchronizes everything when the state changes (i.e, an object is added, an object is moved or the camera parameters change.)
A panda3D renderer should be instanciated with a vpPanda3DRenderParameters object. This object defines:
The camera intrinsics (see vpCameraParameters): As of now, Only parameters for a distortion free model are supported
The image resolution
The near and far clipping plane values. Object parts that are too close (less than the near clipping value) or too far (greater than the far clipping value) will not be rendered.
The creation of the renderer set can be found below
vpPanda3DGeometryRenderer instances allow to retrieve 3D information about the object: these are the surface normals in the object or camera frame, as well as the depth information.
vpPanda3DRGBRenderer objects perform the traditional color rendering. Lighting interaction can be disable, as is the case for the second renderer (diffuse only).
Once they have been added, a call to vpPanda3DBaseRenderer::initFramework() should be performed. Otherwise, no rendering will be performed and objects will not be loaded.
Here you will find the code used to configure the scene:
Each frame, we compute the values of the clipping planes, and update the rendering properties. This will ensure that the target object is visible. Depending on your use case, this may not be necessary.
Once this is done, we can call upon Panda3D to render the object with
Note that under the hood, all subrenderers rely on the same Panda3D "framework": calling renderFrame on one will call it for the others.
To use the renders, we must convert them to ViSP images. To do so, each subrenderer defines its own getRender method, that will perform the conversion from a panda texture to the relevant ViSP data type.
For each render type, we start by getting the correct renderer via the vpPanda3DRendererSet::getRenderer, then call its getRender method.
Now that we have the retrieved the images, we can display them. To do so, we leverage utility functions defined beforehand (see the full code for more information). This may be required in cases where the data cannot be directly displayed. For instance, normals are encoded as 3 32-bit floats, but displays require colors to be represented as 8-bit unsigned characters. The same goes for the depth render, which is mapped back to the 0-255 range, although its value unbound.
Finally, we use the snippet below to move the object, updating the scene. To have a constant velocity, we multiply the displacement by the time that has elapsed between the frame's start and end.
The installer calls makepanda/makepandacore.py and enters in SdkLocateMacOSX() function to try to find the latest MacOSX SDK in one of the following folders:
1.9.1