Visual Servoing Platform  version 3.2.1 under development (2020-01-21)
Tutorial: AprilTag marker real-time detection on iOS


This tutorial follows Tutorial: AprilTag marker detection on iOS and shows how to detect AprilTag markers in real time.

In this tutorial, you will be able to learn how to detect with Swift 4 and get camera intrinsic parameters.

All the material (Xcode project) described in this tutorial is part of ViSP source code and could be downloaded using the following command:

$ svn export

Once downloaded, you have just to drag & drop ViSP and OpenCV frameworks available following Tutorial: Installation from prebuilt packages for iOS devices.

VideoCapture for real time detection

Real-time capture of video can be achieved with AVFundation framework. You just apply the detection process you learned from the previous tutorial to each captured image.

This is capturing code in VideoCapture.swift:

func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
// check image processing flag.
if self.isImgProcessing { return }
// make Pixel Buffer
guard let imagePixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { fatalError() }
CVPixelBufferLockBaseAddress(imagePixelBuffer, [])
// get intrinsic matrix
var matrix = matrix_float3x3.init()
if let camData = CMGetAttachment(sampleBuffer, key: kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, attachmentModeOut: nil) as? Data {
matrix = camData.withUnsafeBytes { $0.pointee }
let px = matrix.columns.0.x
let py = matrix.columns.1.y
// get UIImage
guard let uiImage = imageFromCVPixelBuffer(pixelBuffer: imagePixelBuffer) else { fatalError() }
// process image in main threads
self.isImgProcessing = true
DispatchQueue.main.async {
self.delegate?.imageDidCapture(uiImage, with:px, and: py)
// clear processing flag
self.dataOutputQueue.async {
self.isImgProcessing = false

Pass image to Image process methods in ViewController.swift:

func imageDidCapture(_ uiImage: UIImage, with px: Float, and py: Float) {
self.imageView.image = self.visp.detectAprilTag(uiImage, px:px, py:py)

The camera’s intrinsic parameters can be acquired from each captured image by setting isCameraIntrinsicMatrixDeliveryEnabled to true in the connection settings in VideoCapture.swift:

if videoConnection.isCameraIntrinsicMatrixDeliverySupported {
// Enable Intrinsic parameter
videoConnection.isCameraIntrinsicMatrixDeliveryEnabled = true
print("Intrinsic Matrix is supported on this device :)" )
} else {
print("Intrinsic Matrix is NOT supported on this device :(" )

Note: intrinsic parameters are only supported on some iOS devices with iOS11.

The intrinsic parameters that represent camera features can generally be represented by a matrix of pixel-based focal lengths and principal points (axis centers) in the image. The documentation for Swift is here. Since the principal point almost coincides with the image center, this tutorial uses only the focal length.

Call Objective-C class from Swift

ViSP's AprilTag detection class is currently written in Objective-C. In order to use this Objective-C class from Swift, setting of Bridging-Header is necessary. Look for the "Objective-C Bridging Header" column in the build settings, and make sure that "VispDetector.h", which describes the detector class this time, is set.


The content of VispDetector.h is the following:

#import <UIKit/UIKit.h>
@interface VispDetector : NSObject
- (UIImage *)detectAprilTag: (UIImage*)image px:(float)px py:(float)py;

and in ViewController.swiftyou have something similar to:

class ViewController: UIViewController {
// make Objective-C class instance here.
let visp = VispDetector()
func imageDidCapture(_ uiImage: UIImage, withIntrinsicParam px: Float, and py: Float) {
// You can call method here.
imageView.image = visp.detectAprilTag(uiImage, px:px, py:py)

Distance from camera

Detection and drawing processing is processed in For details on the detection process, see Tutorial: AprilTag marker detection and on how to draw the pose of a tag, see the previous Tutorial: AprilTag marker detection on iOS.

The distance from the iOS device to the marker can be accurately detected if the tag size is properly set. The distance can be obtained using the getTranslationVector() method from the homogeneous transformation matrix (cMo_vec) representing the pose with rotation (R) and position (t) of the marker in camera coordinates. See here for more information vpHomogeneousMatrix class

This is achieved in

for(int i=0; i < NumberOfTags; i++){
vpTranslationVector trans = cMo_vec[i].getTranslationVector();
float x = trans[0];
float y = trans[1];
float z = trans[2];
std::cout << x << y << z << std::endl;

Application output

Like this image below, you can see the tag id, posture and position from the camera in real time with the video captured image.