Tutorial: AprilTag marker real-time detection on iOS

Table of Contents

• Introduction
• VideoCapture for real time detection
• Call Objective-C class from Swift
• Distance from camera
• Application output

Introduction

This tutorial follows Tutorial: AprilTag marker detection on iOS and shows how to detect AprilTag markers in real time.

In this tutorial, you will be able to learn how to detect with Swift 4 and get camera intrinsic parameters.

Note that all the material (Xcode project) described in this tutorial is part of ViSP source code (in tutorial/ios/AprilTagLiveCamera folder) and could be found in https://github.com/lagadic/visp/tree/master/tutorial/ios/AprilTagLiveCamera.

Once downloaded, you have just to drag & drop ViSP and OpenCV frameworks available following Tutorial: Installation from prebuilt packages for iOS devices.

VideoCapture for real time detection

Real-time capture of video can be achieved with AVFundation framework. You just apply the detection process you learned from the previous tutorial to each captured image.

This is capturing code in VideoCapture.swift:

    func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
        
        // check image processing flag.
        if self.isImgProcessing { return }
        
        // make Pixel Buffer
        guard let imagePixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { fatalError() }
        
        
        CVPixelBufferLockBaseAddress(imagePixelBuffer, [])
        
        // get intrinsic matrix
        var matrix = matrix_float3x3.init()
        if let camData = CMGetAttachment(sampleBuffer, key: kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, attachmentModeOut: nil) as? Data {
            matrix = camData.withUnsafeBytes { $0.pointee }
        }
        let px = matrix.columns.0.x
        let py = matrix.columns.1.y
        
        // get UIImage
        guard let uiImage = imageFromCVPixelBuffer(pixelBuffer: imagePixelBuffer) else { fatalError() }
        
        CVPixelBufferUnlockBaseAddress(imagePixelBuffer,[])
        
        // process image in main threads
        self.isImgProcessing = true
        DispatchQueue.main.async {
 
            self.delegate?.imageDidCapture(uiImage, with:px, and: py)
            
            // clear processing flag
            self.dataOutputQueue.async {
                self.isImgProcessing = false
            }
        }
    }

Pass image to Image process methods in ViewController.swift:

    func imageDidCapture(_ uiImage: UIImage, with px: Float, and py: Float) {
        self.imageView.image = self.visp.detectAprilTag(uiImage, px:px, py:py)
    }

The camera’s intrinsic parameters can be acquired from each captured image by setting isCameraIntrinsicMatrixDeliveryEnabled to true in the connection settings in VideoCapture.swift:

        if videoConnection.isCameraIntrinsicMatrixDeliverySupported {
            // Enable Intrinsic parameter
            videoConnection.isCameraIntrinsicMatrixDeliveryEnabled = true
            print("Intrinsic Matrix is supported on this device :)" )
        } else {
            print("Intrinsic Matrix is NOT supported on this device :(" )
        }

Note: intrinsic parameters are only supported on some iOS devices with iOS11.

The intrinsic parameters that represent camera features can generally be represented by a matrix of pixel-based focal lengths and principal points (axis centers) in the image. The documentation for Swift is here. Since the principal point almost coincides with the image center, this tutorial uses only the focal length.

Call Objective-C class from Swift

Let us consider the Xcode project named AprilTagLiveCamera that is part of ViSP source code and located in $VISP_WS/tutorial/ios/AprilTagLiveCamera.

To open this application, if you followed Tutorial: Installation from prebuilt packages for iOS devices simply run:

$ cd $HOME/framework

Donwload the content of https://github.com/lagadic/visp/tree/master/tutorial/ios/AprilTagLiveCamera anr run

$ open AprilTagLiveCamera -a Xcode

or if you already downloaded ViSP following Tutorial: Installation from source for iOS devices run:

$ open $HOME/framework/visp/tutorial/ios/AprilTagLiveCamera -a Xcode

As described in Tutorial: AprilTag marker detection on iOS once opened, you have just to drag & drop ViSP and OpenCV frameworks available in $HOME/framework/ios if you followed Tutorial: Installation from prebuilt packages for iOS devices.

ViSP's AprilTag detection class is currently written in Objective-C. In order to use this Objective-C class from Swift, setting of Bridging-Header is necessary. Look for the "Objective-C Bridging Header" column in the "Build Settings", and make sure that "VispDetector.h", which describes the detector class this time, is set.

The content of VispDetector.h is the following:

#import <UIKit/UIKit.h>
 
@interface VispDetector : NSObject
- (UIImage *)detectAprilTag: (UIImage*)image px:(float)px py:(float)py;
@end
 

and in ViewController.swift you have the following code:

import AVFoundation
 
class ViewController: UIViewController, VideoCaptureDelegate {
    
    @IBOutlet weak var imageView: UIImageView!
    
    let videoCapture = VideoCapture()
    private let visp = VispDetector()
    
    override func viewDidLoad() {
        super.viewDidLoad()
        self.videoCapture.delegate = self
    }
    
    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        self.videoCapture.startCapturing()
    }
    
    // pass image to detector.
    //! [imageDidCapture]
    func imageDidCapture(_ uiImage: UIImage, with px: Float, and py: Float) {
        self.imageView.image = self.visp.detectAprilTag(uiImage, px:px, py:py)
    }
    //! [imageDidCapture]
    
    override func viewWillDisappear(_ animated: Bool) {
        super.viewWillDisappear(animated)
        self.videoCapture.stopCapturing()
    }
    
    override func didReceiveMemoryWarning() {
        super.didReceiveMemoryWarning()
        self.videoCapture.stopCapturing()
    }
    
}
 

Distance from camera

Detection and drawing processing is processed in VispDetector.mm. For details on the detection process, see Tutorial: AprilTag marker detection and on how to draw the pose of a tag, see the previous Tutorial: AprilTag marker detection on iOS.

The distance from the iOS device to the marker can be accurately detected if the tag size is properly set. The distance can be obtained using the getTranslationVector() method from the homogeneous transformation matrix (cMo_vec) representing the pose with rotation (R) and position (t) of the marker in camera coordinates. See here for more information vpHomogeneousMatrix class

This is achieved in VispDetector.mm:

for(int i=0; i < NumberOfTags; i++){
    vpTranslationVector trans = cMo_vec[i].getTranslationVector();
    float x = trans[0];
    float y = trans[1];
    float z = trans[2];
    std::cout << x << y << z << std::endl;
}

Application output

Like this image below, you can see the tag id, posture and position from the camera in real time with the video captured image.