Augmented Reality – Moving cars with virtual buttons

Virtual buttons with ARToolKit5 and Android NDK

I recently came across a very cool Augmented Reality video that demonstrates the use of virtual buttons using ARToolKit5 with Unity3D. Following this idea, I created an example that does something similar but uses the C classes of ARToolKit5 with the Android NDK directly.

We are going to create an app that moves a Ferrari Modena Spider and a Porsche 911 GT3 left and right using virtual buttons.

Getting started

You need to have some basic things up and running before you can follow this tutorial:

• Clone the ARToolKit5 GitHub repository
• Follow these instructions to get ARToolKit5 play nicely with the Android NDK
• Open the ARSimpleNativeCarsProj in Android Studio

Extending the sample

We use the ARSimpleNativeCarsProj example as a basis for your interactive Augmented Reality app.

The whole interaction idea is based on the absence of a marker in the camera stream. This means we implement an app that expects to find two specific markers in the camera stream. If one of these markers is missing we perform the interaction.

First, we add the new markers that we will use for the interactions. To do this we create an additional array that contains the new interaction markers.

Add the markers

#define NUM_MODELS 2
static ARModel models[NUM_MODELS] = {0};

#define NUM_INTERACTION_MODELS 2
static ARModel interactionModels[NUM_INTERACTION_MODELS] = {0};

Then we add our interaction markers to the ARToolKit framework at the end of the demoInitialize()  function.

//Add interaction markers
interactionModels[0].patternID = arwAddMarker("single;Data/left.patt;80");
arwSetMarkerOptionBool(interactionModels[0].patternID, ARW_MARKER_OPTION_SQUARE_USE_CONT_POSE_ESTIMATION, false);
arwSetMarkerOptionBool(interactionModels[0].patternID, ARW_MARKER_OPTION_FILTERED, true);
interactionModels[0].visible = false;

interactionModels[1].patternID = arwAddMarker("single;Data/right.patt;80");
arwSetMarkerOptionBool(interactionModels[1].patternID, ARW_MARKER_OPTION_SQUARE_USE_CONT_POSE_ESTIMATION, false);
arwSetMarkerOptionBool(interactionModels[1].patternID, ARW_MARKER_OPTION_FILTERED, true);
interactionModels[1].visible = false;

Create the virtual button

Now inside the demoDrawFrame function, we check if these interactionModels are visible or not.
If they are not visible we perform the interaction.

interactionModels[0].visible = arwQueryMarkerVisibility(interactionModels[0].patternID);
interactionModels[1].visible = arwQueryMarkerVisibility(interactionModels[1].patternID);

if(!interactionModels[0].visible) {
LOGE("Move Left id: %d", interactionModels[0].patternID);
moveLeft();
}

if(!interactionModels[1].visible) {
LOGE("Move Right id: %d", interactionModels[1].patternID);
moveRight();
}

Bring the virtual button to life

The interaction itself is fairly simple, we only modify the x-Coordinate of the model to move it to the left or to the right. We do this by saving the offset as an additional value on the model and add it to the transformation matrix prior to painting the model in the video stream.

Add up the offset:

void moveLeft(){
//Find out which model we need to modify
for (int i = 0; i < NUM_MODELS; i++) { models[i].visible = arwQueryMarkerTransformation(models[i].patternID, models[i].transformationMatrix); if (models[i].visible) { models[i].offset -= 1.0; } } }

void moveRight(){
//Find out which model we need to modify
for (int i = 0; i < NUM_MODELS; i++) { models[i].visible = arwQueryMarkerTransformation(models[i].patternID, models[i].transformationMatrix); if (models[i].visible) { models[i].offset += 1.0; } } }

Manipulate the transformation matrix:

if (models[i].visible) {
//Manipulate the x-Axis
models[i].transformationMatrix[12] += models[i].offset;
glLoadMatrixf(models[i].transformationMatrix);
}

Going forward

This is a simple example, how to get some interaction going using ARToolKit5 and the Android NDK. As this involves a basic understanding of programming and the C programming language a next step is to port this example to Unity3D to abstract it a bit from the programming part.

Additionally, it would be great if the markers used for the interaction would be smaller so they occupy less space.

Let me know your cool ideas and remarks about this example.

If you are interested in the whole functioning code you can get it from GitHub.