Control a robot with Augmented Reality

How to control a robot with OpenSpace3D

In this tutorial you will learn how you can build an interactive Augmented Reality (AR) app using OpenSpace3D. With their latest release OpenSpace3D included the ARToolKit5 library.

Components for this tutorial

  • OpenSpace3D (on Windows)
    • I used the portable version which is available here.
  • These images which will be used as NFT markers
robot
computer
girlbot
  • And as usual for an Augmented Reality app you need a camera

Setup the model

We are going to start OpenSpace3D and load the supplied robot model.

  1. Start OpenSpace3D
  2. Add the robot model (Andy) to the scene.
    • Right click on the scene root
    • Add mesh
    • Select next
    • Select the robot mesh: “OS3D\_root”\OpenSpace3D\assets\models\common\andy\mesh\andyv2_andy.mesh
    • Select next
    • Select *Add script* and select “OS3D\_root”\OpenSpace3D\assets\models\common\andy\material\andyv2.material -> Open
    • Import

Now you have Andy in. If you select it in the Scene tree you get a little ‘+’ icon next to it which let’s you explore the model and all it’s animations.

You can play the available animations by right click on one of them and select Edit setting. In the following dialog you can see a play icon in the lower left corner. That lets you play the selected animation. You can also select to loop that animation if needed.
You can play the available animations by right click on one of then and select Edit setting. In the following dialog you can see a play icon in the lower left corner. That lets you play the selected animation. You can also select to loop that animation if needed.
You can play the available animations by right click on one of then and select Edit setting. In the following dialog you can see a play icon in the lower left corner. That lets you play the selected animation. You can also select to loop that animation if needed.
You can play the available animations by right click on one of then and select *Edit setting*. In the following dialog you can see a play icon in the lower left corner. That lets you play the selected animation. You can also select to loop that animation if needed.
  1. Add a separate root node to the scene for the Andy model.Right click on the Screen root
    • Select Add dummy
    • Name the dummy ARAndy_dummy
  2. Drag the andy model onto the dummy

Scenetree overview

 

Activate the material

Good, now we have the basic scene. But, our model is not looking very good. We need to enable the material to make it look good.

  1. Right click the Scene root
  2. Set material scheme
  3. In the dialog select basic_mat

Now Andy should appear with a white and blue material with the OpenSpace3D logo on his chest.

Light and shadow

The scene is still a bit dark and also it looks much more realistic if we enable shadows. So let’s just do that.

  1. Right click on ARAndy_dummy
  2. Add light -> Ok
  3. In the dialog ensure you select Cast shadow
  4. Also make your light a direct light.

Now we add a plane to which the shadow is projected onto and enable the shadow mode.

  1. Right click on ARAndy_dummy
  2. Add Mesh -> Next
  3. Select “OS3D_root”\OpenSpace3D\assets\primitives\prim_plane.mash -> Next -> Import
  4. Right click on Scene root / Add resources to scene
  5. Select “OS3D_root”\OpenSpace3D\assets\materials\transshadow.material -> Open
  6. Right click the plane you just added -> Edit materials
  7. For material name select ASSET/TransShadow
  8. Right click on Scene root -> Set shadow method
  9. For Preset method select presetPSSM
That’s good Andy now produces a shadow onto the plane which makes the whole scene much more realistic.
But uhm, Andy is somehow cut in half by the plane. We need to move him up along the y-axis 0.9 units. Also the light is still placed at the center of the scene. Move and rotate it till you are satisfied with the light and the corresponding shadow on the plane.
Andy the robot with a shadow

Augment it

If we run the scene now you can see that Andy is doing some work and that it has a shadow.

You can run the scene using the little play icon at the top left corner of the scene view.

Add the AR camera

Now we want Andy to be augmented onto the robot image.

  1. At the bottom inside the groups tab double click Scene
  2. In the scene tab right click/input/AR capture -> OK
You can configure the settings to your liking. For now the default settings will do.

Add the marker

If you play the scene now you see that Andy is rendered in front of the camera stream.

Now we need to place Andy on a marker:

  1. Right click in the scene tab at the bottom
  2. Input / Add marker
  3. For Bitmap path select the image of the robot. This starts the marker training which may take a bit.
  4. Marker size needs to be changed to 1 or bigger otherwise Andy will be really large. Play around with this setting till Andy appears to your liking. (I set it to 2)
  5. In Object name select the ARAndy-dummy node.

Ok, now Andy is placed on top of the marker when the marker is shown. However, it is also rendered when the marker is not shown which is kind of disturbing. So let’s fix that.

  1. Right click in the scene tab at the bottom
  2. Select object/hide
  3. Set Source name to ARAndy_dummy
  4. Select Hide on start

Now Andy is no longer rendered when we play the scene. To make Andy appear on the marker do the following:

  1. Right click AR marker inst
  2. Select found: Now a line is shown starting at AR marker inst and following your mouse pointer
  3. Right click on hide inst and select show

This makes Andy appear as soon as the marker is detected. Do the same for Lost to hide Andy when the marker is not visible any more.

A video, demonstrating the above steps can be viewed here.

Control Andy

Now that you can show and hide Andy based on the robot marker the next step is to show him what to do based on the other markers.

  1. Add three more markers for girlbot, beerbrake and computer.
  2. Right click in the scene tab at the bottom
  3. Select object/animation switcher
  4. Add the animation you would like to be played for each of the marker (robogirl, beerbrake, computer) to the Animations list.
  5. Right click on one of the new markers. Select found, right click on the animation switcher and select the animation you would like to play when the selected marker is found. Be aware that one animation is set to default so you won’t see the name of this animation in the list.
  6. Repeat this step for all of your markers
  7. Don’t forget to select some animation (or non) when the marker isn’t visible any more (lost)
For each chosen animation it might be good to loop it so that the animation still plays while the marker is shown to Andy.

You can see my final app here:

Conclusion

You have learned how you can setup an augmented reality app using OpenSpace 3D. Additionally this tutorial gives you some background how you can use markers to control 3D objects like our Andy and tell a nice story with it.
Try to think this example further and make real objects react on images shown to them (or to a camera). Maybe using a Raspberry PI or an Arduino.

I’d love to hear from you and see your stories and apps. Let’s keep in contact on Twitter (https://twitter.com/artoolkit_) or @Thor_Bux. Also for questions and feedback please leave a comment below.

2 Comments

  1. homeschooling March 21, 2017 Reply
  2. school education March 21, 2017 Reply

Leave a Reply