Augmented Images developer guide for Android

Learn how to use Augmented Images in your own apps.

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Create an image database

Each image database can store information for up to 1,000 images.

There two ways to create an AugmentedImageDatabase:

• Load a saved image database. Then optionally add more reference images.
• Create a new empty database. Then add reference images one at a time.

Load a saved image database

Use AugmentedImageDatabase.deserialize() to load an existing image database:

AugmentedImageDatabase imageDatabase;
try (InputStream inputStream = this.getAssets().open("example.imgdb")) {
  imageDatabase = AugmentedImageDatabase.deserialize(session, inputStream);
} catch (IOException e) {
  // The Augmented Image database could not be deserialized; handle this error appropriately.
}

val imageDatabase = this.assets.open("example.imgdb").use {
  AugmentedImageDatabase.deserialize(session, it)
}

Image databases can be created using the arcoreimg command line tool during development, or by calling AugmentedImageDatabase.serialize() on a database that contains that is loaded in memory.

Create a new empty database

To create an empty image database at runtime, use the AugmentedImageDatabase constructor:

AugmentedImageDatabase imageDatabase = new AugmentedImageDatabase(session);

val imageDatabase = AugmentedImageDatabase(session)

Add images to an existing database

Add images to your image database by calling AugmentedImageDatabase.addImage() for each image, specifying an optional widthInMeters.

Bitmap bitmap;
try (InputStream bitmapString = getAssets().open("dog.jpg")) {
  bitmap = BitmapFactory.decodeStream(bitmapString);
} catch (IOException e) {
  // The bitmap could not be found in assets; handle this error appropriately.
  throw new AssertionError("The bitmap could not be found in assets.", e);
}

// If the physical size of the image is not known, use addImage(String, Bitmap) instead, at the
// expense of an increased image detection time.
float imageWidthInMeters = 0.10f; // 10 cm
int dogIndex = imageDatabase.addImage("dog", bitmap, imageWidthInMeters);

val bitmap = assets.open("dog.jpg").use { BitmapFactory.decodeStream(it) }
// If the physical size of the image is not known, use addImage(String, Bitmap) instead, at the
// expense of an increased image detection time.
val imageWidthInMeters = 0.10f // 10 cm
val dogIndex = imageDatabase.addImage("dog", bitmap, imageWidthInMeters)

The returned indexes can later be used to identify which reference image was detected.

Enable image tracking

Configure your ARCore session to begin tracking images by setting the session config to one that is configured with the desired image database:

Config config = new Config(session);
config.setAugmentedImageDatabase(imageDatabase);
session.configure(config);

val config = Config(session)
config.augmentedImageDatabase = imageDatabase
session.configure(config)

During the session, ARCore looks for images by matching feature points from the camera image against those in the image database.

To get the matched images, poll for updated AugmentedImages in your frame update loop.

Collection<AugmentedImage> updatedAugmentedImages =
    frame.getUpdatedTrackables(AugmentedImage.class);
for (AugmentedImage img : updatedAugmentedImages) {
  if (img.getTrackingState() == TrackingState.TRACKING) {
    // Use getTrackingMethod() to determine whether the image is currently
    // being tracked by the camera.
    switch (img.getTrackingMethod()) {
      case LAST_KNOWN_POSE:
        // The planar target is currently being tracked based on its last
        // known pose.
        break;
      case FULL_TRACKING:
        // The planar target is being tracked using the current camera image.
        break;
      case NOT_TRACKING:
        // The planar target isn't been tracked.
        break;
    }

    // You can also check which image this is based on img.getName().
    if (img.getIndex() == dogIndex) {
      // TODO: Render a 3D version of a dog in front of img.getCenterPose().
    } else if (img.getIndex() == catIndex) {
      // TODO: Render a 3D version of a cat in front of img.getCenterPose().
    }
  }
}

val updatedAugmentedImages = frame.getUpdatedTrackables(AugmentedImage::class.java)

for (img in updatedAugmentedImages) {
  if (img.trackingState == TrackingState.TRACKING) {
    // Use getTrackingMethod() to determine whether the image is currently
    // being tracked by the camera.
    when (img.trackingMethod) {
      AugmentedImage.TrackingMethod.LAST_KNOWN_POSE -> {
        // The planar target is currently being tracked based on its last known pose.
      }
      AugmentedImage.TrackingMethod.FULL_TRACKING -> {
        // The planar target is being tracked using the current camera image.
      }
      AugmentedImage.TrackingMethod.NOT_TRACKING -> {
        // The planar target isn't been tracked.
      }
    }

    // You can also check which image this is based on AugmentedImage.getName().
    when (img.index) {
      dogIndex -> TODO("Render a 3D version of a dog at img.getCenterPose()")
      catIndex -> TODO("Render a 3D version of a cat at img.getCenterPose()")
    }
  }
}

Supporting different use cases

When ARCore detects an Augmented Image, it creates a Trackable for that Augmented Image and sets TrackingState to TRACKING and TrackingMethod to FULL_TRACKING. When the tracked image moves out of camera view, ARCore changes the TrackingMethod to LAST_KNOWN_POSE while continuing to provide the orientation and position of the image.

Your app should use these enumerations differently depending on the intended use case.

• Fixed images. Most use cases involving images that are fixed in place (that is, not expected to move) can simply use TrackingState to determine whether the image has been detected and whether its location is known. TrackingMethod can be ignored.

• Moving images. If your app needs to track a moving image, use both TrackingState and TrackingMethod to determine whether the image has been detected and whether its position is known.

See also

• The Augmented Images sample projects in the ARCore SDK.

• The ARCore Augmented Images codelab.