Understand the user's environment on Unity's AR Foundation

Learn how to use the Scene Semantics API in your own apps.

The Scene Semantics API enables developers to understand the scene surrounding the user, by providing ML model-based, real-time semantic information. Given an image of an outdoor scene, the API returns a label for each pixel across a set of useful semantic classes, such a sky, building, tree, road, sidewalk, vehicle, person, and more. In addition to pixel labels, the Scene Semantics API also offers confidence values for each pixel label and an easy-to-use way to query the prevalence of a given label in an outdoor scene.

From left to right, examples of an input image, the semantic image of pixel labels, and the corresponding confidence image:

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Enable Scene Semantics

In a new ARCore session, check whether a user's device supports the Scene Semantics API. Not all ARCore-compatible devices support the Scene Semantics API due to processing power constraints.

To save resources, Scene Semantics is disabled by default on ARCore. Enable semantic mode to have your app use the Scene Semantics API.

In your ARCoreExtensionsConfig, set the Semantics Mode to Enabled.

If using iOS, Semantics must also be enabled in the project settings:

1. Navigate to Edit > Project Settings > XR Plug-In Management > ARCore Extensions.
2. Under Optional Features, select Semantics on iOS.

Obtain the semantic image

Once Scene Semantics is enabled, the semantic image can be retrieved. The semantic image is a TextureFormat.R8 image, where each pixel corresponds to a semantic label defined by SemanticLabel.

Use ArSemanticManager.TryGetSemanticTexture() to acquire the semantic image:

if (semanticManager.TryGetSemanticTexture(out Texture2D semanticImage))
{
    using (semanticImage)
    {
        // Use the semantic image here.
    }
}

Output semantic images should be available after about 1-3 frames from the start of the session, depending on the device.

Obtain the confidence image

In addition to the semantic image, which provides a label for each pixel, the API also provides a confidence image of corresponding pixel confidence values. The confidence image is a TextureFormat.Alpha8 image, where each pixel corresponds to a value in the range [0, 255], corresponding to the probability associated with the semantic label for each pixel.

Use ArSemanticManager.TryGetSemanticConfidenceTexture() to acquire the semantic confidence image:

if (semanticManager.TryGetSemanticConfidenceTexture(out Texture2D semanticConfidenceImage))
{
    using (semanticConfidenceImage)
    {
        // Use the semantic confidence image here.
    }
}

Output confidence images should be available after about 1-3 frames from the start of the session, depending on the device.

Query the fraction of pixels for a semantic label

You can also query the fraction of pixels in the current frame that belong to a particular class, such as sky. This query is more efficient than returning the semantic image and performing a pixel-wise search for a specific label. The returned fraction is a float value in the range [0.0, 1.0].

Use ArSemanticManager.GetSemanticLabelFraction() to acquire the fraction for a given label:

var fraction = semanticManager.GetSemanticLabelFraction(SemanticLabel.SKY);