Interactive Segmenter

The MediaPipe Interactive Segmenter task lets you divide an image into two regions: a selected object and everything else. The task is stateful: You first set an image, and then you can efficiently perform segmentation multiple times by providing different strokes (points and brush mode) indicating a location in an image. The task estimates the boundaries of an object at that location, and returns image data defining the area of the object. You can use this task to interactively select an object in an image and use the output to apply effects to the image, such as color overlays highlighting the object or blurring the background around it. This task operates on image data with a machine learning (ML) model.

Open in Web DemosEdit in StackBlitz

Quickstart

Install the SDK for your platform, prepare input data, initialize the task with a model, and run inference.

Web

npmnpm install @mediapipe/tasks-visionWeb demo·Edit in StackBlitz·Setup guide
import { FilesetResolver, InteractiveSegmenter } from "@mediapipe/tasks-vision";

const MODEL_URL = "https://storage.googleapis.com/mediapipe-models/interactive_segmenter_v2/magic_touch/int8/latest/interactive_segmentation.task";

const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const interactiveSegmenter = await InteractiveSegmenter.createFromOptions(vision, {
  baseOptions: { modelAssetPath: MODEL_URL, delegate: "GPU" },
});

const image = document.getElementById("image") as HTMLImageElement;
interactiveSegmenter.setImage(image);

image.addEventListener("click", (event) => {
  const mask = interactiveSegmenter.segment([
    {
      brushMode: 1, // 1 = POSITIVE, 2 = NEGATIVE
      point: [{ x: event.offsetX / image.width, y: event.offsetY / image.height }],
      isCompleted: true,
    },
  ]);
  const maskData = mask.getAsFloat32Array();
  mask.close();
});

Android (Kotlin)

Gradleimplementation("com.google.mediapipe:tasks-vision:latest.release")Sample code·Setup guide
import com.google.mediapipe.framework.image.BitmapImageBuilder
import com.google.mediapipe.framework.image.ByteBufferExtractor
import com.google.mediapipe.framework.image.MPImage
import com.google.mediapipe.tasks.components.containers.NormalizedKeypoint
import com.google.mediapipe.tasks.core.BaseOptions
import com.google.mediapipe.tasks.vision.interactivesegmenter.InteractiveSegmenter
import com.google.mediapipe.tasks.vision.interactivesegmenter.InteractiveSegmenterOptions
import com.google.mediapipe.tasks.vision.interactivesegmenter.Stroke

val baseOptions = BaseOptions.builder()
    .setModelAssetPath("interactive_segmentation.task")
    .build()
val options = InteractiveSegmenterOptions.builder()
    .setBaseOptions(baseOptions)
    .build()

val interactiveSegmenter = InteractiveSegmenter.createFromOptions(context, options)
val mpImage = BitmapImageBuilder(bitmap).build()
interactiveSegmenter.setImage(mpImage)

val strokes = listOf(
    Stroke.builder()
        .setBrushMode(Stroke.BrushMode.POSITIVE)
        .setPoints(listOf(NormalizedKeypoint.create(0.2f, 0.5f), NormalizedKeypoint.create(0.5f, 0.5f)))
        .setCompleted(true)
        .build()
)

val mask: MPImage = interactiveSegmenter.segment(strokes)
val maskBuffer = ByteBufferExtractor.extract(mask).asFloatBuffer()
mask.close()

Android (Java)

Gradleimplementation 'com.google.mediapipe:tasks-vision:latest.release'Sample code·Setup guide
import com.google.mediapipe.framework.image.BitmapImageBuilder;
import com.google.mediapipe.framework.image.ByteBufferExtractor;
import com.google.mediapipe.framework.image.MPImage;
import java.nio.FloatBuffer;
import com.google.mediapipe.tasks.components.containers.NormalizedKeypoint;
import com.google.mediapipe.tasks.core.BaseOptions;
import com.google.mediapipe.tasks.vision.interactivesegmenter.InteractiveSegmenter;
import com.google.mediapipe.tasks.vision.interactivesegmenter.InteractiveSegmenterOptions;
import com.google.mediapipe.tasks.vision.interactivesegmenter.Stroke;
import java.util.Arrays;
import java.util.List;

BaseOptions baseOptions = BaseOptions.builder()
    .setModelAssetPath("interactive_segmentation.task")
    .build();
InteractiveSegmenterOptions options = InteractiveSegmenterOptions.builder()
    .setBaseOptions(baseOptions)
    .build();

InteractiveSegmenter interactiveSegmenter = InteractiveSegmenter.createFromOptions(context, options);
MPImage mpImage = new BitmapImageBuilder(bitmap).build();
interactiveSegmenter.setImage(mpImage);

List<Stroke> strokes = Arrays.asList(
    Stroke.builder()
        .setBrushMode(Stroke.BrushMode.POSITIVE)
        .setPoints(Arrays.asList(NormalizedKeypoint.create(0.2f, 0.5f), NormalizedKeypoint.create(0.5f, 0.5f)))
        .setCompleted(true)
        .build()
);

MPImage mask = interactiveSegmenter.segment(strokes);
FloatBuffer maskBuffer = ByteBufferExtractor.extract(mask).asFloatBuffer();
mask.close();

iOS

SwiftPMhttps://github.com/google-ai-edge/mediapipe → MediaPipeTasksVisionPackage.swift·Sample code·Setup guide
import CoreGraphics
import Foundation
import MediaPipeTasksVision

guard let modelPath = Bundle.main.path(forResource: "interactive_segmentation", ofType: "task") else { return }

let options = InteractiveSegmenterOptions()
options.baseOptions.modelAssetPath = modelPath

let interactiveSegmenter = try InteractiveSegmenter(options: options)
let mpImage = try MPImage(uiImage: image)
try interactiveSegmenter.set(image: mpImage)

let strokes = [
    Stroke(
        points: [NormalizedKeypoint(location: CGPoint(x: 0.5, y: 0.5), label: nil, score: 0.0)],
        brushMode: .positive,
        isCompleted: true
    )
]

let mask = try interactiveSegmenter.segment(strokes: strokes)

Python

pippip install mediapipeSample code·Setup guide
import mediapipe as mp

BaseOptions = mp.tasks.BaseOptions
InteractiveSegmenter = mp.tasks.vision.InteractiveSegmenter
InteractiveSegmenterOptions = mp.tasks.vision.InteractiveSegmenterOptions
BrushMode = mp.tasks.vision.InteractiveSegmenterBrushMode
Stroke = mp.tasks.vision.InteractiveSegmenterStroke
StrokePoint = mp.tasks.vision.InteractiveSegmenterStrokePoint

options = InteractiveSegmenterOptions(
    base_options=BaseOptions(model_asset_path="interactive_segmentation.task")
)

with InteractiveSegmenter.create_from_options(options) as segmenter:
    mp_image = mp.Image.create_from_file("input.jpg")
    segmenter.set_image(mp_image)

    strokes = [
        Stroke(brush_mode=BrushMode.POSITIVE, points=[StrokePoint(x=0.5, y=0.5)], is_completed=True)
    ]
    segmented_mask = segmenter.segment(strokes)

Task details

This section describes the capabilities, inputs, outputs, and configuration options of this task.

Features

  • Input image processing - Processing includes image rotation, resizing, normalization, and color space conversion.
Inputs
  • Strokes (points and brush mode) indicating an area of interest in an image
  • Image file to be processed

See Prepare vision input data for MPImage, mp.Image, and browser media examples.

Outputs
The Interactive Segmenter outputs a segmented mask with pixel values in float32 format. Each pixel value indicates the level of confidence that it is part of the object located at the area of interest.

Configuration options

Web

Option Name Description Value Range Default Value
baseOptions.modelAssetPath Path or URL to the interactive segmentation model bundle (.task). String Required
baseOptions.delegate Hardware acceleration delegate used to execute the model. {CPU, GPU} CPU

Android

Option Name Description Value Range Default Value
baseOptions.modelAssetPath Path or URL to the interactive segmentation model bundle (.task). String Required
baseOptions.delegate Hardware acceleration delegate used to execute the model. {CPU, GPU} CPU

iOS

Option Name Description Value Range Default Value
baseOptions.modelAssetPath Path or URL to the interactive segmentation model bundle (.task). String Required
baseOptions.delegate Hardware acceleration delegate used to execute the model. {CPU, GPU} CPU

Python

Option Name Description Value Range Default Value
base_options.model_asset_path Path or URL to the interactive segmentation model bundle (.task). String Required
base_options.delegate Hardware acceleration delegate used to execute the model. {CPU, GPU} CPU

Task output

Each segment(strokes) call returns a pixel-aligned float32 foreground confidence mask (MPMask on Web, MPImage on Android/iOS, or mp.Image in Python) for the object selected by the cumulative positive and negative Stroke inputs:

Field Type / Range Description
width int (pixels) Width of the output mask in pixels, matching the input image passed to setImage.
height int (pixels) Height of the output mask in pixels, matching the input image passed to setImage.
data Float32Array ([0.0, 1.0]) Row-major single-channel buffer of width × height float32 foreground probabilities (mask.getAsFloat32Array() on Web, ByteBufferExtractor.extract(mask).asFloatBuffer() on Android, or mask.numpy_view() in Python).

The following example shows the foreground confidence mask returned by segment(strokes):

{
  width: 640,
  height: 480,
  data: Float32Array(307200) [0.003, 0.004, 0.964, 0.981 /* ... foreground probability in [0.0, 1.0] per pixel */]
}

Models

Interactive Segmenter pipeline overview

The Interactive Segmenter can be used with more than one ML model. Start with the default, recommended model for your target platform when you start developing with this task. The other available models typically make trade-offs between performance, accuracy, resolution, and resource requirements, and in some cases, include additional features.

This model identifies segments given image coordinates for an area of interest. The model uses a Convolutional Neural Network, similar to a MobileNetV3 architecture, with a customized decoder.

Model name Input shape Quantization type Model Card Versions
MagicTouch 768 x 768 x 3 int8 info Latest

Task benchmarks

Here are the task benchmarks for the whole pipeline based on the preceding pre-trained models. The latency result is the average latency on Pixel 10 using CPU / GPU.

Model name Device CPU latency GPU latency
MagicTouch Pixel 10 208.2ms 580.6ms