GestureRecognizer

public final class GestureRecognizer extends BaseVisionTaskApi

java.lang.Object
   ↳ com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi
     ↳ com.google.mediapipe.tasks.vision.gesturerecognizer.GestureRecognizer

Performs gesture recognition on images.

This API expects a pre-trained hand gesture model asset bundle, or a custom one created using Model Maker. See .

  • Input image MPImage
    • The image that gesture recognition runs on.
  • Output GestureRecognizerResult GestureRecognizerResult
    • A GestureRecognizerResult containing hand landmarks and recognized hand gestures.

Summary

Nested types

Options for setting up an GestureRecognizer.

Builder for GestureRecognizerOptions.

Public methods
static GestureRecognizer
createFromBuffer(Context context, ByteBuffer modelBuffer)

Creates a GestureRecognizer instance from a model buffer and the default GestureRecognizerOptions.
static GestureRecognizer
createFromFile(Context context, File modelFile)

Creates a GestureRecognizer instance from a model file and the default GestureRecognizerOptions.
static GestureRecognizer
createFromFile(Context context, String modelPath)

Creates a GestureRecognizer instance from a model file and the default GestureRecognizerOptions.
static GestureRecognizer
createFromOptions(
    Context context,
    GestureRecognizer.GestureRecognizerOptions recognizerOptions
)

Creates a GestureRecognizer instance from a GestureRecognizerOptions.
GestureRecognizerResult

Performs gesture recognition on the provided single image with default image processing options, i.e. without any rotation applied.
GestureRecognizerResult
recognize(MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs gesture recognition on the provided single image.
void
recognizeAsync(MPImage image, long timestampMs)

Sends live image data to perform gesture recognition with default image processing options, i.e. without any rotation applied, and the results will be available via the ResultListener provided in the GestureRecognizerOptions.
void
recognizeAsync(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

Sends live image data to perform gesture recognition, and the results will be available via the ResultListener provided in the GestureRecognizerOptions.
GestureRecognizerResult
recognizeForVideo(MPImage image, long timestampMs)

Performs gesture recognition on the provided video frame with default image processing options, i.e. without any rotation applied.
GestureRecognizerResult
recognizeForVideo(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

Performs gesture recognition on the provided video frame.

Inherited Constants
From com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi
static final long

Inherited fields
From com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi
final String
final String
final TaskRunner
final RunningMode

Inherited methods
From java.lang.AutoCloseable
abstract void
From com.google.mediapipe.tasks.vision.core.BaseVisionTaskApi
void

Closes and cleans up the MediaPipe vision task.
static RectProto.NormalizedRect
convertToNormalizedRect(
    ImageProcessingOptions imageProcessingOptions,
    MPImage image
)

Converts an ImageProcessingOptions instance into a NormalizedRect protobuf message.
static long

Generates the timestamp of a vision task result object from the vision task running mode and the output packet.
TaskResult
processImageData(Map<String, Packet> inputPackets)

A synchronous method to process single image inputs.
TaskResult
processImageData(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions
)

A synchronous method to process single image inputs.
TaskResult
processVideoData(Map<String, Packet> inputPackets, long timestampMs)

A synchronous method to process continuous video frames.
TaskResult
processVideoData(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

A synchronous method to process continuous video frames.
void
sendLiveStreamData(Map<String, Packet> inputPackets, long timestampMs)

An asynchronous method to send live stream data to the TaskRunner.
void
sendLiveStreamData(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

An asynchronous method to send live stream data to the TaskRunner.

Public methods

createFromBuffer

public static GestureRecognizer createFromBuffer(Context context, ByteBuffer modelBuffer)

Creates a GestureRecognizer instance from a model buffer and the default GestureRecognizerOptions.

Parameters
Context context

an Android Context.
ByteBuffer modelBuffer

a direct ByteBuffer or a MappedByteBuffer of the detection model.
Throws
MediaPipeException

if there is an error during GestureRecognizer creation.

createFromFile

public static GestureRecognizer createFromFile(Context context, File modelFile)

Creates a GestureRecognizer instance from a model file and the default GestureRecognizerOptions.

Parameters
Context context

an Android Context.
File modelFile

the gesture recognition model File instance.
Throws
java.io.IOException

if an I/O error occurs when opening the tflite model file.
MediaPipeException

if there is an error during GestureRecognizer creation.

createFromFile

public static GestureRecognizer createFromFile(Context context, String modelPath)

Creates a GestureRecognizer instance from a model file and the default GestureRecognizerOptions.

Parameters
Context context

an Android Context.
String modelPath

path to the gesture recognition model with metadata in the assets.
Throws
MediaPipeException

if there is an error during GestureRecognizer creation.

createFromOptions

public static GestureRecognizer createFromOptions(
    Context context,
    GestureRecognizer.GestureRecognizerOptions recognizerOptions
)

Creates a GestureRecognizer instance from a GestureRecognizerOptions.

Parameters
Context context

an Android Context.
GestureRecognizer.GestureRecognizerOptions recognizerOptions

a GestureRecognizerOptions instance.
Throws
MediaPipeException

if there is an error during GestureRecognizer creation.

recognize

public GestureRecognizerResult recognize(MPImage image)

Performs gesture recognition on the provided single image with default image processing options, i.e. without any rotation applied. Only use this method when the GestureRecognizer is created with RunningMode.IMAGE. TODO update java doc for input image format.

GestureRecognizer supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.
Throws
MediaPipeException

if there is an internal error.

recognize

public GestureRecognizerResult recognize(MPImage image, ImageProcessingOptions imageProcessingOptions)

Performs gesture recognition on the provided single image. Only use this method when the GestureRecognizer is created with RunningMode.IMAGE. TODO update java doc for input image format.

GestureRecognizer supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.
ImageProcessingOptions imageProcessingOptions

the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying regionOfInterest will result in this method throwing an IllegalArgumentException.
Throws
java.lang.IllegalArgumentException

if the ImageProcessingOptions specify a region-of-interest.
MediaPipeException

if there is an internal error.

recognizeAsync

public void recognizeAsync(MPImage image, long timestampMs)

Sends live image data to perform gesture recognition with default image processing options, i.e. without any rotation applied, and the results will be available via the ResultListener provided in the GestureRecognizerOptions. Only use this method when the GestureRecognition is created with RunningMode.LIVE_STREAM.

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the gesture recognizer. The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.
long timestampMs

the input timestamp (in milliseconds).
Throws
MediaPipeException

if there is an internal error.

recognizeAsync

public void recognizeAsync(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

Sends live image data to perform gesture recognition, and the results will be available via the ResultListener provided in the GestureRecognizerOptions. Only use this method when the GestureRecognition is created with RunningMode.LIVE_STREAM.

It's required to provide a timestamp (in milliseconds) to indicate when the input image is sent to the gesture recognizer. The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.
ImageProcessingOptions imageProcessingOptions

the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying regionOfInterest will result in this method throwing an IllegalArgumentException.
long timestampMs

the input timestamp (in milliseconds).
Throws
java.lang.IllegalArgumentException

if the ImageProcessingOptions specify a region-of-interest.
MediaPipeException

if there is an internal error.

recognizeForVideo

public GestureRecognizerResult recognizeForVideo(MPImage image, long timestampMs)

Performs gesture recognition on the provided video frame with default image processing options, i.e. without any rotation applied. Only use this method when the GestureRecognizer is created with RunningMode.VIDEO.

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.
long timestampMs

the input timestamp (in milliseconds).
Throws
MediaPipeException

if there is an internal error.

recognizeForVideo

public GestureRecognizerResult recognizeForVideo(
    MPImage image,
    ImageProcessingOptions imageProcessingOptions,
    long timestampMs
)

Performs gesture recognition on the provided video frame. Only use this method when the GestureRecognizer is created with RunningMode.VIDEO.

It's required to provide the video frame's timestamp (in milliseconds). The input timestamps must be monotonically increasing.

GestureRecognizer supports the following color space types:

  • Bitmap.Config.ARGB_8888
Parameters
MPImage image

a MediaPipe MPImage object for processing.
ImageProcessingOptions imageProcessingOptions

the ImageProcessingOptions specifying how to process the input image before running inference. Note that region-of-interest is not supported by this task: specifying regionOfInterest will result in this method throwing an IllegalArgumentException.
long timestampMs

the input timestamp (in milliseconds).
Throws
java.lang.IllegalArgumentException

if the ImageProcessingOptions specify a region-of-interest.
MediaPipeException

if there is an internal error.