AudioClassifier

public final class AudioClassifier extends BaseAudioTaskApi

java.lang.Object
   ↳ com.google.mediapipe.tasks.audio.core.BaseAudioTaskApi
     ↳ com.google.mediapipe.tasks.audio.audioclassifier.AudioClassifier

Performs audio classification on audio clips or audio stream.

This API expects a TFLite model with mandatory TFLite Model Metadata that contains the mandatory AudioProperties of the solo input audio tensor and the optional (but recommended) label items as AssociatedFiles with type TENSOR_AXIS_LABELS per output classification tensor.

Input tensor: (kTfLiteFloat32)

  • input audio buffer of size `[batch * samples]`.
  • batch inference is not supported (`batch` is required to be 1).
  • for multi-channel models, the channels need be interleaved.

At least one output tensor with: (kTfLiteFloat32)

  • `[1 x N]` array with `N` represents the number of categories.
  • optional (but recommended) label items as AssociatedFiles with type TENSOR_AXIS_LABELS, containing one label per line. The first such AssociatedFile (if any) is used to fill the `category_name` field of the results. The `display_name` field is filled from the AssociatedFile (if any) whose locale matches the `display_names_locale` field of the `AudioClassifierOptions` used at creation time ("en" by default, i.e. English). If none of these are available, only the `index` field of the results will be filled.

Summary

Nested types

Options for setting up and AudioClassifier.

Builder for AudioClassifierOptions.

Public methods
AudioClassifierResult
classify(AudioData audioClip)
void
classifyAsync(AudioData audioBlock, long timestampMs)
static AudioClassifier
createFromBuffer(Context context, ByteBuffer modelBuffer)

Creates an AudioClassifier instance from a model buffer and default AudioClassifierOptions.
static AudioClassifier
createFromFile(Context context, File modelFile)

Creates an AudioClassifier instance from a model file and default AudioClassifierOptions.
static AudioClassifier
createFromFile(Context context, String modelPath)

Creates an AudioClassifier instance from a model file and default AudioClassifierOptions.
static AudioClassifier

Creates an AudioClassifier instance from an AudioClassifierOptions instance.

Inherited methods
From java.lang.AutoCloseable
abstract void
From com.google.mediapipe.tasks.audio.core.BaseAudioTaskApi
void
checkOrSetSampleRate(double sampleRate)

Checks or sets the audio sample rate in the audio stream mode.
void

Closes and cleans up the MediaPipe audio task.
AudioRecord

Creates an android.media.AudioRecord instance to record audio stream that has mono channel at sample rate at sample rate 16kHz, the sample rate required for models like Yamnet.
AudioRecord
createAudioRecord(
    int numChannels,
    int sampleRate,
    int requiredInputBufferSize
)

Creates an android.media.AudioRecord instance to record audio stream.
TaskResult

A synchronous method to process audio clips.
void
sendAudioStreamData(AudioData audioClip, long timestampMs)

An asynchronous method to send audio stream data to the TaskRunner.

Public methods

classify

public AudioClassifierResult classify(AudioData audioClip)

classifyAsync

public void classifyAsync(AudioData audioBlock, long timestampMs)

createFromBuffer

public static AudioClassifier createFromBuffer(Context context, ByteBuffer modelBuffer)

Creates an AudioClassifier instance from a model buffer and default AudioClassifierOptions.

Parameters
Context context

an Android Context.
ByteBuffer modelBuffer

a direct ByteBuffer or a MappedByteBuffer of the classification model.
Throws
com.google.mediapipe.framework.MediaPipeException

if there is an error during AudioClassifier creation.

createFromFile

public static AudioClassifier createFromFile(Context context, File modelFile)

Creates an AudioClassifier instance from a model file and default AudioClassifierOptions.

Parameters
Context context

an Android Context.
File modelFile

the classification model File instance.
Throws
java.io.IOException

if an I/O error occurs when opening the tflite model file.
com.google.mediapipe.framework.MediaPipeException

if there is an error during AudioClassifier creation.

createFromFile

public static AudioClassifier createFromFile(Context context, String modelPath)

Creates an AudioClassifier instance from a model file and default AudioClassifierOptions.

Parameters
Context context

an Android Context.
String modelPath

path to the classification model in the assets.
Throws
com.google.mediapipe.framework.MediaPipeException

if there is an error during AudioClassifier creation.

createFromOptions

public static AudioClassifier createFromOptions(
    Context context,
    AudioClassifier.AudioClassifierOptions options
)

Creates an AudioClassifier instance from an AudioClassifierOptions instance.

Parameters
Context context

an Android Context.
AudioClassifier.AudioClassifierOptions options

an AudioClassifierOptions instance.
Throws
com.google.mediapipe.framework.MediaPipeException

if there is an error during AudioClassifier creation.