MediaPipeTasksAudio Framework Reference

AudioClassifier

class AudioClassifier: NSObject

Class that performs classification on audio data.

This API expects a TFLite model with mandatory TFLite Model Metadata that contains the mandatory AudioProperties of the solo input audio tensor and the optional (but recommended) category labels as AssociatedFiles with type TENSOR_AXIS_LABELS per output classification tensor.

Input tensor (kTfLiteFloat32) - input audio buffer of size [batch * samples]. - batch inference is not supported (batch is required to be 1). - for multi-channel models, the channels must be interleaved.

At least one output tensor with: (kTfLiteFloat32) - [1 x N] array with N represents the number of categories. - optional (but recommended) category labels as AssociatedFiles with type TENSOR_AXIS_LABELS, containing one label per line. The first such AssociatedFile (if any) is used to fill the category_name field of the results. The display_name field is filled from the AssociatedFile (if any) whose locale matches the display_names_locale field of the AudioClassifierOptions used at creation time ("en" by default, i.e. English). If none of these are available, only the index field of the results will be filled.

  • Creates a new instance of AudioClassifier from an absolute path to a TensorFlow Lite model file stored locally on the device and the default AudioClassifierOptions.

    Declaration

    Swift

    convenience init(modelPath: String) throws

    Parameters

    modelPath

    An absolute path to a TensorFlow Lite model file stored locally on the device.

    Return Value

    A new instance of AudioClassifier with the given model path. This method throws an error if the audio classifier initialization fails.

  • Creates a new instance of AudioClassifier from the given AudioClassifierOptions.

    Declaration

    Swift

    init(options: AudioClassifierOptions) throws

    Parameters

    options

    The options of type AudioClassifierOptions to use for configuring the AudioClassifier.

    Return Value

    A new instance of AudioClassifier with the given options. This method throws an error if the audio classifier initialization fails.

  • Performs classification on the provided audio clip. Only use this method when the AudioClassifier is created with the .audioClips mode.

    The audio clip is represented as a AudioData object The method accepts audio clips with various lengths and audio sample rates. The AudioData object must be initialized with an AudioDataFormat specifying the sample rate and channel count of the audio clip.

    The input audio clip may be longer than what the model is able to process in a single inference. When this occurs, the input audio clip is split into multiple chunks starting at different timestamps. For this reason, the AudioClassifierResult this function returns consists of an array of ClassificationResult objects, each associated with a timestamp corresponding to the start (in milliseconds) of the chunk data that was classified, e.g:

    ClassificationResult #0 (first chunk of data): timestamp_ms: 0 (starts at 0ms) classifications #0 (single head model): category #0: category_name: "Speech" score: 0.6 category #1: category_name: "Music" score: 0.2 ClassificationResult #1 (second chunk of data): timestamp_ms: 800 (starts at 800ms) classifications #0 (single head model): category #0: category_name: "Speech" score: 0.5 category #1: category_name: "Silence" score: 0.1

    Declaration

    Swift

    func classify(audioClip: AudioData) throws -> AudioClassifierResult

    Parameters

    audioClip

    The audio clip of type AudioData on which audio classification is to be performed.

    Return Value

    An AudioClassifierResult object that contains a list of audio classifications. This method throws an error if audio classification on the audio block fails.

  • Sends audio data (a block in a continuous audio stream) to perform audio classification. Only use this method when the AudioClassifier is created with the .audioStream mode. This method is designed to process auido stream data such as the microphone input.

    This method will return immediately after the input audio data is accepted. The results are delivered asynchronously via delegation. The object which needs to be continuously notified of the available results of audio classification must confirm to AudioClassifierLiveStreamDelegate protocol and implement the audioClassifier(_:didFinishClassificationWithResult:timestampInMilliseconds:error:) delegate method.

    The audio block is represented as a AudioData object. The audio data will be resampled, accumulated, and framed to the proper size for the underlying model to consume. The AudioData object must be initialized with an AudioDataFormat specifying the sample rate and channel count of the audio stream as well as a timestamp (in milliseconds) to indicate the start time of the input audio block. The timestamps must be monotonically increasing. The input audio block may be longer than what the model is able to process in a single inference. When this occurs, the input audio block is split into multiple chunks. For this reason, the callback may be called multiple times (once per chunk) for each call to this function.

    Declaration

    Swift

    func classifyAsync(audioBlock: AudioData, timestampInMilliseconds: Int) throws

    Parameters

    audioBlock

    A block of audio data of type AudioData in a continuous audio stream.

    timestampInMilliseconds

    The timestamp (in milliseconds) which indicates the start time of the input audio block

    Return Value

    This method returns successfully if the audio block is submitted for classification. Otherwise, throws an error indicating the reason for failure.

  • Closes and cleans up the MediaPipe audio classifier.

    For audio classifiers initialized with .audioStream mode, ensure that this method is called after all audio blocks in an audio stream are sent for inference using classifyAsync(audioBlock:timestampInMilliseconds:). Otherwise, the audio classifier will not process the last audio block (of type AudioData) in the stream if its bufferLength is shorter than the model's input length. Once an audio classifier is closed, you cannot send any inference requests to it. You must create a new instance of AudioClassifier to send any pending requests. Ensure that you are ready to dispose off the audio classifier before this method is invoked.

    Declaration

    Swift

    func close() throws

    Return Value

    Returns successfully if the task was closed. Otherwise, throws an error indicating the reason for failure.