AudioClassifier
class AudioClassifier: NSObjectClass that performs classification on audio data.
This API expects a TFLite model with mandatory TFLite Model Metadata that contains the mandatory AudioProperties of the solo input audio tensor and the optional (but recommended) category labels as AssociatedFiles with type TENSOR_AXIS_LABELS per output classification tensor.
Input tensor (kTfLiteFloat32) - input audio buffer of size [batch * samples]. - batch inference is not supported (batch is required to be 1). - for multi-channel models, the channels must be interleaved.
At least one output tensor with: (kTfLiteFloat32) - [1 x N] array with N represents the number of categories. - optional (but recommended) category labels as AssociatedFiles with type TENSOR_AXIS_LABELS, containing one label per line. The first such AssociatedFile (if any) is used to fill the category_name field of the results. The display_name field is filled from the AssociatedFile (if any) whose locale matches the display_names_locale field of the AudioClassifierOptions used at creation time ("en" by default, i.e. English). If none of these are available, only the index field of the results will be filled.
-
Creates a new instance of
AudioClassifierfrom an absolute path to a TensorFlow Lite model file stored locally on the device and the defaultAudioClassifierOptions.Declaration
Swift
convenience init(modelPath: String) throwsParameters
modelPathAn absolute path to a TensorFlow Lite model file stored locally on the device.
Return Value
A new instance of
AudioClassifierwith the given model path. This method throws an error if the audio classifier initialization fails. -
Creates a new instance of
AudioClassifierfrom the givenAudioClassifierOptions.Declaration
Swift
init(options: AudioClassifierOptions) throwsParameters
optionsThe options of type
AudioClassifierOptionsto use for configuring theAudioClassifier.Return Value
A new instance of
AudioClassifierwith the given options. This method throws an error if the audio classifier initialization fails. -
Performs classification on the provided audio clip. Only use this method when the
AudioClassifieris created with the.audioClipsmode.The audio clip is represented as a
AudioDataobject The method accepts audio clips with various lengths and audio sample rates. TheAudioDataobject must be initialized with anAudioDataFormatspecifying the sample rate and channel count of the audio clip.The input audio clip may be longer than what the model is able to process in a single inference. When this occurs, the input audio clip is split into multiple chunks starting at different timestamps. For this reason, the
AudioClassifierResultthis function returns consists of an array ofClassificationResultobjects, each associated with a timestamp corresponding to the start (in milliseconds) of the chunk data that was classified, e.g:ClassificationResult #0 (first chunk of data): timestamp_ms: 0 (starts at 0ms) classifications #0 (single head model): category #0: category_name: "Speech" score: 0.6 category #1: category_name: "Music" score: 0.2 ClassificationResult #1 (second chunk of data): timestamp_ms: 800 (starts at 800ms) classifications #0 (single head model): category #0: category_name: "Speech" score: 0.5 category #1: category_name: "Silence" score: 0.1
Declaration
Swift
func classify(audioClip: AudioData) throws -> AudioClassifierResultParameters
audioClipThe audio clip of type
AudioDataon which audio classification is to be performed.Return Value
An
AudioClassifierResultobject that contains a list of audio classifications. This method throws an error if audio classification on the audio block fails. -
Sends audio data (a block in a continuous audio stream) to perform audio classification. Only use this method when the
AudioClassifieris created with the.audioStreammode. This method is designed to process auido stream data such as the microphone input.This method will return immediately after the input audio data is accepted. The results are delivered asynchronously via delegation. The object which needs to be continuously notified of the available results of audio classification must confirm to
AudioClassifierLiveStreamDelegateprotocol and implement theaudioClassifier(_:didFinishClassificationWithResult:timestampInMilliseconds:error:)delegate method.The audio block is represented as a
AudioDataobject. The audio data will be resampled, accumulated, and framed to the proper size for the underlying model to consume. TheAudioDataobject must be initialized with anAudioDataFormatspecifying the sample rate and channel count of the audio stream as well as a timestamp (in milliseconds) to indicate the start time of the input audio block. The timestamps must be monotonically increasing. The input audio block may be longer than what the model is able to process in a single inference. When this occurs, the input audio block is split into multiple chunks. For this reason, the callback may be called multiple times (once per chunk) for each call to this function.Declaration
Swift
func classifyAsync(audioBlock: AudioData, timestampInMilliseconds: Int) throwsParameters
audioBlockA block of audio data of type
AudioDatain a continuous audio stream.timestampInMillisecondsThe timestamp (in milliseconds) which indicates the start time of the input audio block
Return Value
This method returns successfully if the audio block is submitted for classification. Otherwise, throws an error indicating the reason for failure.
-
Closes and cleans up the MediaPipe audio classifier.
For audio classifiers initialized with
.audioStreammode, ensure that this method is called after all audio blocks in an audio stream are sent for inference usingclassifyAsync(audioBlock:timestampInMilliseconds:). Otherwise, the audio classifier will not process the last audio block (of typeAudioData) in the stream if itsbufferLengthis shorter than the model's input length. Once an audio classifier is closed, you cannot send any inference requests to it. You must create a new instance ofAudioClassifierto send any pending requests. Ensure that you are ready to dispose off the audio classifier before this method is invoked.Declaration
Swift
func close() throwsReturn Value
Returns successfully if the task was closed. Otherwise, throws an error indicating the reason for failure.