Handle timed metadata in linear DAI streams

The Interactive Media Ads (IMA) Dynamic Ad Insertion (DAI) SDK relies on metadata information embedded in the stream's media segments (in-band metadata), or in the streaming manifest file (in-manifest metadata) to track viewers' positions and client-side ad events. Metadata is sent in different formats, depending on the type of stream being played.

The video player receives timed metadata in batches. Depending on the player, metadata can be surfaced at the scheduled time, or in batches. Each metadata string has an associated presentation timestamp (PTS) for when it should be triggered.

Your app is responsible for capturing metadata and forwarding it to the IMA DAI SDK. The SDK offers the following methods to pass this information:

onTimedMetadata

This method forwards metadata strings that are ready to be processed to the SDK. It takes a single argument:

metadata: an object containing a key of TXXX with an associated string value that is prefixed by google_.

processMetadata

This method schedules metadata strings to be processed by the SDK after the specified PTS. It takes the following arguments:

type: a string containing the type of event being processed. Accepted values are ID3 for HLS or urn:google:dai:2018 for DASH
data: either a string value prefixed by google_ or a byte array that follows this format ID3:u\0004u\000u\000u\0000TXXXu\0004u\000u\000u\0000google_xxxxxxxx.
timestamp: the timestamp in seconds when data should be processed.

Each stream type supported by the IMA DAI SDK uses a unique form of timed metadata, as described in the following sections.

HLS MPEG2TS streams

Linear DAI HLS streams using the MPEG2TS segments pass timed metadata to the video player through in-band ID3 tags. These ID3 tags are embedded within the MPEG2TS segments and are given the TXXX field name (for custom user-defined text content).

Playback in Safari

Safari processes ID3 tags automatically, as a hidden track, so cuechange events fire at the correct time to process each piece of metadata. It's alright to pass all metadata to the IMA DAI SDK, regardless of content or type. Irrelevant metadata is filtered out automatically.

Here's an example:

videoElement.textTracks.addEventListener('addtrack', (e) => {
  const track = e.track;
  if (track.kind === 'metadata') {
    track.mode = 'hidden';
    track.addEventListener('cuechange', () => {
      for (const cue of track.activeCues) {
        const metadata = {};
        metadata[cue.value.key] = cue.value.data;
        streamManager.onTimedMetadata(metadata);
      }
    });
  }
});
...

HLS.js

HLS.js provides ID3 tags in batches through the FRAG_PARSING_METADATA event, as an array of samples. HLS.js doesn't translate the ID3 data from byte arrays to strings and doesn't offset events to their corresponding PTS. It isn't necessary to decode the sample data from byte array to string, or to filter out irrelevant ID3 tags, as the IMA DAI SDK performs this decoding and filtering automatically.

Here's an example:

hls.on(Hls.Events.FRAG_PARSING_METADATA, (e, data) => {
  if (streamManager && data) {
    data.samples.forEach((sample) => {
      streamManager.processMetadata('ID3', sample.data, sample.pts);
    });
  }
});
...

HLS CMAF streams

Linear DAI HLS streams using the Common Media Application Framework (CMAF) pass timed metadata through in-band eMSGv1 boxes following the ID3 through CMAF standard. These eMSG boxes are embedded at the beginning of each media segment, with each ID3 eMSG containing a PTS relative to the last discontinuity in the stream.

As of the 1.2.0 release of HLS.js, both of our suggested players pass ID3 through CMAF to the user as if they were ID3 tags. For this reason, the following examples are the same as for HLS MPEG2TS streams. However, this might not be the case with all players, so implementing support for HLS CMAF streams could require unique code to parse ID3 through eMSG.

Playback in Safari

Safari treats ID3 through eMSG metadata as pseudo ID3 events, providing them in batches, automatically, as a hidden track, such that cuechange events are fired at the correct time to process each piece of metadata. It is alright to pass all metadata to the IMA DAI SDK, whether relevant to timing or not. Any non-DAI-related metadata are filtered out automatically.

Here's an example:

videoElement.textTracks.addEventListener('addtrack', (e) => {
  const track = e.track;
  if (track.kind === 'metadata') {
    track.mode = 'hidden';
    track.addEventListener('cuechange', () => {
      for (const cue of track.activeCues) {
        const metadata = {};
        metadata[cue.value.key] = cue.value.data;
        streamManager.onTimedMetadata(metadata);
      }
    });
  }
});
...

HLS.js

As of version 1.2.0, HLS.js treats ID3 through eMSG metadata as pseudo ID3 events, providing them in batches, through the FRAG_PARSING_METADATA event, as an array of samples. HLS.js does not translate the ID3 data from byte arrays to strings and does not offset events to their corresponding PTS. It isn't necessary to decode the sample data from byte array to string, as the IMA DAI SDK performs this decoding automatically.

Here's an example:

hls.on(Hls.Events.FRAG_PARSING_METADATA, (e, data) => {
  if (streamManager && data) {
    data.samples.forEach((sample) => {
      streamManager.processMetadata('ID3', sample.data, sample.pts);
    });
  }
});
...

DASH streams

Linear DAI DASH streams pass metadata as manifest events in an event stream with the custom schemeIdUri value urn:google:dai:2018. Each event in these streams contains a text payload, and the PTS.

DASH.js

Dash.js provides custom event handlers named after the schemeIdUri value of each event stream. These custom handlers fire in batches, leaving it up to you to process the PTS value to properly time the event. The IMA DAI SDK can handle this for you, with the streamManager method, processMetadata().

Here's an example:

const dash = dashjs.MediaPlayer().create();
dash.on('urn:google:dai:2018', (payload) => {
  const mediaId = payload.event.messageData;
  const pts = payload.event.calculatedPresentationTime;
  streamManager.processMetadata('urn:google:dai:2018', mediaId, pts);
});
...

Shaka Player

Shaka Player surfaces events as a part of their timelineregionenter event. Due to a formatting incompatibility with Shaka Player, the metadata value must be retrieved raw, through the detail property eventNode.attributes['messageData'].

Here's an example:

player.addEventListener('timelineregionenter', function(event) {
  const detail = event.detail;
  if ( detail.eventNode.attributes &&
       detail.eventNode.attributes['messageData']) {
    const mediaId = detail.eventNode.attributes['messageData'];
    const pts = detail.startTime;
    streamManager.processMetadata("urn:google:dai:2018", mediaId, pts);
  }
});
...

Pod serving

For Pod serving, there are different configurations for passing on timed metadata depending on the following criteria:

Live or VOD stream type
HLS or DASH stream format
The type of player being used
The type of DAI backend being used

HLS stream format (Live and VOD streams, HLS.js player)

If you are using an HLS.js player, listen to the HLS.js FRAG_PARSING_METADATA event to get ID3 metadata and pass it to the SDK with StreamManager.processMetadata().

To automatically play the video after everything is loaded and ready, listen to the HLS.js MANIFEST_PARSED event to trigger playback.

function loadStream(streamID) {
  hls.loadSource(url);
  hls.attachMedia(videoElement);
  
  // Timed metadata is passed HLS stream events to the streamManager.
  hls.on(Hls.Events.FRAG_PARSING_METADATA, parseID3Events);
  hls.on(Hls.Events.MANIFEST_PARSED, startPlayback);
}

function parseID3Events(event, data) {
  if (streamManager && data) {
    // For each ID3 tag in the metadata, pass in the type - ID3, the
    // tag data (a byte array), and the presentation timestamp (PTS).
    data.samples.forEach((sample) => {
      streamManager.processMetadata('ID3', sample.data, sample.pts);
    });
  }
}

function startPlayback() {
  console.log('Video Play');
  videoElement.play();
}

DASH.js (DASH streams format, Live and VOD stream type)

If you are using a DASH.js player, you have to use different strings to listen for ID3 metadata for Live or VOD streams:

Livestreams: 'https://developer.apple.com/streaming/emsg-id3'
VOD streams: 'urn:google:dai:2018'

Pass the ID3 metadata to the SDK with StreamManager.processMetadata().

To automatically show the video controls after everything is loaded and ready, listen to the DASH.js MANIFEST_LOADED event.

const googleLiveSchema = 'https://developer.apple.com/streaming/emsg-id3';
const googleVodSchema = 'urn:google:dai:2018';
dashPlayer.on(googleLiveSchema, processMetadata);
dashPlayer.on(googleVodSchema, processMetadata);
dashPlayer.on(dashjs.MediaPlayer.events.MANIFEST_LOADED, loadlistener);

function processMetadata(metadataEvent) {
  const messageData = metadataEvent.event.messageData;
  const timestamp = metadataEvent.event.calculatedPresentationTime;

  // Use StreamManager.processMetadata() if your video player provides raw
  // ID3 tags, as with dash.js.
  streamManager.processMetadata('ID3', messageData, timestamp);
}

function loadlistener() {
  showControls();

  // This listener must be removed, otherwise it triggers as addional
  // manifests are loaded. The manifest is loaded once for the content,
  // but additional manifests are loaded for upcoming ad breaks.
  dashPlayer.off(dashjs.MediaPlayer.events.MANIFEST_LOADED, loadlistener);
}

Shaka Player with livestreams (DASH streams format)

If you are using Shaka player for livestream playback, use the string 'emsg' to listen for metadata events. Then, use the event message data in your call to StreamManager.onTimedMetadata().

shakaPlayer.addEventListener('emsg', (event) => onEmsgEvent(event));

function onEmsgEvent(metadataEvent) {
  // Use StreamManager.onTimedMetadata() if your video player provides
  // processed metadata, as with Shaka player livestreams.
  streamManager.onTimedMetadata({'TXXX': metadataEvent.detail.messageData});
}

Shaka Player with VOD streams (DASH streams format)

If you are using Shaka player for VOD stream playback, use the string 'timelineregionenter' to listen for metadata events. Then, use the event message data in your call to StreamManager.processMetadata() with the string 'urn:google:dai:2018'.

shakaPlayer.addEventListener('timelineregionenter', (event) => onTimelineEvent(event));

function onTimelineEvent(metadataEvent) {
  const detail = metadataEvent.detail;
  if ( detail.eventElement.attributes &&
       detail.eventElement.attributes['messageData'] &&
       detail.eventElement.attributes['messageData'].value ) {
        const mediaId = detail.eventElement.attributes['messageData'].value;
        const pts = detail.startTime;
        // Use StreamManager.processMetadata() if your video player provides raw
        // ID3 tags, as with Shaka player VOD streams.
        streamManager.processMetadata('urn:google:dai:2018', mediaId, pts);
       }
}