<-AI Modules Go to ToC

Functions
Ref. Architecture
I/O Data
SubAIMs
JSON Metadata
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Television Media Analysis (OSD-TMA) Composite AIM analyses a television programme and produces the Descriptors of the Events of the programme:

Receives Basic Speech Object (OSD-BSO) The speech of the television programme to analyse. Shall validate against the Basic Speech Object schema.
Basic Visual Object (OSD-BVO) The video of the television programme to analyse. Shall validate against the Basic Visual Object schema.
Text Object (OSD-TXO) Auxiliary Text that helps the SubAIMs to improve their performance. Optional.
Produces Multimodal Event Descriptors (OSD-MEV) The analysis of the programme, in terms of Basic Visual Objects and Basic Speech Objects: the relevant Audio, Visual or Multimodal Objects, the IDs of speakers and the IDs of faces with their spatial positions, and the text of the utterances.

OSD-TMA assumes that there is only one speaker at a time. It replaces the Television Media Analysis AI Workflow of MPAI-OSD V1.4: in this version it is a Composite AIM, made of AIMs specified by MPAI-OSD V1.5 and by other standards.

2 Reference Model

Figure 1 depicts the Reference Model of the Television Media Analysis (OSD-TMA) AIM.

[Figure 1 – The Reference Model of the Television Media Analysis (OSD-TMA) AIM: to be drawn]

The boundary of the AIM has the Ports of Table 1, identified by Data Type.

3 I/O Data

Table 1 specifies the Input and Output Data of the Television Media Analysis (OSD-TMA) AIM.

Table 1 – I/O Data of the Television Media Analysis (OSD-TMA) AIM

Input Description
Basic Speech Object (OSD-BSO) The speech of the television programme.
Basic Visual Object (OSD-BVO) The video of the television programme.
Text Object (OSD-TXO) The Auxiliary Text. Optional.
Output Description
Multimodal Event Descriptors (OSD-MEV) The result of the analysis of the Speech, the Visual and the Auxiliary Text.

4 SubAIMs

4.1 Reference Model

Figure 2 depicts the Reference Model of the Television Media Analysis (OSD-TMA) Composite AIM.


Figure 2 – The Reference Model of the Television Media Analysis (OSD-TMA) Composite AIM

4.2 Operation

Processing proceeds asynchronously on the Basic Speech Object, the Basic Visual Object and the Text Object, for the entire duration of the programme:

  1. Visual Change Detection (CVE-VCD) finds the Video frames that convey a scene change and produces, for each, the first changed image, as a Basic Visual Object.
  2. Audio Segmentation (CAE-AUS) segments the speech into Speech Objects, assigns an anonymous Speaker ID to each speaker and produces the Speech Overlap information.
  3. Full to Basic Object (OSD-FBO) gives the Basic Speech Objects of the Speech Object, one by one, to the AIMs that follow.
  4. Face Identity Recognition (PAF-FIR) identifies the faces in the Images and the Text, and gives each a Face ID with its Bounding Box.
  5. Speaker Identity Recognition (MMC-SIR) identifies the speakers from the Speech Objects, the Speech Overlap and the Text, and gives each a Speaker ID.
  6. Automatic Speech Recognition (MMC-ASR) transcribes the speech, taking into account the Text.
  7. Identity Reconciliation (OSD-IDR) reconciles the Face ID and the Speaker ID.
  8. Basic Speech Scene Description (OSD-BSS) and Basic Visual Scene Description (OSD-BVS) produce the Basic Scene Descriptors of the speech and of the images. Multimodal Scene Multiplexing (OSD-SMX) combines them, with the Recognised Text and the User ID, into the Basic Multimodal Scene Descriptors, and Multimodal Event Description (OSD-MED) produces from them the Multimodal Event Descriptors that leave the AIM.

4.3 Functions of SubAIMs

Table 2 provides the functions of the Television Media Analysis (OSD-TMA) SubAIMs. The AIMs specified by other standards are referenced by OSD-TMA: their specification is in those standards.

Table 2 – Functions of the Television Media Analysis (OSD-TMA) SubAIMs

SubAIM Function
Visual Change Detection (CVE-VCD) Detects scene cuts and transitions in the Video and produces the first changed image of each scene, as a Basic Visual Object. Specified by MPAI-CVE V1.0.
Audio Segmentation (CAE-AUS) Finds the speech segments and the speakers, produces Speech Objects with the Speaker Time and the Speech Overlap. Specified by CAE-ASM.
Full to Basic Object (OSD-FBO) Gives access to the Basic Objects of a Full Object. Specified by OSD.
Face Identity Recognition (PAF-FIR) Identifies the faces in the Images with the help of the Text; produces Face ID and Bounding Box. Specified by MPAI-PAF V1.6.
Speaker Identity Recognition (MMC-SIR) Identifies the speakers of the Speech Objects with the help of the Text; produces the Speaker ID. Specified by MPAI-MMC V2.5.
Automatic Speech Recognition (MMC-ASR) Transcribes the speech, with time stamps, taking the Text into account. Specified by MPAI-MMC V2.5.
Identity Reconciliation (OSD-IDR) Reconciles the Face ID and the Speaker ID into one User ID. Specified by OSD.
Basic Speech Scene Description (OSD-BSS) Produces the Basic Speech Scene Descriptors from the Basic Speech Objects. Specified by OSD.
Basic Visual Scene Description (OSD-BVS) Produces the Basic Visual Scene Descriptors from the Basic Visual Objects. Specified by OSD.
Multimodal Scene Multiplexing (OSD-SMX) Combines the Scene Descriptors, the Text and the identifiers into the Basic Multimodal Scene Descriptors. Specified by OSD.
Multimodal Event Description (OSD-MED) Produces the Multimodal Event Descriptors from the Basic Multimodal Scene Descriptors. Specified by OSD.

5 JSON Metadata

https://schemas.mpai.community/OSD/V1.5/AIMs/TelevisionMediaAnalysis.json

6 Profiles

No Profiles.

7 Reference Software

Under development.

8 Conformance Testing

Table 3 provides the Conformance Testing Method for the Television Media Analysis (OSD-TMA) AIM. Conformance Testing of the individual SubAIMs is given by the individual AIM specifications.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 3 – Conformance Testing Method for the Television Media Analysis (OSD-TMA) AIM

Receives Multimodal Object (OSD-MMO) Shall validate against the Multimodal Object schema.
Text Object (OSD-TXO) Shall validate against the Text Object schema.
Produces Multimodal Event Descriptors (OSD-MEV) Shall validate against the Multimodal Event Descriptors schema.

9 Performance Assessment

Not part of this specification.