MPAI-OSD V1.5 AIMs – Basic Multimodal Scene Description

Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Basic Multimodal Scene Description (OSD-BMS) AIM describes a multimodal scene from its Multimodal Object, its Basic Audio Object and its Basic Visual Object, and extracts from it the speech of the input, the Speech Object of the participant, the Face Object, the Face Descriptors and the Body Descriptors.

Receives	Space Time	Space Time of the input Objects.
	Multimodal Object	The acquired Multimodal Object.
	Basic Audio Object	The Basic Audio Object of the scene.
	Basic Visual Object	The Basic Visual Object of the scene.
Produces	Basic Multimodal Scene Descriptors	The description of the multimodal scene.
	Input Speech	The Speech Object of the input.
	Participant Speech Object	The Speech Object of the participant.
	Face Object	The Face Object extracted from the scene.
	Face Descriptors	The Face Descriptors of the participant.
	Body Descriptors	The Body Descriptors of the participant.

2 Reference Model

Figure 1 depicts the Reference Model of the Basic Multimodal Scene Description (OSD-BMS) AIM.

Basic Multimodal Scene Description OSD-BMS AIM

Figure 1 – Reference Model of the Basic Multimodal Scene Description (OSD-BMS) AIM

3 I/O Data

Table 1 specifies the Input and Output Data of the Basic Multimodal Scene Description (OSD-BMS) AIM. Links are to the Data Type specifications.

Table 1 – I/O Data of the Basic Multimodal Scene Description (OSD-BMS) AIM

Input	Data Type	Description
Space Time	Space Time	Space Time of the input Objects.
Multimodal Object	Multimodal Object	The acquired Multimodal Object.
Basic Audio Object	Basic Audio Object	The Basic Audio Object of the scene.
Basic Visual Object	Basic Visual Object	The Basic Visual Object of the scene.
Output	Data Type	Description
Basic Multimodal Scene Descriptors	Basic Multimodal Scene Descriptors	The description of the multimodal scene.
Input Speech	Basic Speech Object (Port Number 1)	The Speech Object of the input.
Participant Speech Object	Basic Speech Object (Port Number 2)	The Speech Object of the participant.
Face Object	Basic Visual Object	The Face Object extracted from the scene.
Face Descriptors	Face Descriptors Object	The Face Descriptors of the participant.
Body Descriptors	Body Descriptors Object	The Body Descriptors of the participant.

4 SubAIMs

No SubAIMs.

5 JSON Metadata

https://schemas.mpai.community/OSD/V1.5/AIMs/BasicMultimodalSceneDescription.json

6 Profiles

No Profiles.

7 Reference Software

Not part of this specification.

8 Conformance Testing

Table 2 provides the Conformance Testing Method for the OSD-BMS AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the OSD-BMS AIM

Receives	Space Time (Space Time)	Shall validate against the Space Time schema.
	Multimodal Object (Multimodal Object)	Shall validate against the Multimodal Object schema.
	Basic Audio Object (Basic Audio Object)	Shall validate against the Basic Audio Object schema.
	Basic Visual Object (Basic Visual Object)	Shall validate against the Basic Visual Object schema.
Produces	Basic Multimodal Scene Descriptors (Basic Multimodal Scene Descriptors)	Shall validate against the Basic Multimodal Scene Descriptors schema.
	Input Speech (Basic Speech Object)	Shall validate against the Basic Speech Object schema.
	Participant Speech Object (Basic Speech Object)	Shall validate against the Basic Speech Object schema.
	Face Object (Basic Visual Object)	Shall validate against the Basic Visual Object schema.
	Face Descriptors (Face Descriptors Object)	Shall validate against the Face Descriptors Object schema.
	Body Descriptors (Body Descriptors Object)	Shall validate against the Body Descriptors Object schema.

9 Performance Assessment