Go to MPAI-OSD V1.5 Data Types

Definition
Functional Requirements
Syntax
Semantics
Conformance Testing
Performance Assessment

1      Definition

Multimodal Scene Descriptors are a Data Type including the Multimodal Objects of a scene, their Sub-Scenes, and their arrangement in the Scene. Multimodal Scene Descriptors may be hierarchical, i.e., they may contain Objects and Multimodal Scene Descriptors.

2      Functional Requirements

An Multimodal Scene Descriptors instance must include:

  1. The Multimodal Scene Descriptors Header.
  2. The ID of the Multimodal Scene Descriptors instance.
  3. The Space/Time of the Multimodal Scene Descriptors instance.

An Multimodal Scene Descriptors instance may include:

  1. The ID of a Virtual Space (M-Instance) where it is or is intended to be located.
  2. The ID of a U-Environment (Real Space) where it is or is intended to be located.
  3. Time when this instance was produced.
  4. Audio, Speech, Visual, 3D Model, and Multimodal Objects in the Multimodal Scene, each with their Space/Time.
  5. Audio, Speech, Visual, 3D Model, and Multimodal Sub-Scenes in the Multimodal Scene, each with their Space/Time.
  6. A mixed array of Basic Multimodal Scene Descriptors and Multimodal Scene Descriptors.
  7. MPAI-PTF Data Exchange Metadata.
  8. Descriptive Metadata.

3      Syntax

https://schemas.mpai.community/OSD/V1.5/data/MultimodalSceneDescriptors.json

4      Semantics

Table 1 – Semantics of the Multimodal Scene Descriptors Data Type

Label
Description
Header Multimodal Scene Descriptors Header – Standard “OSD‑MSD‑Vx.y”.
MInstanceID Identifier of the M-Instance (Virtual Space) where this Multimodal Scene is or is intended to be located.
UEnvironmentID Identifier of the U-Environment (Real Space) where this Multimodal Scene is or is intended to be located.
MultimodalSceneDescriptorsID Unique identifier of this Multimodal Scene Descriptors instance.
MultimodalSceneDescriptorsTime Time this Multimodal Scene Descriptors instance was produced.
MultimodalSceneDescriptorsSpaceTime Space/Time where/when this Multimodal Scene Descriptors instance is located.
AudioObjectCount Number of Audio Objects in the Multimodal Scene.
AudioObjects[] Set of Audio Objects in the Multimodal Scene.
    – AudioObjectSpaceTime Space/Time where/when this Audio Object is located within the Multimodal Scene.
    – AudioObject Either an Audio Object or the ID of an Audio Object.
SpeechObjectCount Number of Speech Objects in the Multimodal Scene.
SpeechObjects[] Set of Speech Objects in the Multimodal Scene.
    – SpeechObjectSpaceTime Space/Time where/when this Speech Object is located within the Multimodal Scene.
    – SpeechObject Either a Speech Object or the ID of a Speech Object.
VisualObjectCount Number of Visual Objects in the Multimodal Scene.
VisualObjects[] Set of Visual Objects in the Multimodal Scene.
    – VisualObjectSpaceTime Space/Time where/when this Visual Object is located within the Multimodal Scene.
    – VisualObject Either a Visual Object or the ID of a Visual Object.
3DModelObjectCount Number of 3D Model Objects in the Multimodal Scene.
3DModelObjects[] Set of 3D Model Objects in the Multimodal Scene.
    – 3DModelObjectSpaceTime Space/Time where/when this 3D Model Object is located within the Multimodal Scene.
    – 3DModelObject Either a 3D Model Object or the ID of a 3D Model Object.
MultimodalObjectCount Number of Multimodal Objects in the Multimodal Scene.
MultimodalObjects[] Set of Multimodal Objects in the Multimodal Scene.
    – MultimodalObjectSpaceTime Space/Time where/when this Multimodal Object is located within the Multimodal Scene.
    – MultimodalObject Either a Multimodal Object or the ID of a Multimodal Object.
SubAudioSceneCount Number of Audio Sub-Scenes in the Multimodal Scene.
SubAudioScenes[] Set of Audio Sub-Scenes in the Multimodal Scene.
    – SubAudioSceneSpaceTime Space/Time where/when this Audio Sub-Scene is located within the Multimodal Scene.
    – SubAudioScene Either an Audio Scene Descriptors instance or its ID.
SubSpeechSceneCount Number of Speech Sub-Scenes in the Multimodal Scene.
SubSpeechScenes[] Set of Speech Sub-Scenes in the Multimodal Scene.
    – SubSpeechSceneSpaceTime Space/Time where/when this Speech Sub-Scene is located within the Multimodal Scene.
    – SubSpeechScene Either a Speech Scene Descriptors instance or its ID.
SubVisualSceneCount Number of Visual Sub-Scenes in the Multimodal Scene.
SubVisualScenes[] Set of Visual Sub-Scenes in the Multimodal Scene.
    – SubVisualSceneSpaceTime Space/Time where/when this Visual Sub-Scene is located within the Multimodal Scene.
    – SubVisualScene Either a Visual Scene Descriptors instance or its ID.
Sub3DModelSceneCount Number of 3D Model Sub-Scenes in the Multimodal Scene.
Sub3DModelScenes[] Set of 3D Model Sub-Scenes in the Multimodal Scene.
    – Sub3DModelSceneSpaceTime Space/Time where/when this 3D Model Sub-Scene is located within the Multimodal Scene.
    – Sub3DModelScene Either a 3D Model Scene Descriptors instance or its ID.
SubMultimodalSceneCount Number of Multimodal Sub-Scenes in the Multimodal Scene.
SubMultimodalScenes[] Set of Multimodal Sub-Scenes in the Multimodal Scene.
    – SubMultimodalSceneSpaceTime Space/Time where/when this Multimodal Sub-Scene is located within the Multimodal Scene.
    – SubMultimodalScene Either a Multimodal Scene Descriptors instance or its ID.
BasicMultimodalSceneOrMultimodalScene A mixed array of Basic Multimodal Scene Descriptors and Multimodal Scene Descriptors.
DataXMData MPAI-PTF Data Exchange Metadata providing provenance, authorisation, legal, security, and confidence information associated with this Multimodal Scene Descriptors instance in a trusted data exchange context.
DescrMetadata Human-readable descriptive metadata of the Multimodal Scene Descriptors instance (plain text, max 2048 characters).

5      Conformance Testing

A Data instance conforms with Multimodal Scene Descriptors (OSD‑MSD) if:

  1. The Data validates against the Multimodal Scene Descriptors’ JSON Schema.
  2. All Data in the Multimodal Scene Descriptors’ JSON Schema:
    1. Have the specified type.
    2. Validate against their JSON Schemas.
    3. Conform with their Data Qualifiers.

6      Performance Assessment

Not part of this specification.

Go to MPAI-OSD V1.5 Data Types