MPAI-OSD V1.5 AIMs – Multimodal Scene Editing

Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Multimodal Scene Editing (OSD‑MSE) AIM composes Scenes and Objects of at least two media (Audio, Visual, Speech, LiDAR, 3D Model, RADAR, Ultrasound, Offline Map) into a Multimodal Scene: a voice over music is a Multimodal Scene of an Audio Scene Descriptors and a Speech Scene Descriptors, or of an Audio Object and a Speech Object. Each of its two Scene Ports takes a Scene of any medium, and each of its two Object Ports an Object of any medium, kept as it arrives. One Multimodal Scene is open at a time: the User Command places the Scenes and Objects, named by the name the User gave them, in its common frame (AddedObjects), moves them, changes where they are, takes them out (RemovedObjects), deletes the Multimodal Scenes named, with all their versions, from where the Module keeps them (DeletedObjects: a Multimodal Scene that is a member of another Scene is not deleted, and the response says so, naming that Scene; a name that is not known fails) and sets where the Multimodal Scene is heard and seen from (the User PoV, carried by the User Command: UserPoV). The Scenes and Objects being edited are followed: a member that is the version last arrived on its Port is replaced by the new version. Every edit is a new version. The AIM outputs the open Multimodal Scene, when its members are of at least two media, and the User Command with its response (UserCommandRS) saying how each action was executed.

Receives	Scene	A Scene of any medium.
	Other Scene	A second Scene of any medium.
	Object	An Object of any medium.
	Other Object	A second Object of any medium.
	User Command	The commands that place, move and remove Scenes and Objects.
Produces	Multimodal Scene	The open Multimodal Scene.
	User Command	The response: how each action of the User Command was executed.

2 Reference Model

Figure 1 depicts the Reference Model of the Multimodal Scene Editing (OSD-MSE) AIM.

Multimodal Scene Editing OSD-MSE AIM

Figure 1 – Reference Model of the Multimodal Scene Editing (OSD-MSE) AIM

3 I/O Data

Table 1 specifies the Input and Output Data of the Multimodal Scene Editing (OSD-MSE) AIM. Links are to the Data Type specifications and to their JSON schemas.

Table 1 – I/O Data of the Multimodal Scene Editing (OSD-MSE) AIM

Input	Data Type	Description
Scene	Audio Scene Descriptors (OSD-ASD) or Visual Scene Descriptors (OSD-VSD) or Speech Scene Descriptors (OSD-SSD) or LiDAR Scene Descriptors (OSD-LSD) or 3D Model Scene Descriptors (OSD-3SD) or RADAR Scene Descriptors (OSD-RSD) or Ultrasound Scene Descriptors (OSD-USD) or Offline Map Scene Descriptors (OSD-OSD) (optional) (Port Number 1)	A Scene of any medium.
Other Scene	Audio Scene Descriptors (OSD-ASD) or Visual Scene Descriptors (OSD-VSD) or Speech Scene Descriptors (OSD-SSD) or LiDAR Scene Descriptors (OSD-LSD) or 3D Model Scene Descriptors (OSD-3SD) or RADAR Scene Descriptors (OSD-RSD) or Ultrasound Scene Descriptors (OSD-USD) or Offline Map Scene Descriptors (OSD-OSD) (optional) (Port Number 2)	A second Scene of any medium.
Object	Audio Object (OSD-AUO) or Visual Object (OSD-VIO) or Speech Object (OSD-SPO) or 3D Model Object (OSD-3DO) or RADAR Object (OSD-RAO) or Offline Map Object (OSD-OMO) (optional) (Port Number 1)	An Object of any medium.
Other Object	Audio Object (OSD-AUO) or Visual Object (OSD-VIO) or Speech Object (OSD-SPO) or 3D Model Object (OSD-3DO) or RADAR Object (OSD-RAO) or Offline Map Object (OSD-OMO) (optional) (Port Number 2)	An Object of any medium.
User Command	User Command (CAE-UCM) RQ (optional)	The commands that place, move and remove Scenes and Objects, with the User Point of View (UserPoV).
Output	Data Type	Description
Multimodal Scene	Multimodal Scene Descriptors (OSD-MSD)	The open Multimodal Scene.
User Command	User Command (CAE-UCM) RS (optional)	How each action of the User Command was executed.

4 SubAIMs

No SubAIMs.

5 JSON Metadata

https://schemas.mpai.community/OSD/V1.5/AIMs/MultimodalSceneEditing.json

6 Profiles

No Profiles.

7 Reference Software

Not part of this specification.

8 Conformance Testing

Table 2 provides the Conformance Testing Method for the OSD-MSE AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the OSD-MSE AIM

Receives	Scene (Audio Scene Descriptors or Visual Scene Descriptors or Speech Scene Descriptors or LiDAR Scene Descriptors or 3D Model Scene Descriptors or RADAR Scene Descriptors or Ultrasound Scene Descriptors or Offline Map Scene Descriptors)	Shall validate against the schema of the Data Type its Header names.
	Other Scene (Audio Scene Descriptors or Visual Scene Descriptors or Speech Scene Descriptors or LiDAR Scene Descriptors or 3D Model Scene Descriptors or RADAR Scene Descriptors or Ultrasound Scene Descriptors or Offline Map Scene Descriptors)	Shall validate against the schema of the Data Type its Header names.
	Object (Audio Object or Visual Object or Speech Object or 3D Model Object or RADAR Object or Offline Map Object)	Shall validate against the schema of the Data Type its Header names.
	Other Object (Audio Object or Visual Object or Speech Object or 3D Model Object or RADAR Object or Offline Map Object)	Shall validate against the schema of the Data Type its Header names.
	User Command (User Command)	Shall validate against the User Command schema, and contain a UserCommandRQ.
Produces	Multimodal Scene (Multimodal Scene Descriptors (OSD-MSD))	Shall validate against the Multimodal Scene Descriptors schema.
	User Command (User Command)	Shall validate against the User Command schema, and contain a UserCommandRS.

9 Performance Assessment