(Informative)
Audio is increasingly produced and consumed as Objects placed in Scenes rather than as fixed channels: each Object carries its sound and what describes it, and a Scene says where each Object is, how the space sounds, and from where it is heard. This makes a presentation adaptable to the user, the device, and the listening position.
1 What Audio Scene Management does
Technical Specification: Context-based Audio Enhancement (MPAI-CAE) – Audio Scene Management (CAE-ASM) V1.0 specifies Audio Scene Management (ASM), a Composite AIM that lets a User build and change an audio presentation made of Objects and Scenes of audio and of speech.
The Objects and Scenes that ASM works on are acquired from a disk or from a network, or created by the User. A User directs ASM with User Commands, one per User action, which the User Agent gives to the AIM they are for. A User Command is a request – move this Object, change its loudness, place it in that Scene, translate that speech, list what there is – that also carries the User Point of View and the Object to acquire (UserPoV, AcquiredObject), and the AIM answers it with a response that says how each action was executed. Three properties make the commands usable by a person as well as by a program:
- Objects and Scenes are identified by the name the User gave them (“violin”, “violin 2”, “concert”), never by an internal key. A name that is taken becomes the name followed by the next free number.
- Every amount in a command – a position, an orientation, a loudness – is a number the command carries, and a command is absolute: it places an Object at a position, it moves it to a position, it rotates it to an orientation. A command that lacks a number is not executed: the lack is reported.
- A command can list: it asks for the names of the Objects or Scenes of a Kind, with their duration, their number of children and their version, or, for a named Scene, its members with their position and orientation as provided in the Scene.
Every edit produces a new version of the Object or Scene, which keeps its name. ASM edits Audio Objects and Audio Scenes, and Speech Objects and Speech Scenes; it composes an Audio Scene and a Speech Scene into a Multimodal Scene. ASM renders nothing: what the User hears is delivered elsewhere, as Scope specifies.
A User can give ASM the commands directly, as User Commands, or by speaking: vocal commands may be issued to ASM through the Voice Command Interface (MMC-VCI), which MPAI-MMC specifies and ASM uses. The person speaks at the boundary of ASM and hears the response, and can read it as text, together with the text of what was heard; the commands that the Voice Command Interface forms go to the editing AIMs, whose responses come back to it.
2 Audio Scene Management in the AI Framework
ASM operates in the AI Framework (MPAI-AIF V3.0). It is a Composite AIM, and as the Module of an application it is run by an AIF Controller, which a User Agent drives. This has consequences for the architecture:
- The AIMs exchange data through Ports. A Port is addressed by the Data Type of what crosses it and, where several Ports have the same Data Type, by its Port Number; never by a name. Each editing AIM has its own User Command Port at the boundary of the Composite AIM, and the connections between the Ports of the AIMs are the Topology of the Composite AIM.
- Each AIM is defined by its Function and by the Data Types of its Ports. The AIMs are independently implementable: an Implementation that meets the Function and the Ports of an AIM can replace any other.
- The AIMs do not handle storage or communication: the Controller does. The versions of the Objects and Scenes are kept in the storage the Controller provides, and the data between AIMs travels on the Channels the Controller sets up.
- The User Agent never touches the AIMs. It asks the Controller to start and stop the Module, gives data to the Ports of the boundary of the Module and obtains data from them. It deals with the real world – acquiring audio from a disk or a network, delivering audio to a loudspeaker – through the Acquisition Units and Delivery Units of its Physical Layer, which are not AIMs.
- Data that an AIM reads and does not write is data of Access, identified by the URI it is published at and by its SHA-256.
- Conformance is claimed per AIM: an Implementation of an AIM conforms if it accepts and produces the Data Types of its Ports as the AIM page specifies.
3 The AIMs of Audio Scene Management
Audio Object Editing (CAE-AOE) edits Audio Objects: one Audio Object is open at a time, and the User Command changes what it is like, where it is heard from and a defined list of its features. The sound itself is not edited.
Audio Scene Editing (CAE-ASE) composes Audio Scenes: one Audio Scene is open at a time, and the User Command adds Audio Objects to it, moves them, changes where they are and removes them.
Speech Object Editing (MMC-SOE), specified by MPAI-MMC V2.5, edits Speech Objects and has them translated.
Speech Scene Editing (MMC-SSE), specified by MPAI-MMC V2.5, composes Speech Scenes as Audio Scene Editing composes Audio Scenes.
Multimodal Scene Editing (OSD-MSE), specified by MPAI-OSD V1.5, composes the Audio Scene Descriptors and the Speech Scene Descriptors into a Multimodal Scene.
Voice Command Interface (MMC-VCI), specified by MPAI-MMC V2.5, is the AIM through which vocal commands may be issued to ASM: it is a Composite AIM that enables a person to issue vocal commands to an ASM and, in the future, to other Modules.
Text and Speech Translation (MMC-TST), specified by MPAI-MMC V2.5, translates the Speech Object that Speech Object Editing hands to it; the translation replaces the original under its name.
