| Function | Ref. Model | I/O Data | SubAIMs | JSON MData | Profiles | Ref. Software | Conformance | Performance |
1. Function
The Audio-Visual Alignment (PGM‑AVA) AIM establishes which Audio Entities and which Visual Entities of a captured Space are the same Entity, describes the resulting Entities in a single spatial frame, and, when the Directive so requires, reconciles the salience each modality assigned to them into a joint ranking over the aligned Entities.
Audio-Visual Alignment:
- Operates on the Enhanced Audio Scene Descriptors produced by Audio Scene Enhancement, the Enhanced Visual Scene Descriptors produced by Visual Scene Enhancement, the AVA CXT Directive received from A‑User Control, the AVA Domain Response resulting from queries made to Domain Access, and the AVA IH Response resulting from queries made to A‑User Storage.
- Performs a multi‑stage alignment process including cross‑modal candidate association, correspondence resolution, scene geometry construction, and, under the AVA CXT Directive, audio-visual salience reconciliation, without deriving new modality‑specific evidence and without modifying the Enhanced Scene Descriptors it receives.
- Produces the AV Scene Geometry sent to User State Description and to Context Description Multiplexing, the AVA CXT Status sent to A‑User Control, the AVA Domain Request when querying Domain Access, and the AVA IH Request when querying A‑User Storage.
The AV Scene Geometry carries the Entity Correspondences established, the Audio and Visual Entities that remained unmatched, the spatial relations holding between the aligned Entities, and the Joint Salience where the Directive required it.
Audio-Visual Alignment is the first AI Module of the A-User Frontend at which the audio and the visual evidence of a capture are held together. Salience determined within a single modality by Audio Salience Mapping and by Visual Salience Mapping ranks Audio Entities and Visual Entities separately and cannot rank the Entity that both describe; reconciling the two is therefore performed here, where the correspondence between them is established.
2. Reference Model
Figure 1 depicts the Reference Model of the Audio-Visual Alignment (PGM‑AVA) AIM.
Figure 1 – Reference Model of the Audio-Visual Alignment (PGM‑AVA) AIM
3. Input/Output Data
Table 1 lists the Input and Output Data of the Audio-Visual Alignment (PGM‑AVA) AIM.
Table 1 – Input/Output Data of the Audio-Visual Alignment (PGM‑AVA) AIM
| Input | Description |
| Enhanced Audio Scene Descriptors | Descriptors of the audio scene, with the Acoustic Profile and the salience determined within the audio modality, produced by Audio Scene Enhancement. |
| Enhanced Visual Scene Descriptors | Descriptors of the visual scene, with the Visual Profile and the salience determined within the visual modality, produced by Visual Scene Enhancement. |
| AVA CXT Directive | Directive from A‑User Control specifying scope, depth, and policy constraints for alignment, and requiring or not requiring the reconciliation of salience across the modalities. |
| AVA Domain Response | Domain‑specific knowledge from Domain Access supporting the association of Entities across modalities and the weight the domain attaches to each in the setting. |
| AVA IH Response | Interaction History response from A‑User Storage supporting the persistence of aligned Entity identifiers across the captures of a session. |
| Output | Description |
| AV Scene Descriptors | Correspondence between the Audio Entities and the Visual Entities of the Space, the Entities that remained unmatched, the spatial relations holding between the aligned Entities, and the Joint Salience where required. |
| AVA CXT Status | Multiplexed status signals from the Audio-Visual Alignment‘s internal Sub-AIMs reporting their processing outcomes. |
| AVA Domain Request | Multiplexed domain requests from the internal Sub-AIMs issued to Domain Access. |
| AVA IH Request | Multiplexed Interaction History requests from the internal Sub-AIMs issued to A‑User Storage. |
4. Sub-AIMs (Informative)
This section is informative. The decomposition into Sub-AIMs described below illustrates one conformant architecture for producing the normative outputs of PGM‑AVA. Implementations may adopt alternative internal structures provided they satisfy the conformance requirements of Section 8.
4.1 Reference Model
Figure 2 depicts the Reference Model of the Audio-Visual Alignment (PGM‑AVA) Composite AIM.
Figure 2 – Reference Model of the Audio-Visual Alignment (PGM‑AVA) Composite AIM
4.2 Operation
The Audio-Visual Alignment AIM receives the Enhanced Audio Scene Descriptors, the Enhanced Visual Scene Descriptors, and the AVA CXT Directive, and progressively transforms them into the AV Scene Geometry through a sequence of specialised Sub-AIMs.
The external inputs and control signals are fed to the appropriate Sub-AIMs:
- The Cross-Modal Candidate Association Sub-AIM proposes the pairings of Audio Entities with Visual Entities that the evidence admits, from their spatial coincidence, from the coincidence in time of voice activity with lip and body motion, and from the correlation of their motion over the capture interval. It produces Association Candidates, in which an Audio Entity and a Visual Entity may each appear in more than one candidate pairing, each carrying an association score. It queries Domain Access via AVA Domain as needed.
- The Correspondence Resolution Sub-AIM operates on the Association Candidates to establish a consistent assignment, resolving the competing claims of two candidates on the same Entity and determining the cardinality that holds, including the Audio Entity that no Visual Entity corresponds to and the Visual Entity that emits no sound. It produces the Entity Correspondences, each carrying its confidence, and the Unmatched Entities. It queries A‑User Storage via AVA IH to retain across the captures of a session the identifier of an Entity already aligned.
- The Scene Geometry Construction Sub-AIM describes the aligned Entities in a single spatial frame, producing the Spatial Relations that hold between them, comprising relative position, distance, and orientation, and, where the two modalities estimate the position of an Entity differently, the estimate it retains and the divergence between the two.
- The Audio-Visual Salience Reconciliation Sub-AIM operates when the AVA CXT Directive requires the reconciliation of salience. It produces the Joint Salience, a ranking over the aligned Entities and the unmatched Entities, from the salience each modality determined, weighted by the confidence each carries and by the conditions under which each Entity was observed, as the Acoustic Profile and the Visual Profile report them. Each rank carries the ground on which it holds: the concurrence of the modalities, the evidence of one modality alone where the other did not observe the Entity, or the divergence of the two where each ranked the Entity differently. It queries Domain Access via AVA Domain as needed.
- The AV Geometry Multiplexing Sub-AIM assembles the Entity Correspondences, the Unmatched Entities, the Spatial Relations, and the Joint Salience into the AV Scene Geometry. It also multiplexes the AVA CXT Status, AVA Domain Request, and AVA IH Request signals from all Sub-AIMs into the corresponding external output signals.
Each of Cross-Modal Candidate Association, Correspondence Resolution, Scene Geometry Construction, and Audio-Visual Salience Reconciliation independently generates an AVA CXT Status, an AVA Domain Request, and an AVA IH Request, which are multiplexed by AV Geometry Multiplexing into the corresponding external output signals.
Audio-Visual Alignment operates in real time. It does not engage the reasoning capability of Basic Knowledge and does not open Model Context Protocol sessions. The Response to a query issued to Domain Access or to A‑User Storage is in general received during a subsequent processing cycle.
4.3 Functions of Sub-AIMs
Table 2 specifies the functions of the Sub-AIMs of the Audio-Visual Alignment (PGM‑AVA) Composite AIM.
Table 2 – Functions of the Sub-AIMs of the Audio-Visual Alignment (PGM‑AVA) Composite AIM
| Name | Function |
| Cross-Modal Candidate Association | Proposes the pairings of Audio Entities with Visual Entities that the spatial, temporal, and motion evidence admits, expressing the result as Association Candidates each carrying an association score, querying Domain Access via AVA Domain as needed. |
| Correspondence Resolution | Establishes a consistent assignment from the Association Candidates, resolving competing claims and determining cardinality, producing the Entity Correspondences with their confidence and the Unmatched Entities, querying A‑User Storage via AVA IH to retain the identifier of an Entity aligned in an earlier capture. |
| Scene Geometry Construction | Describes the aligned Entities in a single spatial frame, producing the Spatial Relations comprising relative position, distance, and orientation, and the divergence between the position the two modalities estimate for the same Entity. |
| Audio-Visual Salience Reconciliation | Produces, when the AVA CXT Directive so requires, the Joint Salience over the aligned and unmatched Entities from the salience each modality determined, weighted by confidence and by the conditions of observation, each rank carrying the ground on which it holds, querying Domain Access via AVA Domain as needed. |
| AV Geometry Multiplexing | Assembles the Entity Correspondences, Unmatched Entities, Spatial Relations, and Joint Salience into the AV Scene Descriptors, and multiplexes the AVA CXT Status, AVA Domain Request, and AVA IH Request signals from all Sub-AIMs into the corresponding external outputs. |
4.4 Input/Output Data of Sub-AIMs
Table 3 lists the Input and Output Data of the Sub-AIMs of the Audio-Visual Alignment (PGM‑AVA) Composite AIM.
Table 3 – Input/Output Data of the Sub-AIMs of the Audio-Visual Alignment (PGM‑AVA) Composite AIM
4.5 AIMs and JSON Metadata
Table 4 gives the Audio-Visual Alignment (AIM1) and its Sub-AIMs (AIM2).
Table 4 – Audio-Visual Alignment (AIM1) and its Sub-AIMs (AIM2)
| AIM1 | AIM2 | Name | JSON |
| PGM‑AVA | Audio-Visual Alignment | X | |
| PGM‑CCA | Cross-Modal Candidate Association | X | |
| PGM‑CRE | Correspondence Resolution | X | |
| PGM‑SGC | Scene Geometry Construction | X | |
| PGM‑AVR | Audio-Visual Salience Reconciliation | X | |
| PGM‑AVM | AV Geometry Multiplexing | X |
5. JSON Metadata
https://schemas.mpai.community/PGM1/V1.0/AIMs/AudioVisualAlignment.json
6. Profiles
No Profiles.
7. Reference Software
Not part of this specification.
8. Conformance Testing
An AVA implementation conforms with Audio-Visual Alignment (PGM‑AVA) if:
- The implementation includes all Sub-AIMs listed in Table 2.
- All I/O Data listed in Table 1 are present and conform with their respective Data Types.
- All Sub-AIM I/O Data listed in Table 3 conform with their respective Data Types.
- The implementation produces an AV Scene Geometry in which every Audio Entity and every Visual Entity of the Enhanced Scene Descriptors received appears either in an Entity Correspondence or among the Unmatched Entities.
- The implementation produces the Joint Salience when, and only when, the AVA CXT Directive requires the reconciliation of salience.
- The implementation does not modify the Enhanced Audio Scene Descriptors and the Enhanced Visual Scene Descriptors it receives.
9. Performance Assessment
Not part of this specification.
