PGM-AUA V1.0 AIMs - Audio-Visual Salience Reconciliation

Go to PGM-AUA AI Modules

Function

Ref. Model

1. Function

The Audio-Visual Salience Reconciliation (PGM‑MSR) AIM reconciles the salience that Audio Salience Mapping and Visual Salience Mapping determined separately into a single ranking, the Joint Salience, over the AV Entities of the captured Space.

Audio-Visual Salience Reconciliation:

Operates on the AV Scene Descriptors produced by Scene Geometry Construction, the Enhanced Audio Scene Descriptors produced by Audio Scene Enhancement, the Enhanced Visual Scene Descriptors produced by Visual Scene Enhancement, the AVA CXT Directive received from A‑User Control, and the AVA Domain Response resulting from queries made to Domain Access.
Operates only when the AVA CXT Directive requires the reconciliation of salience, and then over the scope the Directive sets.
Produces the Joint Salience, the AVA CXT Status, and the AVA Domain Request, all sent to AV Geometry Multiplexing.

Joint Salience is not a merge of two rankings. The audio ranking orders Audio Entities and the visual ranking orders Visual Entities; neither can order the Entity that both describe, nor compare an Entity that one modality perceived with one that only the other perceived. Reconciliation is performed over the AV Entities, where each Entity appears once, and adjudicates between the two rankings where they disagree.

Each rank carries the ground on which it holds, so that the AI Modules downstream can distinguish a rank on which the modalities concur from one resting on a single modality, or on a divergence between them.

When the AVA CXT Directive does not require reconciliation, the AIM produces no Joint Salience and reports in the AVA CXT Status that it did not operate.

2. Reference Model

Figure 1 depicts the Reference Model of the Audio-Visual Salience Reconciliation (PGM‑AVR) AIM.

Figure 1 – Reference Model of the Audio-Visual Salience Reconciliation (PGM‑AVR) AIM

The Audio-Visual Salience Reconciliation (PGM‑AVR) AIM operates as follows:

Receives the AV Scene Descriptors of the capture and the Enhanced Audio Scene Descriptors and Enhanced Visual Scene Descriptors from which they were constructed.
Retrieves, for each AV Entity, through its references to the Audio Entity and the Visual Entity it was constructed from, the salience each modality determined and the confidence attaching to it.
Qualifies each modal salience by the conditions of its observation: the audio salience weighing less under strong reverberation or masking, as the Acoustic Profile reports them, and the visual salience weighing less under occlusion or poor visibility, as the Visual Profile reports them.
Applies the salience criteria the AVA CXT Directive sets, such as relevance to the current exchange, proximity to the A‑User, or the Entities A‑User Control has designated.
Determines the joint salience of each AV Entity according to its case:
- Concurrence, i.e., the Entity is perceived by both modalities and both assign it comparable salience: the two corroborate each other and the joint salience is raised accordingly
- Single modality, i.e., the Entity is perceived by one modality only: the joint salience rests on that modality, without penalty for the absence of the other
- Divergence, i.e., the Entity is perceived by both modalities and they assign it markedly different salience: the joint salience is determined from the qualified salience of each, and the divergence is recorded
Orders the AV Entities by joint salience into a single ranking.
Attaches to each rank its ground, i.e., concurrence, single modality, or divergence, and its confidence.
Outputs the Joint Salience.

The AIM operates in real time. It does not engage the reasoning capability of Basic Knowledge and does not open Model Context Protocol sessions. The Response to a query issued to Domain Access is in general received during a subsequent processing cycle.

The AVA Domain Response supports the reconciliation in the terms of the domain: the weight the setting gives to what is heard relative to what is seen, as in a concert hall compared with a museum, and the classes of Entity the setting makes salient, such as an alarm in a workshop or a speaker at a lectern.

A divergence is not an error to be removed. A person who is silent but approaching the A‑User, or a voice raised by someone out of view, are divergences that carry information, and the Joint Salience preserves them as such.

3. Input/Output Data

Table 1 lists the Input and Output Data of the Audio-Visual Salience Reconciliation (PGM‑AVR) AIM.

Table 1 – Input/Output Data of the Audio-Visual Salience Reconciliation (PGM‑AVR) AIM

Input	Description
AV Scene Descriptors	Description of the Space in which each Entity appears once as an AV Entity, produced by Scene Geometry Construction.
Enhanced Audio Scene Descriptors	Descriptors of the audio scene, with the salience determined within the audio modality and the Acoustic Profile, produced by Audio Scene Enhancement.
Enhanced Visual Scene Descriptors	Descriptors of the visual scene, with the salience determined within the visual modality and the Visual Profile, produced by Visual Scene Enhancement.
AVA CXT Directive	Directive from A‑User Control requiring or not requiring reconciliation, and specifying its scope and salience criteria.
AVA Domain Response	Domain‑specific knowledge from Domain Access on the relative weight of the modalities and the Entities salient in the setting.
Output	Description
Joint Salience	Ranking of the AV Entities by joint salience, each rank with its ground and confidence.
AVA CXT Status	Status information describing the execution and outcome of Audio-Visual Salience Reconciliation processing.
AVA Domain Request	Query to Domain Access for domain‑specific knowledge.

4. SubAIMs

No SubAIMs.

5. JSON Metadata

https://schemas.mpai.community/PGM1/V1.0/AIMs/AudioVisualSalienceReconciliation.json

6. Profiles

No Profiles.

7. Reference Software

Not part of this specification.

8. Conformance Testing

Table 2 provides the Conformance Testing Method for the Audio-Visual Salience Reconciliation (PGM‑AVR) AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the Audio-Visual Salience Reconciliation (PGM‑AVR) AIM

Receives	AV Scene Descriptors	Shall validate against AV Scene Descriptors schema.
	Enhanced Audio Scene Descriptors	Shall validate against Enhanced Audio Scene Descriptors schema.
	Enhanced Visual Scene Descriptors	Shall validate against Enhanced Visual Scene Descriptors schema.
	AVA CXT Directive	Shall validate against AVA CXT Directive schema.
	AVA Domain Response	Shall validate against AVA Domain schema.
Produces	Joint Salience	Shall validate against Joint Salience schema.
	AVA CXT Status	Shall validate against AVA CXT Status schema.
	AVA Domain Request	Shall validate against AVA Domain schema.

9. Performance Assessment