PGM-AUA V1.0 AIMs - Correspondence Resolution

Go to PGM-AUA AI Modules

Function

Ref. Model

1. Function

The Correspondence Resolution (PGM‑CRE) AIM turns the candidate pairings of Audio Entities and Visual Entities into a consistent assignment, establishing which Audio Entities and which Visual Entities of a captured Space are the same Entity and which have no counterpart in the other modality.

Correspondence Resolution:

Operates on the Association Candidates produced by Cross-Modal Candidate Association, the Enhanced Audio Scene Descriptors produced by Audio Scene Enhancement, the Enhanced Visual Scene Descriptors produced by Visual Scene Enhancement, the AVA CXT Directive received from A‑User Control, and the AVA IH Response resulting from queries made to A‑User Storage.
Resolves the competing claims of candidate pairings on the same Entity, determines the cardinality of each correspondence, attaches a confidence to each correspondence it establishes, and retains across the captures of a session the identifier of an Entity already aligned.
Produces the Entity Correspondences sent to Scene Geometry Construction, Audio-Visual Salience Reconciliation, and AV Geometry Multiplexing, the Unmatched Entities sent to Audio-Visual Salience Reconciliation and AV Geometry Multiplexing, and the AVA CXT Status and the AVA IH Request sent to AV Geometry Multiplexing.

Every Audio Entity and every Visual Entity of the Enhanced Scene Descriptors received is placed either in an Entity Correspondence or among the Unmatched Entities. An Unmatched Entity is not a failure of the AIM: a voice with no visible speaker and a person who has not spoken are ordinary states of a Space, and are reported as such.

Where the evidence does not settle which of two correspondences holds, Correspondence Resolution establishes the one the evidence favours, records the alternative with its score, and lowers the confidence of the correspondence it establishes accordingly.

2. Reference Model

Figure 1 depicts the Reference Model of the Correspondence Resolution (PGM‑CRE) AIM.

Figure 1 – Reference Model of the Correspondence Resolution (PGM‑CRE) AIM

The Correspondence Resolution (PGM‑CRE) AIM operates as follows:

Receives the Association Candidates of the capture, together with the Enhanced Audio Scene Descriptors and the Enhanced Visual Scene Descriptors from which they were derived.
Retrieves from the Interaction History the correspondences established in earlier captures of the session, with the identifiers assigned to the aligned Entities.
Determines the cardinality admissible for each Entity from its class and its properties:
- One-to-one, as when one voice corresponds to one person
- Many-to-one, as when several sound sources are emitted by one object, such as the engine and the horn of a vehicle
- One-to-none, as when an Audio Entity has no visible source, or a Visual Entity emits no sound
Resolves competing claims, where two or more candidate pairings claim the same Entity beyond the cardinality admissible for it, by retaining the assignment that maximises the overall association score of the capture, subject to the cardinalities determined and to the constraints of the AVA CXT Directive.
Attaches to each correspondence established a confidence, derived from:
- The association score of the retained pairing
- The margin between that score and the score of the best alternative the assignment excluded
- The agreement of the correspondence with those established in earlier captures of the session
Assigns to each aligned Entity an identifier, retaining the identifier assigned in earlier captures where the correspondence persists, and assigning a new identifier where it does not.
Records:
- The Audio Entities and the Visual Entities left without counterpart, with the reason, such as the absence of any candidate or the loss of a competing claim
- The alternatives excluded by the assignment whose score came close to that of the retained pairing
- The correspondences of earlier captures that the current capture no longer supports
Outputs the Entity Correspondences and the Unmatched Entities.

The AIM operates in real time. It does not engage the reasoning capability of Basic Knowledge and does not open Model Context Protocol sessions. The Response to a query issued to A‑User Storage is in general received during a subsequent processing cycle.

The AVA IH Response gives the aligned Entities of a session their persistence: the same person keeps the same aligned identifier across captures, including when the person is momentarily silent or momentarily out of view. A correspondence that the current evidence contradicts is not retained on the ground of history alone; its end is recorded instead.

3. Input/Output Data

Table 1 lists the Input and Output Data of the Correspondence Resolution (PGM‑CRE) AIM.

Table 1 – Input/Output Data of the Correspondence Resolution (PGM‑CRE) AIM

Input	Description
Association Candidates	Admissible pairings of Audio Entities with Visual Entities, with their association scores, produced by Cross-Modal Candidate Association.
Enhanced Audio Scene Descriptors	Descriptors of the audio scene produced by Audio Scene Enhancement.
Enhanced Visual Scene Descriptors	Descriptors of the visual scene produced by Visual Scene Enhancement.
AVA CXT Directive	Directive from A‑User Control specifying scope, depth, and policy constraints for correspondence resolution.
AVA IH Response	Interaction History from A‑User Storage reporting the correspondences and identifiers established in earlier captures of the session.
Output	Description
Entity Correspondences	Correspondences established between Audio Entities and Visual Entities, each with its cardinality, confidence, aligned identifier, and close alternatives.
Unmatched Entities	Audio Entities and Visual Entities left without counterpart in the other modality, each with the reason.
AVA CXT Status	Status information describing the execution and outcome of Correspondence Resolution processing.
AVA IH Request	Query to A‑User Storage for the correspondences established in earlier captures.

4. SubAIMs

No SubAIMs.

5. JSON Metadata

https://schemas.mpai.community/PGM1/V1.0/AIMs/CorrespondenceResolution.json

6. Profiles

No Profiles.

7. Reference Software

Not part of this specification.

8. Conformance Testing

Table 2 provides the Conformance Testing Method for the Correspondence Resolution (PGM‑CRE) AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the Correspondence Resolution (PGM‑CRE) AIM

Receives	Association Candidates	Shall validate against Association Candidates schema.
	Enhanced Audio Scene Descriptors	Shall validate against Enhanced Audio Scene Descriptors schema.
	Enhanced Visual Scene Descriptors	Shall validate against Enhanced Visual Scene Descriptors schema.
	AVA CXT Directive	Shall validate against AVA CXT Directive schema.
	AVA IH Response	Shall validate against Interaction History schema.
Produces	Entity Correspondences	Shall validate against Entity Correspondences schema.
	Unmatched Entities	Shall validate against Unmatched Entities schema.
	AVA CXT Status	Shall validate against AVA CXT Status schema.
	AVA IH Request	Shall validate against Interaction History schema.

9. Performance Assessment