PGM-AUA V1.0 AIMs - Cross-Modal Candidate Association

Go to PGM-AUA AI Modules

Function

Ref. Model

1. Function

The Cross-Modal Candidate Association (PGM‑CCA) AIM proposes the pairings of the Audio Entities and the Visual Entities of a captured Space that the evidence admits as possibly referring to the same Entity, without committing to any of them.

Cross-Modal Candidate Association:

Operates on the Enhanced Audio Scene Descriptors produced by Audio Scene Enhancement, the Enhanced Visual Scene Descriptors produced by Visual Scene Enhancement, the AVA CXT Directive received from A‑User Control, the AVA Domain Response resulting from queries made to Domain Access, and the AVA IH Response resulting from queries made to A‑User Storage.
Evaluates each admissible pairing of an Audio Entity with a Visual Entity against four kinds of evidence: spatial coincidence, temporal coincidence, motion correlation over the capture interval, and the pairings established in earlier captures of the session.
Produces the Association Candidates sent to Correspondence Resolution and to Audio-Visual Salience Reconciliation, the AVA CXT Status, the AVA Domain Request, and the AVA IH Request sent to AV Geometry Multiplexing.

The association is deliberately many-to-many. An Audio Entity may appear in several candidate pairings, and so may a Visual Entity. Cross-Modal Candidate Association does not decide which pairing holds, does not resolve competing claims on the same Entity, and does not determine that an Entity has no counterpart: these are functions of Correspondence Resolution. Retaining every admissible pairing with its score allows Correspondence Resolution to report how close the alternatives to the pairing it retains were.

2. Reference Model

Figure 1 depicts the Reference Model of the Cross-Modal Candidate Association (PGM‑CCA) AIM.

Figure 1 – Reference Model of the Cross-Modal Candidate Association (PGM‑CCA) AIM

The Cross-Modal Candidate Association (PGM‑CCA) AIM operates as follows:

Receives the Enhanced Audio Scene Descriptors and the Enhanced Visual Scene Descriptors of the same capture.
Determines the admissible pairings, excluding those the AVA CXT Directive places outside its scope and those between Entities that cannot refer to the same source, such as an Audio Entity classified as reverberation or ambient noise and a Visual Entity.
Assesses, for each admissible pairing:
- Spatial coincidence, i.e., the agreement between the direction of arrival and, where available, the distance estimated for the Audio Entity, and the position of the Visual Entity
- Temporal coincidence, i.e., the agreement between the onset, offset, and activity of the Audio Entity, including voice activity, and the lip, facial, and body motion of the Visual Entity
- Motion correlation, i.e., the agreement between the displacement of the Audio Entity and that of the Visual Entity over the capture interval
- Precedence, i.e., whether the same Audio Entity and Visual Entity were paired in earlier captures of the session, as recorded in the Interaction History
Qualifies each assessment by the conditions of the observation, lowering the weight of spatial evidence under strong reverberation, as reported by the Acoustic Profile, and of visual evidence where the Visual Entity is occluded or poorly visible, as reported by the Visual Profile.
Combines the four assessments into an association score for the pairing, retaining the contribution of each.
Records the kinds of evidence that were unavailable for a pairing, such as the absence of motion over the capture interval or of any earlier capture in the session.
Outputs the Association Candidates.

The AIM operates in real time. It does not engage the reasoning capability of Basic Knowledge and does not open Model Context Protocol sessions. The Response to a query issued to Domain Access or to A‑User Storage is in general received during a subsequent processing cycle.

The AVA Domain Response supports the association in the terms of the domain: the object classes that may emit the sounds heard, and the sources that the setting makes probable or excludes. The AVA IH Response supports it with the history of the session: the pairings established in earlier captures, which make the same pairing probable in the current one unless the other evidence contradicts it.

3. Input/Output Data

Table 1 lists the Input and Output Data of the Cross-Modal Candidate Association (PGM‑CCA) AIM.

Table 1 – Input/Output Data of the Cross-Modal Candidate Association (PGM‑CCA) AIM

Input	Description
Enhanced Audio Scene Descriptors	Descriptors of the audio scene produced by Audio Scene Enhancement.
Enhanced Visual Scene Descriptors	Descriptors of the visual scene produced by Visual Scene Enhancement.
AVA CXT Directive	Directive from A‑User Control specifying scope, depth, and policy constraints for candidate association.
AVA Domain Response	Domain‑specific knowledge from Domain Access supporting the association of sound sources with objects.
AVA IH Response	Interaction History from A‑User Storage reporting the pairings established in earlier captures of the session.
Output	Description
Association Candidates	Admissible pairings of Audio Entities with Visual Entities, each with its association score and the contribution of each kind of evidence.
AVA CXT Status	Status information describing the execution and outcome of Cross-Modal Candidate Association processing.
AVA Domain Request	Query to Domain Access for domain‑specific knowledge.
AVA IH Request	Query to A‑User Storage for the pairings established in earlier captures.

4. SubAIMs

No SubAIMs.

5. JSON Metadata

https://schemas.mpai.community/PGM1/V1.0/AIMs/CrossModalCandidateAssociation.json

6. Profiles

No Profiles.

7. Reference Software

Not part of this specification.

8. Conformance Testing

Table 2 provides the Conformance Testing Method for the Cross-Modal Candidate Association (PGM‑CCA) AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the Cross-Modal Candidate Association (PGM‑CCA) AIM

Receives	Enhanced Audio Scene Descriptors	Shall validate against Enhanced Audio Scene Descriptors schema.
	Enhanced Visual Scene Descriptors	Shall validate against Enhanced Visual Scene Descriptors schema.
	AVA CXT Directive	Shall validate against AVA CXT Directive schema.
	AVA Domain Response	Shall validate against AVA Domain schema.
	AVA IH Response	Shall validate against Interaction History schema.
Produces	Association Candidates	Shall validate against Association Candidates schema.
	AVA CXT Status	Shall validate against AVA CXT Status schema.
	AVA Domain Request	Shall validate against AVA Domain schema.
	AVA IH Request	Shall validate against Interaction History schema.

9. Performance Assessment