Go to PGM-AUA AI Modules

Function Ref. Model I/O Data SubAIMs JSON MData Profiles Ref. Software Conformance Performance

1. Function

The Audio‑Visual Scene Descriptors Multiplexing (PGM‑AVM) AIM is the output boundary of the Audio-Visual Alignment (PGM‑AVA) Composite AIM. It joins the Joint Salience, when produced, to the Audio‑Visual Scene Descriptors, and multiplexes the Response and Request signals of the other Sub-AIMs of Audio-Visual Alignment into the corresponding external output signals.

Audio‑Visual Scene Descriptors Multiplexing:

  1. Operates on the Audio‑Visual Scene Descriptors produced by Scene Geometry Construction, the Joint Salience produced by Audio-Visual Salience Reconciliation, the AVA CXT Request received from A‑User Control, and the AVA CXT Response, AVA Domain Request, and AVA Interaction History Request signals issued by the other Sub-AIMs.
  2. Attaches the joint salience of each AV Entity to the AV Entity it ranks, and verifies that what it has received is what the AVA CXT Request required.
  3. Produces the Audio‑Visual Scene Descriptors sent to User State Description and Context Description Multiplexing, the AVA CXT Response sent to A‑User Control through Context Description Multiplexing, the AVA Domain Request sent to Domain Access, and the AVA Interaction History Request sent to Private Storage.

Audio‑Visual Scene Descriptors Multiplexing performs no analysis. It does not alter the content of the Audio‑Visual Scene Descriptors or of the Joint Salience it receives, it does not merge, reorder, or suppress the Requests it multiplexes, and it does not reinterpret the Responses it assembles.

The composite AVA CXT Response conveys, for each Sub-AIM, the goal it understood, the outcome it reached, and the reason, identifying the Sub-AIM from which each report originates, so that A‑User Control can determine which Sub-AIM reported what.

2. Reference Model

Figure 1 depicts the Reference Model of the Audio‑Visual Scene Descriptors Multiplexing (PGM‑AVM) AIM.

 

Figure 1 – Reference Model of the Audio‑Visual Scene Descriptors Multiplexing (PGM‑AVM) AIM

The Audio‑Visual Scene Descriptors Multiplexing (PGM‑AVM) AIM operates as follows:

  1. Receives the Audio‑Visual Scene Descriptors of the capture, the Joint Salience if produced, and the Response and Request signals issued for the same capture by Cross-Modal Candidate Association, Correspondence Resolution, Scene Geometry Construction, and Audio-Visual Salience Reconciliation.
  2. Verifies, against the AVA CXT Request:
    • That the Joint Salience is present when the Request required reconciliation, and absent when it did not
    • That the Joint Salience and the Audio‑Visual Scene Descriptors refer to the same capture and to the same set of AV Entities
  3. Joins the Joint Salience to the Audio‑Visual Scene Descriptors, attaching to each AV Entity its joint rank, the ground on which the rank holds, and its confidence.
  4. Assembles the composite AVA CXT Response from the Responses of the Sub-AIMs, preserving each report together with the identifier of the Sub-AIM that issued it, and adding its own report of the verification performed in step 2.
  5. Multiplexes:
    • The AVA Domain Requests issued by Cross-Modal Candidate Association and Audio-Visual Salience Reconciliation into the external AVA Domain Request
    • The AVA Interaction History Requests issued by Cross-Modal Candidate Association and Correspondence Resolution into the external AVA Interaction History Request

    each multiplexed Request preserving the identifier of the Sub-AIM that issued it, so that the Response can be routed back to it.

  6. Outputs the Audio‑Visual Scene Descriptors, the AVA CXT Response, the AVA Domain Request, and the AVA Interaction History Request.

When a Sub-AIM has produced no output for the capture, Audio‑Visual Scene Descriptors Multiplexing does not substitute one. If the Audio‑Visual Scene Descriptors are missing, it outputs none and reports their absence in the AVA CXT Response. If only the Joint Salience is missing when required, it outputs the Audio‑Visual Scene Descriptors without it and reports the omission.

The AIM operates in real time. It does not engage the reasoning capability of Basic Knowledge and does not open Model Context Protocol sessions.

3. Input/Output Data

Table 1 lists the Input and Output Data of the Audio‑Visual Scene Descriptors Multiplexing (PGM‑AVM) AIM.

Table 1 – Input/Output Data of the Audio‑Visual Scene Descriptors Multiplexing (PGM‑AVM) AIM

Input Description
Audio‑Visual Scene Descriptors Description of the Space in which each Entity appears once as an AV Entity, produced by Scene Geometry Construction.
Joint Salience Ranking of the AV Entities by joint salience, produced by Audio-Visual Salience Reconciliation when the Request requires it.
AVA CXT Request Request from A‑User Control against which the completeness of the outputs is verified.
AVA CXT Response Response signals issued by the other Sub-AIMs of Audio-Visual Alignment.
AVA Domain Request Domain requests issued by Cross-Modal Candidate Association and Audio-Visual Salience Reconciliation.
AVA Interaction History Request Interaction History requests issued by Cross-Modal Candidate Association and Correspondence Resolution.
Output Description
Audio‑Visual Scene Descriptors Audio‑Visual Scene Descriptors with, where produced, the joint rank, ground, and confidence of each AV Entity.
AVA CXT Response Composite status of Audio-Visual Alignment, preserving the report of each Sub-AIM with its origin.
AVA Domain Request Multiplexed domain requests issued to Domain Access.
AVA Interaction History Request Multiplexed Interaction History requests issued to Private Storage.

4. SubAIMs

No SubAIMs.

5. JSON Metadata

https://schemas.mpai.community/PGM1/V1.0/AIMs/AVGeometryMultiplexing.json

6. Profiles

No Profiles.

7. Reference Software

Not part of this specification.

8. Conformance Testing

Table 2 provides the Conformance Testing Method for the Audio‑Visual Scene Descriptors Multiplexing (PGM‑AVM) AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the Audio‑Visual Scene Descriptors Multiplexing (PGM‑AVM) AIM

Receives Audio‑Visual Scene Descriptors Shall validate against Audio‑Visual Scene Descriptors schema.
Joint Salience Shall validate against Joint Salience schema.
AVA CXT Request Shall validate against AVA CXT Request schema.
AVA CXT Response Shall validate against AVA CXT Response schema.
AVA Domain Request Shall validate against AVA Domain schema.
AVA Interaction History Request Shall validate against Interaction History schema.
Produces Audio‑Visual Scene Descriptors Shall validate against Audio‑Visual Scene Descriptors schema.
AVA CXT Response Shall validate against AVA CXT Response schema.
AVA Domain Request Shall validate against AVA Domain schema.
AVA Interaction History Request Shall validate against Interaction History schema.

9. Performance Assessment

Not part of this specification.

Go to PGM-AUA AI Modules