Go to PGM-AUA V1.0 AI Modules

Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Multimodal Input Harmonisation (PGM‑MIH) AI Module assembles the audio, visual, textual, and paralinguistic evidence available about a scene into a single Harmonised Multimodal Context, in which each item of evidence is placed on a common time base and attributed to the User that produced it, by

  • Establishing a common time base across the modality streams.
  • Attributing the Text and the Speech Descriptors to the User that uttered them, using the Audio‑Visual Scene Geometry to carry the attribution across modalities.
  • Reconciling the Entity identifiers used by the several modalities into one harmonised set.
  • Recording the gaps, the disagreements, and the unattributed evidence found while doing so.
  • Producing the Harmonised Multimodal Context.

The module is assembling and reconciling. It should not interpret the evidence it harmonises: it establishes what was observed, when, and by whom, and leaves interpretation of the evidence to the analysis AI Modules of User State Description, which consume its output. This AIM may require a degree of intelligence obtained through contextual training.

Multimodal Input Harmonisation does not repeat the audio‑visual association performed by the Audio‑Visual Alignment (PGM‑AVA) AIM. Rather, it receives the Audio‑Visual Scene Descriptors from the ASE and VSE AIMs and adds textual correspondence.

 

2 Reference Model

Figure 1 depicts the Reference Model of the Multimodal Input Harmonisation (PGM‑MIH) AIM.

Figure 1 – The Multimodal Input Harmonisation (PGM‑MIH) AIM

The Multimodal Input Harmonisation (PGM‑MIH) AI Module operates as follows:

  1. Receives the Enhanced Audio Scene Descriptors, the Enhanced Visual Scene Descriptors, the Text, and the Speech Descriptors.
  2. Establishes a common time base by:
    • Referring the segment boundaries of each stream to the corresponding capture times.
    • Resolving the differing rates at which the streams are produced.
    • Marking the intervals for which a stream is absent.
  3. Establishes referential correspondence by:
    • Attributing each Text segment and each Speech Descriptor to the User that produced the utterance.
    • Carrying that attribution to the corresponding Visual Entity through the Audio‑Visual Scene Descriptors.
    • Binding the harmonised media to the User, identified by the User‑USD Directive.
  4. Reconciles the media identifiers used by the modality streams into one harmonised set, retaining the original identifier of each modality.
  5. Records:
    • Evidence that could not be attributed to any media.
    • Disagreements between modalities about certain media.
    • Streams that are absent, incomplete, or outside the tolerance of the common time base.
  6. Constructs a consistent Harmonised Multimodal Context.
  7. Outputs the Harmonised Multimodal Context for use by downstream AI Modules.

The module operates in real time, without exploiting the reasoning capability of Basic Knowledge and without Model Context Protocol interactions. The AIM queries Domain Access and A-User Storage to obtain the necessary information.

3 I/O Data

Table 1 specifies the Input and Output Data of the Multimodal Input Harmonisation (PGM‑MIH) AIM.

Table 1 – I/O Data of the Multimodal Input Harmonisation (PGM‑MIH) AIM

Input Description
Enhanced Audio Scene Descriptors Audio Scene Descriptors augmented by Audio Scene Enhancement with derived and semantic properties.
Enhanced Visual Scene Descriptors Visual Scene Descriptors augmented by Visual Scene Enhancement with derived and semantic properties.
Audio‑Visual Scene Geometry Correspondence and spatial relations between the Audio Entities and the Visual Entities of the scene, produced by Audio‑Visual Alignment.
Text Text recognised by Automatic Speech Recognition from the speech component of the scene.
Speech Descriptors Paralinguistic features of the speech segment produced by Automatic Speech Recognition.
User‑CXT Directive The control directive specifying scope, tolerance, and policy constraints for harmonisation, and identifying the User where known.
User Domain Response Domain‑specific knowledge supporting the reconciliation of evidence across modalities.
User Interaction History Response Prior interaction‑history content supporting the persistence of harmonised media identifiers across perceptual episodes.
Output Description
Harmonised Multimodal Context Audio, visual, textual, and paralinguistic evidence of the scene, placed on a common time base and attributed to harmonised Entities, with a record of media instances that could not be harmonised.
User‑CXT Status Status information describing the execution and outcome of Multimodal Input Harmonisation processing.
User Domain Request Query to Domain Access for domain‑specific knowledge.
User Interaction History Request Query to A‑User Storage for prior interaction‑history content.

4 SubAIMs

No SubAIMs.

5 JSON Metadata

https://schemas.mpai.community/PGM1/V1.0/AIMs/MultimodalInputHarmonisation.json

6 Profiles

No Profiles.

7 Reference Software

Not part of this specification.

8 Conformance Testing

Table 2 provides the Conformance Testing Method for the Multimodal Input Harmonisation (PGM‑MIH) AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the Multimodal Input Harmonisation (PGM‑MIH) AIM

Receives Enhanced Audio Scene Descriptors Shall validate against Enhanced Audio Scene Descriptors schema.
Enhanced Visual Scene Descriptors Shall validate against Enhanced Visual Scene Descriptors schema.
Audio‑Visual Scene Geometry Shall validate against Audio‑Visual Scene Geometry schema.
Text Shall validate against Text Object schema. Text Data shall conform with Text Qualifier.
Speech Descriptors Shall validate against Speech Descriptors schema.
User‑USD Directive Shall validate against User‑USD Directive schema.
User Domain Response Shall validate against User Domain schema.
User Interaction History Response Shall validate against Interaction History schema.
Produces Harmonised Multimodal Context Shall validate against Harmonised Multimodal Context schema.
User‑USD Status Shall validate against User‑USD Status schema.
User Domain Request Shall validate against User Domain schema.
User Interaction History Request Shall validate against Interaction History schema.

9 Performance Assessment

Not part of this specification.

Go to PGM-AUA V1.0 AI Modules