Go to PGM-AUA V1.0 Data Types

Definition
Functional Requirements
Syntax
Semantics
Conformance Testing
Performance Assessment

Definition

Harmonised Multimodal Context (PGM-MIH): The audio, visual, textual and paralinguistic evidence of the scene placed on a common time base and attributed to harmonised Entities by Multimodal Input Harmonisation, with a record of the evidence that could not be harmonised. It states what was observed, when and by whom, without interpreting it.

Functional Requirements

  1. Header: The Data shall have Header “PGM-MIH-V1.0”.
  2. Identification: The Data shall carry the identifiers of the M‑Instance, of the U‑Environment and of the session, and a unique identifier and a time (HarmonisedMultimodalContextID, HarmonisedMultimodalContextTime).
  3. TimeBase: The Data shall include the common time base and how each stream refers to it.
  4. Entities: The Data shall include the Entities.
  5. Utterances: The Data may include the utterances.
  6. AudioEvidence: The Data may include enhanced Audio Scene Descriptors on the common time base.
  7. VisualEvidence: The Data may include enhanced Visual Scene Descriptors on the common time base.
  8. Unattributed: The Data may include evidence that could not be attributed to any Entity.
  9. Disagreements: The Data may include disagreements between modalities about an Entity.
  10. AbsentStreams: The Data may include streams that are absent, incomplete or outside the tolerance.
  11. Metadata: The Data shall include DataXMData and may include DescrMetadata.

Syntax

https://schemas.mpai.community/PGM1/V1.0/data/HarmonisedMultimodalContext.json

Semantics

Table 1 – Semantics of Harmonised Multimodal Context

Label Description
Header Header of the Data Type, with the version. (required)
MInstance Identifier of the M-Instance. (required)
UEnvironment Identifier of the U-Environment. (required)
SessionID Identifier of the A-User session. (required)
HarmonisedMultimodalContextID Identifier of this instance. (required)
HarmonisedMultimodalContextTime Time at which the Data was produced. (required)
TimeBase The common time base and how each stream refers to it. (required)
– Origin Origin.
– Tolerance Tolerance on the common time base, in seconds.
– Streams How each stream refers to the common time base.
–– Stream The stream. One of: audio, visual, text, speech. (required)
–– Rate Rate at which the stream is produced.
–– Offset Offset of the stream with respect to the common time base.
–– AbsentIntervals Intervals for which the stream is absent.
––– StartTime Time at which the Data was produced.
––– EndTime Time at which the Data was produced.
Entities The Entities. (required)
– HarmonisedEntityID Harmonised Entity ID. (required)
– UserID The User, where identified by the User CXT Request.
– OriginalIDs The identifier of each modality, retained.
–– Audio Identifiers used by the audio modality.
–– Visual Identifiers used by the visual modality.
–– Text Text of the referring expression.
–– Speech Identifiers used by the speech.
Utterances The utterances.
– UtteranceID Utterance ID. (required)
– HarmonisedEntityID Harmonised Entity ID.
– Interval Time interval, on the time base of the capture. (required)
–– StartTime Time at which the Data was produced.
–– EndTime Time at which the Data was produced.
– TextDescriptors Text Descriptors.
– SpeechDescriptors Request: whether the production of the Speech Descriptors is required, optional or not_requested. Response: fulfilment of the production of the Speech Descriptors (produced, partial, missing or not_requested).
AudioEvidence Enhanced Audio Scene Descriptors on the common time base.
VisualEvidence Enhanced Visual Scene Descriptors on the common time base.
Unattributed Evidence that could not be attributed to any Entity.
– Stream The stream. (required)
– ItemID Item ID.
– Interval Time interval, on the time base of the capture.
–– StartTime Time at which the Data was produced.
–– EndTime Time at which the Data was produced.
Disagreements Disagreements between modalities about an Entity.
– Modalities The modalities that disagree.
– HarmonisedEntityID Harmonised Entity ID.
– Description Description of the reason.
AbsentStreams Streams that are absent, incomplete or outside the tolerance. Each one of: audio, visual, text, speech.
DataXMData Data Exchange Metadata of MPAI-PTF: provenance, authorisation, security and confidence of the Data.
DescrMetadata Descriptive metadata (plain text, at most 2048 characters).

Conformance Testing

Data conforms with Harmonised Multimodal Context if it validates against the schema of Section 3. If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema.

Performance Assessment

Not part of this specification.

Go to PGM-AUA V1.0 Data Types