Go to PGM-AUA V1.0 AI Modules

Function Ref. Model I/O Data SubAIMs JSON MData Profiles Ref. Software Conformance Performance

1. Function

The User State Description (PGM‑USD) AIM derives a structured representation of the User State from multimodal perceptual and enhanced contextual inputs.

User State Description:

  1. Operates on the Enhanced Audio Scene Descriptors produced by Audio Scene Enhancement, the Enhanced Visual Scene Descriptors produced by Visual Scene Enhancement, the User CXT Request received from A‑User Control, the User Domain Response resulting from queries made to Domain Access, and the User Interaction History Response resulting from queries made, through the Controller, to Private Storage.
  2. Performs a multi‑stage interpretative process including multimodal harmonisation, linguistic and paralinguistic analysis, behavioural and expressive inference, and cross‑modal interpretation, without performing deliberative reasoning or goal‑directed decision‑making.
  3. Produces the User State sent to Context Description Multiplexing, the User CXT Response, the User Domain Request when querying Domain Access, and the User Interaction History Request when querying Interaction History from Private Storage through the Controller.

The User State carries a structured representation of the cognitive, behavioural, expressive, and interactional aspects of the observed User, under User CXT Request control.

2. Reference Model

Figure 1 depicts the Reference Model of the User State Description (PGM‑USD) AIM.

 

Figure 1 – Reference Model of the User State Description (PGM‑USD) AIM

3. Input/Output Data

Table 1 lists the Input and Output Data of the User State Description (PGM‑USD) AIM.

Table 1 – Input/Output Data of the User State Description (PGM‑USD) AIM

Input Description
Enhanced Audio Scene Descriptors Audio descriptors enriched by Audio Scene Enhancement representing speech, paralinguistic cues, and environmental audio context.
Speech Object Speech of the responsible human or of the User, when the User Agent has recognised that the audio it captured is speech. It goes to Automatic Speech Recognition.
Enhanced Visual Scene Descriptors Visual descriptors enriched by Visual Scene Enhancement representing user posture, gestures, and visual interaction context.
Audio-Visual Scene Descriptors The scene in which the audio and visual objects are aligned, produced by Audio-Visual Alignment. It goes to Multimodal Input Harmonisation and to Audio Scene Object Identification.
Text Object Text typed by the User. It goes to Multimodal Input Harmonisation.
User CXT Request Contextual directive specifying scope, depth, and policy constraints for user state derivation.
User Domain Response Domain‑specific knowledge supporting interpretation and inference stages.
User Interaction History Response Interaction History response providing temporal context for user state derivation.
Output Description
User State Structured representation of the cognitive, behavioural, expressive, and interactional state of the observed User.
User CXT Response Multiplexed status signals from the internal SubAIMs reporting their processing outcomes.
User Domain Request Multiplexed domain requests from the internal SubAIMs.
User Interaction History Request Multiplexed Interaction History requests from the internal SubAIMs.

4. Sub-AIMs (Informative)

This section is informative. The decomposition into Sub-AIMs described below illustrates one conformant architecture for producing the normative outputs of PGM‑USD. Implementations may adopt alternative internal structures provided they satisfy the conformance requirements of Section 8.

4.1 Reference Model

Figure 2 depicts the Reference Model of the User State Description (PGM‑USD) Composite AIM.

 

Figure 2 – Reference Model of the User State Description (PGM‑USD) Composite AIM

4.2 Operation

The User State Description receives Enhanced Audio Scene Descriptors, Enhanced Visual Scene Descriptors, and the AUC-CXT Request, and progressively transforms them into a structured Entity State (User State) through a sequence of specialised Sub-AIMs.

The external inputs and control signals are fed to the appropriate Sub-AIMs:

  1. The Audio Scene Object Identification (CAE-ASI) Sub-AIM classifies the audio objects of the Enhanced Audio Scene Descriptors and requalifies the speech objects as Basic Speech Objects. The audio object that Audio-Visual Alignment pairs with the avatar is the likely speech of the avatar.
  2. The Automatic Speech Recognition Sub-AIM converts the Speech Objects, those of Audio Scene Object Identification or the one received from the User Agent, into a Text Object and Speech Descriptors for use by subsequent Sub-AIMs.
  3. The Multimodal Input Harmonisation Sub-AIM aligns temporal, structural, and referential aspects of the multimodal inputs, producing a Harmonised Multimodal Context suitable for subsequent analysis.
  4. The Linguistic‑Paralinguistic Analysis Sub-AIM processes text and speech descriptors to derive linguistic content and paralinguistic features such as prosody, emphasis, and rhythm, producing Linguistic‑Paralinguistic Evidence.
  5. The Behavioural and Expressive Analysis Sub-AIM interprets linguistic evidence and multimodal context to derive behavioural and expressive indicators reflecting interaction patterns, communicative intent, and expressive signals.
  6. The Cross‑Modal Interpretation Sub-AIM integrates linguistic, behavioural, and multimodal evidence to produce a coherent cross‑modal interpretation of user behaviour and interaction context.
  7. The User State Creation Sub-AIM aggregates all derived evidence into a consistent User State and assembles the multiplexed status and request signals.

Each of Linguistic‑Paralinguistic Analysis, Behavioural and Expressive Analysis, and Cross‑Modal Interpretation independently generates a User CXT Response, a User Domain Request, and a User Interaction History Request, which are multiplexed by User State Creation into the corresponding external output signals.

4.3 Functions of Sub-AIMs

Table 2 specifies the functions of the Sub-AIMs of the User State Description (PGM‑USD) Composite AIM.

Table 2 – Functions of the Sub-AIMs of the User State Description (PGM‑USD) Composite AIM

Name Function
Audio Scene Object Identification (CAE-ASI) Classifies the audio objects of the Enhanced Audio Scene Descriptors and requalifies the speech objects as Basic Speech Objects, which it gives to Automatic Speech Recognition. The audio object paired with the avatar by Audio-Visual Alignment is the likely speech of the avatar.
Automatic Speech Recognition Converts the Speech Objects into a Text Object and Speech Descriptors, under User CXT Request control, reporting processing outcome via User CXT Response.
Multimodal Input Harmonisation Aligns temporal, structural, and referential aspects of multimodal inputs into a Harmonised Multimodal Context suitable for subsequent analysis.
Linguistic‑Paralinguistic Analysis Extracts linguistic meaning and paralinguistic cues such as prosody, emphasis, and rhythm from speech and text, producing Linguistic‑Paralinguistic Evidence.
Behavioural and Expressive Analysis Derives Behavioural and Expressive Indices including behavioural patterns and expressive indicators reflecting interaction patterns, communicative intent, and expressive signals from multimodal evidence.
Cross‑Modal Interpretation Integrates linguistic, behavioural, and multimodal evidence into Cross-Modal Interpretation Evidence containing a coherent cross‑modal interpretation of User behaviour and interaction context.
User State Creation Aggregates all derived evidence into a consistent User State and assembles the multiplexed User CXT Response, User Domain Request, and User Interaction History Request.

4.4 Input/Output Data of Sub-AIMs

Table 3 lists the Input and Output Data of the Sub-AIMs of the User State Description (PGM‑USD) Composite AIM.

Table 3 – Input/Output Data of the Sub-AIMs of the User State Description (PGM‑USD) Composite AIM

Name Input Data Output Data
Audio Scene Object Identification (CAE-ASI) Enhanced Audio Scene Descriptors Basic Speech Object, Basic Audio Object
Automatic Speech Recognition Speech Object
SpeakerID
User CXT Request		 Text Object
Speech Descriptors
User CXT Response
Multimodal Input Harmonisation Enhanced Audio Scene Descriptors
Enhanced Visual Scene Descriptors
User CXT Request
User Domain Response
User Interaction History Response		 User CXT Response
User Domain Request
User Interaction History Request
Harmonised Multimodal Context
Linguistic‑Paralinguistic Analysis Harmonised Multimodal Context
Speech Descriptors
User CXT Request
User Domain Response
User Interaction History Response		 Linguistic‑PL Evidence
User CXT Response
User Domain Request
User Interaction History Request
Behavioural and Expressive Analysis Harmonised Multimodal Context
Linguistic‑PL Evidence
User CXT Request
User Domain Response
User Interaction History Response		 Behavioural & Expressive Indicators
User CXT Response
User Domain Request
User Interaction History Request
Cross‑Modal Interpretation Harmonised Multimodal Context
Linguistic‑PL Evidence
Behavioural & Expressive Indicators
User CXT Request
User Domain Response
User Interaction History Response		 Cross‑Modal Interpretation Evidence
User CXT Response
User Domain Request
User Interaction History Request
User State Creation Harmonised Multimodal Context
Linguistic‑PL Evidence
Behavioural & Expressive Indicators
Cross‑Modal Interpretation Evidence
User CXT Response
User Domain Request
User Interaction History Request
User Text		 User State
User CXT Response
User Domain Request
User Interaction History Request
User Text

4.5 AIMs and JSON Metadata

Table 4 gives the User State Description (AIM1) and its Sub-AIMs (AIM2).

Table 4 – User State Description (AIM1) and its Sub-AIMs (AIM2)

AIM1 AIM2 Name JSON
PGM‑USD User State Description X
PGM‑ASR Automatic Speech Recognition X
PGM‑MIH Multimodal Input Harmonisation X
PGM‑LPA Linguistic‑Paralinguistic Analysis X
PGM‑BEA Behavioural and Expressive Analysis X
PGM‑CMI Cross‑Modal Interpretation X
PGM‑USC User State Creation X

5. JSON Metadata

https://schemas.mpai.community/PGM1/V1.0/AIMs/UserStateDescription.json

6. Profiles

No Profiles.

7. Reference Software

Not part of this specification.

8. Conformance Testing

A USD implementation conforms with User State Description (PGM‑USD) if:

  1. The implementation includes all Sub-AIMs listed in Table 2.
  2. All I/O Data listed in Table 1 are present and conform with their respective Data Types.
  3. All Sub-AIM I/O Data listed in Table 3 conform with their respective Data Types.
  4. The implementation produces a User State that integrates the outputs of Linguistic‑Paralinguistic Analysis, Behavioural and Expressive Analysis, and Cross‑Modal Interpretation.

9. Performance Assessment

Not part of this specification.

Go to PGM-AUA V1.0 AI Modules