Go to PGM-AUA V1.0 AI Modules

Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Behavioural and Expressive Analysis (PGM‑BEA) AI Module derives, from the bodily evidence of the Harmonised Multimodal Context, the indicators borne by what an Entity does and by what its body expresses, by

  • Deriving the expressive indicators borne by the face, the gaze, and the head
  • Deriving the behavioural indicators borne by posture, gesture, and movement in the Space
  • Determining what the Entity attends to and how it is disposed towards the A‑User
  • Recording where the bodily evidence agrees with the Linguistic‑Paralinguistic Evidence and where it does not
  • Producing the Behavioural and Expressive Indicators

The module is analytical. It derives indicators about the conduct of an Entity; it SHALL NOT construct a state of that Entity, which is the function of Entity State Construction.

Each indicator produced carries the identifier of the Entity to which the Harmonised Multimodal Context attributed the evidence it derives from, and a measure of confidence. An indicator derived while the Entity was occluded, out of frame, or otherwise imperfectly observed is qualified by the visibility the Enhanced Visual Scene Descriptors report for it.

2 Reference Model

Figure 1 depicts the Reference Model of the Behavioural and Expressive Analysis (PGM‑BEA) AIM.

Figure 1 – The Behavioural and Expressive Analysis (PGM‑BEA) AIM

The Behavioural and Expressive Analysis (PGM‑BEA) AI Module operates as follows:

  1. Receives the Harmonised Multimodal Context and the Linguistic‑Paralinguistic Evidence
  2. Derives the expressive indicators:
    • Facial expression and its intensity
    • Gaze direction, fixation, and aversion
    • Head pose, nodding, and shaking
    • Blink and eye‑opening behaviour
  3. Derives the behavioural indicators:
    • Posture and its orientation with respect to the A‑User and to the other Entities of the Space
    • Gesture, including deictic gesture and its referent
    • Movement in the Space: approach, withdrawal, and stillness
    • Manipulation of the Objects of the Space, and the affordances thereby exercised
    • Rhythm of the exchange: turn‑taking, overlapping, and waiting
  4. Determines:
    • Attention Target, i.e., the Entity or Object the Entity attends to
    • Engagement, i.e., the degree of the Entity’s participation in the exchange
    • Interpersonal disposition, i.e., the stance the Entity’s conduct takes towards the A‑User
  5. Compares the bodily evidence with the Linguistic‑Paralinguistic Evidence:
    • Marks the intervals in which the two reinforce each other
    • Marks the intervals in which they are at variance
    • Marks the intervals for which one of the two is absent or unusable
  6. Constructs consistent Behavioural and Expressive Indicators
  7. Outputs the Behavioural and Expressive Indicators for use by downstream AI Modules

The module operates in real time, without recourse to the reasoning capability of Basic Knowledge and without Model Context Protocol interactions. Domain knowledge is obtained by query, and the Response to a query is in general received during a subsequent processing cycle.

The Domain Response supports the interpretation of conduct in the terms of the domain: the gestures and postures conventional in the setting, the proximity and gaze customary in it, and the conduct the setting expects or forbids.

3 I/O Data

Table 1 specifies the Input and Output Data of the Behavioural and Expressive Analysis (PGM‑BEA) AIM.

Table 1 – I/O Data of the Behavioural and Expressive Analysis (PGM‑BEA) AIM

Input Description
Harmonised Multimodal Context Audio, visual, textual, and paralinguistic evidence of the scene placed on a common time base and attributed to harmonised Entities.
Linguistic‑Paralinguistic Evidence Evidence borne by the content of the utterances of an Entity and by the manner of their delivery.
User‑USD Directive Control directive specifying scope, depth, or policy constraints for behavioural and expressive analysis.
User Domain Response Domain‑specific knowledge supporting the interpretation of conduct in the terms of the domain.
User Interaction History Response Prior interaction‑history content supporting the assessment of conduct against that observed in earlier captures.
Output Description
Behavioural and Expressive Indicators Indicators borne by the conduct of an Entity and by the expression of its body, with their agreement with the Linguistic‑Paralinguistic Evidence.
User‑USD Status Status information describing the execution and outcome of Behavioural and Expressive Analysis processing.
User Domain Request Query to Domain Access for domain‑specific knowledge.
User Interaction History Request Query to A‑User Storage for prior interaction‑history content.

4 SubAIMs

No SubAIMs.

5 JSON Metadata

https://schemas.mpai.community/PGM1/V1.0/AIMs/BehaviouralAndExpressiveAnalysis.json

6 Profiles

No Profiles.

7 Reference Software

Not part of this specification.

8 Conformance Testing

Table 2 provides the Conformance Testing Method for the Behavioural and Expressive Analysis (PGM‑BEA) AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 2 – Conformance Testing Method for the Behavioural and Expressive Analysis (PGM‑BEA) AIM

Receives Harmonised Multimodal Context Shall validate against Harmonised Multimodal Context schema.
Linguistic‑Paralinguistic Evidence Shall validate against Linguistic‑Paralinguistic Evidence schema.
User‑USD Directive Shall validate against User‑USD Directive schema.
User Domain Response Shall validate against User Domain schema.
User Interaction History Response Shall validate against Interaction History schema.
Produces Behavioural and Expressive Indicators Shall validate against Behavioural and Expressive Indicators schema.
User‑USD Status Shall validate against User‑USD Status schema.
User Domain Request Shall validate against User Domain schema.
User Interaction History Request Shall validate against Interaction History schema.

9 Performance Assessment

Not part of this specification.

Go to PGM-AUA V1.0 AI Modules