<-Go to AI Modules Go to ToC

1 Functions 2 Reference Model 3 I/O Data
4 Functions of AI Modules 5 I/O Data of AI Modules 6 AIW, AIMx, and JSON Metadata
7 Reference Software 8 Conformance Texting 9 Performance Assessment

1 Functions

When humans have a conversation with other humans, they use speech and, in constrained cases, text. Their interlocutors perceive speech and/or text supplemented by visual information related to the speaker’s face and body of a conversing human. Text, speech, face, and Body may convey information about the internal state of the speaker that MPAI calls Personal Status. Therefore, handling of Personal Status information in a human-machine conversation and, in the future, even machine-machine conversation, is a key feature of a machine trying to understand what the speakers’ utterances mean because Personal Status recognition can improve understanding of the speaker’s utterance and help a machine produce better replies.

Conversation with Personal Status (MMC-CPS) is a general Use Case of an Entity – a real human or Digital Human – conversing with and asking questions to a machine. The machine captures and understands Text and Speech, extracts Personal Status from the Text, Speech, Face, and Body Factors, fuses the Factors’ Personal Statuses into an estimated Personal Status of the Entity to achieve a better understanding of the context in which the Entity utters Speech.

2 Reference Model

Figure 1 gives the Conversation with Personal Status Reference Model including the input/output data, the AIMs, and the data exchanged between and among the AIMs.

Conversation with Personal Status MMC-CPS

Figure 1 – Reference Model of Conversation with Personal Status

The operation of the Conversation with Personal Status AIM develops as follows:

  1. Media Selector is used to inform the machine whether the human employs Text or Speech in conversation with the machine.
  2. Multimodal Frontend Description produces the Basic Speech Object, the Face, Body and other Basic Visual Objects and the Basic Visual Scene Descriptors from the Input Visual and the Input Audio.
  3. Entity Body Description produces the Body Descriptors from the Body Object.
  4. Visual Object Identification produces the Instance ID of the Visual Object the human indicates from the Body Descriptors, the Basic Visual Objects and the Basic Visual Scene Descriptors.
  5. Automatic Speech Recognition produces the Recognised Text from the Basic Speech Object.
  6. Natural Language Understanding produces the Text Descriptors and the Refined Text from the Recognised Text or the Input Text, the Instance ID and the Media Selector.
  7. Personal Status Extraction produces the human’s Input Personal Status from the Basic Speech Object, the Face Object, the Body Object and the Text Descriptors.
  8. Entity Dialogue Processing produces the Machine Text and the Machine Personal Status from the Input Personal Status, the Text Descriptors, the Refined Text, the Input Text and the Instance ID.
  9. Portable Avatar Multiplexing produces the Portable Avatar from the Machine Text, the Machine Personal Status and the Machine ID. The Portable Avatar is rendered by the User Agent.

3 I/O Data

Table 1 gives the input and output data of the Conversation with Personal Status AIM.

Table 1 – I/O Data of Conversation with Personal Status

Input Descriptions
Input Text (optional): Basic Text Object Text typed by the human as additional information stream or as a replacement of the Speech.
Input Audio (optional): Basic Audio Object The audio of the scene, including the Speech of the human having a conversation with the machine.
Input Visual: Basic Visual Object Visual information of the human having a conversation with the machine and of the scene.
Media Selector: Selector Data determining the use of Speech vs Text.
Machine ID: string The Identifier of the Machine, the ID of the Avatar in the Portable Avatar.
Output Descriptions
Portable Avatar: Portable Avatar The Portable Avatar produced by the Machine, with the Machine’s Text and Personal Status.

4 Functions of AI Modules

Table 2 provides the functions of the AI Modules of Conversation with Personal Status.

Table 2 – Functions of AI Modules of Conversation with Personal Status

AIM Function
Multimodal Frontend Description 1. Receives the Input Visual and the Input Audio.
2. Provides the Basic Speech Object, the Face, Body and other Basic Visual Objects and the Basic Visual Scene Descriptors.
Entity Body Description 1. Receives the Body Object.
2. Provides the Body Descriptors.
Visual Object Identification 1. Receives the Body Descriptors, the Basic Visual Objects and the Basic Visual Scene Descriptors.
2. Finds the Visual Object that the human body indicates and provides its Instance ID.
Automatic Speech Recognition 1. Receives the Basic Speech Object.
2. Provides the Recognised Text.
Natural Language Understanding 1. Receives the Recognised Text or the Input Text, the Instance ID and the Media Selector.
2. Provides the Refined Text, the Text Descriptors and the Instance ID.
Personal Status Extraction 1. Receives the Basic Speech Object, the Face Object, the Body Object and the Text Descriptors.
2. Provides the Input Personal Status.
Entity Dialogue Processing 1. Receives the Input Personal Status, the Text Descriptors, the Refined Text, the Input Text and the Instance ID.
2. Produces the Machine Text and the Machine Personal Status.
Portable Avatar Multiplexing 1. Receives the Machine Text, the Machine Personal Status and the Machine ID.
2. Provides the Portable Avatar.

5 I/O Data of AI Modules

Table 3 provides the I/O Data of the AI Modules of Conversation with Personal Status.

Table 3 – I/O Data of AI Modules of Conversation with Personal Status

AIM Receives Produces
Multimodal Frontend Description Input Visual: Basic Visual Object
Input Audio: Basic Audio Object
Basic Speech Object: Basic Speech Object
Face Object (1): Basic Visual Object
Body Object (2): Basic Visual Object
Basic Visual Objects (3): Basic Visual Object
Basic Visual Scene Descriptors: Basic Visual Scene Descriptors
Entity Body Description Body Object: Basic Visual Object Body Descriptors: Body Descriptors Object
Visual Object Identification Body Descriptors: Body Descriptors Object
Basic Visual Scene Descriptors: Basic Visual Scene Descriptors
Basic Visual Objects: Basic Visual Object
Instance ID: Instance Identifier
Automatic Speech Recognition Basic Speech Object (3): Basic Speech Object Recognised Text (1): Basic Text Object
Natural Language Understanding Input Text (1): Basic Text Object
Recognised Text (2): Basic Text Object
Instance ID: Instance Identifier
Media Selector: Selector
Text Descriptors: Text Descriptors Object
Refined Text: Basic Text Object
Instance ID: Instance Identifier
Personal Status Extraction Basic Speech Object: Basic Speech Object
Face Object (1): Basic Visual Object
Body Object (2): Basic Visual Object
Text Descriptors: Text Descriptors Object
Input Personal Status: Personal Status
Entity Dialogue Processing Input Personal Status (1): Personal Status
Text Descriptors: Text Descriptors Object
Refined Text (3): Basic Text Object
Input Text (1): Basic Text Object
Instance ID (2): Instance Identifier
Machine Text: Basic Text Object
Machine Personal Status: Personal Status
Portable Avatar Multiplexing Machine Text (2): Basic Text Object
Machine Personal Status: Personal Status
Machine ID: string
Portable Avatar: Portable Avatar

6 AIW, AIMs, and JSON Metadata and AIMs

Table 4 provides the links to the AIM specifications and to the JSON syntaxes. AIMs/1 indicates the Composite AIM and the following columns indicate its SubAIMs.

Table 4 – AIMs and JSON Metadata of Conversation with Personal Status

AIW AIMs/1 AIMs/2 Name JSON
MMC-CPS Conversation with Personal Status X
OSD-MFD Multimodal Frontend Description X
PAF-EBD Entity Body Description X
CVE-VOI Visual Object Identification X
CVE-VDI Visual Direction Identification X
OSD-VOE Visual Object Extraction X
CVE-VII Visual Instance Identification X
MMC-ASR Automatic Speech Recognition X
MMC-NLU Natural Language Understanding X
MMC-PSE Personal Status Extraction X
MMC-ETD Entity Text Description X
MMC-ESD Entity Speech Description X
PAF-EFD Entity Face Description X
PAF-EBD Entity Body Description X
MMC-PTI PS-Text Interpretation X
MMC-PSI PS-Speech Interpretation X
PAF-PFI PS-Face Interpretation X
PAF-PGI PS-Body Interpretation X
MMC-PMX Personal Status Multiplexing X
MMC-EDP Entity Dialogue Processing X
PAF-PMX Portable Avatar Multiplexing X

7 Reference Software

8 Conformance Testing

Table 5 provides the Conformance Testing Method for MMC-CPS AIM.

If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present and conform with the Qualifier, if present.

Table 5 – Conformance Testing Method for MMC-CPS AIM

Receives Input Text: Basic Text Object Shall validate against the Basic Text Object Schema.
Text Data shall conform with the Text Qualifier.
Input Audio: Basic Audio Object Shall validate against the Basic Audio Object Schema.
Audio Data shall conform with the Audio Qualifier.
Input Visual: Basic Visual Object Shall validate against the Basic Visual Object Schema.
Visual Data shall conform with the Visual Qualifier.
Media Selector: Selector Shall validate against the Selector Schema.
Machine ID: string Shall validate against the string Schema.
Produces Portable Avatar: Portable Avatar Shall validate against the Portable Avatar Schema.
Data shall conform with the respective Qualifiers.

9 Performance Assessment

<-Go to AI Modules Go to ToC Conversation with Emotion->