| 1 Functions | 2 Reference Model | 3 I/O Data |
| 4 Functions of AI Modules | 5 I/O Data of AI Modules | 6 AIW, AIMx, and JSON Metadata |
| 7 Reference Software | 8 Conformance Texting | 9 Performance Assessment |
1 Functions
When humans have a conversation with other humans, they use speech and, in constrained cases, text. Their interlocutors perceive speech and/or text supplemented by visual information related to the speaker’s face and body of a conversing human. Text, speech, face, and Body may convey information about the internal state of the speaker that MPAI calls Personal Status. Therefore, handling of Personal Status information in a human-machine conversation and, in the future, even machine-machine conversation, is a key feature of a machine trying to understand what the speakers’ utterances mean because Personal Status recognition can improve understanding of the speaker’s utterance and help a machine produce better replies.
Conversation with Personal Status (MMC-CPS) is a general Use Case of an Entity – a real human or Digital Human – conversing with and asking questions to a machine. The machine captures and understands Text and Speech, extracts Personal Status from the Text, Speech, Face, and Body Factors, fuses the Factors’ Personal Statuses into an estimated Personal Status of the Entity to achieve a better understanding of the context in which the Entity utters Speech.
2 Reference Model
Figure 1 gives the Conversation with Personal Status Reference Model including the input/output data, the AIMs, and the data exchanged between and among the AIMs.

Figure 1 – Reference Model of Conversation with Personal Status
The operation of the Conversation with Personal Status AIM develops as follows:
- Media Selector is used to inform the machine whether the human employs Text or Speech in conversation with the machine.
- Multimodal Frontend Description produces the Basic Speech Object, the Face, Body and other Basic Visual Objects and the Basic Visual Scene Descriptors from the Input Visual and the Input Audio.
- Entity Body Description produces the Body Descriptors from the Body Object.
- Visual Object Identification produces the Instance ID of the Visual Object the human indicates from the Body Descriptors, the Basic Visual Objects and the Basic Visual Scene Descriptors.
- Automatic Speech Recognition produces the Recognised Text from the Basic Speech Object.
- Natural Language Understanding produces the Text Descriptors and the Refined Text from the Recognised Text or the Input Text, the Instance ID and the Media Selector.
- Personal Status Extraction produces the human’s Input Personal Status from the Basic Speech Object, the Face Object, the Body Object and the Text Descriptors.
- Entity Dialogue Processing produces the Machine Text and the Machine Personal Status from the Input Personal Status, the Text Descriptors, the Refined Text, the Input Text and the Instance ID.
- Portable Avatar Multiplexing produces the Portable Avatar from the Machine Text, the Machine Personal Status and the Machine ID. The Portable Avatar is rendered by the User Agent.
3 I/O Data
Table 1 gives the input and output data of the Conversation with Personal Status AIM.
Table 1 – I/O Data of Conversation with Personal Status
| Input | Descriptions |
| Input Text (optional): Basic Text Object | Text typed by the human as additional information stream or as a replacement of the Speech. |
| Input Audio (optional): Basic Audio Object | The audio of the scene, including the Speech of the human having a conversation with the machine. |
| Input Visual: Basic Visual Object | Visual information of the human having a conversation with the machine and of the scene. |
| Media Selector: Selector | Data determining the use of Speech vs Text. |
| Machine ID: string | The Identifier of the Machine, the ID of the Avatar in the Portable Avatar. |
| Output | Descriptions |
| Portable Avatar: Portable Avatar | The Portable Avatar produced by the Machine, with the Machine’s Text and Personal Status. |
4 Functions of AI Modules
Table 2 provides the functions of the AI Modules of Conversation with Personal Status.
Table 2 – Functions of AI Modules of Conversation with Personal Status
| AIM | Function |
| Multimodal Frontend Description | 1. Receives the Input Visual and the Input Audio. 2. Provides the Basic Speech Object, the Face, Body and other Basic Visual Objects and the Basic Visual Scene Descriptors. |
| Entity Body Description | 1. Receives the Body Object. 2. Provides the Body Descriptors. |
| Visual Object Identification | 1. Receives the Body Descriptors, the Basic Visual Objects and the Basic Visual Scene Descriptors. 2. Finds the Visual Object that the human body indicates and provides its Instance ID. |
| Automatic Speech Recognition | 1. Receives the Basic Speech Object. 2. Provides the Recognised Text. |
| Natural Language Understanding | 1. Receives the Recognised Text or the Input Text, the Instance ID and the Media Selector. 2. Provides the Refined Text, the Text Descriptors and the Instance ID. |
| Personal Status Extraction | 1. Receives the Basic Speech Object, the Face Object, the Body Object and the Text Descriptors. 2. Provides the Input Personal Status. |
| Entity Dialogue Processing | 1. Receives the Input Personal Status, the Text Descriptors, the Refined Text, the Input Text and the Instance ID. 2. Produces the Machine Text and the Machine Personal Status. |
| Portable Avatar Multiplexing | 1. Receives the Machine Text, the Machine Personal Status and the Machine ID. 2. Provides the Portable Avatar. |
5 I/O Data of AI Modules
Table 3 provides the I/O Data of the AI Modules of Conversation with Personal Status.
Table 3 – I/O Data of AI Modules of Conversation with Personal Status
6 AIW, AIMs, and JSON Metadata and AIMs
Table 4 provides the links to the AIM specifications and to the JSON syntaxes. AIMs/1 indicates the Composite AIM and the following columns indicate its SubAIMs.
Table 4 – AIMs and JSON Metadata of Conversation with Personal Status
| AIW | AIMs/1 | AIMs/2 | Name | JSON |
| MMC-CPS | Conversation with Personal Status | X | ||
| OSD-MFD | Multimodal Frontend Description | X | ||
| PAF-EBD | Entity Body Description | X | ||
| CVE-VOI | Visual Object Identification | X | ||
| CVE-VDI | Visual Direction Identification | X | ||
| OSD-VOE | Visual Object Extraction | X | ||
| CVE-VII | Visual Instance Identification | X | ||
| MMC-ASR | Automatic Speech Recognition | X | ||
| MMC-NLU | Natural Language Understanding | X | ||
| MMC-PSE | Personal Status Extraction | X | ||
| MMC-ETD | Entity Text Description | X | ||
| MMC-ESD | Entity Speech Description | X | ||
| PAF-EFD | Entity Face Description | X | ||
| PAF-EBD | Entity Body Description | X | ||
| MMC-PTI | PS-Text Interpretation | X | ||
| MMC-PSI | PS-Speech Interpretation | X | ||
| PAF-PFI | PS-Face Interpretation | X | ||
| PAF-PGI | PS-Body Interpretation | X | ||
| MMC-PMX | Personal Status Multiplexing | X | ||
| MMC-EDP | Entity Dialogue Processing | X | ||
| PAF-PMX | Portable Avatar Multiplexing | X |
7 Reference Software
8 Conformance Testing
Table 5 provides the Conformance Testing Method for MMC-CPS AIM.
If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present and conform with the Qualifier, if present.
Table 5 – Conformance Testing Method for MMC-CPS AIM
| Receives | Input Text: Basic Text Object | Shall validate against the Basic Text Object Schema. Text Data shall conform with the Text Qualifier. |
| Input Audio: Basic Audio Object | Shall validate against the Basic Audio Object Schema. Audio Data shall conform with the Audio Qualifier. |
|
| Input Visual: Basic Visual Object | Shall validate against the Basic Visual Object Schema. Visual Data shall conform with the Visual Qualifier. |
|
| Media Selector: Selector | Shall validate against the Selector Schema. | |
| Machine ID: string | Shall validate against the string Schema. | |
| Produces | Portable Avatar: Portable Avatar | Shall validate against the Portable Avatar Schema. Data shall conform with the respective Qualifiers. |