Go to MPAI-MMC V2.5 AI Modules
Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance
1 Functions
The Multimodal Conversation Avatar (MMC‑MCA) AIM conducts a conversation between a human and a machine represented by a Portable Avatar. The human’s turn is spoken (Speech Object) or typed (Text Object); the machine’s answer is given as a Portable Avatar that speaks it, with lip-synchronised Face Descriptors. The conversation is anonymous: the human has no identity, no ID Reconciliation takes place, and no Personal Status is used.
2 Reference Model
Figure 1 depicts the Reference Model of the Multimodal Conversation Avatar (MMC‑MCA) AIM.
Figure 1 – Reference Model of the Multimodal Conversation Avatar (MMC‑MCA) AIM
The operation of the Multimodal Conversation Avatar (MMC‑MCA) AIM develops in the following way:
- A human provides a turn of the conversation as:
- Speech, or
- Text.
- The User Agent provides the Avatar Model that represents the machine and the Machine ID.
- The machine provides the Portable Avatar that utters its response.
3 I/O Data
Table 1 specifies the Input and Output Data of the Multimodal Conversation Avatar (MMC‑MCA) AIM.
| Input | Description |
|---|---|
| Input Speech | Speech of the human’s turn, if spoken. Optional. |
| Input Text | Text of the human’s turn, if typed. Optional. |
| Avatar Model | The 3D model of the avatar that represents the machine. |
| Machine ID | Identifier of the machine. Optional. |
| Summary | The Summary of the conversation so far, as the Module gave it with the previous turn. Optional: absent at the first turn. |
| Output | Description |
| Portable Avatar | The avatar that utters the machine’s response, with its Machine Speech and lip-synchronised Face Descriptors. |
| Edited Summary | The Summary of the conversation with this turn included. The User Agent gives it back with the next turn. |
4 SubAIMs
4.1 Reference Model
Figure 2 depicts the Reference Model of the Multimodal Conversation Avatar (MMC‑MCA) Composite AIM.
Figure 2 – Reference Model of the Multimodal Conversation Avatar (MMC‑MCA) Composite AIM
4.2 Operation
The Multimodal Conversation Avatar (MMC‑MCA) AIM receives the human’s turn as a Speech Object, which Automatic Speech Recognition converts to Text, or as Text. Entity Dialogue Processing produces the machine’s response as Text, using the Summary of the conversation, which it receives with the turn and returns, updated, as the Edited Summary. Text-to-Speech produces the Machine Speech, Generative Face Description the lip-synchronised Machine Face Descriptors, and Portable Avatar Multiplexing combines Speech, Text, Face Descriptors, Avatar Model and Machine ID into the Portable Avatar. Acquisition and delivery are the User Agent’s, not SubAIMs.
4.3 Functions of SubAIMs
Table 2 provides the functions of the Multimodal Conversation Avatar (MMC‑MCA) SubAIMs.
| SubAIM | Function |
|---|---|
| Automatic Speech Recognition | Recognises as Text an input Speech. |
| Entity Dialogue Processing | Receives the text of the human’s turn and produces the machine’s response Text, using the Summary of the conversation received with the turn, which it updates and returns as the Edited Summary. |
| Text-to-Speech | Converts the machine’s Text into Machine Speech. |
| Generative Face Description | Produces the lip-synchronised Machine Face Descriptors from the machine’s Text and Speech. |
| Portable Avatar Multiplexing | Combines Speech, Text, Face Descriptors, Avatar Model and Machine ID into the Portable Avatar. |
4.4 I/O Data of SubAIMs
Table 3 gives the Input and Output Data of the Multimodal Conversation Avatar (MMC‑MCA) SubAIMs.
| SubAIM | Input | Output |
|---|---|---|
| Automatic Speech Recognition | Speech Object | Recognised Text |
| Entity Dialogue Processing | Text of the turn Summary |
Machine Text Edited Summary |
| Text-to-Speech | Machine Text | Machine Speech |
| Generative Face Description | Machine Text Machine Speech |
Machine Face Descriptors |
| Portable Avatar Multiplexing | Machine Speech Machine Text Machine Face Descriptors Avatar Model Machine ID |
Portable Avatar |
4.5 AIMs and JSON Metadata
Table 4 provides the links to the AIM specifications and JSON schemas. AIM1 indicates the Composite AIM and AIM2 its SubAIMs.
| AIM1 | AIM2 | Name | JSON |
|---|---|---|---|
| MMC‑MCA | Multimodal Conversation Avatar | X | |
| MMC‑ASR | Automatic Speech Recognition | X | |
| MMC‑EDP | Entity Dialogue Processing | X | |
| MMC‑TTS | Text-to-Speech | X | |
| PAF‑GFD | Generative Face Description | X | |
| PAF‑PMX | Portable Avatar Multiplexing | X |
5 JSON Metadata
https://schemas.mpai.community/MMC/V2.5/AIMs/MultimodalConversationAvatar.json
6 Profiles
No Profiles.
7 Reference Software
Under development.
8 Conformance Testing
Table 5 provides the Conformance Testing Method for the Multimodal Conversation Avatar (MMC‑MCA) AIM. Conformance Testing of the individual SubAIMs is given by the individual AIM specifications.
If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.
| Receives | Input Speech | Shall validate against Basic Speech Object schema. Speech Data shall conform with Speech Qualifier. |
| Input Text | Shall validate against Basic Text Object schema. Text Data shall conform with Text Qualifier. | |
| Avatar Model | Shall validate against Basic 3D Model Object schema. 3D Model Data shall conform with 3D Model Qualifier. | |
| Machine ID | Shall be a string. | |
| Summary | Shall validate against Summary schema. | |
| Produces | Portable Avatar | Shall validate against Portable Avatar schema. |
| Edited Summary | Shall validate against Summary schema. |
9 Performance Assessment
Not part of this specification.
