Go to MPAI-MMC V2.5 AI Modules

Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Affective Multimodal Conversation (MMC‑AMC) AIM conducts a conversation in which the machine is aware both of what the human says and of how the human feels. It recognises the speech, understands the text, extracts the Personal Status from speech and, if provided, from the face, combines it with the Personal Status of the text, produces the machine’s response and the machine’s own Personal Status, and presents them as a Portable Avatar.

2 Reference Model

Figure 1 depicts the Reference Model of the Affective Multimodal Conversation (MMC‑AMC) AIM.
 
Affective Multimodal Conversation AIM

Figure 1 – Reference Model of the Affective Multimodal Conversation (MMC‑AMC) AIM

The operation of the Affective Multimodal Conversation (MMC‑AMC) AIM develops in the following way:

  1. A human provides:
    1. Speech, or text typed instead of speaking.
    2. Optionally, an image of the face.
  2. The User Agent provides the Avatar Model that represents the machine and the Machine ID.
  3. The machine provides the Portable Avatar that utters its response with its own Personal Status.

3 I/O Data

Table 1 specifies the Input and Output Data of the Affective Multimodal Conversation (MMC‑AMC) AIM.

Table 1 – I/O Data of the Affective Multimodal Conversation (MMC‑AMC) AIM

Input Description
Input Speech Speech of the human’s turn, if spoken.
Input Face Image of the human’s face, from which the Personal Status of the face is extracted. Optional.
Input Text Text typed by the human instead of speaking. Optional.
Avatar Model The 3D model of the avatar that represents the machine.
Machine ID Identifier of the machine. Optional.
Summary The Summary of the conversation so far, as the Module gave it with the previous turn. Optional: absent at the first turn.
Output Description
Portable Avatar The avatar that utters the machine’s response with its Personal Status.
Edited Summary The Summary of the conversation with this turn included. The User Agent gives it back with the next turn.

4 SubAIMs

4.1 Reference Model

Figure 2 depicts the Reference Model of the Affective Multimodal Conversation (MMC‑AMC) Composite AIM.
 
Affective Multimodal Conversation Composite AIM

Figure 2 – Reference Model of the Affective Multimodal Conversation (MMC‑AMC) Composite AIM

4.2 Operation

The Affective Multimodal Conversation (MMC‑AMC) AIM receives the human’s speech, which Automatic Speech Recognition converts to Text. Natural Language Understanding produces the Text Descriptors, the refined Text and the Text Personal Status. Personal Status Extraction reads the Speech Personal Status from the Speech Object, and the Face Personal Status from the face if provided, and multiplexes them with the Text Personal Status into the Personal Status of the human. Entity Dialogue Processing produces the machine’s response Text and Personal Status, aware both of the meaning and of the feeling; the Summary of the conversation reaches it with the turn and leaves it, updated, as the Edited Summary. Response and Scene Rendering renders the Portable Avatar. Acquisition and delivery are the User Agent’s, not SubAIMs.

4.3 Functions of SubAIMs

Table 2 provides the functions of the Affective Multimodal Conversation (MMC‑AMC) SubAIMs.

Table 2 – Functions of the Affective Multimodal Conversation (MMC‑AMC) SubAIMs

SubAIM Function
Automatic Speech Recognition Recognises as Text an input Speech.
Natural Language Understanding Understands the recognised Text: produces the refined Text, its Text Descriptors and the Text Personal Status.
Personal Status Extraction Extracts the Personal Status of the speech and of the face and multiplexes them with the Text Personal Status into the Personal Status of the human.
Entity Dialogue Processing Receives the refined Text and the Personal Status of the human and produces the machine’s response Text and its Personal Status; it uses the Summary of the conversation received with the turn and returns it, updated, as the Edited Summary.
Response and Scene Rendering Receives the machine’s Text and Personal Status, the Avatar Model and the Machine ID and produces the Portable Avatar that speaks the Text with lip-synchronised Machine Face Descriptors.

4.4 I/O Data of SubAIMs

Table 3 gives the Input and Output Data of the Affective Multimodal Conversation (MMC‑AMC) SubAIMs.

Table 3 – I/O Data of the Affective Multimodal Conversation (MMC‑AMC) SubAIMs

SubAIM Input Output
Automatic Speech Recognition Speech Object Recognised Text
Natural Language Understanding Recognised Text Refined Text
Text Personal Status
Personal Status Extraction Speech Object
Face
Text Personal Status		 Personal Status
Entity Dialogue Processing Refined Text
Personal Status
Summary		 Machine Text
Machine Personal Status
Edited Summary
Response and Scene Rendering Machine Text
Machine Personal Status
Avatar Model
Machine ID		 Portable Avatar

4.5 AIMs and JSON Metadata

Table 4 provides the links to the AIM specifications and JSON schemas. AIM1 indicates the Composite AIM and AIM2 its SubAIMs.

Table 4 – AIMs and JSON Metadata of the Affective Multimodal Conversation (MMC‑AMC)

AIM1 AIM2 Name JSON
MMC‑AMC Affective Multimodal Conversation X
MMC‑ASR Automatic Speech Recognition X
MMC‑NLU Natural Language Understanding X
MMC‑PSE Personal Status Extraction X
MMC‑EDP Entity Dialogue Processing X
PAF‑RSR Response and Scene Rendering X

5 JSON Metadata

https://schemas.mpai.community/MMC/V2.5/AIMs/AffectiveMultimodalConversation.json

6 Profiles

No Profiles.

7 Reference Software

Under development.

8 Conformance Testing

Table 5 provides the Conformance Testing Method for the Affective Multimodal Conversation (MMC‑AMC) AIM. Conformance Testing of the individual SubAIMs is given by the individual AIM specifications.
If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 5 – Conformance Testing Method for the Affective Multimodal Conversation (MMC‑AMC) AIM

Receives Input Speech Shall validate against Basic Speech Object schema. Speech Data shall conform with Speech Qualifier.
Input Face Shall validate against Basic Visual Object schema. Visual Data shall conform with Visual Qualifier.
Input Text Shall validate against Basic Text Object schema. Text Data shall conform with Text Qualifier.
Avatar Model Shall validate against Basic 3D Model Object schema. 3D Model Data shall conform with 3D Model Qualifier.
Machine ID Shall be a string.
Summary Shall validate against Summary schema.
Produces Portable Avatar Shall validate against Portable Avatar schema.
Edited Summary Shall validate against Summary schema.

9 Performance Assessment

Not part of this specification.

Go to MPAI-MMC V2.5 AI Modules