This is the home page of the Multi-modal conversation (MPAI-MMC) standard. MPAI-MMC uses AI to enable human-machine conversation emul­ating human-human conversation in completeness and intensity. The MPAI-MMC standard includes 5 Use Cases: Conversation with Emotion, Multimodal Question Answering, Unidirectional Speech Translation, Bidirectional Speech Translation and One-to-Many Unidirectional Speech Translation Use Cases.

The figure below shows the reference models of the MPAI-MMC Use Cases. Note that an Implementation is supposed to run in the MPAI-specified AI Framework (MPAI-AIF).

Conversation with Emotion (CWE) enables a human to holds an audio-visual conver­sation using audio and video with a computational system that is impersonated by a synthetic voice and an animated face, both expressing emotion appropriate to the emotional state of the human.
Figure 1 – Conversation with Emotion
Multimodal Question Answering (MQA) enables a user to request information using speech concerning an object the user displays and to receive the requested information from a computational system via synthetic speech.
Figure 2 – Multimodal Question Answering
Unidirectional Speech Translation (UST) allows a user to select a language different from the one s/he uses and to get a spoken utterance translated into the desired language with a synthetic voice that optionally preserves the personal vocal traits of the spoken utterance.
Figure 3 – Unidirectional Speech Translation
Bidirectional Speech Translation (BST) allows a human to hold a dialogue with another human. Both speech their own language and their translated speech is a synthetic speech that optionally preserves their personal vocal traits.
Figure 4 – Bidirectional Speech Translation
One-to-Many Speech Translation (MST) enables a human to select a number of languages and have their speech translates to the selected languages using a synthetic speech that optionally preserves their personal vocal traits.
Figure 5 – One-to-Many Speech Translation

The MPAI-MMC Technical Specification has been developed by the MMC Development Committee (MMC-DC) chaired by Miran Choi (ETRI). The MPAI-MMC Technical Specification has been approved and is available for public download.

MMC-DC is developing the Reference Software, Conformance Testing and Performance Assessment Specifications. It is also developing MPAI-MMC Version 2. This will contain the Human-CAV Interaction subsystem of Connected Autonomous Vehicles (MPAI-CAV) depicyed .

Figure 6 – Human-CAV Interaction
Human-CAV interaction (HCI) recognises the human CAV rights holder, responds to humans’ commands and queries, provides extended environment representation (Full World Representation) for humans to use, senses human activities during the travel and may activate other Subsystems as required by humans. See MMC-HCI Use Case and Functional Requirements WD0.5

If you wish to participate in this work you have the following options:

  1. Join MPAI
  2. Participate until the Functional Requirements of MMC-HCI are approved (after that only MPAI members can participate) by sending an email to the MPAI Secretariat.
  3. Keep an eye on this page.