Multimodal Conversation (MPAI-MMC)
Multi-modal conversation (MPAI-MMC) Version 1 is an MPAI standard covering Conversation with Emotion supporting audio-visual conversation with a machine impersonated by a synthetic voice and an animated face; Multimodal Question Answering supports request for information about a displayed object; Unidirectional, Bidirectional and One-to-Many Speech Translation support conversational translation using a synthetic voice that preserves the speech features of the human.
MPAI is now developing Multi-modal conversation (MPAI-MMC) Version 2 including 5 use cases:
- Personal Status Extraction: provides an estimate of the Personal Status (PS) – of a human or an avatar – conveyed by Text, Speech, Face, and Gesture. PS is the ensemble of information internal to a person, including Emotion, Cognitive State, and Attitude.
- Personal Status Display: generates an avatar from Text and PS that utters speech with the intended PS while the face and gesture show the intended PS.
- Conversation About a Scene: a human holds a conversation with a machine about objects in a scene. While conversing, the human points their fingers to indicate their interest in a particular object. The machine is helped by the understanding of the human’s PS.
- Human-Connected Autonomous Vehicle (CAV) Interaction: a group of humans converse with a CAV which understands the utterances and the PSs of the humans it converses with and manifests itself as the output of a Personal Status Display.
- Avatar-Based Videoconference: avatars representing humans with a high degree of accuracy participate in a videoconference. A virtual secretary (VS) represented as an avatar displaying PS creates an online summary of the meeting with a quality enhanced by the virtual secretary’s ability to understand the PS of the avatar it converses with.
Read the current version of the MPAI-MMC Use Cases and Functional Requirements
Development of the MPAI-MMC Technical Specification V1 is completed. MPAI is indebted to the following individuals: Miran Choi (ETRI), Gérard Chollet (IMT), Jisu Kang (KLleon), Mark Seligman (SMI) and Fathy Yassa (SMI) for their efforts.
Reference Software, Conformance Testing and Performance Assessment are under development. Use Cases and Functional Requirememnst for MPAI-MMCX V2 are being developed.
- Version 1.1
Visit the About MPAI-MMC page