1     Version

V2.1

2     Functions

The MMC-CPS AI Workflow:

  1. Receives:
    • Audio Input and Visual Input of a human wishing to have a conversation with a Machine
    • Input Text, complementing or replacing the Audio Input and Visual Input.
  2. Understands the information and the Personal Status of the human.
  3. Produces and emits multimodal responses to the humans in the form of a speaking Avatar that displays Personal Status.

3      Reference Model

Figure 1 depicts the MMC-CPS Reference Architecture.

Figure 1 – The Conversation with Personal Status AIW

4      I/O Data

The Input and Output Data of the Human and Machine Communication AIW is specified in Table 1.

Table 1 – I/O Data of Conversation with Personal Status

Input Descriptions
Input Text Text typed by the human as additional information stream or as a replacement of the Speech.
Input Speech Speech of the human having a conversation with the machine.
Input Visual Visual information of the Face and Body of the human having a conversation with the machine.
Input Selector Data determining the use of Speech vs Text.
Output Descriptions
Machine Portable Avatar The representation of the Machine response.

5      SubAIMs

– OSD-VSD Visual Scene Description X
– CAE-ASD Audio Scene Description X
– CAE-AAT Audio Analysis Transform X
– CAE-ASL Audio Source Localisation X
– CAE-ASE Audio Separation and Enhancement X
– CAE-AST Audio Synthesis Transform X
– CAE-AMX Audio Descriptor Multiplexing X
– OSD-VOI Visual Object Identification X
– OSD-VDI Visual Direction Identification X
– OSD-VOE Visual Object Extraction X
– OSD-VII Visual Instance Identification X
– OSD-AVA Audio-Visual Alignment X
– MMC-ASR Automatic Speech Recognition X
– MMC-NLU Natural Language Understanding X
– MMC-PSE Personal Status Extraction X
– MMC-ITD Input Text Description X
– MMC-ISD Input Speech Description X
– PAF-IFD Input Face Description X
– PAF-IBD Input Body Description X
– MMC-PTI PS-Text Interpretation X
– MMC-PSI PS-Speech Interpretation X
– PAF-PFI PS-Face Interpretation X
– PAF-PGI PS-Gesture Interpretation X
– MMC-PMX Personal Status Multiplexing X
– MMC-EDP Entity Dialogue Processing X
– PAF-PSD Personal Status Display X
– MMC-TTS Text-to-Speech X
– PAF-IFD Input Face Description X
– PAF-IBD Input Body Description X
– PAF-PMX Portable Avatar Multiplexing X

6    JSON Metadata

https://schemas.mpai.community/MMC/V2.1/AIWs/ConversationWithPersonalStatus.json