| 1 Functions | 2 Reference Model | 3 I/O Data |
| 4 Functions of AI Modules | 5 I/O Data of AI Modules | 6 AIW, AIMs, and JSON Metadata |
| 7 Reference Software | 8 Conformance Testing | 9 Performance Assessment |
1 Functions
This Use Case addresses the case of a human holding a conversation with a Machine:
- The human converses with the Machine indicating the object in the Environment s/he wishes to talk to or ask questions about it using Speech, Face, and Body.
- The Machine
- Sees and hears an Environment containing a speaking human and some scattered objects.
- Recognises the human’s Speech and obtains the human’s Personal Status by capturing Speech, Face, and Body.
- Understands which object the human is referring to and generates an avatar that:
- Utters Speech conveying a synthetic Personal Status that is relevant to the human’s Personal Status as shown by his/her Speech, Face, and Body, and
- Displays a face conveying a Personal Status that is relevant to the human’s Personal Status and to the response the Machine intends to make.
- Renders the Scene that it perceives from a human-selected Point of View. The objects in the scene are labelled with the Machine’s understanding of their semantics so that the human can understand how the Machine sees the Environment.
2 Reference Model
Figure 1 gives the Conversation About a Scene Reference Model including the input/output data, the AIMs, and the data exchanged between and among the AIMs.

Figure 1 – Reference Model of Conversation About a Scene (MMC-CAS) AIM
The Machine operates according to the following workflow:
- Multimodal Frontend Description produces the Basic Speech Object, the Face, Body and other Basic Visual Objects and the Basic Visual Scene Descriptors from the Input Visual and the Input Audio.
- Entity Body Description produces the Body Descriptors from the Body Object.
- Visual Object Identification produces the Instance ID of the Visual Object the human indicates from the Body Descriptors, the Basic Visual Objects and the Basic Visual Scene Descriptors.
- Automatic Speech Recognition produces the Recognised Text from the Basic Speech Object.
- Natural Language Understanding produces the Text Descriptors and the Refined Text from the Recognised Text and the Instance ID.
- Personal Status Extraction produces the Input Personal Status from the Basic Speech Object, the Face Object, the Body Object and the Text Descriptors.
- Entity Dialogue Processing produces the Machine Text and the Machine Personal Status from the Input Personal Status, the Text Descriptors and the Refined Text.
- Portable Avatar Multiplexing produces the Portable Avatar from the Machine Text, the Machine Personal Status, the Basic Visual Scene Descriptors, the View Selector, the Point of View and the Machine ID. The Portable Avatar is rendered by the User Agent.
3 I/O Data
Table 1 gives the input and output data of the Conversation About a Scene AIM.
Table 1 – I/O Data of Conversation About a Scene
| Input data | From | Description |
| View Selector: Selector | Human | Selects whether the Machine is rendered in the scene (the View element of the Selector: Avatar or Scene). |
| Point of View: Point of View | Human | The point of view of the Multimodal Scene conveyed by the Portable Avatar. |
| Input Visual: Basic Visual Object | Camera | Points to human and scene. |
| Input Audio: Basic Audio Object | Microphone | The audio of the scene, including the Speech of the human. |
| Machine ID: string | Machine | The Identifier of the Machine, the ID of the Avatar in the Portable Avatar. |
| Output data | To | Descriptions |
| Portable Avatar: Portable Avatar | Human | The Portable Avatar produced by the Machine, with the Machine’s Text and Personal Status, the Basic Visual Scene Descriptors of the Scene it perceives, the View Selector and the Point of View. |
4 Functions of AI Modules
Table 2 provides the functions of the AI Modules of Conversation About a Scene.
Table 2 – Functions of AI Modules of Conversation About a Scene
| AIM | Functions |
| Multimodal Frontend Description | 1. Receives the Input Visual and the Input Audio. 2. Provides the Basic Speech Object, the Face, Body and other Basic Visual Objects and the Basic Visual Scene Descriptors. |
| Entity Body Description | 1. Receives the Body Object. 2. Provides the Body Descriptors. |
| Visual Object Identification | 1. Receives the Body Descriptors, the Basic Visual Objects and the Basic Visual Scene Descriptors. 2. Finds the Visual Object that the human body indicates and provides its Instance ID. |
| Automatic Speech Recognition | 1. Receives the Basic Speech Object. 2. Provides the Recognised Text. |
| Natural Language Understanding | 1. Receives the Instance ID and the Recognised Text. 2. Provides the Refined Text and the Text Descriptors. |
| Personal Status Extraction | 1. Receives the Basic Speech Object, the Face Object, the Body Object and the Text Descriptors. 2. Provides the Input Personal Status. |
| Entity Dialogue Processing | 1. Receives the Input Personal Status, the Text Descriptors and the Refined Text. 2. Produces the Machine Text and the Machine Personal Status. |
| Portable Avatar Multiplexing | 1. Receives the Machine Text, the Machine Personal Status, the Basic Visual Scene Descriptors, the View Selector, the Point of View and the Machine ID. 2. Provides the Portable Avatar. |
5 I/O Data of AI Modules
Table 3 provides the I/O Data of the AI Modules of Conversation About a Scene.
Table 3 – I/O Data of AI Modules of Conversation About a Scene
6 AIW, AIMs, and JSON Metadata and AIMs
Table 4 provides the links to the AIM specifications and to the JSON syntaxes. AIM/1 indicates the Composite AIM and the following columns indicate its SubAIMs.
Table 4 – AIMs and JSON Metadata of Conversation About a Scene
| AIM/1 | AIMs/2 | AIMs/3 | Name | JSON |
| MMC-CAS | Conversation About a Scene | X | ||
| OSD-MFD | Multimodal Frontend Description | X | ||
| PAF-EBD | Entity Body Description | X | ||
| CVE-VOI | Visual Object Identification | X | ||
| CVE-VDI | Visual Direction Identification | X | ||
| OSD-VOE | Visual Object Extraction | X | ||
| CVE-VII | Visual Instance Identification | X | ||
| MMC-ASR | Automatic Speech Recognition | X | ||
| MMC-NLU | Natural Language Understanding | X | ||
| MMC-PSE | Personal Status Extraction | X | ||
| MMC-ETD | Entity Text Description | X | ||
| MMC-ESD | Entity Speech Description | X | ||
| PAF-EFD | Entity Face Description | X | ||
| PAF-EBD | Entity Body Description | X | ||
| MMC-PTI | PS-Text Interpretation | X | ||
| MMC-PSI | PS-Speech Interpretation | X | ||
| PAF-PFI | PS-Face Interpretation | X | ||
| PAF-PGI | PS-Body Interpretation | X | ||
| MMC-PMX | Personal Status Multiplexing | X | ||
| MMC-EDP | Entity Dialogue Processing | X | ||
| PAF-PMX | Portable Avatar Multiplexing | X |
7 Reference Software
8 Conformance Testing
Table 5 provides the Conformance Testing Method for MMC-CAS AIM.
If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present and conform with the Qualifier, if present.
Table 5 – Conformance Testing Method for MMC-CAS AIM
| Receives | View Selector: Selector | Shall validate against the Selector Schema. |
| Point of View: Point of View | Shall validate against the Point of View Schema. | |
| Input Visual: Basic Visual Object | Shall validate against the Basic Visual Object Schema. Visual Data shall conform with the Visual Qualifier. |
|
| Input Audio: Basic Audio Object | Shall validate against the Basic Audio Object Schema. Audio Data shall conform with the Audio Qualifier. |
|
| Machine ID: string | Shall validate against the string Schema. | |
| Produces | Portable Avatar: Portable Avatar | Shall validate against the Portable Avatar Schema. Data shall conform with the respective Qualifiers. |
9 Performance Assessment
<-Go to AI Workflow Go to ToC Conversation with Personal Status->