Go to MPAI-MMC V2.5 AI Modules
Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance
1 Functions
The Answer to Multimodal Question (MMC‑AMQ) AIM answers a question that a human asks about an image. The human provides the image and asks the question by speech or by text. The machine answers with a Portable Avatar that utters the answer, and returns the image.
2 Reference Model
Figure 1 depicts the Reference Model of the Answer to Multimodal Question (MMC‑AMQ) AIM.
Figure 1 – Reference Model of the Answer to Multimodal Question (MMC‑AMQ) AIM
The operation of the Answer to Multimodal Question (MMC‑AMQ) AIM develops in the following way:
- The human provides an image and asks a question about it, by speech or by text.
- The User Agent provides the Machine ID.
- The machine recognises the speech question, if any, answers the question about the image, and provides the Portable Avatar that utters the answer, and the image.
3 I/O Data
Table 1 specifies the Input and Output Data of the Answer to Multimodal Question (MMC‑AMQ) AIM.
| Input | Description |
|---|---|
| Input Visual | The image about which a question is asked. |
| Input Speech | The question, if spoken. Optional. |
| Input Text | The question, if typed instead of spoken. Optional. |
| Machine ID | Identifier of the machine. |
| Output | Description |
| Output Visual | The image about which the question was asked. |
| Portable Avatar | The avatar that utters the answer. |
4 SubAIMs
4.1 Reference Model
Figure 2 depicts the Reference Model of the Answer to Multimodal Question (MMC‑AMQ) Composite AIM.
Figure 2 – Reference Model of the Answer to Multimodal Question (MMC‑AMQ) Composite AIM
4.2 Operation
The Answer to Multimodal Question (MMC‑AMQ) AIM receives the Input Visual, and the question as Input Speech or as Input Text. Automatic Speech Recognition turns a spoken question into text. Text and Image Query answers the question, typed or recognised, about the image and returns the answer text and the image. Portable Avatar Multiplexing puts the answer text and the Machine ID into the Portable Avatar. Acquisition and delivery are the User Agent’s, not SubAIMs.
4.3 Functions of SubAIMs
Table 2 provides the functions of the Answer to Multimodal Question (MMC‑AMQ) SubAIMs.
| SubAIM | Function |
|---|---|
| Automatic Speech Recognition | Recognises as Text the Input Speech. |
| Text and Image Query | Receives a question as Text and an image and produces the Text that answers it, and the image. |
| Portable Avatar Multiplexing | Puts the answer Text and the Machine ID in the Portable Avatar. |
4.4 I/O Data of SubAIMs
Table 3 gives the Input and Output Data of the Answer to Multimodal Question (MMC‑AMQ) SubAIMs.
4.5 AIMs and JSON Metadata
Table 4 provides the links to the AIM specifications and JSON schemas. AIM1 indicates the Composite AIM and AIM2 its SubAIMs.
| AIM1 | AIM2 | Name | JSON |
|---|---|---|---|
| MMC‑AMQ | Answer to Multimodal Question | X | |
| MMC‑ASR | Automatic Speech Recognition | X | |
| MMC‑TIQ | Text and Image Query | X | |
| PAF‑PMX | Portable Avatar Multiplexing | X |
5 JSON Metadata
https://schemas.mpai.community/MMC/V2.5/AIMs/AnswerToMultimodalQuestion.json
6 Profiles
No Profiles.
7 Reference Software
Under development.
8 Conformance Testing
Table 5 provides the Conformance Testing Method for the Answer to Multimodal Question (MMC‑AMQ) AIM. Conformance Testing of the individual SubAIMs is given by the individual AIM specifications.
If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.
| Receives | Input Visual | Shall validate against Basic Visual Object schema. Visual Data shall conform with Visual Qualifier. |
| Input Speech | Shall validate against Basic Speech Object schema. Speech Data shall conform with Speech Qualifier. | |
| Input Text | Shall validate against Basic Text Object schema. Text Data shall conform with Text Qualifier. | |
| Machine ID | Shall be a string. | |
| Produces | Output Visual | Shall validate against Basic Visual Object schema. Visual Data shall conform with Visual Qualifier. |
| Portable Avatar | Shall validate against Portable Avatar schema. |
9 Performance Assessment
Not part of this specification.
