Go to MPAI-MMC V2.5 AI Modules

Function
Ref. Model
I/O Data
SubAIMs
JSON MData
Profiles
Ref. Software
Conformance
Performance

1 Functions

The Multimodal Translation Service (MMC‑MTS) AIM translates what a human says or types into the language named by a Language Selector and presents the translation through a Portable Avatar that speaks it with lip-synchronised Face Descriptors in the target language. The translated text is also provided. The service is not affective: no Personal Status is supplied, and the human is anonymous.

2 Reference Model

Figure 1 depicts the Reference Model of the Multimodal Translation Service (MMC‑MTS) AIM.
 

Figure 1 – Reference Model of the Multimodal Translation Service (MMC‑MTS) AIM

The operation of the Multimodal Translation Service (MMC‑MTS) AIM develops in the following way:

  1. A human provides:
    1. Speech, or text typed instead of speaking, to be translated.
    2. A Language Selector naming the target language.
  2. The User Agent provides the Avatar Model that represents the machine and the Machine ID.
  3. The machine provides the Translated Text and the Portable Avatar that speaks it.

3 I/O Data

Table 1 specifies the Input and Output Data of the Multimodal Translation Service (MMC‑MTS) AIM.

Table 1 – I/O Data of the Multimodal Translation Service (MMC‑MTS) AIM

Input Description
Input Speech Speech of the human, if spoken. Optional.
Target Language The language into which the input is translated.
Text to Translate Text typed by the human instead of speaking. Optional.
Avatar Model The 3D model of the avatar that represents the machine.
Machine ID Identifier of the machine. Optional.
Output Description
Translated Text The translation as Text.
Portable Avatar The avatar that speaks the translation, with lip-synchronised Face Descriptors in the target language.

4 SubAIMs

4.1 Reference Model

Figure 2 depicts the Reference Model of the Multimodal Translation Service (MMC‑MTS) Composite AIM.
 

Figure 2 – Reference Model of the Multimodal Translation Service (MMC‑MTS) Composite AIM

4.2 Operation

The Multimodal Translation Service (MMC‑MTS) AIM receives the human’s speech, which Automatic Speech Recognition converts to Text, or text. Text-to-Text Translation translates the Text into the language named by the Language Selector. Response and Scene Rendering speaks the translation and produces the lip-synchronised Machine Face Descriptors in the target language. The translated Text is also an output. Acquisition and delivery are the User Agent’s, not SubAIMs.

4.3 Functions of SubAIMs

Table 2 provides the functions of the Multimodal Translation Service (MMC‑MTS) SubAIMs.

Table 2 – Functions of the Multimodal Translation Service (MMC‑MTS) SubAIMs

SubAIM Function
Automatic Speech Recognition Recognises as Text an input Speech.
Text-to-Text Translation Receives a Text and a Language Selector and produces its translation into the selected language.
Response and Scene Rendering Receives the translated Text, the Avatar Model and the Machine ID and produces the Portable Avatar that speaks the Text with lip-synchronised Machine Face Descriptors.

4.4 I/O Data of SubAIMs

Table 3 gives the Input and Output Data of the Multimodal Translation Service (MMC‑MTS) SubAIMs.

Table 3 – I/O Data of the Multimodal Translation Service (MMC‑MTS) SubAIMs

SubAIM Input Output
Automatic Speech Recognition Speech Object Recognised Text
Text-to-Text Translation Text (recognised or typed)
Language Selector		 Translated Text
Response and Scene Rendering Translated Text
Avatar Model
Machine ID		 Portable Avatar

4.5 AIMs and JSON Metadata

Table 4 provides the links to the AIM specifications and JSON schemas. AIM1 indicates the Composite AIM and AIM2 its SubAIMs.

Table 4 – AIMs and JSON Metadata of the Multimodal Translation Service (MMC‑MTS)

AIM1 AIM2 Name JSON
MMC‑MTS Multimodal Translation Service X
MMC‑ASR Automatic Speech Recognition X
MMC‑TTT Text-to-Text Translation X
PAF‑RSR Response and Scene Rendering X

5 JSON Metadata

https://schemas.mpai.community/MMC/V2.5/AIMs/MultimodalTranslationService.json

6 Profiles

No Profiles.

7 Reference Software

Under development.

8 Conformance Testing

Table 5 provides the Conformance Testing Method for the Multimodal Translation Service (MMC‑MTS) AIM. Conformance Testing of the individual SubAIMs is given by the individual AIM specifications.
If a schema contains references to other schemas, conformance of data for the primary schema implies that any data referencing a secondary schema shall also validate against the relevant schema, if present, and conform with the Qualifier, if present.

Table 5 – Conformance Testing Method for the Multimodal Translation Service (MMC‑MTS) AIM

Receives Input Speech Shall validate against Basic Speech Object schema. Speech Data shall conform with Speech Qualifier.
Target Language Shall validate against Selector schema.
Text to Translate Shall validate against Basic Text Object schema. Text Data shall conform with Text Qualifier.
Avatar Model Shall validate against Basic 3D Model Object schema. 3D Model Data shall conform with 3D Model Qualifier.
Machine ID Shall be a string.
Produces Translated Text Shall validate against Basic Text Object schema. Text Data shall conform with Text Qualifier.
Portable Avatar Shall validate against Portable Avatar schema.

9 Performance Assessment

Not part of this specification.

Go to MPAI-MMC V2.5 AI Modules