MPAI has published a Call for Technologies to be included in its planned MPAI-Multimodal Conversation (MPAI-MMC) V2 standard. The Call targets technologies supporting advanced forms of conversation between humans expressing Personal Status (emotion, cognitive state and attitude) and machines represented by speaking avatars expressing personal status. Please see:

1.  2 min video (YouTube and non-YouTube) illustrating MPAI-MMC V2.
2. slides presented at the online meeting on 2022/07/12.
3. video regcording of the online presentation (Youtube, non-YouTube) made at that 12 July presentation.
4. Call for Technologies, Use Cases and Functional Requirements, Framework Licence. and Template for responses.

The text of the MPAI-MMC V2 Call for Technologies is available as a Word document or below as HTML

Table of Contents of MPAI-MMC V2 Call for Technologies:

1       Introduction.

2       How to submit a response. 

3       Evaluation Criteria and Procedure.

4       Expected development timeline. 

5       References.

Annex A: Information Form..

Annex B: Evaluation Sheet

Annex C: Check list of data formats proposed by a respondent

Annex D: Technologies that may require specific testing.

Annex E: Mandatory text in responses.

1        Introduction

Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) is an international non-profit organisation with the mission of developing standards for Artificial Intelligence (AI)-enabled digital data coding and for technologies that facilitate integration of data coding components into ICT systems [1]. MPAI intends to facilitate the creation of a patent pool that relies on the clear IPR licensing frameworks established by the Framework Licences [2].

MPAI standardisation is based on the following assumptions:

1. Development, implementation, and secure distribution of MPAI Standards require a governed ecosystem [3].
2. MPAI Application Standards are executed in a secure environment called the AI Framework specified by MPAI [4].
3. Use Cases defined by MPAI Application Standards:
   1. Are implemented as AI Workflows (AIW) composed of AI Modules (AIM).
   2. Specify
      1. The input/output data formats and the functions of:
         1. The AIWs implementing the Use Cases
         2. The AIMs that are part of an AIW.
      2. The AIM topology.
   3. Do not specify the AIM internals, which are considered as black boxes.

“Multimodal Conversation” is the title of an MPAI application standard whose goal is to enhance the experience of a user conversing with a machine by leveraging different types of information elements. The approved Multimodal Conversation Version 1.2 (MPAI-MMC V1.2) standard [5] specifies a range of technologies supporting five Use Cases: Conversation with Emotion, Multimodal Question Answering, and three Automatic Speech Translation Use Cases.

With an exhaustive process, MPAI has now identified new Use Cases that extend the functionality of some existing and new Use Cases. New technologies may be needed to support extended or new functionalities.

This is a short list of both types of technology:

1. Audio and Visual Scene Description.
2. Speech and Face Authentication.
3. Speech Recognition and Translation.
4. Object Recognition.
5. Face, Head, Arms, Hand, Finger Description.
6. Personal Status[1]
7. Language Understanding, Dialogue Processing and Summarisation.
8. Personal Status Display.
9. Audio and Visual Scene Creation.
10. Audio-Visual Scene Viewer.

Note that Personal Status Description and Personal Status Display include several other technologies, such as those required to extract Personal Status from Text, Speech, Face and Gesture and to animate an avatar.

This MPAI-MMC V2 Call for Technologies (CfT) invites any party owning technologies that satisfy the MPAI-MMC V2 Use Cases and Functional Requirements of N780 [6] and are willing to release such technologies according to the MPAI-MMC V2 Framework Licence [7], if selected by MPAI for possible modification and inclusion in the MPAI-MMC V2 standard. Any respondent party who is not an MPAI member and had their technologies accepted shall join MPAI or lose the opportunity to have their technologies included.

The MPAI-MMC V2 standard will be developed using technologies that are:

1. Part of an already published MPAI standard, or
2. Satisfy the following mandatory requirements:
   1. To be part of responses to this Call containing acceptance of the MPAI-MMC V2 Framework Licence [7].
   2. To satisfy the MPAI-MMC V2 Use Cases and Functional Requirements[6]. In the future, MPAI may decide to further extend MPAI-MMC to support other Use Cases as a part of this MPAI-MMC V2 or as a future extension of it.
   3. To use, where feasible and desirable, the technologies in the MPAI-MMC V1.2 standard [5] that satisfy MPAI-MMC V2 Use Cases and Functional Requirements [6] or the technologies specified in other relevant MPAI standards published in [1].
   4. To be suitable for implementation as AI Workflows (AIW) composed of AI Modules (AIM) conforming to the MPAI AI Framework V1.1 (MPAI-AIF) standard [4]. Respondents should be aware that MPAI has issued the AI Framework (MPAI-AIF) V2 Call for Technologies [8].

Therefore, the scope of this Call for Technologies is restricted to technologies whose specification is required to implement the input and output interfaces of the AIWs and of the AIMs identified in [6].

However, respondents are welcome to additionally do one or more of the following:

1. Make comments on any technology or architectural component identified in [6].
2. Propose to:
3. Add or remove input/output signals to the identified AIMs:
   1. Justifying the changes.
   2. Identifying the data formats of the new input/output data.

1. Partition the AIMs in an AIW implementing the cases providing:

1. Arguments in support of the proposed partitioning.
2. Detailed specifications of the input and output data of the proposed new AIMs.

1. New fully described Use Cases as done in [6].

3. In general, submit motivated proposals of technologies not included in [6], for inclusion in the MPAI-MMC V2 standard MPAI if they satisfy the Framework Licence [7].

All parties who believe they have relevant technologies satisfying all or most of the requirements of one or more than one Use Case described in [6] are invited to submit proposals for consideration by MPAI. MPAI membership is not a prerequisite for responding to this CfT. However, proponents should be aware that, if their proposal or part thereof is accepted for inclusion in the MPAI-MMC V2 standard, they will be requested to immediately join MPAI, or lose the opportunity to have their accepted technologies included in the standard.

MPAI will select the most suitable technologies based on their technical merits for inclusion in the MPAI-MMC V2 standard. However, MPAI in not obligated, by virtue of this CfT, to select a particular technology or to select any of the proposed technology if those submitted are found inadequate.

Submissions shall be received to the MPAI secretariat (secretariat@mpai.community) by 2021/10/24 T23:59 UTC. The secretariat will acknowledge receipt of the submission via email. Submissions will be reviewed according to the schedule that the 25^th MPAI General Assembly (MPAI-25) will define at its online meeting on 2021/10/26. For details on how non MPAI members who have made a submission can attend the said review sessions should contact the MPAI secretariat (secretariat@mpai.community).

2        How to submit a response

Those planning to respond to this CfT are:

1. Advised that the MPAI-MMC V2 CfT has been presented at two online events held on 2022/07/07 and 2022/07/12. Submitters are encouraged to review the recorded video of the second event.
2. Requested to communicate their intention to respond to this CfT with an initial version of the form of Annex A to the MPAI secretariat (secretariat@mpai.community) by 2021/09/13. Submission of an Annex A helps MPAI to properly plan for the revision of submissions. However, a respondent is not required to send Annex A by 2021/09/13 to make a submission and a response to this Call not preceded by the submission of Annex A will still be accepted.
3. Encouraged to regularly visit the Call for Technologies webpage where relevant additional information will be posted.

Responses to this MPAI-MMC V2 CfT may/shall include:

Table 1 – Optional and mandatory elements of a response

Item	Status
Detailed documentation describing the proposed technologies	mandatory
The final version of Annex A.	mandatory
The text of Annex B duly filled out with the table indicating which Functional Requirements identified in [6] are satisfied. If some of the Functional Requirements of a Use Case are not satisfied, this should be explained.	mandatory
Comments on the completeness and appropriateness of the MPAI-MMC V2 Functional Requirements and any motivated suggestion to amend and/or extend those Requirements.	optional
A preliminary demonstration, with a detailed document describing it.	optional
Any other additional relevant information that may help evaluate the submission, such as additional use cases.	optional
The text of Annex E.	mandatory

Respondents are invited to take advantage of the check list of Annex C before filling out Annex A and submitting their response.

Respondents are mandatorily requested to present their submission at a teleconference meeting to be properly announced to submitters by the MPAI Secretariat. If no presenter of a submission will be in attendance to that meeting, the submission will be discarded.

Respondents are advised that, upon acceptance by MPAI of their submission in whole or in part for further evaluation, MPAI will require that:

• A working implementation, including source code – for use in the development of the MPAI-MMC Reference Software and later publication as an MPAI standard – be made available before the technology is accepted for inclusion in the MPAI-MMC standard. Software may be written in programming languages that can be compiled or interpreted. Hardware Description Language implementations are also accepted.
• The working implementation be suitable for operation in the MPAI MMC Framework (MPAI-MMC).
• A non-MPAI member immediately join MPAI. If the non-MPAI member elects not to do so, their submission will be discarded. Direction on how to join MPAI can be found online.

Further information on MPAI can be obtained from the MPAI website.

3        Evaluation Criteria and Procedure

Proposals will be assessed using the following process:

1. Evaluation panel is created from:
   1. MMC-DC and other members in attendance.
   2. Non-MPAI members who are respondents.
   3. Non respondents/non MPAI member experts invited in a consulting capacity.
2. No one from 1.1.-1.2. is denied membership in the Evaluation panel.
3. Respondents present their proposals.
4. Evaluation Panel members ask questions.
5. If required, subjective and/or objective tests are carried out with the following process:
   1. The required tests are defined.
   2. The required tests are carried out.
   3. A report is produced.
6. If required, at least 2 reviewers are appointed to review and report about specific points in a proposal.
7. Evaluation panel members fill out Annex B for each proposal.
8. Respondents respond to evaluations.
9. Proposal evaluation report is produced.

4        Expected development timeline

Timeline of the CfT, deadlines and response evaluation:

Table 2 – Dates and deadlines

 

Step	Date	Time
Online presentation of MPAI-MMC V2	2022/07/07-12	15:00 UTC
Call for Technologies	2022/06/19	17:00 UTC
Notification of intention to submit proposal	2022/09/13	23.59 UTC
Submission deadline	2022/10/24	23.59 UTC
Evaluation of responses will start	2022/10/26 (MPAI-25)

Evaluation to be carried out during 2-hour online sessions according to the calendar agreed at MPAI-25.

5        References

1. MPAI Standards Resources; https://mpai.community/standards/resources/.
2. MPAI Patent Policy; https://mpai.community/about/the-mpai-patent-policy/.
3. Governance of the MPAI Ecosystem (MPAI-GME) V1; https://mpai.community/standards/resources/#GME.
4. AI Framework (MPAI-AIF) V1.1; https://mpai.community/standards/resources/#AIF
5. Multimodal Conversation (MPAI-MMC) V1.2; https://mpai.community/standards/resources/#MMC.
6. Multimodal Conversation (MPAI-MMC) V2 Use Cases and Functional Requirements; MPAI N780; https://mpai.community/standards/mpai-mmc/use-cases-and-functional-requirements/mpai-mmc-v2-use-cases-and-functional-requirements/
7. Multimodal Conversation (MPAI-MMC) V2 Framework Licence; MPAI N800; https://mpai.community/standards/mpai-mmc/framework-licence/
8. AI Framework (MPAI-AIF) V2 Call for Technologies, MPAI N768;  https://mpai.community/standards/mpai-aif/call-for-technologies/mpai-aif-v2-call-for-technologies/
9. Presentation of MPAI-MMC V2 Use Cases and Functional Requirements; https://platform.wim.tv/#/webtv/convenor/vod/18b06e08-a31f-43a1-a338-6fff065695bc.

Annex A: Information Form

This information form is to be filled in by a Respondent to this MPAI-MMC V2 Call for Technologies.

1. Title of the proposal.
2. Organisation: company name, position, e-mail of contact person.
3. What are the main functionalities of your proposal?
4. Does your proposal provide or describe a formal specification and APIs?
5. Will you provide a demonstration to show how your proposal meets the evaluation criteria?

Annex B: Evaluation Sheet

NB: This evaluation sheet will be filled out by Evaluation Team members.

Proposal title:

Main functionalities:

 Response summary: (a few lines)

Comments on relevance to the CfT (Requirements):

Comments on possible MPAI-MMC profiles[2]

Evaluation table:

Table 3 – Assessment of submission features

Note 1	The semantics of submission features is provided by Table 4.
Note 2	Evaluation Elements indicate the elements used by the evaluator in assessing the submission.
Note 3	Final Assessment indicates the ultimate assessment based on the Evaluation Elements.

 

Submission features	Evaluation Elements	Final Assessment
Completeness of description
Understandability
Extensibility
Use of standard technology
Efficiency
Test cases
Maturity of reference implementation
Relative complexity
Support of MPAI use cases
Support of non-MPAI use cases

 

Content of the criteria table cells:

Evaluation facts should mention:

• Not supported / partially supported / fully supported.
• What supports these facts: submission/presentation/demo.
• The summary of the facts themselves, e.g., very good in one way, but weak in another.

Final assessment should mention:

• Possibilities to improve or add to the proposal, e.g., any missing or weak features.
• How sure the evaluators are, i.e., evidence shown, very likely, very hard to tell, etc.
• Global evaluation (Not Applicable/ –/ – / + / ++)

 New Use Cases/Requirements Identified:

(please describe)

•  Evaluation summary:
•  Main strong points, qualitatively:
•  Main weak points, qualitatively:
• Overall evaluation: (0/1/2/3/4/5)

0: could not be evaluated

1: proposal is not relevant.

2: proposal is relevant, but requires significant more work.

3: proposal is relevant, but with a few changes.

4: proposal has some very good points, so it is a good candidate for standard.

5: proposal is superior in its category, very strongly recommended for inclusion in standard.

Additional remarks: (points of importance not covered above.)

The submission features in Table 3 are explained in the following Table 4.

Table 4 – Explanation of submission features

Submission features	Criteria
Completeness of description	Evaluators should: 1.     Compare the list of requirements (Annex C of the CfT) with the submission. 2.     Check if respondents have described in sufficient detail how the requirements are supported by the proposal. Note1: Completeness of a proposal for a Use Case is a merit because reviewers can assess how the components are integrated. Note2: Submissions will be judged for the merit of what is proposed. A submission on a single technology that is excellent may be considered instead of a submission that is complete but has a less performing technology.
Understandability	Evaluators should identify items that are demonstrably unclear (incon­sistencies, sentences with dubious meaning etc.)
Extensibility	Evaluators should check if respondent has proposed extensions to the Use Cases. Note: Extensibility is the capability of the proposed solution to support use cases that are not supported by current requirements.
Use of standard Technology	Evaluators should check if new technologies are proposed where widely adopted technologies exist. If this is the case, the merit of the new tech­nology shall be proved.
Efficiency	Evaluators should assess power consumption, computational speed, computational complexity.
Test cases	Evaluators should report whether a proposal contains suggestions for testing the technologies proposed.
Maturity of reference implementation	Evaluators should assess the maturity of the proposal. Note1: Maturity is measured by the completeness, i.e., having all the necessary information and appropriate parts of the HW/SW implementation of the submission disclosed. Note2: If there are parts of the implementation that are not disclosed but demonstrated, they will be considered if and only if such components are replicable.
Relative complexity	Evaluators should identify issues that would make it difficult to implement the proposal compared to the state of the art.
Support of MPAI-MMC use cases	Evaluators should check how many composite AIMs and use cases are supported in the submission.
Support of non MPAI-MMC use cases	Evaluators should check whether the technologies proposed can demonstrably be used in other significantly different use cases.

Annex C: Check list of data formats proposed by a respondent

Table 5 is a suggested check list to inform MPAI about the data formats contained in a response.

Table 5 – List of data formats in MPAI-MMC Use Cases and Functional Requirements [6]

Note: The numbers in the first column refer to the section numbers of [6].

Sections	Data formats	Response
7.1	Digital representation of analogue signals
7.1.1	Microphone Array Audio	Y/N
7.1.2	2D Video	Y/N
7.1.3	3D Video	Y/N
7.2	Natively digital data formats
7.2.1	Text	Y/N
7.2.2	Recognised Text	Y/N
7.2.3	Language Understanding (Text)	Y/N
7.2.4	Summary	Y/N
7.2.5	Environment Model	Y/N
7.2.6	Avatar Model	Y/N
7.2.7	Human Object	Y/N
7.2.8	Face Object	Y/N
7.2.9	Head Object	Y/N
7.2.10	Arm Object	Y/N
7.2.11	Hand Object	Y/N
7.2.12	Finger Object	Y/N
7.2.13	Audio-Visual Scene Presentation	Y/N
7.2.14	Spatial Attitude	Y/N
7.3	Descriptors
7.3.1	Audio Scene Descriptors	Y/N
7.3.2	Visual Scene Descriptors	Y/N
7.3.3	Audio-Visual Scene Descriptors	Y/N
7.3.4	Avatar Descriptors	Y/N
7.3.5	Face Descriptors	Y/N
7.3.6	Gesture Descriptors	Y/N
7.3.7	Head Descriptors	Y/N
7.3.8	Arm Descriptors	Y/N
7.3.9	Hand Descriptors	Y/N
7.3.10	Finger Descriptors	Y/N
7.3.11	Speech Descriptors	Y/N
7.3.12	Text Descriptors	Y/N
7.4	Interpretations
7.4.1	Emotion	Y/N
7.4.2	Cognitive State	Y/N
7.4.3	Attitude	Y/N
7.4.4	Personal Status	Y/N
7.4.5	Meaning	Y/N
7.4.6	Object Identifier	Y/N

 

Respondent should in any case review the equivalent list in the table of contents of [6].

Annex D: Technologies that may require specific testing

Table 6 will be compiled based on the responses received.

Table 6 – Data formats that may require specific testing

Section	Technology	Nature of Test

Annex E: Mandatory text in responses

A response to this MPAI-MMC CfT shall mandatorily include the following text

<Company/Member> submits this technical document in response to MPAI Call for Technologies for Multimodal Conversation (MPAI-MMC) V2 (N780).

 <Company/Member> explicitly agrees to the steps of the MPAI standards development process defined in Annex 1 to the MPAI Statutes (N421), in particular <Company/Member> declares that  <Company/Member> or its successors will make available the terms of the Licence related to its Essential Patents according to the MPAI-MMC V2 Framework Licence (N800), alone or jointly with other IPR holders after the approval of the MPAI-MMC Technical Specification by the General Assembly and in no event after commercial implementations of the MPAI-MMC Technical Specification become available on the market.

In case the respondent is a non-MPAI member, the submission shall mandatorily include the following text

If (a part of) this submission is identified for inclusion in a specification, <Company>  understands that  <Company> will be requested to immediately join MPAI and that, if  <Company> elects not to join MPAI, this submission will be discarded.

Subsequent technical contribution shall mandatorily include this text

<Member> submits this document to MPAI-MMC Development Committee (MMC-DC) as a con­tribution to the development of the MPAI-MMC Technical Specification.

 <Member> explicitly agrees to the steps of the MPAI standards development process defined in Annex 1 to the MPAI Statutes (N421), in particular  <Company> declares that <Company> or its successors will make available the terms of the Licence related to its Essential Patents according to the MPAI-MMC V2 Framework Licence (N800), alone or jointly with other IPR holders after the approval of the MPAI-MMC Technical Specification by the General Assembly and in no event after commercial implementations of the MPAI-MMC Technical Specification become available on the market.

 

[1] The ensemble of information internal to a person, including Emotion, Cognitive State, and Attitude. They are elements of the internal status of a human resulting from the interaction with the Environment or subsets of it, such as “Angry”, and “Sad” (Emotion), “Confused” or “Dubious” or “Convinced” (Cognitive State), and “Confrontational”, “Respectful” (Attitude).

[2] Profile of a standard is a particular subset of the technologies that are used in a standard and, where applicable, the classes, subsets, options and parameters relevan for the subset.