Functional Requirements – Application Note

Use Cases and Functional Requirements of MPAI AI Framework (MPAI-AIF)

1 Introduction

Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) is an international association with the mission to develop AI-enabled data coding standards. Artificial Intelligence (AI) technologies have shown they can offer more efficient data coding than existing technol­ogies.

MPAI has analysed six use cases covering applic­ation areas benefiting from AI technol­ogies. Even though use cases are disparate, each of them can be implemented with a combination of processing modules performing functions that concur to achieving the inten­ded result.

MPAI has assessed that, leaving it to the market to develop individual implementations, would multiply costs and delay adoption of AI technologies, while modules with standard interfaces, combined and executed within the MPAI-specified AI Framework, will favour the emergence of horizontal markets where proprietary and competing module implemen­tations exposing standard interfaces will reduce cost, promote adoption and incite progress of AI technologies. MPAI calls these modules AI Modules (AIM).

MPAI calls the planned AI Framework standard as MPAI-AIF. As AI is a fast-moving field, MPAI expects that MPAI-AIF will be extended as new use cases will bring new requirements and new technologies will reach maturity.

To avoid the deadlock experienced in other high-technology fields, before engaging in the development of the the MPAI-AIF standard, MPAI will develop a Frame­work Licence (FWL) associated with the MPAI-AIF Architecture and Functional Requir­ements defined in this document. The FWL, essentially the business model that standard essential patent (SEP) holders will apply to monetise their Intellectual Properties (IP), but without values such as the amount or percentage of royalties or dates due, will act as Commercial Requirements for the standard and provide a clear IPR licensing framework.

This document contains a summary description of the six use cases (Section 2) followed by a section describing the architecture expected to become normative (Section 3). Section 4 lists the normative requirements identified so far.

2        Use Cases

The six use cases considered cover a broad area of application. Therefore, it is expected that the MPAI-AIF architecture can support a wide range of use cases of practical interest.

Each case is identified by its name and the acronym identifying the future MPAI standard. More information about MPAI-AIF can be found in [1].

2.1 Context-based Audio Enhancement (MPAI-CAE)

2.2 Integrative Genomic/Sensor Analysis (MPAI-GSA)

2.3 AI-Enhanced Video Coding (MPAI-EVC)

2.4 Server-based Predictive Multiplayer Gaming (MPAI-SPG)

2.5 Multi-Modal Conversation (MPAI-MMC)

2.6 Compression and Understanding of Industrial data (MPAI-CUI)

3        Architecture

The normative MPAI-AIF architecture enables the creation and automation of mixed ML-AI-DP processing and inference workflows at scale for the use cases considered above. It includes six basic normative elements of the Architecture called Components addressing different modalities of operation – AI, Machine Learning (ML) and Data Processing (DP), data pipelines jungles and computing resource allocations including constrained hardware scenarios of edge AI devices.

The normative reference diagram of MPAI-AIF is given by the following figure where APIs be­tween different Components at different level are shown.

Figure 3 – Proposed normative MPAI-AIF Architecture

  1. Management and Control

Management concerns the activation/disactivation/suspensions of AIMs, while Control supports complex application scenarios.

Management and Control handles simple orchestration tasks (i.e. represented by the execution of a script) and much more complex tasks with a topology of networked AIMs that can be syn­chronised according to a given time base and full ML life cycles.

  1. Execution

The environment where AIMs operate. It is interfaced with Management and Control and with Communication and Storage. It receives external inputs and produces the requested outputs both of which are application specific.

  1. AI Modules (AIM)

AIMs are units comprised of at least the following 3 functions:

  1. The processing element (ML or traditional DP)
  2. Interface to Communication and Storage
  3. Input and output interfaces (function specific)

AIMs can implement auto-configuration or reconfiguration of their ML-based computational models.

  1. Communication

Communication is required in several cases and can be implemented accordingly, e.g. by means of a service bus. Components can communicate among themselves and with outputs and Storage.

The Management and Control API implements one- and two-way signalling for computational workflow initialisation and control.

  1. Storage

Storage encompasses traditional storage and is referred to a variety of data types, e.g.:

  1. Inputs and outputs of the individual AIMs
  2. Data from the AIM’s state, e.g. with respect to traditional and continuous learning
  3. Data from the AIM’s intermediary results
  4. Shared data among AIMs
  5. Information used by Management and Control.
  6. Access

Access represents the access to static or slowly changing data that are required by the application such as domain knowledge data, data models, etc.

4 Requirements

4.1 Component requirements

  1. The MPAI-AIF standard shall include specifications of the interfaces of 6 Components
    1. Management and Control
    2. Execution
    3. AI Modules (AIM)
    4. Communication
    5. Storage
    6. Access
  2. MPAI-AIF shall support configurations where Components are distributed in the cloud and at the edge
  3. Management and Control shall enable operations on the general ML life cycle: the sequence of steps that and/or traditional data processing life cycle of
    1. Single AIMs, e.g. instantiation-configuration-removal, internal state dumping/retrieval, start-suspend-stop, train-retrain-update, enforcement of resource limits
    2. Combinations of AIMs, e.g. initialisation of the overall computational model, instan­tiation-removal-configuration of AIMs, manual, automatic, dynamic and adaptive configuration of interfaces with Components.
  4. Management and Control shall support
    1. Architectures that allow application-scenario dependent hierarchical execution of workflows, i.e. a combination of AIMs into computational graphs
    2. Supervised, unsupervised and reinforcement-based learning paradigms
    3. Computational graphs, such as Direct Acyclic Graph (DAG) as a minimum
    4. Initialisation of signalling patterns, communication and security policies between AIMs
  5. Storage shall support protocols to specify application-dependent requirements such as access time, retention, read/write throughput
  6. Access shall provide
    1. Static or slowly changing data with standard formats
    2. Data with proprietary formats

4.2 Systems requirements

The following requirements are not intended to apply to the MPAI-AIF standard, but should be used for assessing technologies

  1. Management and Control shall support asynchronous and time-based synchronous operation depending on application
  2. The Architecture shall support dynamic update of the ML models with seamless or minimal impact on its operation
  3. ML-based AIMs shall support time sharing operation enabling use of the same ML-based AIM in multiple concurrent applications
  4. AIMs may be aggregations of AIMs exposing new interfaces
  5. Complexity and performance shall be scalable to cope with different scenarios, e.g. from small MCUs to complex distributed systems
  6. The Architecture shall support workflows of a mixture of AI/ML-based and DP technology-based AIMs.

4.3       General requirements

The MPAI-AIF standard may include profiles for specific (sets of) requirements

5 Conclusions

When the definition of the MPAI-AIF Framework Licence will be completed, MPAI will issue a Call for Technologies that support the AI Framework with the requirem­ents given in this document.

Respondents will be requested to state in their submissions their intention to adhere to the Frame­work Licence developed for MPAI-AIF when licensing their technologies if they have been inc­luded in the MPAI-AIF standard.

The MPAI-AIF Framework Licence will be developed, as for all other MPAI Framework Licences, in compliance with the gener­ally accepted principles of competition law.

6        References

[1] MPAI Application Note#4 – MPAI-AIF Artificial Intelligence Framework

[2] MPAI Application Note#1 R1 – MPAI-CAE Context-based Audio Enhancement

[3] MPAI Application Note#2 R1 – MPAI-GSA Integrative Genomic/Sensor Analysis

[4] MPAI Application Note#3 R1 – MPAI-EVC AI-Enhanced Video Coding

[5] MPAI Application Note#5 R1 – MPAI-SPG Server-based Predictive Multiplayer Gaming

[6] MPAI Application Note#6 R1 – MPAI-MMC Multi-Modal Conversation

[7] MPAI-CAE Functional Requirements work programme

[8] MPAI-GSA Functional Requirements work programme

[9] MPAI-MMC Functional Requirements work programme

[10] MPAI-EVC Use Cases and Requirements

[11] Collaborative Evidence Conditions for MPAI-EVC Evidence Project R1

[12] Operational Guidelines for MPAI-EVC Evidence Project

Functional Requirements – Application Note

MPAI Application Note #4

Artificial Intelligence Framework (MPAI-AIF)

Proponent: Andrea Basso.

 Description: The purpose of the MPAI framework is to enable the creation and automation of mixed ML-AI-DP processing and inference workflows at scale. The key components of the framework should address different modalities of operation (AI, ML and DP), data pipeline jungles and computing resource allocations including constrained hardware scenarios of edge AI devices.

The framework is depicted in Figure 1. It is composed by:

  1. Data Storage component
  2. Orchestrator
  3. Processing modules (PM), traditional or ML algorithms

Figure 1 – MPAI Framework

  1. MPAI Processing Modules (PM)

PMs are composed by 4 components as indicated in Figure 1:

  1. The processing element PM (ML or Traditional Data Processor)
  2. Interface to the common data storage format
  3. input interfaces
  4. output interfaces
  5. Orchestrator

The PMs in Figure 1 need to be executed and orchestrated, so that are being executed in the correct order and the timing where needed is respected e.g. that inputs to a PM are computed before the PM is executed. As reported in [6] the glue code between PMs of a complex ML system is often brittle and that custom scripts don’t scale beyond specific cases.

Therefore, it is important to have a specific orchestrator component that supports the implementation of several MPAI application scenarios and ease the efficient usage of PMs.

 As shown in figure 1 the orchestrator is characterized by:

  1. Interface to PMs
  2. Interface to the input
  3. Interface to the common data storage format

Note that the Orchestrator should be able to handle from simple orchestration tasks (i.e. represented by the execution of a script) to much more complex tasks with a topology of networked PMs that need to be synchronized according to a given time base.

  1. Data Storage

The Data Storage component encompass traditional storage and is referred to a variety of data types. Some examples: it stores the inputs and outputs of the individual PMs, data from the orchestrator PM’s state and intermediary results, shared data among PMs, information used by the orches­trator, orchestrator procedures as well domain knowledge data, data models, etc.




Examples of PMs in the MPAI-CAE application scenario are


  • process the signals from the environment captured by the microphones
  • performing dynamic signal equalization
  • selectively recognize and allow relevant environment sounds (i.e. the horn of a car).


Examples of PMs in the MPAI-MMC scenario are

  • Speech recognition
  • Speech synthesis
  • Emotion recognition
  • Intention understanding
  • Gesture recognition
  • Knowledge fusion from different sources such as speech, facial expression, gestures, etc

An illustrative application of the MPAI-AIF Framework is given below

Figure 2 – MPAI-MMC in the MPAI-AIF Framework


MPAI has identified the following initial requirements:

  • Shall allow to orchestrate and automate the execution of AI mixed workflows for a variety of application scenarios
  • Agnostic to AI, ML or DP technology: The architecture should be general enough avoiding the imposition of limitations in terms of algorithmic structure, storage and communication and allow full interoperability of its components.
  • Allow uninterrupted functionality while algorithms or ML models are updated or retained. ML models once deployed may need to update often as more data is made available through their usage. Deployment of the updated models should happen seamlessly with no or minimal impact.
  • Support for distributed computing including combination of cloud and edge AI components.
  • Scalable: can be used in scenarios of different complexity
  • Efficient use of the computing and communication resources e.g. by supporting task sharing also for ML processing modules.
  • Support a common interface for processing modules
  • Support common data representation for storage
  • Support parallel and sequential combination of PMs
  • Support real time processing

 Object of standard:

MPAI has identified the following areas of possible standardization:

  • Architecture that specifies the roles of the components of the architecture
  • The modalities of the creation of the pipelines of PMs
  • Common PM interfaces specified in the application standards s8ch as MPAI-CAE, etc ..
  • Data storage interfaces

The current framework has been made as general as possible taking into consideration also current MPAI application standards. We are expecting the architecture to be enriched and extended according to the proposals of the MPAI contributors.


MPAI-AIF will bring benefits positively affecting

  1. Technology providers need not develop full applications to put to good use their technol­ogies. They can concentrate on improving the AI technologies of the PMs. Further, their technologies can find a much broader use in application domains beyond those they are accustomed to deal with.
  2. Equipment manufacturers and application vendors can tap from the set of technologies made available according to the MPAI-AIF standard from different competing sources, integrate them and satisfy their specific needs
  3. Service providers can deliver complex optimizations and thus superior user experience with minimal time to market as the MPAI-AIF framework enables easy combination of 3rd party components from both a technical and licensing perspective. Their services can deliver a high quality, consistent user audio experience with minimal dependency on the source by selecting the optimal delivery method
  4. End users enjoy a competitive market that provides constantly improved user exper­iences and controlled cost of AI-based products.

 Bottlenecks: the full potential of AI in MPAI-AIF would be limited by a market of AI-friendly processing units and by the introduction of the vast amount of AI technologies into products and services.

 Social aspects:

MPAI-AIF will enable the availability of a variety of AI-based products at lower price and faster.

Success criteria:

MPAI-AIF should create a competitive market of AI-based components expos­ing standard interfaces, processing units available to manufacturers, a variety of end user devices and trigger the implicit need felt by a user to have the best experience whatever the context.


[1] W. Wang, J. Gao, M. Zhang, S. Wang, G. Chen, T. K. Ng, B. C. Ooi, J. Shao, and M. Reyad, “Rafiki: machine learning as an analytics service system,” Proceedings of the VLDB Endowment, vol. 12, no. 2, pp. 128–140, 2018.

[2] Y. Lee, A. Scolari, B.-G. Chun, M. D. Santambrogio, M. Weimer, and M. Interlandi, “PRETZEL: Opening the black box of machine learning prediction serving systems,” in 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI18), pp. 611–626, 2018.

[3] “ML.NET [ONLINE].”

[69] D. Crankshaw, X. Wang, G. Zhou, M. J. Franklin, J. E. Gonzalez, and I. Stoica, “Clipper: A low-latency online prediction serving system.,” in NSDI, pp. 613–627, 2017.

[4] S. Zhao, M. Talasila, G. Jacobson, C. Borcea, S. A. Aftab, and J. F. Murray, “Packaging and sharing machine learning models via the acumos ai open platform,” in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 841–846, IEEE, 2018.

[5] “Apache Prediction I/O.”

[6] D.Sculley, G.Holt,D.Golovin,E. Davydov,T.Phillips,D.Ebner,V. Chaudhary,M. Young, J. Crespo, D. Dennison “Hidden technical debt in Machine learning systems Share” on NIPS’15: Proceedings of the 28th International Conference on Neural Information Processing Systems – Volume 2, December 2015, Pages 2503–2511