Archives: 2024-05-29

MPAI has recently published two main new standards with a request for Community Comments. In MPAI lingo this means that the standards are mature, but MPAI asks the Community to review the drafts before publication.

The two standards are Connected Autonomous Vehicle – Technologies (CAV-TEC) V1.0 and MPAI Metaverse Model – Technologies (MMM-TEC) V2.0. They are not “new” MPAI standards as they have already been published in earlier version, but the new versions represent significant improvements.

You may ask “Why is this topic handled in a single paper when we are talking about two standards that have – apparently – so little in common”? If you ask this question, you may want to continue reading and discover an essential aspect of MPAI standardisation. A standard for an application domain is (almost) never a monolith but is often made of components shared with standards of other domains, often unrelated.

Let’s first have a scan of the two standards.

Connected Autonomous Vehicle (CAV-TEC V1.0) is a standard for the ICT (Information and Communication Technology) part of a vehicle that can move itself in the physical world to reach a destination. The standard assumes that a CAV is composed of four functionally separated but interconnected subsystems:

1. The human-CAV Interaction (HCI): enables a human to establish a dialogue with the CAV to issue a variety of commands to it, such as moving to a destination or have conversations where the human – and the CAV – can express their internal status – e.g., emotion – whether real or fictitious.
2. The Environment Sensing Subsystem (EES): leverages the sensors on board to create the Basic Scene Descriptors (BED), the most accurate digital representation of the external environment possible with the available sensors.
3. The Autonomous Motion Subsystem (AMS): receives the BED and improves its accuracy by exchanging portions of their Environment Descriptors with CAVs in range. Then it analyses or reviews the situation and decides how to issue commands to implement the route decided by the human.
4. The Motion Actuation Subsystem converts a general request to move the CAV potentially by a few meters to specific commands for brakes, motors, and wheel and to report about the implementation of the command.

The four subsystems are implemented as AI Workflows per the AI Framework standard. Each AI Workflow includes several AI Modules that exchange Data specifies by CAV-TEC V1.0 or by other MPAI standards. Most AIMs and data types of:

1. HCI: is specified not by CAV-TEC but by Multimodal Conversation (MPAI-MMC) V2.3 . The rest is specified by Object and Scene Description (MPAI-OSD) V1.3, Portable Avatar Format (MPAI-PAF) V1.4, Data Types, Formats, and Attributes (MPAI-TFA) V1.3, and CAV-TEC.
2. ESS, AMS, and MAS: are specified by CAV-TEC, MPAI-OSD,  MPAI-PAF, and MPAI-TFA.

A concise description of the operation of an implementation of CAV-TEC is available here.

MPAI Metaverse Model (MMM-TEC V2.0) specifies a virtual space composed of processes is provided operating on an ICT platform and executing or requesting other processes to execute actions on items, i.e., MMM-TEC V2.0-specified data types. Processes may be rendered as avatars.

MMM-TEC does not use AIMs but only data types. It defines 27 actions and a language enabling processes to communicate using speech acts. Many are specified by MPAI-MMC, MPAI-OSD, and MPAI-PAF.

Example of actions are MM-Embed applied on an avatar or an object to place it somewhere in the metaverse, UM-Capture applied to media information in the physical world acquired for use in the metaverse, Identify applied to data captured to convert it to an item with an identifier, MM-Anim applied to an item, e.g., an avatar, to animate it, and MU-Render applied to an item in the metaverse to render it in the universe. A concise description of the operation of an implementation of MMM-TEC is available here.

As already said, both draw a sizeable part of their data types (and CAV-TEC of its AIMs) from three other standards: MPAI-OSD V1.3, MPAI-PAF V1.4, and MPAI-TFA V1.3. Some of these AIMs and data types are new in the mentioned versions of the three standards. They were developed in response to the CAV-TEC and MMM-TEC needs.

Why not develop them in CAV-TEC and MMM-TEC directly, then? Because the three standards address a specific area of standardisation that is also required by many other MPAI standards: objects and scenes, 3D graphics, and qualifiers. Therefore, MPAI-OSD V1.3, MPAI-PAF V1.4, and MPAI-TFA V1.3 are also published with a request for Community Comments.

Anybody is invited to send comments on any of the five standards to the MPAI Secretariat by April 13 at 23:59 UTC.

Geneva, Switzerland – 19^th March 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 54^th General Assembly (MPAI-54) with the release of the Connected Autonomous Vehicle and the MPAI Metaverse Model standards for Community Comments and the start of the Up-sampling Filter for Video Applications (EVC-UFV) V1.0 standard project.

MPAI has been working on the Connected Autonomous Vehicle (MPAI-CAV) project since its early days and released the Architecture Specification (CAV-TEC). Today, MPAI-54 released the Technology specification (CAV-TEC) V1.0 that builds on the Architecture Specification adding subsystems, components, and data types. The specification is released with a request for Community Comments and is available from the MPAI website. Anybody is invited to send comments on MMM-CAV V1.0 to the MPAI Secretariat by April 13 at 23:59 UTC.

MPAI has been working on the MPAI Metaverse Model (MPAI-MMM) project since January 2022. So far, two technical Reports and two Technical Specifications on Architecture and Technologies were published. Today, MPAI-54 released the Technology specification (MMM-TEC) V2.0 integrating the Architecture and Technologies specifications and the Reference Software published by MPAI-53. The specification is released with a request for Community Comments and is available from the MPAI website. Anybody is invited to send comments on MMM-TEC V2.0 to the MPAI Secretariat by April 13 at 23:59 UTC.

Both CAV-TEC V1.0 and MMM-TEC V2.0 reuse technology specifications that are shared with other MPAI standard, namely Object and Scene Description (MPAI-OSD) V1.3, Portable Avatar Format (MPAI-PAF) V1.4, and Data Types, Formats and Attributes (MPAI-TFA) V1.3. The specifications – available from the MPAI-OSD, MPAI-PAF, and MPAI-TFA web pages, respectively – were released with a request for Community Comments. Anybody is invited to send comments on any of the three standards to the MPAI Secretariat by April 13 at 23:59 UTC.

MPAI has identified the need for a standard that specifies an up-sampling filter for video applications with improved performance compared to the currently used up-sampling filters. MPAI-54 decided to kick off the new Up-sampling Filter for Video applications (EVC-UFV) V1.0 standard project based on the received responses to the Call for Technologies.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

Geneva, Switzerland – 19^th February 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 53^rd General Assembly (MPAI-53) releasing the first version of the MPAI Metaverse Model Open-Source Reference Software and kicking off the new project Compression and Understanding of Industrial Data (MPAI-CUI) – Company Performance Prediction (CUI-CPP) V2.0.

MPAI has been working on MPAI Metaverse Model (MPAI-MMM) standards since January 2022 and published Two technical Reports and two Technical Specifications  on Architecture and Technologies. The Reference Software released today implements a significant number of the MMM functionalities and uses a set of Unity instances to realise the different environments of the metaverse instance. You can find the MMM software at http://bit.ly/41J0wsj (REST API web server) and https://bit.ly/4ituw0R (Unity web server).

The Company Performance Prediction (CUI-CPP) project intends to provide a solution to a problem afflicting all companies: given the governance structure and the financial situation of a company and various types of risks that may affect it, what is the impact of the components of the governance structure on the governance, finance, and risk on the probability of default? MPAI has developed a set of functional requirements and a framework licence. A Call for Technologies was published and the standard will be collaboratively developed based on the responses to the Call.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

The 52^nd MPAI General Assembly (MPAI-52) has approved Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0. It is a Technical Report that provides a methodology to predict the game state of an online gaming server when some controller data is lost. The Prediction is obtained by applying Machine Learning algorithms based on historical data of the online game.

An online Multiplayer Game is based on a server. When this maintains consistency among all clients’ game instances it is called authoritative. It updates and broadcasts the game state using the controller data of all the clients. This function is harmed when controller data are not correctly received or are maliciously modified.

There are several techniques currently used to cure this situation. In Client Prediction, client game state is updated locally using predicted or interpolated data while waiting for the server data; in Time Delay, the server buffers the game state updates to synchronise all clients; and in Time Warp the server rolls back the game state to when controller data was sent by a client and acts as if the action was taken then, reconciling this new game state with the current game state.

These three methods have shortcomings. Client Prediction causes perceptible delay, Time Delay affects responsiveness, and Time Warp disadvantages other players because the new game state likely differs from the previous one.

Figure 1 depicts the arrangement proposed by SPG-MDL. The right-hand side represents the online game server with the Game State Engine tasked to produce the game state using all clients’ controller data and specialised engines. The Engines of the figure are:

1. Behaviour Engine, orchestrating actions from players and non-player entities.
2. Rules Engine, ensuring adherence to game mechanics.
3. Physics Engine, responsible for physical interactions within the game environment.

Figure 1 – Server Prediction

Whenever a client’s controller data is lost, the server requests the SPG-MDL module (the left-hand side of Figure 1) to compute the next game state. In this way, even if a client is experiencing network latency, the other clients maintain a continuous playing environment. The more accurate the predictions, the less noticeable the effect of the synchronisation process on the lagging client when the network resumes normal operations. A latency-affected client will still receive the results of its actions with a delay, but the server will send the new game state before receiving the action from the client, effectively halving the wait time. Of course, it is possible to further mitigate the effects of this problem by implementing additional client-side techniques, for example Client Prediction.

When some controller data is lost, the process begins with the last correct game state being fed into the SPG-MDL’s Game State Demultiplexer, which deconstructs it into discrete Game Messages* ( ). To differentiate the Game Messages inside the “twin” game server from the ones inside the game server, the ‘*’ symbol is attached to the SPG-MDL Game Messages. Each Game Message* is then processed by its respective Engine AI, leveraging a Neural Network Model to produce a predicted Game Message* ( ).  These predictions are assembled by the Predicted Game State Multiplexer into the predicted Game State ( ), which is then sent back to the game server for the next iteration of Game State computation.

While the SPG-MDL module operates, the server independently computes its updated Game State (GS_t+1) using the available Client Data. The server utilises the predicted Game State as follows. If any Client Data is missing, the server uses the predicted game state to compensate for the missing data shortfall from one or more clients. Note that the online game server architecture is a reference model and that the three engines are not a requirement for a specific game applying the MPAI-SPG methodology. For example, some games may not have a Physics Engine because physical-based behaviour is not required.

SPG-MDL provides a 10-step procedure to develop an SPG-MDL that is applicable to any authoritative game server.

The first 4 steps are required to outline the game setup to enable informed decisions for the implementation of SPG.

1. Select the game.
2. Define the Entities (to identify NN Model parameters):
   1. Environment
   2. Human-controlled players (HPC) and Non-player characters (NPC)
3. Define the Game State and relevant Entities.
4. Design the training dataset.
5. Collect the training dataset.
6. Train prediction NN Models defining viable architectures and training parameters and comparing the training results of different architectures.
7. Implement SPG-MDL.
8. Evaluate SPG-MDL to select the model yielding the best predictions.
9. Implement modules which simulate the disturbances.
10. Evaluate the SPG-MDL-enabled game experience with human players.

For each of the ten steps, the Technical Report provides:

1. High-level guidelines to outline the actions required.
2. An example of how the guidelines are implemented using a car racing game (Figure 2).

Figure 2 – Modular tiles and an example of a racetrack

The game was developed using the Unity game engine, and the networking features implemented through the open-source game networking library Mirror.

The following components are provided:

• The car racing game.
• Four different categories of Agent Players trained using Unity’s ML-Agents library.
• The dataset used for training generated by simulating game sessions played by the Agent Players.
• Jupyter Notebooks for training experiments and results.
• The trained models used by the AI-Behaviour Engine.

The software is available online. Please contact the MPAI secretariat to access the repository. Details on how to use the material are provided in the repository’s README page.

Geneva, Switzerland – 22^nd January 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 52^nd General Assembly (MPAI-52) approving publication of Technical Report: Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0.

Technical Report: Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0 addresses the effect of controller data latency in online multiplayer gaming. When controller data from a player does not reach the server on time, the server is unable to update and distribute a correct game state. The Technical Report provides guidelines on the design and use of Neural Networks that produce reliable and accurate predictions making up for the absence of players’ control data in multiplayer gaming contexts based on authoritative servers. An example Reference Software allows experimenters to test the suggested guidelines in a practical case.

MPAI will make an online presentation of the main results of the SPG-MDL V1.0 Technical Report on 12^th of February 2025 at 15 UTC. Register at https://encr.pw/CzO72 to attend.

See also the MPAI presentation to LA SIGGRAPH by Leonardo Chiariglione, Marina Bosi (Six Degrees of Freedom Audio), Andrea Bottino (MPAI Metaverse Model), Mark Seligman (Multimodal Conversation), and Ed Lantz (XR Venues)

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): waiting for responses on 11 February.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and Yoube.

The 51st MPAI General Assembly has decide to develop a new version V2.0 of Compression and Understanding of Financial Data (MPAI-CUI) – Company Performance Prediction (CUI-CPP) and issued a Call for Technologies to acquire relevant technologies. Register to attend online event where the Call will be presented on 2025/01/08 T15:00 UTC.

Compression and Understanding of Industrial Data (MPAI-CUI) was one of the first (2021) MPAI standards. The MPAI-CUI V1.0 Company Performance Prediction Use Case was based on the notion that the future of a company largely depends on structure, financial state, and risks it may face in the future. The solution used to address the complex task of creating a standard to predict the future of a company using such variables was successfully achieved was based on:

1. Governance Data, Financial Data, and Risk Assessment Data.
2. Conversion of Governance Data and Financial Data into Descriptors.
3. Conversion of Risk Assessment Data into a Risk Matrix.
4. Passing the Governance and Financial Descriptors to an Organisation Assessment and Default Prediction neural network.
5. Perturbing the Default Probability from the neural network with the Risk Matrix to obtain the Discontinuity Prediction.

Governance Descriptors used as input to the neural network were: #Stakeholder Individuals, #Stakeholder Companies, Shareholder Share, Shareholders Gender, Decision-Makers Gender, #Decision-Makers, Members of the Revision And Advisory Board, Presence Of The Advisory Company, #Decision-Makers By The Same Family, Company Phase (Age).

Financial Descriptors used as input to the neural network were: Revenues, EBITDA Margin, EBITDA, Quick Ratio, Current Ratio, Net Working Capital, Net Financial Position, Net Short-Term Assets, Shareholder Funds-Fixed Assets, Long-Term Liability Ratio, Coverage Of Fixed Assets, Amortisation Rate, Debt On Sales, Interest Coverage Ratio, Average Stock Turnover, Stock Coverage Days, Return On Investments (ROI), Return On Assets (ROA), Return On Sales (ROS), Return On Equity (ROE), Cash Flow, Interest On Sales, Type Of Financial Statement.

Risk Matrix included the following characteristics: Occurrence (3 values), Business Impact (3 values), Gravity (5 values), Risk retention (portion of the risk that the Company decides to retain).

The Governance and Financial Descriptors together with the Prediction Horizon fed to Neural Network provided an Organisational Model Index and a Default Probability. Default Probability and Risk Matrix fed to a Prediction Result Perturbation AIM which perturbed the Default Probability and produced the Business Discontinuity Probability.

Figure 1 depicts the AI Workflow that performs as described above.

Figure 1 – The Company Performance Prediction AI Workflow of MPAI-CUI V1.1

At the 51^st General Assembly (MPAI-51) MPAI issued a Call for Technologies for substantially more ambitious goal for MPAI-CUI V2.0 because it Company performance prediction targets:

• A more precise identification of Cyber, Digitisation, Climate, and Business risks.
• A definition of risks to be organised as follows:
  • Risk name, Risk type: (cyber, etc.),
  • Target regulation,
  • Vector of inputs including, e.g. for Cyber Risks:
    • Name of input: IP address, Denial of service;
    • Time: time the attack was detected;
    • Source: provider of input vector; Type: image, text, category, etc.;
    • Value: depends on type.

With this additional information, the task is to define a (set of) neural network(s) that receive(s) Risk Data in addition to Governance and Financial Descriptors. The Assessment and Prediction network of Figure 2 are not only numbers but Descriptors that include index or probability but also information on which input elements have more influence on index and probabilities.

Figure 2 – The Company Performance Prediction AI Workflow of MPAI-CUI V1.1

Risk Assessment and Risk Matrix are used when sufficient data are not available to train the Neural Network or when the Neural Network may not be used because it does not comply with relevant regulations.

The Call for Technologies is directed to all parties having rights to technologies satisfying the Use Cases and Functional Requirements and the Framework Licence of the planned Technical Specification MPAI-CUI V2.0 are invited to respond to the Call for Technologies, preferably using the Template for Responses. Submissions received by 2024/02/11 will be assessed and considered for use in the development of said MPAI-CUI Technical Specification.

Geneva, Switzerland – 18^th December 2024. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 51^st General Assembly (MPAI-51) approving publication of the Compression and Understanding of Industrial data (MPAI-CUI) V2.0 Call for Technologies.

Call for Technologies: Compression and Understanding of Industrial data (MPAI-CUI) V2.0 invites any party able and wishing to contribute to the development of the planned MPAI-CUI V2.0 Technical Specification to submit a response by 11^th of February 2025. The new standard will extend the current company organisation index and default/discontinuity probabilities with descriptors and information on the compliance of the Machine Learning Models used.

MPAI-51 also approved as MPAI standards:

1. Neural Network Traceability (MPAI-NNT) V1.0 to evaluate the ability to trace back to its source a neural network that has been modified, the computational cost of injecting, extracting, detecting, decoding, or matching data from a neural network, and the impact on the performance of a neural network with inserted traceability data and its inference.
2. Human and Machine Communication (MPAI-HMC) V2.0 that enables advanced forms of communication between humans in a real space or represented in a Virtual Space, and Machines represented as humanoids in a Virtual Space or rendered as humanoids in a real space.
3. Context-based Audio Enhancement (MPAI-CAE) V2.3, Multimodal Conversation (MPAI MMC) V2.3, Object and Scene Descriptors (MPAI-OSD) V1.2, and: Portable Avatar Format (MPAI-PAF) V1.3.

The MPAI-CUI V2.0 Call for Technologies will be presented online on 8^th of January 2025 at 15 UTC. Register at https://tinyurl.com/4vdps8f3 to attend.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): waiting for responses on 11 February.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and Yoube.

MPAI-50 has published the Call for Technologies: Up-sampling Filters for Video applications (MPAI-UFV) V1.0 requesting parties having rights to technologies satisfying the Use Cases and Functional Requirements and the Framework Licence of the planned Technical Specification: Up-sampling Filter for Video Applications (MPAI-UFV) V1.0 to respond to this Call for Technologies preferably using the Template for responses.

The goal of MPAI-UFV V1.0 is to develop a standard up-sampling filter that provides optimal performance when applied to a video to generate a video with a higher number of lines and pixels.

The submissions received will be assessed, collaboratively improved, if found suitable, and used in the development of the planned MPAI-UFV Technical Specification.

MPAI membership is not a prerequisite for responding to this Call for Technologies. However, if a (part of a) submission that is accepted for inclusion in the planned MPAI-UFV TS, the Proponent must join MPAI, or lose the opportunity to have the accepted technologies included in the TS.

MPAI will select the most suitable technologies based on their technical merits. However, MPAI is not obligated to select a particular technology or any of the proposed technologies if those submitted are found to be inadequate.

Proponents shall mandatorily:

1. Submit a complete description of the proposed up-sampling filter with a level of detail allowing an expert in the field to develop and implement the proposed filter.
2. Upload the following software to the MPAI storage:
   1. Docker image that contains the encoding and decoding environments, encoder, decoder, and bitstreams.
   2. Up-sampling filter in source code (preferred) or executable form. Note that proponents of accepted proposals will be requested to provide the source code of the up-sampling filter.
   3. Python scripts to enable testers to carry out the Performance Test.
3. Submit the following results:
   1. Tables of objective quality results obtained by the submitter with their proposed solution.
   2. The decoded Test Sequences.
   3. The up-sampling results for SD to HD and HD to 4K obtained with the proposed solution.
   4. The VMAF-BD Rate assessment provided with a graph for each QP and a table with minimum, maximum and average value for each sequence.
   5. A Complexity assessment using MAC/pixel and number of parameters of the submitted up-sampling filter. Use of SADL (6) is recommended.

Submissions will be evaluated by an Evaluation Team created from:

1. MPAI Member representatives in attendance.
2. Non-MPAI Member representatives who are respondents to any of the received submissions.
3. Non respondent experts/non MPAI Member representatives invited in a consulting capacity.
4. No one from 1. and 2. will be denied membership in the Evaluation Panel if they request it.

Submissions will be evaluated by an Evaluation Team created from:

1. MPAI Member representatives in attendance.
2. Non-MPAI Member representatives who are respondents to any of the received submissions.
3. Non respondent experts/non MPAI Member representatives invited in a consulting capacity.
4. No one from 1. and 2. will be denied membership in the Evaluation Panel if they request it.

Proposals will be assessed using the following process:

1. The objectively computed Quality Tests will use the Test Sequences provided to MPAI by an independent academic after the proposal deadline and distributed to Respondents.
2. Each Respondent presents their proposal.
3. Evaluation Team members ask questions.
4. Evaluation Team organises the Tests.
5. A volunteer member of the Evaluation Team executes the docker image of a Respondent and computes the values obtained using the test set provided by the respected academic.
6. The Objective Quality Evaluation will use the VMAF-BD Rate metrics to compare the sequences obtained by up-sampling with the bicubic filter the AVC, HEVC and VVC co-decoded with QP Values: 22, 27, 32, 37, 42 with the same co-decoded sequences up-sampled with the proposed up-sampling algorithm.
7. Latency is the number of frames used by the up-sampling process. Note that the actual number of frames used to produce the response should be specified.
8. The Complexity evaluation will use MAC/pixel and number of parameters.

The timeline of proposal submission is:

Those intending to propose should become familiar with the four documents mentioned above and posted here. An online presentation will be made on 2024/11/27 at 14UTC. Register at https://bit.ly/3V0sdt1 to attend.

Geneva, Switzerland – 21^st November 2024. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, and unaffiliated organisation developing AI-based data coding standards – has concluded its 49^th General Assembly (MPAI-49) by approving for publication:

1. Call for Technologies: Up-sampling Filter for Video applications (MPAI-UFV)
2. Technical Specification: Neural Network Traceability (MPAI-NNT) V1.0
3. Technical Specification: Human and Machine Communication (MPAI-HMC) V2.0
4. New versions of four standards with added Conformance Testing

Call for Technologies: Up-sampling Filter for Video applications (MPAI-UFV) invites any party able and wishing to contribute to the development of the planned MPAI-UFV Technical Specification to submit a response. MPAI-UFV is expected to leverage AI to provide higher video quality when up-sampling a video from Standard Definition to HDTV and from HDTV to 4K.

Technical Specification: Neural Network Traceability (MPAI-NNT) V1.0 specifies methods to evaluate the ability to trace back to its source a neural network that has been modified, the computational cost of injecting, extracting, detecting, decoding, or matching data from a neural network, and the impact on the performance of a neural network with inserted traceability data and its inference.

Technical Specification: Human and Machine Communication (MPAI-HMC) V2.0 enables advanced forms of communication between humans in a real space or represented in a Virtual Space, and Machines represented as humanoids in a Virtual Space or rendered as humanoids in a real space. This new version leverages MPAI-CAE V2.3, MPAI MMC V2.3, MPAI-OSD V1.2, MPAI-PAF V1.3 and Technical Specification: Data Types, Formats, and Attributes (MPAI-TFA) V1.2 – just published as an MPAI standard.

The Call and the standards will be presented online according to the schedule reported below.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing use cases and functional requirements for MPAI-CUI V2.0 supporting a wide range of corporate risks.
6. End-to-End Video Coding (MPAI-EEV): video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Cillaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

We think that the publication of the MPAI Metaverse Model (MPAI-MMM) standard last week is an important step in making the metaverse a viable proposition because it provides practical means for an implementer to develop a metaverse instance (M-Instance) that interoperates with another similarly developed M-Instance.

This post has the moderate ambition of just describing the high-level content of the standard looking at the Table of Contents.

Foreword obviously deals with general matters, such as the AI Framework (MPAI-AIF) standard and the MPAI Store, but also with the recent notion of Qualifier introduced by the Data Types, Formats, and Attributes (MPAI-TFA) that enables a machine to communicate its technology choices to another machine, a vital component if interoperability (you van have a look at A new type of “data about data”).

Introduction includes a summary of the work carried out by MPAI in the MMM projects: two technical reports and two standards.

Scope formalises the meaning of metaverse interoperability.

Definitions defines some 150 terms used by the MMM standard.

References provides a set of normative and informative references.

Operation despite of it being an informative section, it lays down the main elements of the MMM operation.

Processes, Actions, and Items are the steel, cement, and bricks of which the MMM building is composed. Processes perform Actions on Items (data) to achieve a goal and usually need the help of other Processes to reach it.

Scripting Language (MMM Script) helps a Processes to communicate its needs and requests in an efficient way. It also provides a rigorous human readable method to describe use case using Processes, Actions, and Items.

Profiles provide a simple way to achieve interoperability avoiding the need of possibly costly and useless (for the application) technologies.

Verification Use Cases is the moment of truth for MPAI-MMM. MMM Script is used to describe nine rather complex use cases covering widely different domains to show that the MMM-specified technologies can support the identified used cases.

Not sure that the above will convince you to read the MPAI-MMM standard from page 1 to the end (but MPAI-MMM is published as a series of connected web pages -:), I hope you can dedicate 15 more minutes to read Operation where you will learn more about the internals of MPAI-MMM.