Archives: 2024-12-18

Geneva, Switzerland – 19^th February 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 53^rd General Assembly (MPAI-53) releasing the first version of the MPAI Metaverse Model Open-Source Reference Software and kicking off the new project Compression and Understanding of Industrial Data (MPAI-CUI) – Company Performance Prediction (CUI-CPP) V2.0.

MPAI has been working on MPAI Metaverse Model (MPAI-MMM) standards since January 2022 and published Two technical Reports and two Technical Specifications  on Architecture and Technologies. The Reference Software released today implements a significant number of the MMM functionalities and uses a set of Unity instances to realise the different environments of the metaverse instance.

The Company Performance Prediction (CUI-CPP) project intends to provide a solution to a problem afflicting all companies: given the governance structure and the financial situation of a company and various types of risks that may affect it, what is the impact of the components of the governance structure on the governance, finance, and risk on the probability of default? MPAI has developed a set of functional requirements and a framework licence. A Call for Technologies was published and the standard will be collaboratively developed based on the responses to the Call.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

The 52^nd MPAI General Assembly (MPAI-52) has approved Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0. It is a Technical Report that provides a methodology to predict the game state of an online gaming server when some controller data is lost. The Prediction is obtained by applying Machine Learning algorithms based on historical data of the online game.

An online Multiplayer Game is based on a server. When this maintains consistency among all clients’ game instances it is called authoritative. It updates and broadcasts the game state using the controller data of all the clients. This function is harmed when controller data are not correctly received or are maliciously modified.

There are several techniques currently used to cure this situation. In Client Prediction, client game state is updated locally using predicted or interpolated data while waiting for the server data; in Time Delay, the server buffers the game state updates to synchronise all clients; and in Time Warp the server rolls back the game state to when controller data was sent by a client and acts as if the action was taken then, reconciling this new game state with the current game state.

These three methods have shortcomings. Client Prediction causes perceptible delay, Time Delay affects responsiveness, and Time Warp disadvantages other players because the new game state likely differs from the previous one.

Figure 1 depicts the arrangement proposed by SPG-MDL. The right-hand side represents the online game server with the Game State Engine tasked to produce the game state using all clients’ controller data and specialised engines. The Engines of the figure are:

1. Behaviour Engine, orchestrating actions from players and non-player entities.
2. Rules Engine, ensuring adherence to game mechanics.
3. Physics Engine, responsible for physical interactions within the game environment.

Figure 1 – Server Prediction

Whenever a client’s controller data is lost, the server requests the SPG-MDL module (the left-hand side of Figure 1) to compute the next game state. In this way, even if a client is experiencing network latency, the other clients maintain a continuous playing environment. The more accurate the predictions, the less noticeable the effect of the synchronisation process on the lagging client when the network resumes normal operations. A latency-affected client will still receive the results of its actions with a delay, but the server will send the new game state before receiving the action from the client, effectively halving the wait time. Of course, it is possible to further mitigate the effects of this problem by implementing additional client-side techniques, for example Client Prediction.

When some controller data is lost, the process begins with the last correct game state being fed into the SPG-MDL’s Game State Demultiplexer, which deconstructs it into discrete Game Messages* ( ). To differentiate the Game Messages inside the “twin” game server from the ones inside the game server, the ‘*’ symbol is attached to the SPG-MDL Game Messages. Each Game Message* is then processed by its respective Engine AI, leveraging a Neural Network Model to produce a predicted Game Message* ( ).  These predictions are assembled by the Predicted Game State Multiplexer into the predicted Game State ( ), which is then sent back to the game server for the next iteration of Game State computation.

While the SPG-MDL module operates, the server independently computes its updated Game State (GS_t+1) using the available Client Data. The server utilises the predicted Game State as follows. If any Client Data is missing, the server uses the predicted game state to compensate for the missing data shortfall from one or more clients. Note that the online game server architecture is a reference model and that the three engines are not a requirement for a specific game applying the MPAI-SPG methodology. For example, some games may not have a Physics Engine because physical-based behaviour is not required.

SPG-MDL provides a 10-step procedure to develop an SPG-MDL that is applicable to any authoritative game server.

The first 4 steps are required to outline the game setup to enable informed decisions for the implementation of SPG.

1. Select the game.
2. Define the Entities (to identify NN Model parameters):
   1. Environment
   2. Human-controlled players (HPC) and Non-player characters (NPC)
3. Define the Game State and relevant Entities.
4. Design the training dataset.
5. Collect the training dataset.
6. Train prediction NN Models defining viable architectures and training parameters and comparing the training results of different architectures.
7. Implement SPG-MDL.
8. Evaluate SPG-MDL to select the model yielding the best predictions.
9. Implement modules which simulate the disturbances.
10. Evaluate the SPG-MDL-enabled game experience with human players.

For each of the ten steps, the Technical Report provides:

1. High-level guidelines to outline the actions required.
2. An example of how the guidelines are implemented using a car racing game (Figure 2).

Figure 2 – Modular tiles and an example of a racetrack

The game was developed using the Unity game engine, and the networking features implemented through the open-source game networking library Mirror.

The following components are provided:

• The car racing game.
• Four different categories of Agent Players trained using Unity’s ML-Agents library.
• The dataset used for training generated by simulating game sessions played by the Agent Players.
• Jupyter Notebooks for training experiments and results.
• The trained models used by the AI-Behaviour Engine.

The software is available online. Please contact the MPAI secretariat to access the repository. Details on how to use the material are provided in the repository’s README page.

Geneva, Switzerland – 22^nd January 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 52^nd General Assembly (MPAI-52) approving publication of Technical Report: Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0.

Technical Report: Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0 addresses the effect of controller data latency in online multiplayer gaming. When controller data from a player does not reach the server on time, the server is unable to update and distribute a correct game state. The Technical Report provides guidelines on the design and use of Neural Networks that produce reliable and accurate predictions making up for the absence of players’ control data in multiplayer gaming contexts based on authoritative servers. An example Reference Software allows experimenters to test the suggested guidelines in a practical case.

MPAI will make an online presentation of the main results of the SPG-MDL V1.0 Technical Report on 12^th of February 2025 at 15 UTC. Register at https://encr.pw/CzO72 to attend.

See also the MPAI presentation to LA SIGGRAPH by Leonardo Chiariglione, Marina Bosi (Six Degrees of Freedom Audio), Andrea Bottino (MPAI Metaverse Model), Mark Seligman (Multimodal Conversation), and Ed Lantz (XR Venues)

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): waiting for responses on 11 February.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and Yoube.

The 51st MPAI General Assembly has decide to develop a new version V2.0 of Compression and Understanding of Financial Data (MPAI-CUI) – Company Performance Prediction (CUI-CPP) and issued a Call for Technologies to acquire relevant technologies. Register to attend online event where the Call will be presented on 2025/01/08 T15:00 UTC.

Compression and Understanding of Industrial Data (MPAI-CUI) was one of the first (2021) MPAI standards. The MPAI-CUI V1.0 Company Performance Prediction Use Case was based on the notion that the future of a company largely depends on structure, financial state, and risks it may face in the future. The solution used to address the complex task of creating a standard to predict the future of a company using such variables was successfully achieved was based on:

1. Governance Data, Financial Data, and Risk Assessment Data.
2. Conversion of Governance Data and Financial Data into Descriptors.
3. Conversion of Risk Assessment Data into a Risk Matrix.
4. Passing the Governance and Financial Descriptors to an Organisation Assessment and Default Prediction neural network.
5. Perturbing the Default Probability from the neural network with the Risk Matrix to obtain the Discontinuity Prediction.

Governance Descriptors used as input to the neural network were: #Stakeholder Individuals, #Stakeholder Companies, Shareholder Share, Shareholders Gender, Decision-Makers Gender, #Decision-Makers, Members of the Revision And Advisory Board, Presence Of The Advisory Company, #Decision-Makers By The Same Family, Company Phase (Age).

Financial Descriptors used as input to the neural network were: Revenues, EBITDA Margin, EBITDA, Quick Ratio, Current Ratio, Net Working Capital, Net Financial Position, Net Short-Term Assets, Shareholder Funds-Fixed Assets, Long-Term Liability Ratio, Coverage Of Fixed Assets, Amortisation Rate, Debt On Sales, Interest Coverage Ratio, Average Stock Turnover, Stock Coverage Days, Return On Investments (ROI), Return On Assets (ROA), Return On Sales (ROS), Return On Equity (ROE), Cash Flow, Interest On Sales, Type Of Financial Statement.

Risk Matrix included the following characteristics: Occurrence (3 values), Business Impact (3 values), Gravity (5 values), Risk retention (portion of the risk that the Company decides to retain).

The Governance and Financial Descriptors together with the Prediction Horizon fed to Neural Network provided an Organisational Model Index and a Default Probability. Default Probability and Risk Matrix fed to a Prediction Result Perturbation AIM which perturbed the Default Probability and produced the Business Discontinuity Probability.

Figure 1 depicts the AI Workflow that performs as described above.

Figure 1 – The Company Performance Prediction AI Workflow of MPAI-CUI V1.1

At the 51^st General Assembly (MPAI-51) MPAI issued a Call for Technologies for substantially more ambitious goal for MPAI-CUI V2.0 because it Company performance prediction targets:

• A more precise identification of Cyber, Digitisation, Climate, and Business risks.
• A definition of risks to be organised as follows:
  • Risk name, Risk type: (cyber, etc.),
  • Target regulation,
  • Vector of inputs including, e.g. for Cyber Risks:
    • Name of input: IP address, Denial of service;
    • Time: time the attack was detected;
    • Source: provider of input vector; Type: image, text, category, etc.;
    • Value: depends on type.

With this additional information, the task is to define a (set of) neural network(s) that receive(s) Risk Data in addition to Governance and Financial Descriptors. The Assessment and Prediction network of Figure 2 are not only numbers but Descriptors that include index or probability but also information on which input elements have more influence on index and probabilities.

Figure 2 – The Company Performance Prediction AI Workflow of MPAI-CUI V1.1

Risk Assessment and Risk Matrix are used when sufficient data are not available to train the Neural Network or when the Neural Network may not be used because it does not comply with relevant regulations.

The Call for Technologies is directed to all parties having rights to technologies satisfying the Use Cases and Functional Requirements and the Framework Licence of the planned Technical Specification MPAI-CUI V2.0 are invited to respond to the Call for Technologies, preferably using the Template for Responses. Submissions received by 2024/02/11 will be assessed and considered for use in the development of said MPAI-CUI Technical Specification.

Geneva, Switzerland – 18^th December 2024. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 51^st General Assembly (MPAI-51) approving publication of the Compression and Understanding of Industrial data (MPAI-CUI) V2.0 Call for Technologies.

Call for Technologies: Compression and Understanding of Industrial data (MPAI-CUI) V2.0 invites any party able and wishing to contribute to the development of the planned MPAI-CUI V2.0 Technical Specification to submit a response by 11^th of February 2025. The new standard will extend the current company organisation index and default/discontinuity probabilities with descriptors and information on the compliance of the Machine Learning Models used.

MPAI-51 also approved as MPAI standards:

1. Neural Network Traceability (MPAI-NNT) V1.0 to evaluate the ability to trace back to its source a neural network that has been modified, the computational cost of injecting, extracting, detecting, decoding, or matching data from a neural network, and the impact on the performance of a neural network with inserted traceability data and its inference.
2. Human and Machine Communication (MPAI-HMC) V2.0 that enables advanced forms of communication between humans in a real space or represented in a Virtual Space, and Machines represented as humanoids in a Virtual Space or rendered as humanoids in a real space.
3. Context-based Audio Enhancement (MPAI-CAE) V2.3, Multimodal Conversation (MPAI MMC) V2.3, Object and Scene Descriptors (MPAI-OSD) V1.2, and: Portable Avatar Format (MPAI-PAF) V1.3.

The MPAI-CUI V2.0 Call for Technologies will be presented online on 8^th of January 2025 at 15 UTC. Register at https://tinyurl.com/4vdps8f3 to attend.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): waiting for responses on 11 February.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and Yoube.

MPAI-50 has published the Call for Technologies: Up-sampling Filters for Video applications (MPAI-UFV) V1.0 requesting parties having rights to technologies satisfying the Use Cases and Functional Requirements and the Framework Licence of the planned Technical Specification: Up-sampling Filter for Video Applications (MPAI-UFV) V1.0 to respond to this Call for Technologies preferably using the Template for responses.

The goal of MPAI-UFV V1.0 is to develop a standard up-sampling filter that provides optimal performance when applied to a video to generate a video with a higher number of lines and pixels.

The submissions received will be assessed, collaboratively improved, if found suitable, and used in the development of the planned MPAI-UFV Technical Specification.

MPAI membership is not a prerequisite for responding to this Call for Technologies. However, if a (part of a) submission that is accepted for inclusion in the planned MPAI-UFV TS, the Proponent must join MPAI, or lose the opportunity to have the accepted technologies included in the TS.

MPAI will select the most suitable technologies based on their technical merits. However, MPAI is not obligated to select a particular technology or any of the proposed technologies if those submitted are found to be inadequate.

Proponents shall mandatorily:

1. Submit a complete description of the proposed up-sampling filter with a level of detail allowing an expert in the field to develop and implement the proposed filter.
2. Upload the following software to the MPAI storage:
   1. Docker image that contains the encoding and decoding environments, encoder, decoder, and bitstreams.
   2. Up-sampling filter in source code (preferred) or executable form. Note that proponents of accepted proposals will be requested to provide the source code of the up-sampling filter.
   3. Python scripts to enable testers to carry out the Performance Test.
3. Submit the following results:
   1. Tables of objective quality results obtained by the submitter with their proposed solution.
   2. The decoded Test Sequences.
   3. The up-sampling results for SD to HD and HD to 4K obtained with the proposed solution.
   4. The VMAF-BD Rate assessment provided with a graph for each QP and a table with minimum, maximum and average value for each sequence.
   5. A Complexity assessment using MAC/pixel and number of parameters of the submitted up-sampling filter. Use of SADL (6) is recommended.

Submissions will be evaluated by an Evaluation Team created from:

1. MPAI Member representatives in attendance.
2. Non-MPAI Member representatives who are respondents to any of the received submissions.
3. Non respondent experts/non MPAI Member representatives invited in a consulting capacity.
4. No one from 1. and 2. will be denied membership in the Evaluation Panel if they request it.

Submissions will be evaluated by an Evaluation Team created from:

1. MPAI Member representatives in attendance.
2. Non-MPAI Member representatives who are respondents to any of the received submissions.
3. Non respondent experts/non MPAI Member representatives invited in a consulting capacity.
4. No one from 1. and 2. will be denied membership in the Evaluation Panel if they request it.

Proposals will be assessed using the following process:

1. The objectively computed Quality Tests will use the Test Sequences provided to MPAI by an independent academic after the proposal deadline and distributed to Respondents.
2. Each Respondent presents their proposal.
3. Evaluation Team members ask questions.
4. Evaluation Team organises the Tests.
5. A volunteer member of the Evaluation Team executes the docker image of a Respondent and computes the values obtained using the test set provided by the respected academic.
6. The Objective Quality Evaluation will use the VMAF-BD Rate metrics to compare the sequences obtained by up-sampling with the bicubic filter the AVC, HEVC and VVC co-decoded with QP Values: 22, 27, 32, 37, 42 with the same co-decoded sequences up-sampled with the proposed up-sampling algorithm.
7. Latency is the number of frames used by the up-sampling process. Note that the actual number of frames used to produce the response should be specified.
8. The Complexity evaluation will use MAC/pixel and number of parameters.

The timeline of proposal submission is:

Those intending to propose should become familiar with the four documents mentioned above and posted here. An online presentation will be made on 2024/11/27 at 14UTC. Register at https://bit.ly/3V0sdt1 to attend.

Geneva, Switzerland – 21^st November 2024. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, and unaffiliated organisation developing AI-based data coding standards – has concluded its 49^th General Assembly (MPAI-49) by approving for publication:

1. Call for Technologies: Up-sampling Filter for Video applications (MPAI-UFV)
2. Technical Specification: Neural Network Traceability (MPAI-NNT) V1.0
3. Technical Specification: Human and Machine Communication (MPAI-HMC) V2.0
4. New versions of four standards with added Conformance Testing

Call for Technologies: Up-sampling Filter for Video applications (MPAI-UFV) invites any party able and wishing to contribute to the development of the planned MPAI-UFV Technical Specification to submit a response. MPAI-UFV is expected to leverage AI to provide higher video quality when up-sampling a video from Standard Definition to HDTV and from HDTV to 4K.

Technical Specification: Neural Network Traceability (MPAI-NNT) V1.0 specifies methods to evaluate the ability to trace back to its source a neural network that has been modified, the computational cost of injecting, extracting, detecting, decoding, or matching data from a neural network, and the impact on the performance of a neural network with inserted traceability data and its inference.

Technical Specification: Human and Machine Communication (MPAI-HMC) V2.0 enables advanced forms of communication between humans in a real space or represented in a Virtual Space, and Machines represented as humanoids in a Virtual Space or rendered as humanoids in a real space. This new version leverages MPAI-CAE V2.3, MPAI MMC V2.3, MPAI-OSD V1.2, MPAI-PAF V1.3 and Technical Specification: Data Types, Formats, and Attributes (MPAI-TFA) V1.2 – just published as an MPAI standard.

The Call and the standards will be presented online according to the schedule reported below.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing use cases and functional requirements for MPAI-CUI V2.0 supporting a wide range of corporate risks.
6. End-to-End Video Coding (MPAI-EEV): video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Cillaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

We think that the publication of the MPAI Metaverse Model (MPAI-MMM) standard last week is an important step in making the metaverse a viable proposition because it provides practical means for an implementer to develop a metaverse instance (M-Instance) that interoperates with another similarly developed M-Instance.

This post has the moderate ambition of just describing the high-level content of the standard looking at the Table of Contents.

Foreword obviously deals with general matters, such as the AI Framework (MPAI-AIF) standard and the MPAI Store, but also with the recent notion of Qualifier introduced by the Data Types, Formats, and Attributes (MPAI-TFA) that enables a machine to communicate its technology choices to another machine, a vital component if interoperability (you van have a look at A new type of “data about data”).

Introduction includes a summary of the work carried out by MPAI in the MMM projects: two technical reports and two standards.

Scope formalises the meaning of metaverse interoperability.

Definitions defines some 150 terms used by the MMM standard.

References provides a set of normative and informative references.

Operation despite of it being an informative section, it lays down the main elements of the MMM operation.

Processes, Actions, and Items are the steel, cement, and bricks of which the MMM building is composed. Processes perform Actions on Items (data) to achieve a goal and usually need the help of other Processes to reach it.

Scripting Language (MMM Script) helps a Processes to communicate its needs and requests in an efficient way. It also provides a rigorous human readable method to describe use case using Processes, Actions, and Items.

Profiles provide a simple way to achieve interoperability avoiding the need of possibly costly and useless (for the application) technologies.

Verification Use Cases is the moment of truth for MPAI-MMM. MMM Script is used to describe nine rather complex use cases covering widely different domains to show that the MMM-specified technologies can support the identified used cases.

Not sure that the above will convince you to read the MPAI-MMM standard from page 1 to the end (but MPAI-MMM is published as a series of connected web pages -:), I hope you can dedicate 15 more minutes to read Operation where you will learn more about the internals of MPAI-MMM.

Geneva, Switzerland – 30^th September 2024. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, and unaffiliated organisation developing AI-based data coding standards – has concluded its 48^th General Assembly (MPAI-48) by approving for publication

1. The combined MPAI Metaverse Model – Architecture (MMM-ARC) V1.2 and Technologies (MMM-TEC) V1.0 standards
2. The set of Multimodal Conversation (MPAI-MMC) V2.2; Object and Scene Description (MPAI-OSD) V1.1; Portable Avatar Format (MPAI-PAF) V1.2; and Data Types, Formats, and Attributes (MPAI-TFA) V1.0 standards.
3. The Open-Source Reference Implementation of the Television Media Analysis (OSD-TMA) V1.1 Use case.

All new standards are released using the MPAI full web-based publication method providing the specification and, where available, the reference software, the conformance testing, and the performance assessment.

MPAI-48 also learned that four MPAI standards have been adopted as IEEE standards: two new (MPAI-MMM and MPAI-PAF) and two revised (MPAI-AIF and MPAI-MMC) bringing the total to eight.

The combined Technical Specification: MPAI Metaverse Model (MPAI-MMM) –  Architecture (MMM-ARC) V1.2 and Technical Specification: MPAI Metaverse Model (MPAI-MMM) –  Technologies (MMM-ARC) V1.0 specify five types of Processes operating in an M-Instance, thirty Actions that Processes can perform, 65 Data Types and their Qualifiers that Processes can Act on to achieve client and M-Instance interoperability.

Technical Specification: Multimodal Conversation (MPAI-MMC) V2.2 specifies 23 Data Types that enable more human-like and content-rich forms of human-machine conversation, applies them to eight use cases in different domains and specifies 23 AI Modules. MPAI-MMC reuses Data Types and AI Modules from other MPAI standards.

Technical Specification: Object and Scene Description (MPAI-OSD) V1.1 specifies 27 Data Types enabling the digital representation of spatial information of Audio and Visual Objects and Scenes, applies them to the Television Media Analysis (OSD-TMA) use case and specifies 15 AI Modules. MPAI-OSD reuses Data Types and AI Modules from other MPAI standards.

Technical Specification: Portable Avatar Format (MPAI-PAF) V1.2 specifies five Data Types enabling a receiving party to render a digital human as intended by the sending party, applies them to the Avatar-Based Videoconference Use Case, and specifies 13 AI Modules. MPAI-PAF reuses Data Types and AI Modules from other MPAI standards.

Technical Specification: Data Types, Formats, and Attributes (MPAI-TFA) V1.0 specifies Qualifiers – a Data Type containing Sub-Types, Formats, and Attributes – associated to “media” Data Types – currently Text, Speech, Audio, Visual, 3D Model, and Audio-Visual – that facilitate/enable the operation of an AI Module receiving a Data Type instance. MPAI-48 has published Version 1.1 of the standard with Data Qualifiers for the MPAI Metaverse Model.

The reference software implementation of the Television Media Analysis (OSD-TMA) V1.1 Use Case produces a description of a TV program that includes the audio and visual objects, the IDs of the speakers and the faces with their space and time information, and the text of the speaker utterances.

Online presentations of MPAI-MMC, MPAI-MMM, MPAI-OSD, MPAI-PAF, and the reference software implementation of OSD-TMA will be made according to the following timetable:

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF).
4. Connected Aonomous Vehicle (MPAI-CAV): developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard and updating the Architecture part.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing use cases and functional requirements for MPAI-CUI V2.0 supporting a wide range of corporate risks.
6. End-to-End Video Coding (MPAI-EEV): video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): working on a Call for Technologies for up-sampling filter for video.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Cillaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribe to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and Yoube.

AI Modules (AIM) organised in AI Workflows (AIW) executed in the AI Framework enabling initialisation, dynamic configuration, and control of AIWs are a key element of the MPAI approach to AI-based Data Coding standards as depicted in Figure 1. AIMs communicate to other AIMs in the AIW the Data obtained by executing specific functions.

The effectiveness of the functions performed by the AIMs is improved if they know more about the capabilities of the AIMs they are connected to and the Data they receive as demonstrated by natural language processing. An instance of the MPAI Natural Language Processing (MMC-NLU) AIM can produce the recognised text and Meaning using three levels of information:

1.Just the input text.

2.Also the object identifiers referenced in the text.

3.Additionally, the object context in a relevant space.

The accuracy of the refined text and Meaning produced  by an MPAI-NLU AIM is expected to improve when moving from the first to the third case. The cases correspond with different levels of AIM capabilities.

Technical Specification: AI Module Profiles (MPAI-PRF) enables an AIM instance to signal its Attributes – such as input data, output data, and functionality – and Sub-Attributes – such as languages supported by a Text and Speech Translation AIM – that uniquely characterise the AIM. Currently, MPAI-PRF defines the Attributes of eight AIMs but Profiles for more AIMs are likely to be defined in the future.

The effectiveness of the functions performed by an AIM can also be enabled or enhanced if the AIM knows more about the characteristics of the Data received. Examples of characteristics include:

• The CIE 1931 colour space of an instance of the Visual Data Type.
• The MP3 format of a speech segment.
• The WAV file format of an audio segment.
• The gamma correction applied to the device that produced a video.
• The ID of an object instance in an audio segment.
• The Text conveyed by a speech segment.

Technical Specification: Data Types, Formats, and Attributes (MPAI-TFA) V1.0 specifies a new Data Type called Qualifier, a container that can be used to represent, for instance, that a Visual Data Type instance:

• Uses a given colour space (Sub-Type)
• Was produced by an AVC codec (Format).
• Is described by Dublin Core Metadata (Attribute).

The current versions of MPAI Technical Specifications generally assume that most of the Media Objects exchanged by AIMs are composed of “Content” and “Qualifiers”.

Therefore, Qualifiers are a specialised type of metadata intended to support the operation of AIMs receiving data from other AIMs and conveying information on Sub-Types, Formats, and Attributes related to the Content. Qualifiers are  intended to convey information for use by an AIM. They are human-readable but intended only to be used by AIMs.

MPAI also provides a standard method to attach information to a Data Type instance called Annotation defined as Data attached to an Object or a Scene. As opposed to Qualifier that describes intrinsic properties of a Data Type, an Annotation is spatially and temporally local and changeable.

Future versions of MPAI-TFA will likely be published because of the large variety of application needs that will require the specification of Qualifiers for additional Data Types. MPAI-TFA users are invited to communicate their need for extension of existing and specification of additional Data Types in MPAI-TFA to the MPAI Secretariat. Therefore, versioning of Qualifiers is a critical component of MPAI-TFA.