Archives: 2021-08-27

The MPAI Metaverse Model – Technologies standard – in short, MMM-TEC V2.0 – is the first open metaverse standard enabling independently designed and implemented defines a metaverse instance (that MMM-TEC calls M-Instances) and clients to interoperate. These are the main MMM-TEC elements:

• The Architecture is based on Processes acting on Items based on the Rights they hold.
• Items represent any abstract and concrete objects in an M-Instance.
• Processes – possibly in different M-Instances – communicate using the Inter-Process Protocol (IPP).
• Process Actions represent the payload of a message sent by a Process.
• Qualifiers are containers of technology-specific information of an Item.
• The MPAI-MMM API enables fast development of M-Instances.
• Verification Use Cases verify the completeness of the standard.
• The MMM Open-Source Software implementation can easily be installed.

Below is a more extended introduction to MMM-TEC. Here is the text of Technical Specification: MPAI Metaverse Model (MPAI-MMM) – Technologies (MMM-TEC) V2.0.

MMM-TEC  defines an M-Instance as an Information and Communication Technologies platform populated by Processes. They perform a range of activities, such as operate with various degrees of autonomy and interactivity, sense data from the real world, produce various types of entities called Items, perform or request other Processes to perform activities represented by Process Actions or request other Processes – possibly in other M-Instances, hold or acquire Rights on Items, and act on the real world on a variety of ways. They can perform Process Actions based on Rights they may hold, acquire, or be granted.

Processes may be characterised as:

1. Services providing specific functionalities, such as content authoring.
2. Devices connecting the Universe to the M-Instance and the M-Instance to the Universe.
3. Apps running on Devices.
4. Users representing and acting on behalf of human entities residing in the Universe. A User is rendered as a Persona, i.e., an avatar.

Figure 1 depicts the main elements on which the MMM-TEC Specification is based: human, Devices, Apps, Users, Services, and Personae.

Figure 1 – Main elements of an M-Instance

Processes Sense Data from U-Environments, i.e., portions of the Universe and may produce three types of Items, i.e., Data that has been Identified in – and thus recognised by – the M-Instance:

1. Digitised– i.e., sensed from the Universe – possibly animated by activities in the Universe.
2. Virtual– i.e., imported from the Universe as Data or internally generated – possibly autonomous or driven by activities in the Universe.
3. Mixed – Digitised and Virtual.

Processes Perform – either on their initiative, or driven by the actions of humans or machines in the Universe – Process Actions that combine:

1. An Action, possibly prepended by:
   1. MM:to indicate Actions performed inside the M-Instance, e.g., MM-Animate using a stream to animate a 3D Model with a Spatial Attitude (defined as Position, Orientation, and their velocities and accelerations).
   2. MU:to indicate Actions in the M-Instance influencing the Universe, e.g., MU-Actuate to render one of its Items to a U-Location as Media with a Spatial Attitude.
   3. UM: to indicate Actions in the Universe influencing the M-Instance, e.g., UM-Embed to place an Item produced by Identifying a scene, UM-Captured at a U-Location, at an M-Location with a Spatial Attitude.
2. Items on which the Action is performed or are required for performance, such as Asset, 3D Model, Audio Object, Audio-Visual Scene, etc.
3. M-Locations and/or U-Locationswhere the Process Action is performed.
4. Processes with which the Action is performed.
5. Time(s) during which the Process Action is requested to be and is performed.

Processes may hold Rights on an Item, i.e., they may perform the set of Process Actions listed in their Rights. An Item may include Rights signalling which Processes may perform Process Actions on it. Processes affect U-Environments and/or M-Instances using Items in ways that are Consistent with the goals of the M-Instance as expressed by the Rules, within the M-Capabilities of the M-Instance, e.g., to support Transactions, and respecting applicable laws and regulations.

Processes perform activities strictly inside the M-Instance or have various degrees of interaction with Data sensed from and/or actuated in the Universe.

Processes may request other Processes to perform Process Actions on their behalf by using the Inter-Process Protocol, possibly after Transacting a Value (i.e., an Amount in a Currency) to a Wallet.

An M-Instance is managed by an M-Instance Manager. At the initial time, the M-Instance Manager has Rights covering the M-Instance and may decide to define certain subsets inside the M-Instance – called M-Environments – on which it has Rights and attach Rights to them.

A Registering human may:

1. Request to Register to open an Account of a certain class.
2. Be requested to provide their Personal Profile and possibly to perform a Transaction to open an Account.
3. Obtain in exchange a set of Rights that their Processes may perform. Rights have Levels indicating that the Rights are
   1. Internal, g., assigned by the M-Instance at Registration time according to the M-Instance Rules and the Account type.
   2. Acquired, g., obtained by initiative of the Process.
   3. Granted to the Process by another Process.

MMM-TEC V2.0 does not specify how an M-Instance verifies that the Process Actions performed by a Process comply with the Process’s Rights or the M-Instance Rules.  An M-Instance may decide to verify the full set of Activity Data (the log of performed Process Actions), or to make verifications based on claims by another Process, to make random verifications, or to not make any verification at all. Therefore, MMM-TEC V2.0 does not specify how a M-Instance Manager can sanction non-complying Processes.

In some cases, an M-Instance could be wastefully too costly as an undertaking if all the technologies required by the MMM Technical Specification were mandatorily to be implemented, even if a specific M-Instance had limited scope. MMM-TEC V2.0 specifies Profiles to facilitate the take-off of M-Instance implementations that conform to the MMM-TEC V2.0 specification without unduly burdening some other implementations.

A Profile includes only a subset of the Process Actions that are expected to be needed and are shared by a sizeable number of applications. MMM-TEC v2.0 defines four Profiles (see Figure 2:

1. Baseline Profile enables basic applications such as lecture, meeting, and hang-out.
2. Finance Profile enables trading activities.
3. Management Profile includes enables a controlled ecosystem with more advanced functionalities.
4. High Profile enables all the functionalities of the Management Profile with a few additional functionalities of its own.

Figure 2 – MMM-TEC V2.0 Profiles

MPAI developed and used some use cases in the two MPAI-MMM Technical Reports (1 and 2) published in 2023 to develop the MMM-ARC and MMM-TEC Technical Specifications. However, MMM-TEC V2.0 includes various Verification Use Cases that use Process Actions to verify that the currently specified Actions and Items completely support those Use Cases.

The fast development of certain technology areas is one of the issues that has so far prevented the development of metaverse interoperability standards. MMM-TEC deals with this issue by providing JSON syntax and semantics for all Items. When needed, the JSON syntax references Qualifiers, MPAI-defined Data Types that supply additional information to the Data in the form of:

1. Sub-Type (e.g., the colour space of a Visual Data Type).
2. Format (e.g., the compression or the file/streaming format of Speech).
3. Attributes (e.g., the Binaural Cues of an Audio Object).

For instance, a Process receiving an Object can understand from the Qualifier referenced in the Object whether it has the required technology to process it, or else it has to rely on a Conversion Service to obtain a version of the Object matching its P-Capabilities. This approach should help to prolong the life of the MMM-TEC specification as in many cases only the Qualifier specification will need to be updated, not the MMM-TEC specification.

Finally, MMM-TEC V2.0 specifies the MPAI-MMM API. By calling the APIs, a developer can easily develop M-Instances and applications.

Geneva, Switzerland – 16^th April 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 55^th General Assembly (MPAI-55) with the final release of the Connected Autonomous Vehicle and the MPAI Metaverse Model standards.

The MPAI Metaverse Model (MPAI-MMM) – Technologies (MMM-TEC) V2.0 standard specifies:

1. The Functional Requirements of the Processesoperating in a metaverse instance (M-Instance).
2. The Items, i.e., the Data Types and their Qualifiers recognised in an M-Instance.
3. The Process Actionsthat a Process can perform on Items.
4. The Protocolsenabling a Process to communicate with another Process.
5. The MPAI Metaverse Model
6. The MPAI-MMM API.

Availability of APIs enable the rapid development of M-Instances and clients that interoperate with M-Instances conforming with the MMM-TEC V2.0 standard.

An online presentation of the MMM-TEC V2.0 standard will be presented on 9 May at 15 UTC. Register at https://tinyurl.com/5a2d4ucv.

The Connected Autonomous Vehicle (MPAI-CAV) – Technologies (CAV-TEC) V1.0 standard specifies the Reference Model partitioning a CAV into subsystems and components. The standard Reference Model promotes CAV componentisation by enabling:

1. Researchersto optimise component technologies.
2. Component manufacturersto bring their standard-conforming components to an open market.
3. Car manufacturersto access a global market of interchangeable components.
4. Regulatorsto oversee conformance testing of components following standard procedures.
5. Usersto rely on Connected Autonomous Vehicles whose operation they can explain.

An online presentation of the CAV-TEC V1.0 standard will be presented on 8 May at 15 UTC. Register at https://tinyurl.com/372739sa.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Autonomous Vehicle (MPAI-CAV): investigating extensions of the current CAV-TEC standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): developing the Up-sampling Filter for Video applications (EVC-UFV) standard.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MMM-TEC specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

MPAI has recently published two main new standards with a request for Community Comments. In MPAI lingo this means that the standards are mature, but MPAI asks the Community to review the drafts before publication.

The two standards are Connected Autonomous Vehicle – Technologies (CAV-TEC) V1.0 and MPAI Metaverse Model – Technologies (MMM-TEC) V2.0. They are not “new” MPAI standards as they have already been published in earlier version, but the new versions represent significant improvements.

You may ask “Why is this topic handled in a single paper when we are talking about two standards that have – apparently – so little in common”? If you ask this question, you may want to continue reading and discover an essential aspect of MPAI standardisation. A standard for an application domain is (almost) never a monolith but is often made of components shared with standards of other domains, often unrelated.

Let’s first have a scan of the two standards.

Connected Autonomous Vehicle (CAV-TEC V1.0) is a standard for the ICT (Information and Communication Technology) part of a vehicle that can move itself in the physical world to reach a destination. The standard assumes that a CAV is composed of four functionally separated but interconnected subsystems:

1. The human-CAV Interaction (HCI): enables a human to establish a dialogue with the CAV to issue a variety of commands to it, such as moving to a destination or have conversations where the human – and the CAV – can express their internal status – e.g., emotion – whether real or fictitious.
2. The Environment Sensing Subsystem (EES): leverages the sensors on board to create the Basic Scene Descriptors (BED), the most accurate digital representation of the external environment possible with the available sensors.
3. The Autonomous Motion Subsystem (AMS): receives the BED and improves its accuracy by exchanging portions of their Environment Descriptors with CAVs in range. Then it analyses or reviews the situation and decides how to issue commands to implement the route decided by the human.
4. The Motion Actuation Subsystem converts a general request to move the CAV potentially by a few meters to specific commands for brakes, motors, and wheel and to report about the implementation of the command.

The four subsystems are implemented as AI Workflows per the AI Framework standard. Each AI Workflow includes several AI Modules that exchange Data specifies by CAV-TEC V1.0 or by other MPAI standards. Most AIMs and data types of:

1. HCI: is specified not by CAV-TEC but by Multimodal Conversation (MPAI-MMC) V2.3 . The rest is specified by Object and Scene Description (MPAI-OSD) V1.3, Portable Avatar Format (MPAI-PAF) V1.4, Data Types, Formats, and Attributes (MPAI-TFA) V1.3, and CAV-TEC.
2. ESS, AMS, and MAS: are specified by CAV-TEC, MPAI-OSD,  MPAI-PAF, and MPAI-TFA.

A concise description of the operation of an implementation of CAV-TEC is available here.

MPAI Metaverse Model (MMM-TEC V2.0) specifies a virtual space composed of processes is provided operating on an ICT platform and executing or requesting other processes to execute actions on items, i.e., MMM-TEC V2.0-specified data types. Processes may be rendered as avatars.

MMM-TEC does not use AIMs but only data types. It defines 27 actions and a language enabling processes to communicate using speech acts. Many are specified by MPAI-MMC, MPAI-OSD, and MPAI-PAF.

Example of actions are MM-Embed applied on an avatar or an object to place it somewhere in the metaverse, UM-Capture applied to media information in the physical world acquired for use in the metaverse, Identify applied to data captured to convert it to an item with an identifier, MM-Anim applied to an item, e.g., an avatar, to animate it, and MU-Render applied to an item in the metaverse to render it in the universe. A concise description of the operation of an implementation of MMM-TEC is available here.

As already said, both draw a sizeable part of their data types (and CAV-TEC of its AIMs) from three other standards: MPAI-OSD V1.3, MPAI-PAF V1.4, and MPAI-TFA V1.3. Some of these AIMs and data types are new in the mentioned versions of the three standards. They were developed in response to the CAV-TEC and MMM-TEC needs.

Why not develop them in CAV-TEC and MMM-TEC directly, then? Because the three standards address a specific area of standardisation that is also required by many other MPAI standards: objects and scenes, 3D graphics, and qualifiers. Therefore, MPAI-OSD V1.3, MPAI-PAF V1.4, and MPAI-TFA V1.3 are also published with a request for Community Comments.

Anybody is invited to send comments on any of the five standards to the MPAI Secretariat by April 13 at 23:59 UTC.

Geneva, Switzerland – 19^th March 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 54^th General Assembly (MPAI-54) with the release of the Connected Autonomous Vehicle and the MPAI Metaverse Model standards for Community Comments and the start of the Up-sampling Filter for Video Applications (EVC-UFV) V1.0 standard project.

MPAI has been working on the Connected Autonomous Vehicle (MPAI-CAV) project since its early days and released the Architecture Specification (CAV-TEC). Today, MPAI-54 released the Technology specification (CAV-TEC) V1.0 that builds on the Architecture Specification adding subsystems, components, and data types. The specification is released with a request for Community Comments and is available from the MPAI website. Anybody is invited to send comments on MMM-CAV V1.0 to the MPAI Secretariat by April 13 at 23:59 UTC.

MPAI has been working on the MPAI Metaverse Model (MPAI-MMM) project since January 2022. So far, two technical Reports and two Technical Specifications on Architecture and Technologies were published. Today, MPAI-54 released the Technology specification (MMM-TEC) V2.0 integrating the Architecture and Technologies specifications and the Reference Software published by MPAI-53. The specification is released with a request for Community Comments and is available from the MPAI website. Anybody is invited to send comments on MMM-TEC V2.0 to the MPAI Secretariat by April 13 at 23:59 UTC.

Both CAV-TEC V1.0 and MMM-TEC V2.0 reuse technology specifications that are shared with other MPAI standard, namely Object and Scene Description (MPAI-OSD) V1.3, Portable Avatar Format (MPAI-PAF) V1.4, and Data Types, Formats and Attributes (MPAI-TFA) V1.3. The specifications – available from the MPAI-OSD, MPAI-PAF, and MPAI-TFA web pages, respectively – were released with a request for Community Comments. Anybody is invited to send comments on any of the three standards to the MPAI Secretariat by April 13 at 23:59 UTC.

MPAI has identified the need for a standard that specifies an up-sampling filter for video applications with improved performance compared to the currently used up-sampling filters. MPAI-54 decided to kick off the new Up-sampling Filter for Video applications (EVC-UFV) V1.0 standard project based on the received responses to the Call for Technologies.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

Geneva, Switzerland – 19^th February 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 53^rd General Assembly (MPAI-53) releasing the first version of the MPAI Metaverse Model Open-Source Reference Software and kicking off the new project Compression and Understanding of Industrial Data (MPAI-CUI) – Company Performance Prediction (CUI-CPP) V2.0.

MPAI has been working on MPAI Metaverse Model (MPAI-MMM) standards since January 2022 and published Two technical Reports and two Technical Specifications  on Architecture and Technologies. The Reference Software released today implements a significant number of the MMM functionalities and uses a set of Unity instances to realise the different environments of the metaverse instance. You can find the MMM software at http://bit.ly/41J0wsj (REST API web server) and https://bit.ly/4ituw0R (Unity web server).

The Company Performance Prediction (CUI-CPP) project intends to provide a solution to a problem afflicting all companies: given the governance structure and the financial situation of a company and various types of risks that may affect it, what is the impact of the components of the governance structure on the governance, finance, and risk on the probability of default? MPAI has developed a set of functional requirements and a framework licence. A Call for Technologies was published and the standard will be collaboratively developed based on the responses to the Call.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

The 52^nd MPAI General Assembly (MPAI-52) has approved Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0. It is a Technical Report that provides a methodology to predict the game state of an online gaming server when some controller data is lost. The Prediction is obtained by applying Machine Learning algorithms based on historical data of the online game.

An online Multiplayer Game is based on a server. When this maintains consistency among all clients’ game instances it is called authoritative. It updates and broadcasts the game state using the controller data of all the clients. This function is harmed when controller data are not correctly received or are maliciously modified.

There are several techniques currently used to cure this situation. In Client Prediction, client game state is updated locally using predicted or interpolated data while waiting for the server data; in Time Delay, the server buffers the game state updates to synchronise all clients; and in Time Warp the server rolls back the game state to when controller data was sent by a client and acts as if the action was taken then, reconciling this new game state with the current game state.

These three methods have shortcomings. Client Prediction causes perceptible delay, Time Delay affects responsiveness, and Time Warp disadvantages other players because the new game state likely differs from the previous one.

Figure 1 depicts the arrangement proposed by SPG-MDL. The right-hand side represents the online game server with the Game State Engine tasked to produce the game state using all clients’ controller data and specialised engines. The Engines of the figure are:

1. Behaviour Engine, orchestrating actions from players and non-player entities.
2. Rules Engine, ensuring adherence to game mechanics.
3. Physics Engine, responsible for physical interactions within the game environment.

Figure 1 – Server Prediction

Whenever a client’s controller data is lost, the server requests the SPG-MDL module (the left-hand side of Figure 1) to compute the next game state. In this way, even if a client is experiencing network latency, the other clients maintain a continuous playing environment. The more accurate the predictions, the less noticeable the effect of the synchronisation process on the lagging client when the network resumes normal operations. A latency-affected client will still receive the results of its actions with a delay, but the server will send the new game state before receiving the action from the client, effectively halving the wait time. Of course, it is possible to further mitigate the effects of this problem by implementing additional client-side techniques, for example Client Prediction.

When some controller data is lost, the process begins with the last correct game state being fed into the SPG-MDL’s Game State Demultiplexer, which deconstructs it into discrete Game Messages* ( ). To differentiate the Game Messages inside the “twin” game server from the ones inside the game server, the ‘*’ symbol is attached to the SPG-MDL Game Messages. Each Game Message* is then processed by its respective Engine AI, leveraging a Neural Network Model to produce a predicted Game Message* ( ).  These predictions are assembled by the Predicted Game State Multiplexer into the predicted Game State ( ), which is then sent back to the game server for the next iteration of Game State computation.

While the SPG-MDL module operates, the server independently computes its updated Game State (GS_t+1) using the available Client Data. The server utilises the predicted Game State as follows. If any Client Data is missing, the server uses the predicted game state to compensate for the missing data shortfall from one or more clients. Note that the online game server architecture is a reference model and that the three engines are not a requirement for a specific game applying the MPAI-SPG methodology. For example, some games may not have a Physics Engine because physical-based behaviour is not required.

SPG-MDL provides a 10-step procedure to develop an SPG-MDL that is applicable to any authoritative game server.

The first 4 steps are required to outline the game setup to enable informed decisions for the implementation of SPG.

1. Select the game.
2. Define the Entities (to identify NN Model parameters):
   1. Environment
   2. Human-controlled players (HPC) and Non-player characters (NPC)
3. Define the Game State and relevant Entities.
4. Design the training dataset.
5. Collect the training dataset.
6. Train prediction NN Models defining viable architectures and training parameters and comparing the training results of different architectures.
7. Implement SPG-MDL.
8. Evaluate SPG-MDL to select the model yielding the best predictions.
9. Implement modules which simulate the disturbances.
10. Evaluate the SPG-MDL-enabled game experience with human players.

For each of the ten steps, the Technical Report provides:

1. High-level guidelines to outline the actions required.
2. An example of how the guidelines are implemented using a car racing game (Figure 2).

Figure 2 – Modular tiles and an example of a racetrack

The game was developed using the Unity game engine, and the networking features implemented through the open-source game networking library Mirror.

The following components are provided:

• The car racing game.
• Four different categories of Agent Players trained using Unity’s ML-Agents library.
• The dataset used for training generated by simulating game sessions played by the Agent Players.
• Jupyter Notebooks for training experiments and results.
• The trained models used by the AI-Behaviour Engine.

The software is available online. Please contact the MPAI secretariat to access the repository. Details on how to use the material are provided in the repository’s README page.

Geneva, Switzerland – 22^nd January 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 52^nd General Assembly (MPAI-52) approving publication of Technical Report: Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0.

Technical Report: Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0 addresses the effect of controller data latency in online multiplayer gaming. When controller data from a player does not reach the server on time, the server is unable to update and distribute a correct game state. The Technical Report provides guidelines on the design and use of Neural Networks that produce reliable and accurate predictions making up for the absence of players’ control data in multiplayer gaming contexts based on authoritative servers. An example Reference Software allows experimenters to test the suggested guidelines in a practical case.

MPAI will make an online presentation of the main results of the SPG-MDL V1.0 Technical Report on 12^th of February 2025 at 15 UTC. Register at https://encr.pw/CzO72 to attend.

See also the MPAI presentation to LA SIGGRAPH by Leonardo Chiariglione, Marina Bosi (Six Degrees of Freedom Audio), Andrea Bottino (MPAI Metaverse Model), Mark Seligman (Multimodal Conversation), and Ed Lantz (XR Venues)

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): waiting for responses on 11 February.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and Yoube.

The 51st MPAI General Assembly has decide to develop a new version V2.0 of Compression and Understanding of Financial Data (MPAI-CUI) – Company Performance Prediction (CUI-CPP) and issued a Call for Technologies to acquire relevant technologies. Register to attend online event where the Call will be presented on 2025/01/08 T15:00 UTC.

Compression and Understanding of Industrial Data (MPAI-CUI) was one of the first (2021) MPAI standards. The MPAI-CUI V1.0 Company Performance Prediction Use Case was based on the notion that the future of a company largely depends on structure, financial state, and risks it may face in the future. The solution used to address the complex task of creating a standard to predict the future of a company using such variables was successfully achieved was based on:

1. Governance Data, Financial Data, and Risk Assessment Data.
2. Conversion of Governance Data and Financial Data into Descriptors.
3. Conversion of Risk Assessment Data into a Risk Matrix.
4. Passing the Governance and Financial Descriptors to an Organisation Assessment and Default Prediction neural network.
5. Perturbing the Default Probability from the neural network with the Risk Matrix to obtain the Discontinuity Prediction.

Governance Descriptors used as input to the neural network were: #Stakeholder Individuals, #Stakeholder Companies, Shareholder Share, Shareholders Gender, Decision-Makers Gender, #Decision-Makers, Members of the Revision And Advisory Board, Presence Of The Advisory Company, #Decision-Makers By The Same Family, Company Phase (Age).

Financial Descriptors used as input to the neural network were: Revenues, EBITDA Margin, EBITDA, Quick Ratio, Current Ratio, Net Working Capital, Net Financial Position, Net Short-Term Assets, Shareholder Funds-Fixed Assets, Long-Term Liability Ratio, Coverage Of Fixed Assets, Amortisation Rate, Debt On Sales, Interest Coverage Ratio, Average Stock Turnover, Stock Coverage Days, Return On Investments (ROI), Return On Assets (ROA), Return On Sales (ROS), Return On Equity (ROE), Cash Flow, Interest On Sales, Type Of Financial Statement.

Risk Matrix included the following characteristics: Occurrence (3 values), Business Impact (3 values), Gravity (5 values), Risk retention (portion of the risk that the Company decides to retain).

The Governance and Financial Descriptors together with the Prediction Horizon fed to Neural Network provided an Organisational Model Index and a Default Probability. Default Probability and Risk Matrix fed to a Prediction Result Perturbation AIM which perturbed the Default Probability and produced the Business Discontinuity Probability.

Figure 1 depicts the AI Workflow that performs as described above.

Figure 1 – The Company Performance Prediction AI Workflow of MPAI-CUI V1.1

At the 51^st General Assembly (MPAI-51) MPAI issued a Call for Technologies for substantially more ambitious goal for MPAI-CUI V2.0 because it Company performance prediction targets:

• A more precise identification of Cyber, Digitisation, Climate, and Business risks.
• A definition of risks to be organised as follows:
  • Risk name, Risk type: (cyber, etc.),
  • Target regulation,
  • Vector of inputs including, e.g. for Cyber Risks:
    • Name of input: IP address, Denial of service;
    • Time: time the attack was detected;
    • Source: provider of input vector; Type: image, text, category, etc.;
    • Value: depends on type.

With this additional information, the task is to define a (set of) neural network(s) that receive(s) Risk Data in addition to Governance and Financial Descriptors. The Assessment and Prediction network of Figure 2 are not only numbers but Descriptors that include index or probability but also information on which input elements have more influence on index and probabilities.

Figure 2 – The Company Performance Prediction AI Workflow of MPAI-CUI V1.1

Risk Assessment and Risk Matrix are used when sufficient data are not available to train the Neural Network or when the Neural Network may not be used because it does not comply with relevant regulations.

The Call for Technologies is directed to all parties having rights to technologies satisfying the Use Cases and Functional Requirements and the Framework Licence of the planned Technical Specification MPAI-CUI V2.0 are invited to respond to the Call for Technologies, preferably using the Template for Responses. Submissions received by 2024/02/11 will be assessed and considered for use in the development of said MPAI-CUI Technical Specification.

Geneva, Switzerland – 18^th December 2024. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 51^st General Assembly (MPAI-51) approving publication of the Compression and Understanding of Industrial data (MPAI-CUI) V2.0 Call for Technologies.

Call for Technologies: Compression and Understanding of Industrial data (MPAI-CUI) V2.0 invites any party able and wishing to contribute to the development of the planned MPAI-CUI V2.0 Technical Specification to submit a response by 11^th of February 2025. The new standard will extend the current company organisation index and default/discontinuity probabilities with descriptors and information on the compliance of the Machine Learning Models used.

MPAI-51 also approved as MPAI standards:

1. Neural Network Traceability (MPAI-NNT) V1.0 to evaluate the ability to trace back to its source a neural network that has been modified, the computational cost of injecting, extracting, detecting, decoding, or matching data from a neural network, and the impact on the performance of a neural network with inserted traceability data and its inference.
2. Human and Machine Communication (MPAI-HMC) V2.0 that enables advanced forms of communication between humans in a real space or represented in a Virtual Space, and Machines represented as humanoids in a Virtual Space or rendered as humanoids in a real space.
3. Context-based Audio Enhancement (MPAI-CAE) V2.3, Multimodal Conversation (MPAI MMC) V2.3, Object and Scene Descriptors (MPAI-OSD) V1.2, and: Portable Avatar Format (MPAI-PAF) V1.3.

The MPAI-CUI V2.0 Call for Technologies will be presented online on 8^th of January 2025 at 15 UTC. Register at https://tinyurl.com/4vdps8f3 to attend.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): waiting for responses on 11 February.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and Yoube.

MPAI-50 has published the Call for Technologies: Up-sampling Filters for Video applications (MPAI-UFV) V1.0 requesting parties having rights to technologies satisfying the Use Cases and Functional Requirements and the Framework Licence of the planned Technical Specification: Up-sampling Filter for Video Applications (MPAI-UFV) V1.0 to respond to this Call for Technologies preferably using the Template for responses.

The goal of MPAI-UFV V1.0 is to develop a standard up-sampling filter that provides optimal performance when applied to a video to generate a video with a higher number of lines and pixels.

The submissions received will be assessed, collaboratively improved, if found suitable, and used in the development of the planned MPAI-UFV Technical Specification.

MPAI membership is not a prerequisite for responding to this Call for Technologies. However, if a (part of a) submission that is accepted for inclusion in the planned MPAI-UFV TS, the Proponent must join MPAI, or lose the opportunity to have the accepted technologies included in the TS.

MPAI will select the most suitable technologies based on their technical merits. However, MPAI is not obligated to select a particular technology or any of the proposed technologies if those submitted are found to be inadequate.

Proponents shall mandatorily:

1. Submit a complete description of the proposed up-sampling filter with a level of detail allowing an expert in the field to develop and implement the proposed filter.
2. Upload the following software to the MPAI storage:
   1. Docker image that contains the encoding and decoding environments, encoder, decoder, and bitstreams.
   2. Up-sampling filter in source code (preferred) or executable form. Note that proponents of accepted proposals will be requested to provide the source code of the up-sampling filter.
   3. Python scripts to enable testers to carry out the Performance Test.
3. Submit the following results:
   1. Tables of objective quality results obtained by the submitter with their proposed solution.
   2. The decoded Test Sequences.
   3. The up-sampling results for SD to HD and HD to 4K obtained with the proposed solution.
   4. The VMAF-BD Rate assessment provided with a graph for each QP and a table with minimum, maximum and average value for each sequence.
   5. A Complexity assessment using MAC/pixel and number of parameters of the submitted up-sampling filter. Use of SADL (6) is recommended.

Submissions will be evaluated by an Evaluation Team created from:

1. MPAI Member representatives in attendance.
2. Non-MPAI Member representatives who are respondents to any of the received submissions.
3. Non respondent experts/non MPAI Member representatives invited in a consulting capacity.
4. No one from 1. and 2. will be denied membership in the Evaluation Panel if they request it.

Submissions will be evaluated by an Evaluation Team created from:

1. MPAI Member representatives in attendance.
2. Non-MPAI Member representatives who are respondents to any of the received submissions.
3. Non respondent experts/non MPAI Member representatives invited in a consulting capacity.
4. No one from 1. and 2. will be denied membership in the Evaluation Panel if they request it.

Proposals will be assessed using the following process:

1. The objectively computed Quality Tests will use the Test Sequences provided to MPAI by an independent academic after the proposal deadline and distributed to Respondents.
2. Each Respondent presents their proposal.
3. Evaluation Team members ask questions.
4. Evaluation Team organises the Tests.
5. A volunteer member of the Evaluation Team executes the docker image of a Respondent and computes the values obtained using the test set provided by the respected academic.
6. The Objective Quality Evaluation will use the VMAF-BD Rate metrics to compare the sequences obtained by up-sampling with the bicubic filter the AVC, HEVC and VVC co-decoded with QP Values: 22, 27, 32, 37, 42 with the same co-decoded sequences up-sampled with the proposed up-sampling algorithm.
7. Latency is the number of frames used by the up-sampling process. Note that the actual number of frames used to produce the response should be specified.
8. The Complexity evaluation will use MAC/pixel and number of parameters.

The timeline of proposal submission is:

Those intending to propose should become familiar with the four documents mentioned above and posted here. An online presentation will be made on 2024/11/27 at 14UTC. Register at https://bit.ly/3V0sdt1 to attend.