Abstract

In five years, MPAI – Moving Picture, Audio, and Data Coding by Artificial Intelligence, the international, unaffiliated, not-for-profit association developing standards for AI-based data coding – has been carrying out its mission, making its results widely known and its status recognised. However, not all are clear why MPAI was established, which are its distinctive features, and why it stands out among the organisations developing standards.

This is the first of a series of articles that will revisit the driving force that allowed MPAI to develop 15 standards in a variety of fields where AI can make the difference and have half a score under development. The articles will pave the way for the next five years of MPAI.

1          Standards have a divine nature

MPAI was established as a Swiss not-for-profit organisation on 30 September 2020 with the mission of developing data coding standards based on Artificial Intelligence. The first element that has driven MPAI into existence is the very word STANDARD. If you ask people “what is a standard?” you are likely to obtain a wide range of responses. The simplest, most effective, and although rather obscure answer is instead “a tool that connects minds”. A standard establishes a paradigm that allows a mind to interpret what another mind expresses in words or by other means of communication.

The easiest example of standard is language itself. A language assigns labels – words – to physical and intellectual objects. Apparently, the language-defined labels are what make humans different from animals, even though there are innumerable forms of language used by animals that are less rich than the human one.

Words have a divine nature. So, it is no surprise that St. John’s Gospel starts with the sentence “In the beginning was the Word, and the Word was with God, and God was the Word”.

Is this divine nature only applicable to language? Well, no. If we say that 20 mm of rain fell in 4 hours, we are using the language to convey information about rain that would be void of a quantitative value if there was not a standard for length called meter and a standard for time called hour.

Next to these lofty goals, standards have other practical purposes, A technical standard enacted by a country may be used as a tool to limit and sometimes even outlaw products not conforming to that standard, in that country. The existence of standards used for this purpose was the main driver toward the establishment of the Moving Picture Experts Group (MPEG): a single digital video standard that eventually uprooted scores of analogue television standards, with myriads of sub-standards. The intention was to nullify this malignant use of standards.

The goal of MPEG was to serve humans by enabling them to hear and see, thanks to machines that were able to “understand” the bits generated by other machines. In other words, it was necessary that machines be made able to understand the “words” of other machines.

Come 2020, Artificial Intelligence was not yet in the headlines, but the direction was clear. AI systems would be endowed with more and more “intelligence”, communicate between each other, and let humans communicate with them. What forms the “word” would take were not clear, but that the forms would be manifold was.

While MPEG had ushered in a new spirit in standardisation, it had also fostered a transformation in the way the need for standards would take shape and how standards would be exploited. In the early MPEG days, most industries still had research laboratories where the advancement of technology was monitored and fostered with a view of exploiting technology for new products and services. New findings would be patented and new patent-based products launched. The market would bless the winner among different products doing more or less the same thing with different technologies.

MPEG rendered useless the costly step of battling on patents in the market bringing to the standards committee a battle on standards. As a patent in a successful standard was highly remunerative, it did not take much time for the industry to invest in any patentable research. By industry, we intend “any” industry, actually not even industries that had a business in products and services. The idea was that a day would come that a patent would be needed in “a” standard. That single patent would repay the tens-hundreds-thousands of unused patents.

This was significant progress in terms of optimising efforts to generate innovation, and letting more actors join the fray. That progress, however, came at a cost, the disconnect of exploitation from generating innovation. An industry that had made an innovation to achieve an investment had every interest to see that innovation deployed. A company that has a portfolio of 10,000 patents may decide not to license a patent (in practice, by dragging its feet in releasing a patent) if that helps it license a more remunerative patent instead.

Certainly, this epochal evolution has resulted in a more efficient generation of innovation but is actually hindering exploitation of valuable innovations.

2 Inside MPAI

MPAI has been established to uphold the social value of standards, to adapt the notion of standard to the new Artificial Intelligence domain, and to rescue the right of society to exploit innovation. Let’s review how this has been implemented.

Laws pay much attention to standards because the patents that enable them give monopoly rights to exploit the patents to holders for a significant time span. This is the reason why a standards body requests that a proposal of a standard be accompanied by a declaration that, in simplified form, boils down to answering one of the three possibilities: if there are patent rights to the enabling technologies, does the submitter agree to allow the use of the patents for free, or are royalties applied, or is the use of patents not allowed? Setting aside the 1^st and 3^rd cases, in the second case the submitter is asked to declare that it will license the patents “to an unrestricted number of applicants on a worldwide, non-discriminatory basis and on reasonable terms and conditions to make, use and sell implementations of the above document” (i.e., the standard).

In past ages this used to be good enough, but is this still OK in an age when technology moves fast? Is it OK if it takes ten years for the declaration to be implemented and if, ten years later, a user discovers that the licensing terns are not acceptable?

A reasonable answer is that, no, this handling of patents in standards is not acceptable because society is deprived of technologies that may bring significant benefits to it. It may also not be acceptable to some patent holders because they are deprived of their ability to monetise the fruits of their efforts.

The MPAI handling of patents is exactly the opposite on this point. MPAI Members engaged in the development of a standard agree on a set of statements called “Framework Licence” that remove some uncertainty on the time and terms of the eventual licence.

Why only “some” and not all uncertainty? Because there are so-called antitrust laws that forbid this full clarification as this could be maliciously used to distort the market. As Monsieur de Montesquieu used to say: “Better is the enemy of good”, MPAI wants to make things that are good and not perfect things that create problems.

MPAI has developed a model process to develop standards which is based on a combination of openness – when the purpose of a standard is developed – and restriction – when the technologies making up the standard are developed. Figure 1 depicts the eight steps of a standard.

Figure 1 – The MPAI standard life cycle

A standard may be proposed by anybody (Stage 0). To define the scope of the standard, use cases are developed (Stage 1). To define what the standard should exactly do, Functional Requirements are developed (Stage 2). Participation in these stages is open to anybody.

To define the conditions of use of the standard, Commercial Requirements are developed (Stage 3). Of course, MPAI is not engaged in any sort of commerce. “Commercial Requirements” are what above have been called Framework Licence. When the two types of requirements are ready, a Call for Technologies is issued (Stage 4).

Anybody can answer the call (Stage 5), but submitted technologies must satisfy the requirements. Proposed technologies are used to develop the exclusive standard with the participation of MPAI Members and the respondents to the call if they have joined MPAI. If they do not, their submission is discarded.

When the standard has reached a stability plateau, MPAI may publish the current draft before final publication (Stage 6). Anybody is entitled to send their comments to the MPAI Secretariat. Comments will be considered for inclusion in the published standard (Stage 7).

Stage 7 is divided in four Steps. The first Step concerns the publication of the standard, but a standard usually includes other Steps. Step #2 is the reference software, a conforming implementation of the standard that is included in the standard itself with reference to the MPAI Git from where the software – published with a BSD 3 Clause licence – can be downloaded. The third Step is Conformance Testing. An implementation of an AI Workflow conforms with MPAI-MMC if it accepts as input and produces as output Data and/or Data Objects (the combination of Data of a Data Type and its Qualifier) conforming with those specified by MPAI-MMC (see later for more about Qualifiers).

The last Step is Performance Assessment. Performance is a multidimensional entity because it can have various connotations, and the Performance Assessment Specification should provide methods to measure how well an AIW performs its function, using a metric that depends on the nature of the function, such as:

1. Quality: the Performance of an AI System that Answers Questions can measure how well it answers a question related to an image.
2. Bias: Performance of the same AI System can measure the quality of responses depending on the type of images, i.e., the ability of the System to provide balanced unbiased answers.
3. Legalcompliance: the Performance of an AIW can measure the compliance of the AI System to a regulation, e.g., the European AI Act.
4. Ethicalcompliance: the Performance Assessment of an AI System can measure its compliance to a target ethical standard.

Geneva, Switzerland – 14^th May 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 56^th General Assembly (MPAI-56) with the publication of the Technical Report: AIW and AIM Implementation Guidelines (MPAI-WMG) for Community Comments.

A significant share of MPAI standards rely on AI Framework (MPAI-AIF) that supports componentisation of processing elements called AI Modules (AIM) organised in an AI Framework. As AIMs are agnostic of the technologies used in their implementation – traditional data processing or various types of AI technologies, the AIW and AIM implementation Guidelines Technical Report includes an analysis of a significant number of already specified AIMs from the implementation perspective. MPAI-WMG also includes an analysis of the use of the Perceptive and Agentive AI (PAAI) Model that may provide a new basis for new forms of AI Frameworks.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Autonomous Vehicle (MPAI-CAV): investigating extensions of the current CAV-TEC standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): developing the Up-sampling Filter for Video applications (EVC-UFV) standard.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MMM-TEC specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

1. Introduction

The Connected Autonomous Vehicle (CAV) specified by CAV-TEC is a system that instructs a vehicle with at least three wheels to reach a Destination from a current Pose at the request of a human or a process. It does so by exploiting information captured and processed by the vehicle and communicated by other CAVs while respecting the local traffic law, Figure 1 represents an example of the type of environment that a CAV is requested to traverse and Figure 2 depicts the four subsystems of which a CAV is composed, although this partitioning is not a functional requirement as components of a subsystem may be located in another subsystem, provided the interfaces specified by CAV-TEC are preserved.

Figure 1 – An example of an environment traversed by a CAV

In Figure 2, a human approaches a CAV and requests the Human-CAV Interaction Subsystem (HCI) to be taken to a destination using a combination of four media – Text, Speech, Face, and Gesture. Alternatively, a remote process may make a similar request to the CAV.

Figure 2 – The subsystems of a CAV

Either request is passed to the Autonomous Motion Subsystem (AMS), which requests the Environment Sensing Subsystem (ESS) to provide the current CAV Pose. With current Pose information, the Destination, the AMS can propose one or more Routes by accessing Offline Maps. Eventually the human or process can select a Route.

With the human aboard, the AMS continues to receive environment information from the ESS – possibly complemented by information received from other CAVs in range – and instructs the Motion Actuation Subsystem to make appropriate motions.

2. Human-CAV Interaction

The CAV-HCI Reference Model of Figure 3 explains the operation of the HCI in its interaction with humans.

Figure 3 – Reference Model of CAV-HCI

The Audio-Visual Scene Description (AVS) monitors the environment and produces Audio-Visual Scene Descriptors from which it extracts Speech Scene Descriptors and from these, Speech Objects corresponding to any speaking humans in the environment surrounding the CAV. Visual Scene Descriptors may also be extracted in the form of Face and Body Descriptors of all humans present.

The CAV activates Automatic Speech Recognition (ASR) to have the speech of each human recognised and converted into Recognised Text. Each Speech Object is identified according to their position in space. The CAV also activates the Visual Object Identification (VOI) that is able to produce the Instance IDs of Visual Objects as indicated by humans.

Natural Language Understanding (NLU) processes the Speech Objects, produces Refined Text, and extracts Meaning from the Text of each input Speech. This process is facilitated by the use of the IDs of the Visual Objects provided by VOI.

Speaker Identity Recognition (SIR) and Face Identity Recognition (FIR) help the CAV to reliably obtain the Identifiers of the the humans the HCI is interacting with. If the Face ID(s) provided by FIR correspond to the ID(s) provided by SIR, the CAV may proceed to attend to further requests. Especially with humans aboard, Personal Status Extraction (PSE) provides useful information regarding the humans’ state of mind by extracting their Personal Status.

The CAV interacts with humans through Entity Dialogue Processing (EDP). When a human requests to be taken to a Destination, the EDP interprets and communicates the request to the Autonomous Motion Subsystem (AMS). A dialogue may then ensue where the AMS may offer different choices to satisfy potentially different human needs (e.g., a long but comfortable Route or short but less predictable).

Then, while the CAV moves to the Destination, the HCI may have a conversation with the humans, show the Full Environment Descriptors developed by the AMS to the passengers, and may communicate information about the CAV from the Ego AMS or more generally from the HCIs of remote CAVs.

The HCI responds using the two main outputs of the EDP: Text and Personal Status. These are used by the Personal Status Display (PSD) to produce the Portable Avatar of the HCI conveying Speech, Face, and Gesture synthesised to render the HCI Text and Personal Status. Audio-Visual Scene Rendering (AVR) renders Audio, Speech, and Visual information using the HCI Portable Avatar. Alternatively, it can display the AMS’s Full Environment Descriptors from the Point of View selected by the human.

The HCI interacts with passengers in several ways:

1. By responding to commands/queries from one or more humans at the same time, e.g.:
   1. Commands to go to a waypoint, park at a place, etc.
   2. Commands with an effect in the cabin, e.g., turn off air conditioning, turn on the radio, call a person, open a window or door, search for information, etc.
2. By conversing with and responding to questions from one or more humans at the same time about travel-related issues, e.g.:
   1. Humans request information, e.g., time to destination, route conditions, weather at destination, etc.
   2. Humans ask questions about objects in the cabin.
3. By following the conversation on travel matters held by humans in the cabin if
   1. The passengers allow the HCI to do so, and
   2. The processing is carried out privately inside the CAV.

3. Environment Sensing Subsystem

The operation of the Environment Sensing Subsystem (ESS) is best explained using the Reference Model of the CAV-ESS subsystem depicted in Figure 4.

Figure 4 – Reference Model of CAV-ESS

When the CAV is activated in response to a request by a human owner or renter or by a process, Spatial Attitude Generation continuously computes the CAV’s Spatial Attitude relying on the initial Motion Actuation Subsystem’s Spatial Attitude, and information from the Global Navigation Satellite Systems (GNSS), if available.

An ESS may be equipped with a variety of Environment Sensing Technologies (EST). CAV-TEC assumes they are (but not required to all be supported by an ESS implementation) Audio, LiDAR, RADAR, Ultrasound, and Visual. Offline Map is considered as an EST.

An EST-specific Scene Description receives EST-specific Data Objects, produces EST specific Scene Descriptors which are integrated into the Basic Environment Descriptors (BED) by the Basic Environment Description using all available sensing technologies, Weather Data, Road State, and possibly the Full Environment Descriptors of previous instants provided by the AMS. Note that, although in Figure 4 each sensing technology is processed by an individual EST, an implementation may combine two or more Scene Description AIMs to handle two or more ESTs, provided the relevant interfaces are preserved. An EST-specific Scene Description may need to access the BED of previous instants and may produce Alerts that are immediately communicated to AMS.

The Objects in the BEDs may carry Annotations specifically related to traffic signalling, e.g.: Position and Orientation of traffic signals in the environment, Traffic Policemen, Road signs (lanes, turn right/left on the road, one way, stop signs, words painted on the road), Traffic signs – vertical signalisation (signs above the road, signs on objects, poles with signs), Traffic lights, Walkways, and Traffic sounds (siren, whistle, horn).

4. Autonomous Motion Subsystem

The Reference Model of the CAV-AMS subsystem depicted in Figure 5 explains the operation of the Autonomous Motion Subsystem (AMS) is.

Figure 5 – Reference Model of CAV-AMS

When the HCI sends the AMS a human or process request to move the CAV to a Destination, Route Planning uses the Basic Scene Descriptors from the ESS and produces a set of Waypoints starting from the current Pose up to the Destination.

When the CAV is in motion, Route Planning causes Path Selection Planning to generate a set of Poses to reach the next Waypoint. Full Environment Description may request the AMSs of Remote CAVs to send (subsets of) their Scene Descriptors and integrates all sources of Environment Descriptors into its Full Environment Descriptors (FED), and may also respond to similar requests from Remote CAVs.

Motion Selection Planning generates a Trajectory to reach the next Pose in each Path. Traffic Obstacle Avoidance receives the Trajectory and checks if any Alert was received that would cause a collision with the current Trajectory. If a potential collision is detected, Traffic Obstacle Avoidance requests a new Trajectory from Motion Planner, otherwise Traffic Obstacle Avoidance issues an AMS-MAS Message to Motion Actuation Subsystem (MAS).

The MAS sends an AMS-MAS Message to AMS informing it about the execution of the AMS-MAS Message received. The AMS, based on the received AMS-MAS Messages, may discontinue the execution of the earlier AMS-MAS Message, issue a new AMS-MAS Message, and inform Traffic Obstacle Avoidance. The decision of each element of the chain may be recorded in the AMS Memory (“black box”).

5. Motion Actuation Subsystem

The operation of the (MAS) is explained using the Reference Model of the CAV-MAS subsystem depicted in Figure 6.

Figure 6 – Reference Model of CAV-AMS

When the AMS Message Interpretation receives the AMS-MAS Message from the AMS, it interprets the Messages, partitions it into commands, and sends them to the Brake, Motor, and Wheel mechanical subsystems. CAV-TEC is silent on how the three mechanical subsystems process the commands but specifies the format of the commands issued to AMS Message Interpretation. The result of the interpretation is sent as an AMS-MAS Message to AMS.

MAS includes two more AIMs. Spatial Attitude Generation computes the initial Ego CAV’s Spatial Attitude using the Spatial Data provided by Odometer, Speedometer, Accelerometer, and Inclinometer. This initial Spatial Attitude is sent to the ESS to integrate its GSNN-based Spatial Attitude. Ice Condition Analysis augments the Weather Data by analysing the Brake, Motor, and Wheel mechanical subsystems’ responses and sends the augmented Weather Data to the ESS.

6. Conclusions

CAVs promise to bring benefits that will positively affect industry, society, and the environment, for example:

1. Saving lives and reducing injuries by removing human error thanks to a machine less prone to errors.
2. Giving humans more time for rewarding activities, such as interpersonal communication.
3. Optimising the use of vehicles and infrastructure.
4. Reducing congestion and pollution.
5. Supporting elderly and disabled people.

Therefore, society and individuals will be positively impacted by the transformation of today’s “niche market” into tomorrow’s vibrant “mass market” of Connected Autonomous Vehicles. A market of interchangeable components conforming with the CAV-TEC V2.0 specifications can offer affordable and safe Connected Autonomous Vehicles sooner and more efficiently than waiting for market forces to produce “monolithic” cars with progressively higher SAE Levels.

The CAV-TEC Reference Model can create a competitive market offering increasingly performing components until a new, more powerful reference model will eventually replace the model with another, initiating a new sequence of performance improvements. The Reference Model will help:

1. Researchers to optimise component technologies.
2. Component manufacturers to bring their standard-conforming components to market once they are mature.
3. Car manufacturers to access an open global market of interchangeable components.
4. Regulators to oversee conformance testing of components following standard procedures.
5. Users to rely on Connected Autonomous Vehicles whose operation they can explain to a large extent.

An avatar is generally intended as a representation of a real or fictitious human in a virtual space. Research has dedicated much effort to creating and animating realistic avatars. However, the scope of use is typically assumed to be a closed environment such as a proprietary video game. Therefore, the portability of avatars has seldom been a priority.

Some 30 years ago, the Humanoid Animation (H-Anim) standard was developed that defined a human skeleton composed of joints, segments, and sites exhibiting four levels of articulation and the default skeleton pose. The latest versions of the H-Anim standards are ISO/IEC 19774-1:2019, ISO/IEC 19774-2:2019).

An avatar is a basic but not the only element in an application. What if you want to convey to a third party a speaking avatar immersed in an environment with all its features so that it is displayed as you intended?

Technical Specification: Portable Avatar Format (MPAI-PAF), whose Version 1.4 has recently been approved offers a solution to this problem for a broad range of applications called Portable Avatar. Personal Status is a package of data conveying the following information:

1. The ID of the virtual space (M-Instance) where the Portable Avatar is to be placed.
2. The space and time information of the “environment” to be placed in the M-Instance.
3. The Audio-Visual Scene representing the “environment”.
4. The space and time information of the Avatar in the scene.
5. The Avatar represented as a 3D Model, its Face Descriptors and Body Descriptors.
6. The Language Preference of the Avatar.
7. The Text Object the Avatar is associated with, or which will be converted into a Speech Object.
8. The Speech Model used to synthesise the Text Object.
9. The Speech Object alternative to the Text Object that the Avatar utters.
10. The Personal Status of the Avatar.

Here is a brief description of the Portable Avatar components.

1. The ID of the virtual space (M-Instance). This is the ID of a virtual space where the Portable Avatar is to be placed. It can be a metaverse (for MPAI, this would be an M-Instance of MMM-TEC).
2. The space and time information of the “environment”. MPAI has defined a data type called Space-Time that defines:
   1. The space information as Spatial Attitude, Position, Orientation, and optionally their velocities and acceleration.
   2. Time defined as either absolute (from 1970/01/01T00:00 or from an arbitrary origin of time.
3. The Audio-Visual Scene. MPAI has defined Scene as a data type that describes a scene as composed of scenes and objects with their Space-Time information. The MPAI scene is thus hierarchical (see Audio-Visual Scene Descriptors).
4. The space and time information of the Avatar considered a particular type of object in the scene. The Avatar data type has its own space-time information. This is overridden by the scene time information, if different.
5. The Avatar. The MPAI Avatar data type is a structure that includes:
   1. The space-time information (that is overridden by the space-time information of the scene).
   2. The 3D Model Object composed of data and Qualifier giving additional information to the data, e.g., the format.
   3. The Face Descriptors. MPAI has adopted the Actions Units of the Facial Action Coding System (FACS).
   4. The Body Descriptors. MPAI has adopted the H-Anim standard, but the 3D Model Qualifier allows the use of other standards.
6. The Language Preference. MPAI supports the signalling of a large number of language codes.
7. The Text Object. An avatar may have a textual description represented by a Text Object (text data and Qualifier providing various types of information on the text, e.g., language and character code).
8. The Text Object may be used to synthesise a Speech Object (speech data and Qualifier). The Portable Avatar can convey a neural network speech model to synthesise the text. In MPAI, a neural network model (more generally, a Machine Learning Model) has an associated Qualifier providing various types of information such as the conformity of the model to a particular regulation.
9. The Speech Object. In some cases, the speech associated with the avatar is conveyed by the Portable Avatar. Same as for the Text Object, a Speech Object includes speech data and a Qualifier that may be used to provide information on the language, compression format, speaker identity etc.
10. The Personal Status. The Personal Status is an MPAI data type including information on the Cognitive State, Emotion, and Social Attitude of the Text, Speech, Face, and Gesture of an Entity (in MPAI Entity is used to indicate either a human or the process animating an avatar).
    1. Cognitive State represents the internal state of an Entity that has knowledge of the context such as “surprised” or “interested”.
    2. Emotion represents the internal state of an Entity such as that resulting from its interaction with the context, such as “angry” or “sad”.
    3. Social Attitude represents the internal state of an Entity related to the way it intends to position itself vis-à-vis the context, e.g., “respectful” or “soothing”.

The Portable Avatar is an essential component of a variety of use cases. It is typically used as input to the Audio-Visual Scene Rendering AI Module that produces Speech, Audio, and Visual Objects from Portable Avatar, Audio-Visual Scene Descriptors (in case one is not available in the Portable Avatar), and a Point of View as depicted in Figure 1.

Figure 1 – Audio-Visual Scene Rendering AI Module

The Personal Status Display (PAF-PSD) AIM produces a Portable Avatar corresponding to an Avatar Model uttering a Speech Object synthesised from a Text Object with a Speech Model and displaying a Personal Status:

Figure 2 – Personal Status Display

Here, the input is a Text Object, a Neural Network Speech Model (in case the PAF-PSD does not have one embedded), an Avatar Model, and a Personal Status:

1. The Text is used to synthesise speech modulated with the Speech component of the input Personal Status.
2. The Speech, input Text and Face component of the input Personal Status, and the input Avatar Model are used to synthesise a face;
3. The Text, the Gesture component of the input Personal Status, and the input Avatar Model are used to synthesise the body.

In MPAI, the Lego approach to avatar deployment in applications is a reality.

The MPAI Metaverse Model – Technologies standard – in short, MMM-TEC V2.0 – is the first open metaverse standard enabling independently designed and implemented defines a metaverse instance (that MMM-TEC calls M-Instances) and clients to interoperate. These are the main MMM-TEC elements:

• The Architecture is based on Processes acting on Items based on the Rights they hold.
• Items represent any abstract and concrete objects in an M-Instance.
• Processes – possibly in different M-Instances – communicate using the Inter-Process Protocol (IPP).
• Process Actions represent the payload of a message sent by a Process.
• Qualifiers are containers of technology-specific information of an Item.
• The MPAI-MMM API enables fast development of M-Instances.
• Verification Use Cases verify the completeness of the standard.
• The MMM Open-Source Software implementation can easily be installed.

Below is a more extended introduction to MMM-TEC. Here is the text of Technical Specification: MPAI Metaverse Model (MPAI-MMM) – Technologies (MMM-TEC) V2.0.

MMM-TEC  defines an M-Instance as an Information and Communication Technologies platform populated by Processes. They perform a range of activities, such as operate with various degrees of autonomy and interactivity, sense data from the real world, produce various types of entities called Items, perform or request other Processes to perform activities represented by Process Actions or request other Processes – possibly in other M-Instances, hold or acquire Rights on Items, and act on the real world on a variety of ways. They can perform Process Actions based on Rights they may hold, acquire, or be granted.

Processes may be characterised as:

1. Services providing specific functionalities, such as content authoring.
2. Devices connecting the Universe to the M-Instance and the M-Instance to the Universe.
3. Apps running on Devices.
4. Users representing and acting on behalf of human entities residing in the Universe. A User is rendered as a Persona, i.e., an avatar.

Figure 1 depicts the main elements on which the MMM-TEC Specification is based: human, Devices, Apps, Users, Services, and Personae.

Figure 1 – Main elements of an M-Instance

Processes Sense Data from U-Environments, i.e., portions of the Universe and may produce three types of Items, i.e., Data that has been Identified in – and thus recognised by – the M-Instance:

1. Digitised– i.e., sensed from the Universe – possibly animated by activities in the Universe.
2. Virtual– i.e., imported from the Universe as Data or internally generated – possibly autonomous or driven by activities in the Universe.
3. Mixed – Digitised and Virtual.

Processes Perform – either on their initiative, or driven by the actions of humans or machines in the Universe – Process Actions that combine:

1. An Action, possibly prepended by:
   1. MM:to indicate Actions performed inside the M-Instance, e.g., MM-Animate using a stream to animate a 3D Model with a Spatial Attitude (defined as Position, Orientation, and their velocities and accelerations).
   2. MU:to indicate Actions in the M-Instance influencing the Universe, e.g., MU-Actuate to render one of its Items to a U-Location as Media with a Spatial Attitude.
   3. UM: to indicate Actions in the Universe influencing the M-Instance, e.g., UM-Embed to place an Item produced by Identifying a scene, UM-Captured at a U-Location, at an M-Location with a Spatial Attitude.
2. Items on which the Action is performed or are required for performance, such as Asset, 3D Model, Audio Object, Audio-Visual Scene, etc.
3. M-Locations and/or U-Locationswhere the Process Action is performed.
4. Processes with which the Action is performed.
5. Time(s) during which the Process Action is requested to be and is performed.

Processes may hold Rights on an Item, i.e., they may perform the set of Process Actions listed in their Rights. An Item may include Rights signalling which Processes may perform Process Actions on it. Processes affect U-Environments and/or M-Instances using Items in ways that are Consistent with the goals of the M-Instance as expressed by the Rules, within the M-Capabilities of the M-Instance, e.g., to support Transactions, and respecting applicable laws and regulations.

Processes perform activities strictly inside the M-Instance or have various degrees of interaction with Data sensed from and/or actuated in the Universe.

Processes may request other Processes to perform Process Actions on their behalf by using the Inter-Process Protocol, possibly after Transacting a Value (i.e., an Amount in a Currency) to a Wallet.

An M-Instance is managed by an M-Instance Manager. At the initial time, the M-Instance Manager has Rights covering the M-Instance and may decide to define certain subsets inside the M-Instance – called M-Environments – on which it has Rights and attach Rights to them.

A Registering human may:

1. Request to Register to open an Account of a certain class.
2. Be requested to provide their Personal Profile and possibly to perform a Transaction to open an Account.
3. Obtain in exchange a set of Rights that their Processes may perform. Rights have Levels indicating that the Rights are
   1. Internal, g., assigned by the M-Instance at Registration time according to the M-Instance Rules and the Account type.
   2. Acquired, g., obtained by initiative of the Process.
   3. Granted to the Process by another Process.

MMM-TEC V2.0 does not specify how an M-Instance verifies that the Process Actions performed by a Process comply with the Process’s Rights or the M-Instance Rules.  An M-Instance may decide to verify the full set of Activity Data (the log of performed Process Actions), or to make verifications based on claims by another Process, to make random verifications, or to not make any verification at all. Therefore, MMM-TEC V2.0 does not specify how a M-Instance Manager can sanction non-complying Processes.

In some cases, an M-Instance could be wastefully too costly as an undertaking if all the technologies required by the MMM Technical Specification were mandatorily to be implemented, even if a specific M-Instance had limited scope. MMM-TEC V2.0 specifies Profiles to facilitate the take-off of M-Instance implementations that conform to the MMM-TEC V2.0 specification without unduly burdening some other implementations.

A Profile includes only a subset of the Process Actions that are expected to be needed and are shared by a sizeable number of applications. MMM-TEC v2.0 defines four Profiles (see Figure 2:

1. Baseline Profile enables basic applications such as lecture, meeting, and hang-out.
2. Finance Profile enables trading activities.
3. Management Profile includes enables a controlled ecosystem with more advanced functionalities.
4. High Profile enables all the functionalities of the Management Profile with a few additional functionalities of its own.

Figure 2 – MMM-TEC V2.0 Profiles

MPAI developed and used some use cases in the two MPAI-MMM Technical Reports (1 and 2) published in 2023 to develop the MMM-ARC and MMM-TEC Technical Specifications. However, MMM-TEC V2.0 includes various Verification Use Cases that use Process Actions to verify that the currently specified Actions and Items completely support those Use Cases.

The fast development of certain technology areas is one of the issues that has so far prevented the development of metaverse interoperability standards. MMM-TEC deals with this issue by providing JSON syntax and semantics for all Items. When needed, the JSON syntax references Qualifiers, MPAI-defined Data Types that supply additional information to the Data in the form of:

1. Sub-Type (e.g., the colour space of a Visual Data Type).
2. Format (e.g., the compression or the file/streaming format of Speech).
3. Attributes (e.g., the Binaural Cues of an Audio Object).

For instance, a Process receiving an Object can understand from the Qualifier referenced in the Object whether it has the required technology to process it, or else it has to rely on a Conversion Service to obtain a version of the Object matching its P-Capabilities. This approach should help to prolong the life of the MMM-TEC specification as in many cases only the Qualifier specification will need to be updated, not the MMM-TEC specification.

Finally, MMM-TEC V2.0 specifies the MPAI-MMM API. By calling the APIs, a developer can easily develop M-Instances and applications.

Geneva, Switzerland – 16^th April 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 55^th General Assembly (MPAI-55) with the final release of the Connected Autonomous Vehicle and the MPAI Metaverse Model standards.

The MPAI Metaverse Model (MPAI-MMM) – Technologies (MMM-TEC) V2.0 standard specifies:

1. The Functional Requirements of the Processesoperating in a metaverse instance (M-Instance).
2. The Items, i.e., the Data Types and their Qualifiers recognised in an M-Instance.
3. The Process Actionsthat a Process can perform on Items.
4. The Protocolsenabling a Process to communicate with another Process.
5. The MPAI Metaverse Model
6. The MPAI-MMM API.

Availability of APIs enable the rapid development of M-Instances and clients that interoperate with M-Instances conforming with the MMM-TEC V2.0 standard.

An online presentation of the MMM-TEC V2.0 standard will be presented on 9 May at 15 UTC. Register at https://tinyurl.com/5a2d4ucv.

The Connected Autonomous Vehicle (MPAI-CAV) – Technologies (CAV-TEC) V1.0 standard specifies the Reference Model partitioning a CAV into subsystems and components. The standard Reference Model promotes CAV componentisation by enabling:

1. Researchersto optimise component technologies.
2. Component manufacturersto bring their standard-conforming components to an open market.
3. Car manufacturersto access a global market of interchangeable components.
4. Regulatorsto oversee conformance testing of components following standard procedures.
5. Usersto rely on Connected Autonomous Vehicles whose operation they can explain.

An online presentation of the CAV-TEC V1.0 standard will be presented on 8 May at 15 UTC. Register at https://tinyurl.com/372739sa.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Autonomous Vehicle (MPAI-CAV): investigating extensions of the current CAV-TEC standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): developing the Up-sampling Filter for Video applications (EVC-UFV) standard.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MMM-TEC specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

MPAI has recently published two main new standards with a request for Community Comments. In MPAI lingo this means that the standards are mature, but MPAI asks the Community to review the drafts before publication.

The two standards are Connected Autonomous Vehicle – Technologies (CAV-TEC) V1.0 and MPAI Metaverse Model – Technologies (MMM-TEC) V2.0. They are not “new” MPAI standards as they have already been published in earlier version, but the new versions represent significant improvements.

You may ask “Why is this topic handled in a single paper when we are talking about two standards that have – apparently – so little in common”? If you ask this question, you may want to continue reading and discover an essential aspect of MPAI standardisation. A standard for an application domain is (almost) never a monolith but is often made of components shared with standards of other domains, often unrelated.

Let’s first have a scan of the two standards.

Connected Autonomous Vehicle (CAV-TEC V1.0) is a standard for the ICT (Information and Communication Technology) part of a vehicle that can move itself in the physical world to reach a destination. The standard assumes that a CAV is composed of four functionally separated but interconnected subsystems:

1. The human-CAV Interaction (HCI): enables a human to establish a dialogue with the CAV to issue a variety of commands to it, such as moving to a destination or have conversations where the human – and the CAV – can express their internal status – e.g., emotion – whether real or fictitious.
2. The Environment Sensing Subsystem (EES): leverages the sensors on board to create the Basic Scene Descriptors (BED), the most accurate digital representation of the external environment possible with the available sensors.
3. The Autonomous Motion Subsystem (AMS): receives the BED and improves its accuracy by exchanging portions of their Environment Descriptors with CAVs in range. Then it analyses or reviews the situation and decides how to issue commands to implement the route decided by the human.
4. The Motion Actuation Subsystem converts a general request to move the CAV potentially by a few meters to specific commands for brakes, motors, and wheel and to report about the implementation of the command.

The four subsystems are implemented as AI Workflows per the AI Framework standard. Each AI Workflow includes several AI Modules that exchange Data specifies by CAV-TEC V1.0 or by other MPAI standards. Most AIMs and data types of:

1. HCI: is specified not by CAV-TEC but by Multimodal Conversation (MPAI-MMC) V2.3 . The rest is specified by Object and Scene Description (MPAI-OSD) V1.3, Portable Avatar Format (MPAI-PAF) V1.4, Data Types, Formats, and Attributes (MPAI-TFA) V1.3, and CAV-TEC.
2. ESS, AMS, and MAS: are specified by CAV-TEC, MPAI-OSD,  MPAI-PAF, and MPAI-TFA.

A concise description of the operation of an implementation of CAV-TEC is available here.

MPAI Metaverse Model (MMM-TEC V2.0) specifies a virtual space composed of processes is provided operating on an ICT platform and executing or requesting other processes to execute actions on items, i.e., MMM-TEC V2.0-specified data types. Processes may be rendered as avatars.

MMM-TEC does not use AIMs but only data types. It defines 27 actions and a language enabling processes to communicate using speech acts. Many are specified by MPAI-MMC, MPAI-OSD, and MPAI-PAF.

Example of actions are MM-Embed applied on an avatar or an object to place it somewhere in the metaverse, UM-Capture applied to media information in the physical world acquired for use in the metaverse, Identify applied to data captured to convert it to an item with an identifier, MM-Anim applied to an item, e.g., an avatar, to animate it, and MU-Render applied to an item in the metaverse to render it in the universe. A concise description of the operation of an implementation of MMM-TEC is available here.

As already said, both draw a sizeable part of their data types (and CAV-TEC of its AIMs) from three other standards: MPAI-OSD V1.3, MPAI-PAF V1.4, and MPAI-TFA V1.3. Some of these AIMs and data types are new in the mentioned versions of the three standards. They were developed in response to the CAV-TEC and MMM-TEC needs.

Why not develop them in CAV-TEC and MMM-TEC directly, then? Because the three standards address a specific area of standardisation that is also required by many other MPAI standards: objects and scenes, 3D graphics, and qualifiers. Therefore, MPAI-OSD V1.3, MPAI-PAF V1.4, and MPAI-TFA V1.3 are also published with a request for Community Comments.

Anybody is invited to send comments on any of the five standards to the MPAI Secretariat by April 13 at 23:59 UTC.

Geneva, Switzerland – 19^th March 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 54^th General Assembly (MPAI-54) with the release of the Connected Autonomous Vehicle and the MPAI Metaverse Model standards for Community Comments and the start of the Up-sampling Filter for Video Applications (EVC-UFV) V1.0 standard project.

MPAI has been working on the Connected Autonomous Vehicle (MPAI-CAV) project since its early days and released the Architecture Specification (CAV-TEC). Today, MPAI-54 released the Technology specification (CAV-TEC) V1.0 that builds on the Architecture Specification adding subsystems, components, and data types. The specification is released with a request for Community Comments and is available from the MPAI website. Anybody is invited to send comments on MMM-CAV V1.0 to the MPAI Secretariat by April 13 at 23:59 UTC.

MPAI has been working on the MPAI Metaverse Model (MPAI-MMM) project since January 2022. So far, two technical Reports and two Technical Specifications on Architecture and Technologies were published. Today, MPAI-54 released the Technology specification (MMM-TEC) V2.0 integrating the Architecture and Technologies specifications and the Reference Software published by MPAI-53. The specification is released with a request for Community Comments and is available from the MPAI website. Anybody is invited to send comments on MMM-TEC V2.0 to the MPAI Secretariat by April 13 at 23:59 UTC.

Both CAV-TEC V1.0 and MMM-TEC V2.0 reuse technology specifications that are shared with other MPAI standard, namely Object and Scene Description (MPAI-OSD) V1.3, Portable Avatar Format (MPAI-PAF) V1.4, and Data Types, Formats and Attributes (MPAI-TFA) V1.3. The specifications – available from the MPAI-OSD, MPAI-PAF, and MPAI-TFA web pages, respectively – were released with a request for Community Comments. Anybody is invited to send comments on any of the three standards to the MPAI Secretariat by April 13 at 23:59 UTC.

MPAI has identified the need for a standard that specifies an up-sampling filter for video applications with improved performance compared to the currently used up-sampling filters. MPAI-54 decided to kick off the new Up-sampling Filter for Video applications (EVC-UFV) V1.0 standard project based on the received responses to the Call for Technologies.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

Geneva, Switzerland – 19^th February 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 53^rd General Assembly (MPAI-53) releasing the first version of the MPAI Metaverse Model Open-Source Reference Software and kicking off the new project Compression and Understanding of Industrial Data (MPAI-CUI) – Company Performance Prediction (CUI-CPP) V2.0.

MPAI has been working on MPAI Metaverse Model (MPAI-MMM) standards since January 2022 and published Two technical Reports and two Technical Specifications  on Architecture and Technologies. The Reference Software released today implements a significant number of the MMM functionalities and uses a set of Unity instances to realise the different environments of the metaverse instance. You can find the MMM software at http://bit.ly/41J0wsj (REST API web server) and https://bit.ly/4ituw0R (Unity web server).

The Company Performance Prediction (CUI-CPP) project intends to provide a solution to a problem afflicting all companies: given the governance structure and the financial situation of a company and various types of risks that may affect it, what is the impact of the components of the governance structure on the governance, finance, and risk on the probability of default? MPAI has developed a set of functional requirements and a framework licence. A Call for Technologies was published and the standard will be collaboratively developed based on the responses to the Call.

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) standard.
4. Connected Aonomous Vehicle (MPAI-CAV): updating the MPAI-CAV Architecture part and developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing Company Performance Prediction standard V2.0.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): waiting for responses to the Call for Technologies for video up-sampling filter on 11 February.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Collaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

The 52^nd MPAI General Assembly (MPAI-52) has approved Server-based Predictive Multiplayer Gaming (MPAI-SPG) – Mitigation of Data Loss effects (SPG-MDL) V1.0. It is a Technical Report that provides a methodology to predict the game state of an online gaming server when some controller data is lost. The Prediction is obtained by applying Machine Learning algorithms based on historical data of the online game.

An online Multiplayer Game is based on a server. When this maintains consistency among all clients’ game instances it is called authoritative. It updates and broadcasts the game state using the controller data of all the clients. This function is harmed when controller data are not correctly received or are maliciously modified.

There are several techniques currently used to cure this situation. In Client Prediction, client game state is updated locally using predicted or interpolated data while waiting for the server data; in Time Delay, the server buffers the game state updates to synchronise all clients; and in Time Warp the server rolls back the game state to when controller data was sent by a client and acts as if the action was taken then, reconciling this new game state with the current game state.

These three methods have shortcomings. Client Prediction causes perceptible delay, Time Delay affects responsiveness, and Time Warp disadvantages other players because the new game state likely differs from the previous one.

Figure 1 depicts the arrangement proposed by SPG-MDL. The right-hand side represents the online game server with the Game State Engine tasked to produce the game state using all clients’ controller data and specialised engines. The Engines of the figure are:

1. Behaviour Engine, orchestrating actions from players and non-player entities.
2. Rules Engine, ensuring adherence to game mechanics.
3. Physics Engine, responsible for physical interactions within the game environment.

Figure 1 – Server Prediction

Whenever a client’s controller data is lost, the server requests the SPG-MDL module (the left-hand side of Figure 1) to compute the next game state. In this way, even if a client is experiencing network latency, the other clients maintain a continuous playing environment. The more accurate the predictions, the less noticeable the effect of the synchronisation process on the lagging client when the network resumes normal operations. A latency-affected client will still receive the results of its actions with a delay, but the server will send the new game state before receiving the action from the client, effectively halving the wait time. Of course, it is possible to further mitigate the effects of this problem by implementing additional client-side techniques, for example Client Prediction.

When some controller data is lost, the process begins with the last correct game state being fed into the SPG-MDL’s Game State Demultiplexer, which deconstructs it into discrete Game Messages* ( ). To differentiate the Game Messages inside the “twin” game server from the ones inside the game server, the ‘*’ symbol is attached to the SPG-MDL Game Messages. Each Game Message* is then processed by its respective Engine AI, leveraging a Neural Network Model to produce a predicted Game Message* ( ).  These predictions are assembled by the Predicted Game State Multiplexer into the predicted Game State ( ), which is then sent back to the game server for the next iteration of Game State computation.

While the SPG-MDL module operates, the server independently computes its updated Game State (GS_t+1) using the available Client Data. The server utilises the predicted Game State as follows. If any Client Data is missing, the server uses the predicted game state to compensate for the missing data shortfall from one or more clients. Note that the online game server architecture is a reference model and that the three engines are not a requirement for a specific game applying the MPAI-SPG methodology. For example, some games may not have a Physics Engine because physical-based behaviour is not required.

SPG-MDL provides a 10-step procedure to develop an SPG-MDL that is applicable to any authoritative game server.

The first 4 steps are required to outline the game setup to enable informed decisions for the implementation of SPG.

1. Select the game.
2. Define the Entities (to identify NN Model parameters):
   1. Environment
   2. Human-controlled players (HPC) and Non-player characters (NPC)
3. Define the Game State and relevant Entities.
4. Design the training dataset.
5. Collect the training dataset.
6. Train prediction NN Models defining viable architectures and training parameters and comparing the training results of different architectures.
7. Implement SPG-MDL.
8. Evaluate SPG-MDL to select the model yielding the best predictions.
9. Implement modules which simulate the disturbances.
10. Evaluate the SPG-MDL-enabled game experience with human players.

For each of the ten steps, the Technical Report provides:

1. High-level guidelines to outline the actions required.
2. An example of how the guidelines are implemented using a car racing game (Figure 2).

Figure 2 – Modular tiles and an example of a racetrack

The game was developed using the Unity game engine, and the networking features implemented through the open-source game networking library Mirror.

The following components are provided:

• The car racing game.
• Four different categories of Agent Players trained using Unity’s ML-Agents library.
• The dataset used for training generated by simulating game sessions played by the Agent Players.
• Jupyter Notebooks for training experiments and results.
• The trained models used by the AI-Behaviour Engine.

The software is available online. Please contact the MPAI secretariat to access the repository. Details on how to use the material are provided in the repository’s README page.