Archives: 2021-01-24

The concept of virtual reality is now well established, with the metaverse concept as an important variant. Accordingly, MPAI has established a related standard, the MPAI Metaverse Model – Technologies (MMM-TEC) standard. However, standards for the contents of an MPAI metaverse instance (M-Instance) are still in progress. This document introduces the current status of these efforts and invites participation.

The contents include Processes representing entities with agency, called Users, and other entities lacking agency – essentially, various things populating an M-Instance – called Items.

Some Users represent humans. These may be directly operated by humans (and are called H-Users), or may have a high degree of operational autonomy (and are called A-Users, or informally, agents). Both types may be rendered as avatars called Personae.

The MMM-TEC standard specifies technologies enabling Users to perform various Actions on Items (things) in an M-Instance. For example, Users may sense data from the real world or may move Items in the M-Instance, possibly in combination with other Processes. However, MMM-TEC does not yet specify how an A-User decides to perform an Action.

Thus MPAI is developing a new standard covering such decisions: what does an A-User do when deciding to do something to achieve a Goal in an M-Instance? MPAI has assembled numerous relevant technologies, but more are needed. Therefore, the 61^st MPAI General Assembly (MPAI-61) has published the Call for Technologies Pursuing Goals in metaverse (MPAI-PGM) – Autonomous User Architecture (AUA). The Call requests interested parties – irrespective of their membership in MPAI – to submit responses that may enable MPAI to develop a robust A-User Architecture standard attractive to implementers and users.

The planned standard’s scope is as follows: PGM-AUA will specify functions and interfaces by which an A-User interacts with another User, either an A-User or an H-User. (Again, the term “User” means “conversational partner in the metaverse”, whether autonomous or driven by a human.) A-Users can capture text and audio-visual information originated by, or surrounding, the User; extract the User State, i.e., snapshots of the User’s cognitive, emotional, and interactional states; produce an appropriate multimodal response, rendered as a speaking Avatar; and move appropriately in the relevant virtual space.

One possible way to model an A-User’s interactions with other Users might be to train a very powerful unitary Large Language Model, able to use spatial and media information. However, because such a model would be unwieldy and difficult to manage, MPAI instead assumes the use of a relatively simple Large Language Model with basic language and reasoning capabilities. Spatial, audio-visual, and User description information will be passed to and from this Basic Model in natural language.

To handle this integration, MPAI proposes the MPAI AI Framework (MPAI-AIF) standard. This standard provides the necessary infrastructure to define a foundation for an A-User to which the necessary technologies can be added. MPAI-AIF enables specification of an AI Workflow (AIW) composed of AI Modules (AIMs). In this case, these can jointly represent an A-User in a manner that is modular, i.e., able to swap or update modules independently from other modules; transparent, i.e., able to perform clear roles and expose well-defined interfaces; and extensible, i.e., able to add or replace specific competences as needed.

The following figure represents a tentative diagram of the A-User architecture.

The model represents a largely autonomous A-User’s (“agent’s”) interactions with another User (A-User or H-User) at a given instant. It would thus be invoked repeatedly for extended interactions.

At a high level, we see an executive element (A-User Control), which can receive as input a human command or the response to some Action, and which delivers as output its status in response to the relevant command; any related action; and any request that it may itself deliver.

NOTE: While an A-User is defined as a relatively autonomous Process, a human may take over or modify its operation via the A-User Control.

More formally, the executive A-User Control AIM (AI Module) drives A-User operation by controlling how it interacts with the environment and performs Actions and Process Actions by:

1. Performing an Action or requesting another Process to perform one.
2. Controlling the operation of AIMs, in particular A-User Rendering.

Here is a full accounting of the input and output, separated from the full diagram to remove distractions:

This input-output diagram summarizes in two lines the Responses from, and Commands to, the A-User’s six current AIMs, or AI Modules, which jointly enable the A-User’s actions. They represent perception and reasoning about what is perceived; knowledge and processing about the current domain, e.g., surgery or a particular game; and composition of an appropriate response, including the A-User’s simulated emotion, cognitive state, and social attitudes, alignment with the agent’s simulated personality, and rendering of the resulting response’s visual and audio aspects. Most of these modules can consult the A-User’s Large Language Model, the Basic Knowledge (LLM).

Keeping this overview in mind, we can survey the individual modules in greater detail.

Context Capture

The Context Capture AIM, prompted by the A-User Control, perceives a particular location of the M-Instance – called an M-Location – where the A-User’s conversation partner, here designated as User, is rendering its Avatar. The result of the capture is called the Context, a time-stamped structured snapshot representing the A-User’s initial understanding of the M-Location. Context is composed of

1. Audio-Visual Scene Descriptors describing the spatial content.
2. User State, describing the User’s cognitive, emotional, and attentional posture within the environment.

Spatial Reasoning

The Spatial Reasoning AIM analyses the Context and:

1. Sends Audio and Visual Spatial Output, i.e., spatial relationships, referent resolutions, and interaction constraints, to the Domain Access AIM seeking additional domain-specific information.
2. Sends Audio and Visual Spatial Guides, i.e., audio source relevance, directionality, and proximity (Audio) and object relevance, orientation, proximity, and affordance (Visual) to the Prompt Creation AIM. The objective is to enrich the A-User’s spoken or written input with additional information about A-User before sending a prompt created by Prompt Creation (a PC-Prompt) to Basic Knowledge, a basic LLM. PC-Prompt includes
   1. User Text and User State (from Context Capture).
   2. Audio and Visual Spatial Guide (from Spatial Reasoning).

Spatial Reasoning is specified as two separate AIMs, one for the Audio and the other for the Visual component. 3D Graphics inputs are also handled by the Visual component.

Domain Access

The Basic Knowledge LLM sends to the Domain Access module an Initial Response containing a direct response to the PC-Prompt, along with general reasoning based on foundational LLM capabilities.

The Domain Access module then

1. Processes and responds to two flows:
   1. Spatial Output (audio and visual, from Spatial Reasoning):
      1. Accesses domain-specific models, ontologies, or M-Instance services.
      2. Returns Audio and Visual Spatial Directive to inject contextual background, scene-specific logic, and task relevance into the reasoning loop of Spatial Reasoning to improve the fidelity of its spatial interpretation.
   2. Initial Response (from the Basic Knowledge LLM):
      1. Accesses domain-specific models, ontologies, or M-Instance services to retrieve:
         1. Scene-specific object roles (e.g., “this is a surgical tool”)
         2. Task-specific constraints (e.g., “only authorised Users may interact”)
         3. Semantic affordances (e.g., “this object can be grasped”)
      2. Returns to Basic Knowledge LLM a DA-Prompt that includes initial reasoning, spatial semantics, domain overlays, and User/task constraints.
2. Performs further interchanges:
   1. Sends Refined Context Guide (to User State Refinement):
      1. Includes a structured object with:
         1. Updated User descriptors
         2. Scene salience and relevance
         3. Interaction history and inferred goals

User State Refinement

Uses Refined Context Guide from the Domain Access module to update User State and to generate a UR-Prompt for the Basic Knowledge LLM, reflecting the refined understanding.

Domain Access

The Basic Knowledge LLM sends to the Domain Access module an Initial Response containing a direct response to the PC-Prompt, along with general reasoning based on foundational LLM capabilities.

The Domain Access module then

1. Processes and responds to two flows:
   1. Spatial Output (audio and visual, from Spatial Reasoning):
      1. Accesses domain-specific models, ontologies, or M-Instance services.
      2. Returns Audio and Visual Spatial Directive to inject contextual background, scene-specific logic, and task relevance into the reasoning loop of Spatial Reasoning to improve the fidelity of its spatial interpretation.
   2. Initial Response (from the Basic Knowledge LLM):
      1. Accesses domain-specific models, ontologies, or M-Instance services to retrieve:
         1. Scene-specific object roles (e.g., “this is a surgical tool”)
         2. Task-specific constraints (e.g., “only authorised Users may interact”)
         3. Semantic affordances (e.g., “this object can be grasped”)
      2. Returns to Basic Knowledge LLM a DA-Prompt that includes initial reasoning, spatial semantics, domain overlays, and User/task constraints.
2. Performs further interchanges:
   1. Sends Refined Context Guide (to User State Refinement):
      1. Includes a structured object with:
         1. Updated User descriptors
         2. Scene salience and relevance
         3. Interaction history and inferred goals

User State Refinement

Uses Refined Context Guide from the Domain Access module to update User State and to generate a UR-Prompt for the Basic Knowledge LLM, reflecting the refined understanding.

Basic Knowledge produces and sends to Personality Alignment a Refined Response.

Personality Alignment

1. Receives (1) a Personality-based Refined Response and (2) an Expressive State Guide, both conveying a variety of elements such as:
   1. Expressivity, e.g.:
      1. Tone, e.g., formal, casual, empathetic, assertive
      2. Tempo, e.g., fast, slow, rhythmic
      3. Gesture style, e.g., expansive, restrained, animated
      4. Facial dynamics, e.g., smile frequency, gaze behaviour, eyebrow movement
      5. Etc.
   2. Behavioural Traits, e.g.:
      1. Verbosity level
      2. Use of metaphors or humour
      3. Degree of emotional expressiveness
   3. Type of role: assistant, mentor, negotiator, entertainer, etc.
2. Formulates and sends
   1. An A-User Personal Status (using the MPAI-specified Personal Status structure, containing specification of emotion, cognitive state, and social attitudes) reflecting the Personality to A-User Rendering.
   2. A PA-Prompt to Basic Knowledge reflecting:
      1. Speech modulation instructions (e.g., pitch, emphasis)
      2. Facial expression timing and intensity
   3. Gesture choreography
   4. Synchronisation cues across modalities.

A-User Rendering

The Basic Knowledge LLM sends a Final Response to the A-User Rendering module that conveys semantic content, contextual integration, expressive framing, and personality consistency.

As seen below, the A-User Rendering module (1) uses the Final Response and A-User Personal Status to synthesise a speaking Avatar  and (2) employs an A-User Control Command from A-User Control to refine the speaking Avatar.

Extended Call for Technologies

The complexity of the MMM-TEC model has prompted MPAI to extend its usual practice for Calls for Technologies. In addition to the usual Call for Technologies, Use Cases and Functional Requirements, Framework Licence, and Template for Responses, the Call also refers to a Tentative Technical Specification, a document drafted as if it were an actual Technical Specification. Respondents to the Call are free to comment on, change, or extend the Tentative Technical Specification or to make any other proposals judged relevant to the Call.

Anyone, irrespective of MPAI membership status, may respond to the Call. Responses shall reach the MPAI Secretariat by 2026/01/21T23:59.

Appropriate MPAI working groups will thoroughly review the Responses and retain those deemed appropriate for the future PGM-AUA standard. MPAI may select suitable technologies from those submitted in Responses, but is not obligated to select any proposal. Respondents will be encouraged to join MPAI. If Respondents whose Responses are accepted in full or in part do not join MPAI, MPAI will discontinue consideration of their proposed technologies.

MPAI is organizing two online presentations with similar content on 2025/11/17. To attend register:

• here for the 9 UTC presentation.
• here for the 16 UTC presentation.

Geneva, Switzerland – 29^th October 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 61^st General Assembly (MPAI-61) approving the publication of a Call for Autonomous User Architecture Technologies.

With this Call for Technologies, formally “Pursuing Goals in metaverse (MPAI-PGM) – Autonomous User Architecture (PGM-AUA)”, MPAI is aiming at a standard enabling Autonomous Users to perform activities such as moving around and conversing with other Users. These are processes representing humans in a metaverse conforming with the MPAI Metaverse Model Technologies standard (MMM-TEC). They can either operate with a high degree of autonomy (A-Users) or be directly controlled by humans (H-Users).

PGM-AUA will rely on the friendly MMM-TEC environment and many relevant technologies already available in the 16 approved MPAI standards. However, the ambitious PGM-AUA goal requires many new technologies that the Call is designed to secure.

The text of the call and associated document is available. Responses are due to the MPAI Secretariat by 2025/01/21T23:59.

Register to attend the online presentation of the call and its associated documents on 17 November at  9 UTC and 16 UTC.

MPAI-61 has also approved the new versions of standards previously posted for Community Comments:

• AI Framework
• Human and Machine Communication
• Multimodal Conversation.

MPAI is continuing the development of its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): developing a new MPAI-AIF specification that facilitates the creation of new workflows using available AIMs.
2. AI for Health (MPAI-AIH): developing the specification of a system receiving and processing licenses AI Health Data and enabling clients to improve health processing models via federated learning.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) and Audio Object Scene Rendering (CAE-AOR) specifications.
4. Connected Autonomous Vehicle (MPAI-CAV): investigating extensions of the current CAV-TEC specification.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing the Company Performance Prediction V2.0 specification.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): finalising the Up-sampling Filter for Video applications (EVC-UFV) standard.
8. Governance of the MPAI Ecosystem (MPAI-GME): operating the MPAI Ecosystem per the MPAI-GME Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): discussing the conversational part of the PGM-AUA Call for Technologies.
11. MPAI Metaverse Model (MPAI-MMM): developing support for security in the MMM-TEC specs.
12. Neural Network Watermarking (MPAI-NNW): Reviewing the responses to the Call on Neural Network Traceability Technologies.
13. Object and Scene Description (MPAI-OSD): discussing the spatial part of the PGM-AUA Call for Technologies.
14. Portable Avatar Format (MPAI-PAF): discussing the rendering part of the PGM-AUA Call for Technologies.
15. AI Module Profiles (MPAI-PRF): extending the scope of the current version of AI Module Profiles.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attributes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., automotive, health, and metaverse).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execution of Live Theatrical Performances.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI Secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

Where there are organisations counting years of existence in decades or centuries, there should not be much to celebrate for an organisation that only reaches as few as five years of existence. But there are years and years – even days and days – like in one day as a lion or a hundred years as a sheep.

The last five were not the years of a sheep but as one day as a lion.

We started with the idea of an organisation dedicated to standards for AI-based data coding because we thought that standards would bring benefits to a domain mostly alien to it. Not like some standards that look more like legal tools designed to oppress users but standards offering fair opportunities to all parties in the chain extending from innovators to end users.

An ambitious organisation like MPAI could not operate like four friends in a bar. The MPAI operation rules were developed and are now enshrined in the MPAI Patent Policy. The ambitions of MPAI were further enhanced by the definition of the MPAI Ecosystem extending from MPAI to implementers, integrators, and end users with the introduction of a new actor called MPAI Store, now incorporated in Scotland as a company limited by guarantee. There is a standard – Governance of the MPAI Ecosystem (MPAI-GME) setting the rules of operation of the Ecosystem.

The idea of a mission was there but what about implementing it? We acted as lions and posited that opaque monolithic AI should become component-based AI. Now a large share of our standards are based on the AI Framework (MPAI-AIF) standard, specifying an environment where AI Workflows composed of AI Modules can be initialised, dynamically configured, and controlled. MPAI-AIF also provided a stimulus to adoption of JSON Schema as a “language” to represent data types, AI Modules, and AI Workflows in MPAI standards. Today there is virtually no MPAI standard that does not use that language.

Having laid down the technical foundations, we started the buildings. One was designed to host the quite representative area of human and machine conversation extending beyond the “word” to cover other sometimes ethereal but information-carrying sensations and feelings. The standard called Multimodal Conversation (MPAI-MMC) is the first attempt at digitally representing this ethereal information with the Personal Status data type and Human-Machine Communication (MPAI-HMC) is an excellent example of its application.

Another investigation stream since the early MPAI days is audio sitting at the MPAI table as “Context-based Audio Enhancement” leading to the Context-based Audio Enhancement – Use Cases (MPAI-CAE) standard. Finally, with Compression and Understanding of Industrial Data (MPAI-CUI), MPAI demonstrated that data from so far unexplored domains like finance could benefit from standards.

Just one year after its establishment, MPAI could claim success by publishing its first three standards:  MPAI-CUI, MPAI-GME, and MPAI-MMC and, by the end of 2021, another two: MPAI-AIF and MPAI-CAE.

Since its early days, MPAI was convinced that standards should have as much visibility as possible. For this reason, it established a successful cooperation with the Institute of Electric and Electronic Engineers (IEEE) – Standard Association (SA). Today, starting from three standards in 2022, nine MPAI standards have been adopted by IEEE without modifications and three more are in the pipeline.

The creation of MPAI Development Committees and Working Groups and their activity continued unrelenting. The use of watermarking and then fingerprinting to trace the use of neural networks let to the development of Neural Network Watermarking – Traceability (NNW-NNT). Connected Autonomous Vehicles was started in late 2020 and is now a standard with the name Connected Autonomous Vehicle – Technologies (CAV-TEC). MPAI was probably the first to engage in activities leading to a metaverse standard and now it can claim to have a solid candidate to lead the move to interoperable metaverses with MPAI Metaverse Model – Technologies (MMM-TEC). Since its early days, MPAI worked on online gaming, producing the Server-based Predictive multiplayer Gaming – Mitigation of Data Loss Effects (SPG-MDL) standard where a set of AI Modules predicts the game state of an online multiplayer game.

MPAI abhors the attitude of other standards bodies who develop unnecessarily “siloed” standards where technologies are treated exclusively from the point of view of the domain of that standard without considering similar technologies in other domains. Object and Scene Description (MPAI-OSD) and Portable Avatar Format (MPAI-PAF) do specify AI Workflows specific to their domains but their AI Modules and Data Types were specified for wide reuse in many other MPAI standards. This attitude is not confined to these two standards as the same can be said of MPAI-CAE and MPAI-MMC.

Atypical – but no less important – standards are AI Module Profiles (MPAI-PRF) establishing a machine-readable description to identify AI Module Profiles and Data Types, Formats, and Attributes (MPAI-TFA) providing a standard way to add information about data for processing by a machine.

Last comes a standard that embodies probably the very first activity – AI for video. AI-Enhanced Video Coding – Up-sampling Filter for Video applications (EVC-UFV) offers an AI super-resolution filter vastly superior to currently used filters.

Five years ago, MPAI was very bold in targeting standards for AI, then just a nice technology to talk about. In five years, however, AI is all over the place and much talked about. What will the future offer for MPAI?

Some answers are clear:

• With its impressive portfolio of 15 standards, there will be much maintenance and enhancement work to do.
• Two new standards are being developed and should be completed in a short time: AI for Health – Health Secure Platform and XR Venues – Live Theatrical Performance.
• One project – End-to-End Video coding has still to go through the Call for Technologies phase
• A Call for Technologies is open, and responses are expected: Neural Network Watermarking – Technologies.
• A new Call for Technologies on Pursuing Goals in the metaverse is being prepared. This will require the development of a significant number of “behaviours” on top of a “baseline” Small Language Model.
• Development of reference implementations to enhance the value and attractiveness of existing standards.

AI continues its lightning speed of development and MPAI will continue watching and identifying standardisation opportunities in different domains.

Long live MPAI!

Geneva, Switzerland – 30^th September 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has celebrated its fifth anniversary at its 60^th General Assembly (MPAI-60).

Established 5 years ago on 30 September 2020, MPAI has created the organisation, given itself rigorous procedures of work, developed 15 standards and two technical reports, obtained adoption of eight of its standards without modification by IEEE Standards Association, and is setting sights on next challenges targeting both extensions and new standards.

In line with its mission of AI-based data coding, MPAI standards cover execution of AI applications, audio enhancement, connected autonomous vehicles, finance, human and machine conversation, metaverse, objects and scenes, avatars, and many others.

MPAI-60 has approved final publication of new versions of existing standards:

• Governance of the MPAI Ecosystem0
• MPAI Metaverse Model Technologies1
• Object and Scene Description4
• Portable Avatar Format5
• Data Types, Formats, and Attributes4

and is publishing the following standards for Community Comments

• AI Framework2
• Human and Machine Communication1
• Multimodal Conversation4.

MPAI is continuing the development of its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): developing a new MPAI-AIF specification that facilitates the creation of new workflows using available AIMs.
2. AI for Health (MPAI-AIH): developing the specification of a system receiving and processing licenses AI Health Data and enabling clients to improve health processing models via federated learning.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) and Audio Object Scene Rendering (CAE-AOR) specifications.
4. Connected Autonomous Vehicle (MPAI-CAV): investigating extensions of the current CAV-TEC specification.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing the Company Performance Prediction V2.0 specification.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): refining the Up-sampling Filter for Video applications (EVC-UFV) standard.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing the notion of Perceptive and Agentive AI (PAAI) capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the capabilities of the MMM-TEC specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): Issuing a Call on Neural Network Traceability Technologies.
13. Object and Scene Description (MPAI-OSD): extending the capabilities of the MPAI-OSD V1.3 to support more applications.
14. Portable Avatar Format (MPAI-PAF): extending the capabilities of the MPAI-PAF V1.4 to support more applications.
15. AI Module Profiles (MPAI-PRF): extending the scope of the current version of AI Module Profiles.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attributes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., automotive, health, and metaverse).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execution of Live Theatrical Performances.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI Secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

The MPAI Metaverse Model (MPAI-MMM) – Technologies (MMM-TEC) specification is based on an innovative approach. As in the real world (Universe) we have animate and inanimate things, in an MPAI Metaverse (M-Instance) we have Processes and Items. Processes can animate Items (things) in the metaverse but can also act as a bridge between metaverse and universe. For convenience, MMM-TEC defines four classes of Processes: Apps, Devices, Services, and Users.

Geneva, Switzerland – 20^th August 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 59^th General Assembly (MPAI-59) approving the publication of the MPAI Metaverse Model – Technologies V2.1 with a request for Community Comments.

The earlier 2.0 Version of Technical Specification: MPAI Metaverse Model (MMM) – Technologies (MMM-TEC) already supported digital twinning of real-world environments and their blending with MMM-TEC-specified virtual environments. The new MMM-TEC V2.1 supports “analogue twinning” of virtual- with real-world environments opening attractive industrial metaverse applications. This is achieved by introducing new “Process Actions” (speech acts of an MMM-TEC process sent to another process) and the notion of R-Item (real object) that can be MU-Added (placed at a U-Location, a location in the real world), MU-Moved (moved from a U-Location to another U-Location along a Trajectory), and MU-Animated (animated) in sync with a Persona (the rendering of a Process as an avatar) in the metaverse.

Among the several other innovations included in MMM-TEC V2.1, we mention Change Property, a Process Action whereby a Process changes – if it holds the Rights – the place where and object is located; its properties such as perceptibility, size, mass, gravity, and texture; audio properties such as reflectivity, reverberation, diffusion and absorption; an audio or light source; and the emotional state of an avatar.

MPAI standards are best described as a web of interconnected specifications. The new technologies needed by MMM-TEC are partly specified by Object and Scene Descriptors (MPAI-OSD), Portable Avatar Format (MPAI-PAF), and Data Types, Formats and Attributes (MPAI-TFA). They are now at versions V1.4, V1.5, and V1.4, respectively.

Register to attend the presentations of the many innovations added to

• The MMM-TEC1 V2.1 standard on 12 September at 15 UTC (link).
• The MPAI-OSD V1.4 and MPAI-PAF V1.5 standards on 12 September at 10 UTC (link).
• The MPAI-TFA V1.4 standard on Wednesday 17 September at 15 UTC (link)
• The MPAI-GME V2.0 standard on Friday 26 September at 14 UTC (link).

MPAI is continuing the development of its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): developing a new MPAI-AIF specification that facilitates the creation of new workflows using available AIMs.
2. AI for Health (MPAI-AIH): developing the specification of a system receiving and processing licenses AI Health Data and enabling clients to improve health processing models via federated learning.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) and Audio Object Scene Rendering (CAE-AOR) specifications.
4. Connected Autonomous Vehicle (MPAI-CAV): investigating extensions of the current CAV-TEC specification.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing the Company Performance Prediction V2.0 specification.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): refining the Up-sampling Filter for Video applications (EVC-UFV) standard.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing the notion of Perceptive and Agentive AI (PAAI) capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the capabilities of the MMM-TEC specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): Issuing a Call on Neural Network Traceability Technologies.
13. Object and Scene Description (MPAI-OSD): extending the capabilities of the MPAI-OSD V1.3 to support more applications.
14. Portable Avatar Format (MPAI-PAF): extending the capabilities of the MPAI-PAF V1.4 to support more applications.
15. AI Module Profiles (MPAI-PRF): extending the scope of the current version of AI Module Profiles.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attributes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., automotive, health, and metaverse).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execution of Live Theatrical Performances.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI Secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

MPAI has approved Technical Specification: AI-Enhanced Video Coding (MPAI-EVC) – Up-sampling Filter for Video applications (EVC-UFV).

The standard includes a general procedure to design video up-sampling filters based on super resolution techniques and a method to reduce the complexity of the designed filters without significant performance loss. The standard also provides the parameters of specific filters for standard definition to high definition and high definition to ultra-high definition, for the complexity-reduced and original cases.

The standard will be presented online on 23 July at 13 UTC. Register here to attend the presentation.

The standard is not in final form. It is published with a request for Community Comments according to MPAI procedures. Comments should be sent the MPAI Secretariat by 2025/08/18 T23:59 UTC.

A method typically used in video coding is to down-sample to half the input video frame before encoding. This reduces the computational cost but requires an up-sampling filter to recover the original video resolution in the decoded video to reduce as much as possible the loss in visual quality. Currently used filters are bicubic and Lanczos,

Figure 1 – Up-sampling Filters for Video application (EVC-UFV)

In the last few years, Artificial Intelligence (AI), Machine Learning (ML), and especially Deep Learning (DL) techniques, have demonstrated their capability to enhance the performance of various image and video processing tasks.  MPAI has performed an investigation to assess how video coding performance could be improved by replacing traditional coding blocks with deep-learning ones. The outcome of this study has shown that deep-learning based up-sampling filters significantly improve the performance of existing video codecs.

MPAI issued a Call for Technologies for up-sampling filters for video applications in October 2024. This was followed by an intense phase of development that enabled MPAI to approve Technical Specification: AI-Enhanced Video Coding (MPAI-EVC) – Up-sampling Filter for Video application (EVC-UFV) V1.0 with a request for Community Comments at its 58^th General Assembly (MPAI-58).

EVC-UFV standard enables efficient and low complexity up-sampling filters applied to video with different bit-depth of 8 and 10 bit per pixels per component, in standard YCbCr colour space with 4:2:0 sub-sampling, encoded with a variety of encoding technologies using different encoding features such as random access and low delay.

As depicted in Figure 2, the filter is a Densely Residual Laplacian Super-Resolution network (DRLN), offering a novel deep-learning approach.

Figure 2 – Densely Residual Laplacian Super-Resolution network (DRLN).

The complexity of the filter is reduced in two steps. First, a drastic simplification of the deep-learning structure that reduces the numbers of blocks provides a much lighter network while keeping similar performances of the baseline DRLN. This is achieved by identifying the DRLN’s principal components and understanding the impact of each component on the output video frame quality, memory size, and computational costs.

As shown in Figure 2, the main component of the DRLN architecture is a Residual Block which is composed of the Densely Residual Laplacian Modules (DRLM) and a convolutional layer. Each DRLM contains three Residual Units, as well as one compression unit and one Laplacian attention unit (a set of Convolutional Layers with a square filter size and Dilation that is greater than or equal the filter size). Each Residual Unit consists of two convolutional layers and two ReLU Layers. All DRLM modules in each Residual Block and all Residual Units in each DRLM are densely connected. The Laplacian attention unit consists of three convolutional layers with filter size 3×3 and dilation (a technique for expanding a convolutional kernel by inserting holes or gaps between its elements) equal to 3, 5, 7. All convolutional layers in the network, except the Laplacian one, have filter size 3×3 with dilation equal to 1. Throughout the network, the number of feature maps (the outputs of convolutional layers) is 64.

Based on this structural analysis, reducing the number of the main Residual Blocks, adding more DRLMs, and reducing the complexity of the Residual Unit and the number of hidden convolutional layers and features map drastically accelerates execution speed and reduces memory management but does not substantially affect the network’s visual quality performance.

Figure 3 depicts the resulting EVC-UFV Up-sampling Filter,

Figure 3 – Structure of the EVC-UFV Up-sampling Filter

The parameters of the original and complexity-reduced network are given in Table 1.

Table 1 – Parameters of the original and the complexity-reduced network

Further, by pruning the parameters and weights of the network, the network complexity is reduced by 40%. The loss in performance is less than 1% in BD-rate. This is achieved, by first using the well-known DeepGraph technique, modified to work with deep-learning based up-sampling filter, understanding the dependency of the different components’ layers of the simplified deep-learning network. This facilitates grouping components sharing a common pruning approach that can be applied without introducing dimensional inconsistencies among the inputs and outputs of the layers.

Verification Tests of the technology has been performed on:

Results show an impressive improvement for all coding technologies, and encoding options for all three objective metrics when compared with the currently used traditional bicubic interpolation. The results of Table 2 have been obtained foe the low-delay coding mode.

Table 2 – Performance of the EVC-UFV Up-sampling Filter

All results are obtained with the 40% pruned network.

Geneva, Switzerland – 9^th July 2025. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, unaffiliated organisation developing AI-based data coding standards – has concluded its 58^th General Assembly (MPAI-58) approving the publication of the Up-sampling Filter for Video applications standard with a request for Community Comments.

The approved Technical Specification: AI-Enhanced Video Coding (MPAI-EVC) – Up-sampling Filter for Video applications (EVC-UFV) specifies a general procedure to design video up-sampling filters based on super resolution techniques. Additionally, it specifies a method to reduce the complexity of the designed filters without significant loss in performance. The standard also provides the parameters of specific filters for standard definition to high definition and high definition to ultra-high definition, for the complexity-reduced and original cases. The standard is not in final form. It is published with a request for Community Comments according to MPAI procedures. Comments should be sent the MPAI Secretariat by 2025/08/18 T23:59 UTC.

The standard will be presented online on 23 July at 13 UTC. Register here to attend the presentation.

MPAI is continuing the development of its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): developing a new MPAI-AIF specification that facilitates the creation of new workflows using available AIMs.
2. AI for Health (MPAI-AIH): developing the specification of a system receiving and processing licenses AI Health Data and enabling clients to improve health processing models via federated learning.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF) and Audio Object Scene Rendering (CAE-AOR) specifications.
4. Connected Autonomous Vehicle (MPAI-CAV): investigating extensions of the current CAV-TEC specification.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing the Company Performance Prediction V2.0 specification.
6. End-to-End Video Coding (MPAI-EEV): exploring the potential of AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): refining the Up-sampling Filter for Video applications (EVC-UFV) standard.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software and performance assessment.
10. Multimodal Conversation (MPAI-MMC): Developing the notion of Perceptive and Agentive AI (PAAI) capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the capabilities of the MMM-TEC specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): Issuing a Call on Neural Network Traceability Technologies.
13. Object and Scene Description (MPAI-OSD): extending the capabilities of the MPAI-OSD V1.3 to support more applications.
14. Portable Avatar Format (MPAI-PAF): extending the capabilities of the MPAI-PAF V1.4 to support more applications.
15. AI Module Profiles (MPAI-PRF): extending the scope of the current version of AI Module Profiles.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): exploring new standard opportunities in the domain.
17. Data Types, Formats, and Attributes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., automotive, health, and metaverse).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execution of Live Theatrical Performances.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI Secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

2025 July 10 T09-13 – Campus Biotech Innovation Park, Av. de Sécheron 15, 1202 Genève, Switzerland (CH)

To join online: Link to Workshop

 Context, Scope and Objectives

This workshop has been conceived and organized by a group of people from the University of Geneva, the University of Zurich and the MPAI (https://mpai.community/), in collaboration with members of the Institute for AI International Governance of Tsinghua University, I-AIIG). The workshop is open to invited scholars and experts in the fields of AI related technical, social and legal research.

During the past few years, the theme of Global AI Governance has gained broad public and policy attention favoring a wide review of the impacts of current AI development path. We understand that AI’s underlying technologies and AI systems’ performance continue to develop at an exponential pace. This provides significant opportunities as well as substantial risks. Here we focus on risks, and particularly extreme or “existential” risks.

Following the arguments and calls made by the world’s most eminent AI experts, we share key Concerns regarding the development of Advanced AI, i.e.: Safety, Alignment, Transparency, Trustworthiness, Fairness.

Many efforts have already been made to address some of these Concerns, and we are fully supportive of them. However, there is a substantial gap between the efforts made to improve AI systems’ performance (the “AI race”) and efforts made to strengthen AI safety and to address the other Concerns. This is due to a variety of reasons such as the limited focus and investment by companies, IPR issues, underestimation of the risks by Governments, difficulty of managing Generative AI through regulatory frameworks, and the geopolitical context.

In addition, based on our knowledge, there is insufficient attention to integrate standardization in the AI safety field with the AI systems’ research and development cycle.

While we have some initial ideas on the necessary starting point to address this gap, the main objective of the workshop is to openly discuss those ideas, verify if there is consensus on them and, in case, find ways to address the risks that the rapid development of AI poses to humanity. The objective is in no way dismissive of the efforts undertaken by a plurality of actors in this domain: efforts that, in general, we fully support. We believe that our proposal is complementary and should help strengthen and expand the global action required to address the Concerns.

Based on the discussions from the workshop, we hope to draft an outcome document/paper to be published. This explains the format of the workshop below.

Workshop Agenda

Introduction

During the last decade, Neural Networks have been deployed in an increasing variety of domains and the production of Neural Networks became costly, in terms of both resources (GPUs, CPUs, memory) and time. Moreover, users of Neural Network based services more and more express their needs for a certified service quality.

NN Traceability offers solutions to satisfy both needs, ensuring that a deployed Neural Network is traceable and any tampering detected.

Inherited from the multimedia realm, watermarking assembles a family of methodological and application tools allowing to imperceptibly and persistently insert some metadata (payload) into an original NN model. Subsequently, detecting/decoding this metadata from the model itself or from any of its inferences provides the means to trace the source and to verify the authenticity.

An additional traceability technology is fingerprinting that relates to a family of methodological and applicative tools allowing to extract some salient information from the original NN model (a fingerprint) and to subsequently identify that model based on the extracted information.

Therefore, MPAI has found the application area called “Neural Network Watermarking” to be relevant for MPAI standardization as there is a need for both Nural Network Traceability technologies and for assessing the performances such technologies.

MPAI available standards

In response to these needs, MPAI has established the Neural Network Watermarking Development Committee (NNW-DC). The DC has developed Technical Specification: Neural Network Watermarking (MPAI-NNW) – Traceability (NNW-NNT) V1.0. This specifies methods to evaluate the following aspects of Active (Watermarking) and Passive (Fingerprinting) Neural Network Traceability Methods:

• The ability of a Neural Network Traceability Detector/Decoder to detect/decode/match Traceability Data when the traced Neural Network has been modified,
• The computational cost of injecting, extracting, detecting, decoding, or matching Traceability Data,
• Specifically for active tracing methods, the impact of inserted Traceability Data on the performance of a neural network and on its inference.

MPAI-NNW Future Standards

During its 57th GA held on June 11th, MPAI released a Call for Neural Network Watermarking (MPAI-NNW) – Technologies (NNW-TEC). This call requests Neural Network Traceability Technologies that make it possible to:

• verify that the data provided by an Actor, and received by another Actor is not compromised, i.e. it can be used for the intended scope,
• identify the Actors providing and receiving the data, and
• evaluate the quality of solutions supporting the previous two items.

An Actor is a process producing, providing, processing, or consuming information.

MPAI NNT Actors

Four types of Actors are identified as playing a traceability-related role in the use cases.

• NN owner – the developer of the NN, who needs to ensure that ownership of NN can be claimed.
• NN traceability provider – the developer of the traceability technology able to carry a payload in a neural network or in an inference.
• NN customer – the user who needs the NN owner’s NN to make a product or offer a service.
• NN end-user – the user who buys an NN-based product or subscribes to an NN-based service.

Examples of Actors are:

• Edge-devices and software
• Application devices and software
• Network devices and software
• Network services

MPAI NNT Use cases

MPAI use cases relate to both the NN per se (i.e., to the data representation of the model) and to the inference of that NN (i.e., to the result produced by the network when fed with some input data), as illustrated in Figure 1.

Figure 1: Synopsis of NNT generic use cases: Identify the ownership of an NN, Identify an NN (e.g. DOI) and Verify integrity of an NN

The NNW-TEC use cases document is available; it includes sequence diagrams describing the positions and actions of the four main Actors in the workflow.

MPAI NNT Service and application scenarios

MPAI NNT is relevant for services and applications benefitting from one or several conventional NN tasks such as:

1. Video/image/audio/speech/text classification
2. Video/image/audio/speech/text segmentation
3. Video/image/audio/speech/text generation
4. Video/image/audio/speech decoding

Figures 2, 3 and 4 present three typologies of services and applications aggregating the generic use cases presented above.

The first example (Traceable newsletter service – Figure 2) covers the case where an end-user subscribes to a newsletter that is produced by a Generative AI service (provided by an NN customer), according to the end-user profile. In such a use case, a malicious user might try to temper the very production of the personalized content or to modify it during its transmission.

The second example (Autonomous vehicle services – Figure 3) deals with the traceability and authenticity of the multimodal content that is exchanged in various ways: (1) the car A (acting as an NN end-user) sends to a server (acting as an NN customer or owner) acquired signals for data-processing, (2) An embedded AI transmits instructions such as braking, turning, or accelerating to the car (NN owner and end-user), (3) Another vehicle B in the environment transmits environmental information to vehicle A. Various types of malicious attacks with critical consequences can be envisaged: AI interception and corruption (e.g. adversarial learning), les données can be corrupted in their transmission from and/or to the autonomous vehicle (forced connection interruption or data modification).

The third example (AI generated or processed information services – Figure 4) shows how NNT can be beneficial when real images are modified by a deepfake process. A user capturing a video sequence by a connected camera would like to appear as the archetype secret agent (say, James Bond) by interacting with a generative AI service remotely accessible in the network. This module synthesizes novel audiovisual content, which is then rendered on a large display for the user to enjoy. Such services are not immune from security threats: the attacker can intercept the encoded stream prior to its arrival at the trusted AI server and processed is through a malicious edge-deployed generative AI or it can affect the very trusted generative AI service (e.g. by employing some adversarial training techniques).

Figure 2: Autonomous vehicle services

Figure 3: AI-generated newsletter example

Figure 4: AI generated or processed information services