On its fourth anniversary MPAI publishes the first specification supporting metaverse interoperability

Geneva, Switzerland – 30^th September 2024. MPAI – Moving Picture, Audio and Data Coding by Artificial Intelligence – the international, non-profit, and unaffiliated organisation developing AI-based data coding standards – has concluded its 48^th General Assembly (MPAI-48) by approving for publication

1. The combined MPAI Metaverse Model – Architecture (MMM-ARC) V1.2 and Technologies (MMM-TEC) V1.0 standards
2. The set of Multimodal Conversation (MPAI-MMC) V2.2; Object and Scene Description (MPAI-OSD) V1.1; Portable Avatar Format (MPAI-PAF) V1.2; and Data Types, Formats, and Attributes (MPAI-TFA) V1.0 standards.
3. The Open-Source Reference Implementation of the Television Media Analysis (OSD-TMA) V1.1 Use case.

All new standards are released using the MPAI full web-based publication method providing the specification and, where available, the reference software, the conformance testing, and the performance assessment.

MPAI-48 also learned that four MPAI standards have been adopted as IEEE standards: two new (MPAI-MMM and MPAI-PAF) and two revised (MPAI-AIF and MPAI-MMC) bringing the total to eight.

The combined Technical Specification: MPAI Metaverse Model (MPAI-MMM) –  Architecture (MMM-ARC) V1.2 and Technical Specification: MPAI Metaverse Model (MPAI-MMM) –  Technologies (MMM-ARC) V1.0 specify five types of Processes operating in an M-Instance, thirty Actions that Processes can perform, 65 Data Types and their Qualifiers that Processes can Act on to achieve client and M-Instance interoperability.

Technical Specification: Multimodal Conversation (MPAI-MMC) V2.2 specifies 23 Data Types that enable more human-like and content-rich forms of human-machine conversation, applies them to eight use cases in different domains and specifies 23 AI Modules. MPAI-MMC reuses Data Types and AI Modules from other MPAI standards.

Technical Specification: Object and Scene Description (MPAI-OSD) V1.1 specifies 27 Data Types enabling the digital representation of spatial information of Audio and Visual Objects and Scenes, applies them to the Television Media Analysis (OSD-TMA) use case and specifies 15 AI Modules. MPAI-OSD reuses Data Types and AI Modules from other MPAI standards.

Technical Specification: Portable Avatar Format (MPAI-PAF) V1.2 specifies five Data Types enabling a receiving party to render a digital human as intended by the sending party, applies them to the Avatar-Based Videoconference Use Case, and specifies 13 AI Modules. MPAI-PAF reuses Data Types and AI Modules from other MPAI standards.

Technical Specification: Data Types, Formats, and Attributes (MPAI-TFA) V1.0 specifies Qualifiers – a Data Type containing Sub-Types, Formats, and Attributes – associated to “media” Data Types – currently Text, Speech, Audio, Visual, 3D Model, and Audio-Visual – that facilitate/enable the operation of an AI Module receiving a Data Type instance. MPAI-48 has published Version 1.1 of the standard with Data Qualifiers for the MPAI Metaverse Model.

The reference software implementation of the Television Media Analysis (OSD-TMA) V1.1 Use Case produces a description of a TV program that includes the audio and visual objects, the IDs of the speakers and the faces with their space and time information, and the text of the speaker utterances.

Online presentations of MPAI-MMC, MPAI-MMM, MPAI-OSD, MPAI-PAF, and the reference software implementation of OSD-TMA will be made according to the following timetable:

MPAI is continuing its work plan that involves the following activities:

1. AI Framework (MPAI-AIF): building a community of MPAI-AIF-based implementers.
2. AI for Health (MPAI-AIH): developing the specification of a system enabling clients to improve models processing health data and federated learning to share the training.
3. Context-based Audio Enhancement (CAE-DC): developing the Audio Six Degrees of Freedom (CAE-6DF).
4. Connected Aonomous Vehicle (MPAI-CAV): developing the new MPAI-CAV Technologies (CAV-TEC) part of the standard and updating the Architecture part.
5. Compression and Understanding of Industrial Data (MPAI-CUI): developing use cases and functional requirements for MPAI-CUI V2.0 supporting a wide range of corporate risks.
6. End-to-End Video Coding (MPAI-EEV): video coding using AI-based End-to-End Video coding.
7. AI-Enhanced Video Coding (MPAI-EVC): working on a Call for Technologies for up-sampling filter for video.
8. Governance of the MPAI Ecosystem (MPAI-GME): working on version 2.0 of the Specification.
9. Human and Machine Communication (MPAI-HMC): developing reference software.
10. Multimodal Conversation (MPAI-MMC): Developing technologies for more Natural-Language-based user interfaces capable of handling more complex questions.
11. MPAI Metaverse Model (MPAI-MMM): extending the MPAI-MMM specs to support more applications.
12. Neural Network Watermarking (MPAI-NNW): studying the use of fingerprinting as a technology for neural network traceability.
13. Object and Scene Description (MPAI-PAF): studying applications requiring more space-time handling applications.
14. Portable Avatar Format (MPAI-PAF): studying more applications using digital humans needing new technologies.
15. AI Module Profiles (MPAI-PRF): specifying which features AI Workflow or more AI Modules need to support.
16. Server-based Predictive Multiplayer Gaming (MPAI-SPG): developing technical report on mitigation of data loss.
17. Data Types, Formats, and Attribes (MPAI-TFA) extending the standard to data types used by MPAI standards (e.g., aomotive and health).
18. XR Venues (MPAI-XRV): developing the standard for improved development and execion of Live Theatrical Performances and studying the prospects of Cillaborative Immersive Laboratories.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribe to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and Yoube.