In parts I and II of this series of posts, we have highlighted:

1. The basic elements that enable operation of an M-Instance, especially Processes, Items, Actions and Data Type. In particular, Processes can take the shape of a User (a representative of a human in an M-Instance), a Device (to enable the connection of an M-Instance with the real world, called Universe), and a Service.
2. The functional requirements of an initial list of Actions that enable a Process to do useful things in an M-Instance.

We are now going to identify the functional requirements of an initial list of Items that enable a Process to do useful things in an M-Instance. For convenience, Items will be grouped in classes.

Remembers the online presentations at 8 and 15 UTC on 23 June where you can know more about the MPAI Metaverse Model Architecture and the Call for Technologies. Register here for the first and here for the second presentation.

Here are Items with a general applicability.

Here are Items related to the interaction between Processes.

Here are Items related to the use of an M-Instance.

Here are Items with a financial impact.

Here are Items specifically used to access a group of Services.

Here are Items producing a perceptible experience.

Here are Item with a spatial impact.

Of course, more Items can be identified but those introduced above have been tested to cover a significantly large number of use cases in a variety of application domains.

In part I of this series of posts, we have highlighted the basic elements that enable operation of an M-Instance, especially Processes, Items, Actions and Data Type. In particular, Processes can take the shape of a User (a representative of a human in an M-Instance), a Device (to enable the connection of an M-Instance with the real world, called Universe), and a Service.

We are now going to identify the functional requirements of an initial list of Actions that enable a Process to do useful things in an M-Instance.

Of course, more Actions can be identified but those introduced above have been tested to cover a significantly large number of use cases in a variety of application domains.

To know more about the MPAI Metaverse Model Architecture and the Call for Technologies, join the online presentations at 8 and 15 UTC on 23 June. Register here for the first and here for the second presentation.

This is the first of a series of posts that illustrate the Call for Technologies: MPAI Metaverse Model – Architecture, a document inviting interested parties to submit comments to and proposals for Use Cases and Functional Requirements: MPAI Metaverse Model – Architecture. The goal is to facilitate the task of those who wish to contribute to the first “Metaverse Architecture” standard ever attempted by a standards body.

Before starting, let’s clarify why is MPAI, a standards body developing standards for AI-based data coding, engaged in the “metaverse”? The answer is manifold:

1. The metaverse will require a range of technologies that deal with data and their transformations, i.e., data coding.
2. The metaverse is thus and excellent source of valuable standard projects.
3. MPAI has already a few standards that respond to specific metaverse needs.
4. The metaverse is thus also an excellent technology integration platform.

Note that the word metaverse is used here to mean the “metaverse notion” while Metaverse Instance (M-Instance) indicated a “specific implementation” of the MPAI Metaverse Model Architecture (in the following: MPAI-MMM). An M-Instance is considered as a set of Processes providing some or all the following functions:

1. To sense data from U-Locations.
2. To process the sensed data and produce Data from the sensed data and/or autonomously.
3. To produce one or more M-Environments populated by Objects that can be either digitised or virtual, the latter with or without autonomy.
4. To process Objects from the M-Instance or potentially from other M-Instances to affect U- and/or M-Environments using Object in ways that are:
   • Consistent with the goals set for the M-Instance.
   • Effected within:
     • The capabilities of the M-Instance
     • The Rules set for the M-Instance.

The above gives the opportunity to highlight the convention that word beginning with a capital letter are defined here while those beginning with a small letter have the normal meaning of the context.

Some terms are more important than others, so let’s report a few of them here:

1. A Process is Program and Metadata that can be executed in the M-Instance to perform Actions on Items.
2. An Action is a supported Functionality that is performed in an M-Instance.
3. An Item is Data and Metadata supported by the M-Instance where the Item exists.
4. Metadata adds information on a Process of an Item and may include Rights.
5. Rights define:
   • The ability of a Process to perform Actions on Items.
   • The possibility that an Item be subjected to an Action by a Process.
6. An Item may include Rights held by User and Rights that it may be possible to acquire on the Item.
7. Data Types are data referenced by Actions and Items.

We are now able to list the first Functionalities of an M-Instance:

An M-Instance is a set of Processes providing some or all the following functions:

1. Senses data from U-Location.
2. Produces Items autonomously or by processing the sensed data.
3. Hosts one or more M-Environments populated by Objects that can be either digitised or virtual, the latter with or without autonomy.
4. Processes Objects from the M-Instance or potentially from other M-Instances to affect U- and/or M-Environments using Objects in ways that are:
   • Consistent with the goals set for the M-Instance.
   • Effected within the capabilities and Rules of the M-Instance, and in accordance with applicable laws and regulations.
5. Identifies Processes and Items with one or more than one Identifier each of which uniquely refers to one Process or Item.
6. May contain one or more M-Environments each of which:
   • Includes an Identifier.
   • May include M-Locations with space and time attributes.
   • May require a Registration specific to the M-Environment.
7. May make available information regarding its Capabilities.
8. May require Registration for use:
   • A human can request to deploy one or more Users and one or more Personae in an M-Instance.
   • An M-Instance may request a subset of the Personal Profile of the Registering human.
9. Establishes Rules that human’s Users in the M-Instance shall comply with.
10. May penalise a User for lack of compliance with the Rules.

MPAI-MMM – Architecture identifies the following types of Process (see Figure 1):

1. User represents a human rendered as:
   • A Model (Persona) animated by a stream generated by the human or by an autonomous agent. A User may be rendered by one or more Personae.
   • An Object rendering the human.
2. Device connects User with a human or a U-Location:
   • From the Universe to an M-Instance: captures a scene as Media and Provides Media as Data and Metadata:
   • From an M-Instance to the Universe: receives an Entity and renders it as Media with a Spatial Attitude.
3. Service provides Functionalities.
4. App is a Program executed on a Device.

Figure 1 – Relationship of Human-Device-User-Service-Persona

Figure 1 highlights a few basic aspects:

1. A human is connected to one of its potentially many digital counterparts (Users).
2. An object has one or more digital correspondents (Objects).
3. A User can be rendered as a Persona (possibly more than one).
4. Processes, in particular Users, can interact with one another.

Processes play an important role. Here are some features of a Process:

1. Performs Actions on Items if it has the Rights to do that.
2. Can make available information about its Capabilities.
   • The Actions it can Perform.
   • The Items on which Actions can be performed.
   • The time during which they can be performed.
   • The M-Locations where they can be performed.
3. Can request another Process to perform Actions on Items by transmitting to it a Request-Action Item.
4. Can be requested to perform an Action and it does so if:
   • The requesting Process has the Rights required to perform that request, e.g., it has made a Transaction to acquire the Rights, the Rights is part of the set of Rights assigned at Registration time, etc.
   • The requested Process has the Rights to perform the requested Action on the Item.
5. Can respond to the Process requesting an Action with a Response-Action Item (see Figure 2).
6. Uses a supported format:
   • To request another Process to perform Actions on Items (Request-Action).
   • To respond to another Process that has requested an Action (Respond-Action).
7. May perform, or to request other Processes to perform, Actions on Items even in the absence of Rights, if the Rules so allow.
8. May need to be certified by the M-Instance Manager for use in an M-Instance.

Figure 2 – Processes requesting Action and responding to request

An M-Instance is typically administered by a single Manager that makes decisions about the technologies that fit the best with the entity’s needs. Two independent Managers need not make the same technology choices. The following workflow enables interoperability between Metaverse_A and Metaverse_B when Process_A in Metaverse_A requests Process_B in Metaverse_B to perform Action on an Item_A.1, the following (Note: RS=Resolution Service, CS=Conversion Service).

1. Process_A transmits Request-Action₁ to R­S_A.
2. RS_A transmits Request-Action₁ to RS_B.
3. RS_B transmits Item₁ to CS.
4. CS produces and transmit Item₂ containing Converted Data to RS_B.
5. RS_B transmits the new Request-Action₂ to Process_B.
6. Process_B
   • Performs the Action specified in Request-Action₂ using Item_A.2.
   • Produces Response-Action₂.
   • Requests RS_B to transmit to RS_A Response-Action₂ containing Item_A.3 (result of performing Request-Action_A.2).
   • RS_B transmits Response-Action₂ to RS_A.
   • RS_A transmits Item₃ to CS.
   • CS produces and transmits to RS_A Item₄, corresponding to Item_A.3 with converted Data.
   • RS_A produces and transmits to Process_A a new Response-Action₄ that references Item_A.4.

It should be noted that an M-Instance may allow Processes to communicate directly with Processes in other M-Instances without calling ResolutionService_A.

Given the above, what is then the scope of the MPAI-MMM Architecture Call for Technologies?

To make comments on, add functionalities to, or propose new elements to the MPAI Metaverse Model Architecture.

Join the online presentation on 23 June at 08 and 15 UTC. Register here for the first or here for the second presentation.

Geneva, Switzerland – 14 June 2023. Today, the international, non-profit, and unaffiliated Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) organisation developing AI-based data coding standards has concluded its 33^rd General Assembly (MPAI-33) approving the Call for Technologies: MPAI Metaverse Model (MPAI-MMM) – Architecture. Responses are due by 10 July. Two online presentations of the Call will be made on 23 June at 8 and 15 UTC.

After publishing two Technical Reports on Functionalities and Functionality Profiles of the MPAI Metaverse Model, MPAI is now kicking off an ambitious plan to develop a Technical Specification on MPAI Metaverse Model – Architecture. This is a project that no standards body has ever attempted so far.

The first step of the plan is the publication of the Call for Technologies, as mandated by the MPAI standard development process. Note that the Call does not address data formats, only metaverse functionalities.

Three documents are attached to the Call: the first is Use Cases and Functional Requirements. It includes a reference to some thirty metaverse use cases explored by MPAI, a set of Functionalities that the Architecture should provide, and the functional requirements of its key elements: Processes, Items, Actions and Data Types.

The second document is the Framework Licence designed to facilitate the timely access to IP that is essential to implement the planned MPAI-MMM – Architecture standard. Finally, the third document is a Template for responses that respondents to the Call may wish to use in their responses.

Anybody may respond to the Call. However, non-members should join MPAI to participate in the development of the MPAI-MMM – Architecture standard.

MPAI is continuing its work plan comprising the development of the following Technical Specifications:

1. The AI Framework (MPAI-AIF) V2 Technical Specification will enable an implementer to establish a secure AIF environment to execute AI Workflows (AIW) composed of AI Modules (AIM).
2. The Avatar Representation and Animation (MPAI-ARA) V1 Technical Specification will support creation and animation of interoperable human-like avatar models able to understand and express a Personal Status.
3. The Multimodal Conversation (MPAI-MMC) V2 Technical Specification will generalise the notion of Emotion by adding Cognitive State and Social Attitude and specify a new data type called Personal Status.

The MPAI work plan also includes exploratory activities, some of which are close to becoming standard or technical report projects:

1. AI Health (MPAI-AIH). Targets an architecture where smartphones store users’ health data processed using AI and AI Models are updated using Federated Learning.
2. Connected Autonomous Vehicles (MPAI-CAV). Targets the Human-CAV Interaction Environment Sensing, Autonomous Motion, and Motion Actuation subsystems implemented as AI Workflows.
3. End-to-End Video Coding (MPAI-EEV). Extends the video coding frontiers using AI-based End-to-End Video coding.
4. AI-Enhanced Video Coding (MPAI-EVC). Improves existing video coding with AI tools for short-to-medium term applications.
5. Server-based Predictive Multiplayer Gaming (MPAI-SPG). Uses AI to train neural networks that help an online gaming server to compensate data losses and detects false data.
6. XR Venues (MPAI-XRV). Identifies common AI Modules used across various XR-enabled and AI-enhanced use cases where venues may be both real and virtual.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

1. Introduction

Many use the metaverse word with other people, but it is unlikely that they all mean the same. In general one can say that a metaverse instance is a rather complex communication and interaction environment with features, such as synchronous and persistent experiences and virtual reality features such as avatars that may or may not be controlled by humans or objects of the real world.

MPAI Metaverse Model – MPAI-MMM – is the MPAI project developing technical documents – so far Technical Specifications – that can be applied to as many kinds of metaverse instances as possible and enable varied metaverse implementations to interoperate.

The first document, Technical Report – MPAI Metaverse Model – Functionalities, collects the functionalities that potential metaverse users expect a metaverse instance to provide, rather than trying to define what the metaverse is. It includes definitions, assumptions guiding the project, potential sources of functionalities, an organised list of commented functionalities, and an analysis of some of the main technology areas underpinning the development of the metaverse.

The MPAI-MMM is based on the idea of using the notion of Profiles and Levels that digital media standardisation has successfully employed for three decades to cope with the wide variety of expected application domains. As some metaverse technologies are not yet available, the second document, Technical Report – MPAI Metaverse Model – Functionality Profiles, develops Functionality Profiles, a new notion in standardisation that defines profiles for what they do (“functionalities”) rather than for how they do it (“technologies”).

The second document reaches another important milestone by:

1. Extending the existing collection of definitions.
2. Developing a functional metaverse operation model based on Sources requesting Destinations to perform Actions on Items both containing Data Types.
3. Specifying the Actions that Sources request Destinations to perform on Items and the responses of Destinations.
4. Specifying the Metadata of the Items but not their Data Formats, in line with the Functionality approach.
5. Developing nine Use Cases to test the suitability of Actions and Items.
6. Developing four Functionality Profiles.
7. MPAI-MMM Functional Operation Model

As it is hard to describe the many terms defined in the document, we will rely on the common meaning of the words. When in doubt about the meaning of a term (starting with a capital letter), please use the search window.

Figure 1 shows a simple example of the connection between the real world (right-hand side, called Universe) and the representations of U-Environments in M-Instances on the left-hand side. Green indicates that the User/Objects represents real-world humans/objects. Users are visualised as Personae: light blue indicates that a Persona or Object is driven by an autonomous agent and brick red that the Persona moves according to its real twin’s movements.

Figure 1 – An example of Metaverse Scenario

An M-Instance is populated by Processes, e.g., a real or virtual Persona is driven by a Process. A Process may request another Process to perform an Action by sending it a Request-Action and receiving a Response-Action. The Request-Action is an Item, i.e., Data and Metadata, possibly with Rights. The Item contains the Time the request was issued and Source Process, Destination Process, Action requested, InItems provided as input and their InLocations, OutLocations of the output Items, and requested OutRights to Act on the produced Items.

Figure 2 – Processes interacting within and without M-Instances

So far, the following elements have been identified and specified:

1. 4 Processes: App, Device, Service, and User.
2. 27 Actions, such as Authenticate an Entity (an Item that can be perceived), Discover (request a Service to find Items responding to certain criteria), MM-Embed (place and make perceptible an Entity at an M-Location), UM-Animate (animate an Entity with data from the real world), etc.
3. 33 Items such as Account, Asset (an Item that can be Transacted), Map (a list of connections between U-Locations and M-Locations), Model (a representation of an object ready to be UM-Animated by a Stream or to be MM-Animated by an autonomous agent), Rights (a description of what Actions can be done on an Item), etc.
4. 13 Data Types such as, Currency, Emotion, Spatial Attitude, Time, etc.

3. Use Cases

Nine use cases have been developed. Here a simple use case showing the descriptive capabilities of the MPAI-MMM scene description language.

Figure 3 – The Virtual lecture use case

Here is a description of the workflow.

1. The meeting manager authors and embeds a virtual classroom.
2. The student
   1. Connects its place in the M-Instance (“home”) with the place where the human is.
   2. Pays for the right to attend the lecture and save the Experience of the lecture.
   3. Places its Persona in the virtual classroom and stops the rendering of the Persona at home.
3. The teacher
   1. Does likewise (but does not pay),
   2. Places a 3D Model used in the lecture and animates it.
4. The student
   1. Moves close to the teacher’s desk without changing the display of its Persona to feels the audio, visual, and haptic components of the 3D Model.
   2. Saves the lecture how they experienced it.
5. The meeting manager pays lecture fees to the teacher.
6. Both student and teacher go back home.

The other use cases are: Virtual Meeting, Hybrid Working, eSports Tournament, Virtual Performance, AR Tourist Guide, Virtual Dance, Virtual Car Showroom, and Drive a Connected Autonomous Vehicle.

4. Functionality Profiles

The structure of the Metaverse Functionality Profiles is derived from the Use Cases and includes hierarchical Profiles and independent Profiles. Profiles may have Levels. As depicted in Figure 3, the currently identified Profiles are Baseline, Management, Finance, and High. The currently identified Levels for Baseline, Management, and High Profiles are Audio only, Audio-Visual, and Audio-Visual-Haptic. The Finance Profile does not have Levels.

Figure 4 – The currently identified Functionality Profiles

5. What is next

MPAI has now laid down the basic elements and can start from the development of the Technical Specification – Metaverse Architecture. This will contain the main components of an M-Instance, their interconnections and the types of data exchanged. It will also contain the APIs called by the Processes to enable implementation of M-Instances.

Abstract

1       Introduction. 1

2       A functional operation model 2

3       Actions. 3

4       Items. 5

5       Data Types. 6

6       Use cases. 7

6.1        AR Tourist Guide. 7

6.2        Virtual Dance. 8

7       Initial functionality profiles. 10

8       Conclusions. 11

9       References. 11

Abstract

MPAI has developed a roadmap for Metaverse interoperability. The published Technical Report – MPAI Metaverse Model – Functionalities [1] claims that, as standards for a market as vast as the one expected for the Metaverse are difficult to develop, functional profiles should be developed. The published draft Technical Report – MPAI Metaverse Model – Functionality Profiles [2] develops a Metaverse functional operation model, applies it to 7 Use Cases and proposes 4 initial functionality profiles. This document is a concise summary of [2].

1        Introduction

The MPAI Metaverse Model (MPAI-MMM) aims to provide Technical Reports and Technical Specifications that apply to as many kinds of Metaverse instances as possible and enable varied Metaverse implementations to interoperate.

At present, achieving interoperability is difficult because:

1. There is no common understanding of what a Metaverse is or should be, in detail.
2. There is an abundance of existing and potential Metaverse Use Cases.
3. Some independently designed Metaverse implementations are very successful.
4. Some important technologies enabling more advanced and even unforeseen forms of the Metaverse may be uncovered in the next several years.

MPAI has developed a roadmap to deal with this challenging situation. The first milestone of the roadmap is based on the idea of collecting the functionalities that potential Users expect the Metaverse to provide, instead of trying to define what the Metaverse is. The first Technical Report of this roadmap [1] includes definitions, makes assumptions for the MPAI-MMM project, identified sources that can generate functionalities, develops an organised list of commented functionalities, and analyses the main enabling technologies. Reference [3] provides a summary of [1].

Potential Metaverse Users with different needs might require different technologies to support these needs. Therefore, an approach that tried to achieve the goal of making every M-Instance be able to interoperate with every other M-Instance would force implementers to take technologies on board that are useless for their needs and potentially costly.

Reference [1] posits that Metaverse standardisation should be based on the notion of Profiles[1] and Levels[2] successfully adopted by digital media standardisation. A Metaverse Standard that includes Profiles and Levels would enable Metaverse developers to use only the technologies they need that are offered by whatever profile is most suitable to them.

The notion of profile can mitigate the impact of having many disparate Metaverse Users with diverse requirements. Unfortunately, that notion cannot be currently implemented because some key technologies are not yet available and at this time it is unclear which technologies, existing or otherwise, will eventually be adopted. To cope with this situation, [2] only targets Functionality Profiles, i.e., profiles that are defined by the functionalities they offer, not by technologies implementing them. Functionality Profiles are not meant to fully address the interoperability problem, but rather to allow a technology-independent definition of profiles based on the functional value they provide rather than on the “influence” of specific technologies.

2        A functional operation model

A Metaverse instance – called M-Instance – is composed of interacting Processes within and without the M-Instance. An M-Instance interoperates with another M-Instance to the extent its Processes interoperate.

MPAI has identified 3 classes of Process:

1. Users: Processes that represent humans using data from the real world (U-Locations) or autonomous agents (both need not be human-like).
2. Devices: Processes interconnecting U-Locations to M-Locations and vice-versa.
3. Services: Processes providing Functionalities.

A Process performs or requests another Process to perform Actions on Items. Item is data with attached metadata possibly including Rightss enabling a Process to perform Actions on. Rights express the ability to perform an Action on an Item.

The names of some Actions are prefixed depending on where Action is performed:

1. MM- if the interAction takes place in Metaverse.
2. UM- if the interAction is between Universe to Metaverse (Universe is the real world).
3. MU- if the interAction is between Metaverse to Universe.

An interAction of Process #1 with Process #2 unfolds as follows:

1. Process #1 requests Process #2 to performs Actions on Items.
2. Process #2 executes the request if Process #1 has Rightss to call Process #2.
3. Process #2 responds to Process #1’s request.

Note that Processes and Items need not be in the same M-Instance. However, the possibility of having Actions performed on Items may be more limited if they are not in the same M-Instance.

Figure 1 depicts a Metaverse configuration with interacting Processes.

Figure 1- A Metaverse configuration

Devices connect humans and objects in a U-Location (a location in the real world – called Universe) to one or more Metaverse Service generating M-Instances as depicted in Figure 2.

Figure 2 – The relationship between the real world (Universe) and M-Instances

A Device UM-Captures scenes with its Sensors and MU-Render Entities (i.e., Item that can be perceived) though their Actuators.

To enable its User(s) to perform Actions in an M-Instance, a human may be asked to Register and provide a subset of their User data, e.g., Persona(e) and ID(s) of Wallet. The M-Instance provides an Account that univocally associates a Registered human with their Items with the following features:

1. A human may have more than one Account in more than one M-Instance.
2. A User has Rightss to act in the M-Instance associated with the human’s Account.
3. A human may have more than one Account and more than one User per Account.
4. A User exists after a human Registers with an M-Instance.
5. Different Users of an Account may have different Rightss.

The Rules of an M-Instance express the obligations undertaken by the Registering human represented by the User and the terms and conditions under which a User exists in an M-Instance and operates either there or in another M-Instance. Depending on the Rules, a User of a human Registered on an M-Instance may or may not interact with another M-Instance and Rightss enforcement on some Actions performed on some Items may be forfeited.

Data entering an M-Instance, e.g., by Reading or MM-Capturing may include metadata and the Rightss granted to a Process to perform Actions on the data.

Items bear an Identifier is uniquely associated to that Item. However, an Item may bear more than one identifier. It is assumed that identifiers have the following structure:

[M-InstanceID] [ItemID]; [M-InstanceID] [M-LocationID] [ItemID].

3        Actions

A User can call a Process to perform Actions on Entities. An Entity can be:

1. Authored, i.e., the User calls an Authoring Tool Service with an accompanying request to obtain Rights to act on the authored Entity.
2. MM-Added, i.e., the User requests that an Entity be added to an M-Location with a Spatial Attitude.
3. MM-Enabled, i.e., the User requests that a Process be allowed to MM-Capture an Entity that is MM-Added at an M-Location.
4. MM-Embedded, i.e., MM-Added and MM-Enabled in one stroke.
5. MM-Captured, i.e., the User requests that an Entity MM-Embedded at an M-Location be MM-Sent to a Process.
6. UM-Animated, i.e., the User requests that a Process change the features of an Entity using a Stream of data.
7. MM-Disabled, i.e., the User requests that the MM-Enabling of the Entity be stopped.
8. Authenticated, i.e., the User calls a Service to make sure that an Item is what it claims to be.
9. Interpreted, i.e., the User calls a Service to obtain an interpretation of Items in an M-Instance, e.g., translate a Speech Object MM-Embedded at an M-Location into a specific language.
10. Informed, i.e., the User calls a Service to obtain data about an Item.

An example of composite Action, i.e., an Action that involves a plurality of Actions and a plurality of Items is Track that enables a User to request Services to:

1. MM-Add a Persona at an M-Location with a Spatial Attitude.
2. UM-Animate the Persona MM-Added at an M-Location.
3. MU-Render specified Entities at the M-Location to a U-Location with Spatial Attitudes.

So far, the following Actions have been identified:

Of course, the Actions identified are not intended to be the only ones that will be needed by all M-Instances.

A Process requests another Process to perform an Action sending the following payload.

The requested Process will respond by sending the following payload:

4        Items

An Item can be:

1. Created: from data and metadata. Metadata may include Rights.
2. Changed: its Rightss are modified.
3. Discovered by calling an appropriate Service.
4. Written, i.e., stored.
5. Posted as an Asset (i.e., an Item that can be the object of a Transaction to a Marketplace) to a marketplace.
6. Transacted as an Asset.

An Item can belong to one of six categories:

1. Items characterised by the fact that they can be MM-Captured by a User.
2. Items that can cause an Entity to change its perceptible features.
3. Items that have space and time attributes.
4. Items that are finance related.
5. Items that are non-perceptible.
6. Items that are Process-related.

Entities are Items that can be perceived. Here are some relevant Items.

1. Event: the set of Entities that are MM-Embedded at an M-Location from Start Time until End Time.
2. Experience: An Event as a User MM-Captured it and the User’s Interactions with the Entities belonging to the Entity that spawned the Event.
3. Object: the representation of an object and its features. Currently, only the following object types are considered: Audio, Visual, and Haptic.
4. Model: An Object that can be UM-Animated by a Stream or a Process. A Persona is the Model of a human.
5. Scene: a dynamic composition of Objects described by Times and Spatial Attitudes.

The finance-related Items are:

1. Asset: An Item that may be the object of a Transaction and is embedded at an M-Location or Posted to a Service.
2. Ledger: the list of Transactions executed on Assets.
3. Provenance: the Ledger of an Asset.
4. Transaction: Item representing the changed state of the Accounts and the Rights of a seller User and a buyer User on an Asset and optionally of the Service facilitating/enabling the Transaction.
5. Value: An Amount expressed in a Currency.
6. Wallet: A container of Currency units.

So far, the following Items have been identified:

Of course, more Items will be identified as more application domains will be taken into consideration.

Each Item is specified by the following table. It should be noted that, in line with the assumptions made by the MPAI Metaverse roadmap and the current focus on functionalities, the formats of the Item Data are not specified.

5        Data Types

Data Types are data that are referenced in Actions or in Items. Currently, the following data types have been identified.

1. Address
2. Amount
3. Coordinates
4. Currency
5. Personal status
   1. Cognitive state
   2. Emotion
   3. Social attitude
6. Point
7. Spatial attitude
   1. position
   2. orientation
8. Time

6        Use cases

So far 7 Use Cases have been developed to verify the completeness of Actions, Items and data types specified. Use cases are also a tool that facilitates the development of functionality profiles.

To cope with the fact that Use Cases may involve several locations and Items, a notation has been developed exemplified by the following:

1. User_i MM-Embeds Persona₁, Persona_i.2, etc.
2. User_i Calls Process₁, Process_i.2, etc.
3. User_i MM-Embeds Persona_j, at M-Location_i.1, M-Location_i.2, etc.
4. User_i MU-Renders Entity_j at U-Location_i.1, U-Location_i.2, etc.
5. User_i MM-Sends object₂ to User_j.

The following is to be noted:

Here only two of the seven Use Cases of [2] will be presented.

6.1       AR Tourist Guide

6.1.1      Description

This Use Case describes a human who intends to develop a tourist application through an App that alerts the holder of a smart phone where the App is installed and lets them view Entities and talk to autonomous agents residing at M-Locations:

1. User₁
   1. Buys M-Location₁ (parcel) in an M-Environment.
   2. Creates Entity₁ (landscape suitable for a virtual path through n sub-M-Locations).
   3. Embeds Entity₁ (landscape) on M-Location_1.1 (parcel).
   4. Sells Entity₁ (landscape) and M-Location_1.1 (parcel) to a User₂.
2. User₂
   1. Authors Entity₁ to Entity_2.n for the M-Locations.
   2. Embeds the Entities at M-Location₁ to M-Location_2.n.
   3. Sells the result to User₃.
3. human₄
   1. Develops
      1. Map recording the pairs M-Location_i – U-Location_2.i
      2. App alerting a human₅ holding the Device with the App installed that a key U-Location has been reached.
   2. Sells Map and App to human₃.
4. User₃ MM-Embeds one or more autonomous Personae at M-Location₁ to M-Location_2.n.
5. When human₅ gets close to a key U-Location:
   • App prompts Device to Request User₃ to MU-Render the Entity_i MM-Embedded at M-Location_2.i to the key U-Location_2.i.
   • human₅ interacts with MU-Rendered Entity_i that may include an MM-Animated Persona_2.i.

6.1.2      Workflow

Table 1 – AR Tourist Guide workflow.

6.1.3      Actions, Items, and Data Types

6.2       Virtual Dance

6.2.1      Description

1. User₂ (dance teacher)
   • Teaches dance at a virtual classroom.
   • Works at M-Location₁ where its digital twin Persona_2.1 is Audio-Visually MM-Embedded.
   • MM-Embeds and MM-Animates Persona₂ (A-V) (another of its Personae) at M-Location_2.2 as virtual secretary to attends to students coming to learn dance.
2. User₁ (dance student #1):
   • MM-Embeds its Persona₁ (A-V) at Location_1.1 (its “home”).
   • Audio-Visual-Haptically MM-Embeds Persona₁ (A-V-H) at Location_1.2 close to Location_2.2.
   • Sends Object₁ (A) to Persona_2.2 (greets virtual secretary).
3. Virtual secretary:
   • Sends Object₁ (A) to dance students #1 (reciprocates greeting).
   • Send Object₂ (A) to call regular dance teacher’s Persona_2.1.
4. Dance teacher MM-Embeds (A-V-H) Persona₁ at Location_2.3 (classroom) where it dances with Persona_1.1 (dance student #1).
5. While Persona₁ (student #1) and Persona_2.1 (teacher) dance, User₃ (dance student #2):
   • MM-Embeds (A-V) Persona₁ (its digital twin) at Location_3.1 (its “home”).
   • MM-Embeds (A-V-H) Persona₁ to Location_3.2 (close to Location_2.2 where the secretary is located).
6. After a while, User₂ (dance teacher):
   • MM-Embeds (A-V-H) Persona₁ at Location_2.4, (close to Location_3.2 of dance student #2).
   • MM-Disables Persona₁ from Location_2.3 where it was dancing with Persona_1.1 (student #1).
   • MM-Embeds (A-V-H) and MM-Animates an autonomous Persona₃ replacing Persona_2.1 from Location_2.3 so that student #1 can continue practising dance.
   • Dances with Persona₁ (student #2).

6.2.2      Workflow

Table 2 – Virtual Dance workflow.

6.2.3      Actions, Items, and Data Types

7       Initial functionality profiles

The structure of the Metaverse functionality profiles derived from the above includes hierarchical profiles and one independent profile. Profiles may have levels. As depicted in Figure 2, the currently identified profiles are baseline, management, finance, and high. The currently identified levels for baseline, management, and high profiles are audio only, audio-visual, and audio-visual-haptic.

Figure 3 – the currently identified Functionality Profiles

As an example, the baseline functionality profile enables a human equipped with a Device to allow their Users to:

1. Author Entities, e.g., object models.
2. Sense a scene at a U-Location:
   • UM-Capture a scene.
   • UM-Send data.
3. MM-Embed Personae and objects:
   • MM-Add Persona and object.
   • UM-Animate a Persona, with a stream (UM-Animate) or using a Process (MM-Animate).
4. MU-Render an Entity at an M-Location to a U-Location.
5. MM-Capture Entities at an M-Location
6. MM-Disable an Entity.

This Profile supports baseline lecture, meeting, and hang-out Use Cases. TransActions and User management are not supported.

Table 3 lists the Actions, Entities, and Data Types of the Baseline Functionality Profile.

Table 3 – Actions, Entities, and Data Types of the Baseline Functionality Profile

The identified four profiles serve well the needs conveyed by the identified functionalities. As more functionalities will be added, the number of profiles and potentially of levels, is likely to increase.

8       Conclusions

Reference [2] demonstrates the feasibility of the first two milestones of the proposed MPAI roadmap to Metaverse interoperability. Currently, four functionality profiles supporting the selected functionalities have been identified and specified. As more basic Metaverse elements are added, however, more functional profiles are likely to be found necessary. Functionality profiles can be extended and restructured as more Functionalities will be added.

The next step of the MPAI roadmap to Metaverse interoperability is the development of Technical Specification – MPAI Metaverse Model (MPAI-MMM) – Metaverse Architecture.

9        References

1. MPAI; Technical Report – MPAI Metaverse Model – Functionalities (MPAI-MMM); January 2023; https://mpai.community/standards/mpai-mmm/mpai-Metaverse-model/mmm-functionalities/
2. MPAI; Technical Report – MPAI Metaverse Model – Functionality Profiles (MPAI-MMM); March 2023; https://mmm.mpai.community/
3. MPAI; A roadmap to Metaverse Interoperability; February 2023; FGMV-I-012

[1] A Profile is sets of one or more base standards and, if applicable, chosen classes, subsets, options, and parameters of those standards that are necessary for accomplishing a particular function.

[2] A Level is a subdivision of a Profile indicating the completeness of the User experience.

1. Introduction

Metaverse is expected to create new jobs, opportunities, and experiences and transform virtually all sectors of human interaction. To harness its potential, however, there are hurdles to overcome:

1. There is no common agreement on what a “metaverse” is or should be.
2. The potential users of the metaverse  are too disparate.
3. Many successful independent implementations of “metaverse” already exist.
4. Some important enabling technologies may be years away.

To seamlessly use metaverses, standards are needed. Before engaging in metaverse standardisation, we should find a way to achieve metaverse standardisation, even without:

1. Reaching an agreement on what a metaverse is.
2. Disenfranchising potential users.
3. Alienating existing initiatives.
4. Dealing with technologies (for now)

2. The MPAI proposal

MPAI – Moving Picture, Audio, and Data Coding by Artificial Intelligence – believes that developing a (set of) metaverse standards is very challenging goal. Metaverse standardisation requires that we should:

1. Start small and grow.
2. Be creative and devise a new working method.
3. Test the method.
4. Gather confidence in the method.
5. Achieve a wide consensus on the method.

Then, we could develop a (set of) metaverse standards.

MPAI has developed an initial roadmap:

1. Build a metaverse terminology.
2. Agree on basic assumptions.
3. Collect metaverse functionalities.
4. Develop functionality profiles.
5. Develop a metaverse architecture.
6. Develop functional requirements of the metaverse architecture data types.
7. Develop a Table of Contents of Common Metaverse Specifications.
8. Map technologies to the ToC of the Common Metaverse Specifications.

Step #1 – develop a common terminology

We need no convincing of the importance of this step as many are developing metaverse terminologies. Unfortunately, there is no attempt at converging to an industry-wide terminology.

The terminology should:

• Have an agreed scope.
• Be technology- and business-agnostic.
• Not use one industry’s terms if they are used by more than one industry.

The terminology is:

• Intimately connected with the standard that will use it.
• Functional to the following milestones of the roadmap.

MPAI has already defined some 150 classified metaverse terms and encourages the convergence of existing terminology initiatives. The MPAI terminology can be found here.

Step #2 – Agree on basic assumptions

Assumptions are needed for a multi-stakeholder project because designing a roadmap depends on the goal and on the methods used to reach it.

MPAI has laid down 16 assumptions which it proposes for a discussion. Here the first 3 assumptions are presented. All the assumptions can be found here.

• Assumption #1: As there is no agreement on what a metaverse is, let’s accept all legitimate requests of “metaverse” functionalities.
  • Note: an accepted functionality does not imply that a Metaverse Instance shall support it.
• Assumptions#2: Common Metaverse Specifications (CMS) will be developed.
  • Note: they will provide the technologies supporting identified Functionalities.
• Assumption#3: CMS Technologies will be grouped into Technology Profiles in response to industry needs.
  • Note: A profile shall maximise the number technologies supported by specific groups of industries.

The notion of profile was well known and used in digital media standardisation and is defined here:

A set of one or more base standards, and, if applicable, their chosen classes, subsets, options and parameters, necessary for accomplishing a particular function.

Step #3 – Collect metaverse functionalities

The number of industries potentially interested in deploying metaverse is very large and MPAI has explored 18 of them. See here. A metaverse implementation is also likely to use external service providers – interfaces should be defined. See here.

MPAI has collected > 150 functionalities organised in: Areas – Subareas – Titles. See here.

This is not an accomplished task, but its beginning. Collecting metaverse functionalities should be a continuous task.

Step #4 – Which profiles?

The notion of profile is not currently implementable because some key technologies are not yet available and it is not clear which technologies, exisiting or otherwise, will eventually be selected.

MPAI proposes to introduce a new type of profile – functionality profile, characterised by the functionalities offered, not by the technologies implementing them. By dealing only with functionalities and not technologies, profile definition is not “contaminated” by technology considerations. MPAI is in the process of developing:

Technical Report – MPAI Metaverse Model (MPAI-MMM) – Functionality Profiles.

It is expected that the Technical Report will be published for Community Comments on the 26^th of March and finally adopted on the 19^th of April 2023. It will contain the following table of contents:

1. A scalable Metaverse Operational Model.
2. Actions (what you do in the metaverse):
   1. Purpose – what the Action is for.
   2. Payload – data to the Metaverse.
   3. Response – data from the Metaverse.
3. Items (on what you do Actions):
   1. Purpose – what the Item is for.
   2. Data – functional requirements.
   3. Metadata – functional requirements.
4. Example functionality profiles.

The Technical Report will not contain nor make reference to technologies.

3. The next steps of the MPAI poposal

The Technical Report will demonstrate that it is possible to develop metaverse functionality profiles (and levels) that do not make reference to technologies, only to functionalities.

Step #5 – Develop a metaverse architecture.

The goal is to specify of a Metaverse Architecture, including the main functional blocks and the data types exchanged between blocks.

Step #6 – Develop functional requirements of the metaverse architecture data types.

The goal is to develop the functional requirements of the data types exchanged between functional blocks of the metaverse architecture.

Step #7 – Develop the Table of Contents of the Common Metaverse Specifications.

The goal is to produce an initial Table of Contents (ToC) of Common Metaverse Specifications to have a clear understanding of which Technologies are needed for which purpose in which parts of the metaverse architecture to achieve interoperability.

Step #8 – Map technologies to the ToC of the Common Metaverse Specifications.

MPAI intends to map its relevant technologies and see how they fit in the Common Metaverse Specification architecture. Other SDOs are invited to join the effort.

4. Conclusions

Of course, step #8 there will not provide the metaverse specifications but a tested method to produce them. MPAI envisages to reach step #8 in December 2023. It is a good price to pay before engaging in a perilous project.

1. Introduction

The metaverse is expected to create new jobs, opportunities, and experiences with transformational impacts on virtually all sectors of human interaction. Therefore, metaverse standards are important, but:

1. There is no common agreement on what a “metaverse” is or should be.
2. There are many potential users of the metaverse.
3. There are many successful independent implementations of “metaverse”.
4. Some important enabling technologies may be years away.

We need to agree on how to tackle metaverse standardisation, without necessarily:

1. Reaching an agreement on what a metaverse is.
2. Disenfranchising potential users.
3. Alienating existing initiatives
4. Dealing with technologies (for now)

2. The MPAI proposal

MPAI – Moving Picture, Audio, and Data Coding by Artificial Intelligence – believes that developing a (set of) metaverse standards is very challenging goal. Metaverse standardiosation requires that we should:

1. Start small and grow.
2. Be creative and devise a new working method.
3. Test the method.
4. Gather confidence in the method.
5. Gather a wide consensus on the method.

Then, we could develop a (set of) metaverse standards.

MPAI has developed an initial roadmap:

1. Build a metaverse terminology.
2. Agree on basic assumptions.
3. Collect metaverse functionalities.
4. Develop functionality profiles.
5. Develop a metaverse architecture.
6. Develop functional requirements of the metaverse architecture data types.
7. Develop a Table of Contents of Common Metaverse Specifications.
8. Map technologies to the ToC of the Common Metaverse Specifications.

Step #1 – develop a common terminology

We need no convincing of the importance of this step as many are developing metaverse terminologies. Unfortunately, there is no attempt at converging to an industry-wide terminology.

The terminology should:

• Have an agreed scope.
• Be technology- and business-agnostic.
• Not use one industry’s terms if they are used by more than one industry.

The terminology is:

• Intimately connected with the standard that will use it.
• Functional to the following milestones of the roadmap.

MPAI has already defined some 150 classified metaverse terms and encourages the convergence of existing terminology initiatives. The MPAI terminology can be found here.

Step #2 – Agree on basic assumptions

Assumptions are needed for a multi-stakeholder project because designing a roadmap depends on the goal and on the methods used to reach it.

MPAI has laid down 16 assumptions which it proposes for a discussion. Here the first 3 assumptions are presented. All the assumptions can be found here.

• Assumption #1: As there is no agreement on what a metaverse is, let’s accept all legitimate requests of “metaverse” functionalities.
  • Note: an accepted functionality does not imply that a Metaverse Instance shall support it.
• Assumptions#2: Common Metaverse Specifications (CMS) will be developed.
  • Note: they will provide the technologies supporting identified Functionalities.
• Assumption#3: CMS Technologies will be grouped into Technology Profiles in response to industry needs.
  • Note: A profile shall maximise the number technologies supported by specific groups of industries.

The notion of profile was well known and used in digital media standardisation and is defined here:

A set of one or more base standards, and, if applicable, their chosen classes, subsets, options and parameters, necessary for accomplishing a particular function.

Step #3 – Collect metaverse functionalities

The number of industries potentially interested in deploying metaverse is very large and MPAI has explored 18 of them. See here. A metaverse implementation is also likely to use external service providers – interfaces should be defined. See here.

MPAI has collected > 150 functionalities organised in: Areas – Subareas – Titles. See here.

This is not an accomplished task, but its beginning. Collecting metaverse functionalities should be a continuous task.

Step #4 – Which profiles?

The notion of profile is not currently implementable because some key technologies are not yet available and it is not clear which technologies, exisiting or otherwise, will eventually be selected.

MPAI proposes to introduce a new type of profile – functionality profile, characterised by the functionalities offered, not by the technologies implementing them. By dealing only with functionalities and not technologies, profile definition is not “contaminated” by technology considerations. MPAI is in the process of developing:

Technical Report – MPAI Metaverse Model (MPAI-MMM) – Functionality Profiles.

It is expected that the Technical Report will be published for Community Comments on the 26^th of March and finally adopted on the 19^th of April 2023. It will contain the following table of contents:

1. A scalable Metaverse Operational Model.
2. Actions (what you do in the metaverse):
   1. Purpose – what the Action is for.
   2. Payload – data to the Metaverse.
   3. Response – data from the Metaverse.
3. Items (on what you do Actions):
   1. Purpose – what the Item is for.
   2. Data – functional requirements.
   3. Metadata – functional requirements.
4. Example functionality profiles.

The Technical Report will not contain nor make reference to technologies.

3. The next steps of the MPAI poposal

The Technical Report will demonstrate that it is possible to develop metaverse functionality profiles (and levels) that do not make reference to technologies, only to functionalities.

Step #5 – Develop a metaverse architecture.

The goal is to specify of a Metaverse Architecture, including the main functional blocks and the data types exchanged between blocks.

Step #6 – Develop functional requirements of the metaverse architecture data types.

The goal is to develop the functional requirements of the data types exchanged between functional blocks of the metaverse architecture.

Step #7 – Develop the Table of Contents of the Common Metaverse Specifications.

The goal is to produce an initial Table of Contents (ToC) of Common Metaverse Specifications to have a clear understanding of which Technologies are needed for which purpose in which parts of the metaverse architecture to achieve interoperability.

Step #8 – Map technologies to the ToC of the Common Metaverse Specifications.

MPAI intends to map its relevant technologies and see how they fit in the Common Metaverse Specification architecture. Other SDOs are invited to join the effort.

4. Conclusions

Of course, step #8 there will not provide the metaverse specifications but a tested method to produce them. MPAI envisages to reach step #8 in December 2023. It is a good price to pay before engaging in a perilous project.

Geneva, Switzerland – 22 February 2023. Today the international, non-profit, unaffiliated Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) standards developing organisation has concluded its 29^th General Assembly (MPAI-29) approving a new version of its Audio Enhancement (MPAI-CAE) Technical Specification posted for Community Comments and the Neural Network Watermarking (MPAI-NNW) Reference Software.

Version 2 of the Context-based Audio Enhancement (MPAI-CAE) Technical Specification, besides supporting the functionalities of Version 1, specifies new technologies to enable a device to describe an audio scene in terms of audio objects and their directions. MPAI uses this Technical Specification to enable human interaction with autonomous vehicles, avatar-based videoconference and metaverse applications. The document is posted with a request for Community Comments to be sent to secretariat@mpai.community until the 20^th of March 2023.

The Reference Software of Neural Network Watermarking (MPAI-NNW) provides the means, including the software, to evaluate the performance of neural network-based watermarking solutions in terms of imperceptibility, robustness, and computational cost. The version of the software is specific for image classification but can be extended to other application areas.

MPAI is continuing its work plan including the development of the following Technical Specifications:

1. The AI Framework (MPAI-AIF) V2 Technical Specification will enable an implementer to establish a secure AIF environment to execute AI Workflows (AIW) composed of AI Modules (AIM).
2. The Avatar Representation and Animation (MPAI-ARA) V1 Technical Specification will support creation and animation of interoperable human-like avatar models expressing a Personal Status.
3. The Multimodal Conversation (MPAI-MMC) V2 Technical Specification will generalise the notion of Emotion by adding Cognitive State and Social Attitude and specify a new data type called Standard for Personal Status.

The MPAI work plan also includes exploratory activities, some of which are close to becoming standard or technical report projects:

1. AI Health (MPAI-AIH). Targets an architecture where smartphones store users’ health data processed using AI and AI Models are updated using Federated Learning.
2. Connected Autonomous Vehicles (MPAI-CAV). Targets the Human-CAV Interaction Environment Sensing, Autonomous Motion, and Motion Actuation subsystems implemented as AI Workflows.
3. End-to-End Video Coding (MPAI-EEV). Extends the video coding frontiers using AI-based End-to-End Video coding.
4. AI-Enhanced Video Coding (MPAI-EVC). Improves existing video coding with AI tools for short-to-medium term applications.
5. Server-based Predictive Multiplayer Gaming (MPAI-SPG). Uses AI to train neural networks that help an online gaming server to compensate data losses and detects false data.
6. XR Venues (MPAI-XRV). Identifies common AI Modules used across various XR-enabled and AI-enhanced use cases where venues may be both real and virtual.

It is still a good opportunity for legal entities supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data to join MPAI.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

Cannes, France –  9 February 2023. The Cannes Neurons jury, composed by 12 world-renowned AI experts, has awarded MPAI Member Audio Innova srl  the Cannes Neurons Award 2023 Palm d’Or  for the best creative AI project. Audio archive preservation  based on the MPAI Audio Recording Preservation (ARP) standard is the Audio Innova project coordinated by Sergio Canazza and developed by Nadir Dalla Pozza and Niccolò Pretto. The project was  selected among 6 finalists shortlisted from many other candidates to honour the most innovative and impactful Artificial Intelligence (AI) projects.

The Cannes Neurons Award is the the prestigious prize part of the WAIFC, the world #1 event for AI in business and society which takes place every year in Cannes. To know more, see here.

In the figure: the Audio Innova team with the Palm d’Or in the Salon des Ambassadeurs at the Palais des festivals, Cannes. From left: Cristina Paulon, Sergio Canazza, Alessandro Russo, Michele Patella, Nadir Dalla Pozza (kneeling).