Moving Picture, Audio and Data Coding
by Artificial Intelligence

An overview of Connected Autonomous Vehicle (MPAI-CAV) – Architecture

Connected Autonomous Vehicles (CAV) promise to replace human errors with a lower machine errors rate, give more time to human brains for rewarding activities, optimise use of vehicles, infrastructure, and traffic management, reduce congestion and pollution, and help elderly and disabled people have a better life.

MPAI believes that standards can accelerate the coming of CAVs as an established reality and so the first MPAI standard for this is “Connected Autonomous Vehicles – Architecture”. It specifies a CAV Reference Model broken down into Subsystems for which it specifies the Functions and the data exchanged between subsystems. Each subsystem is further broken down into components for which it specifies the Functions, the data exchanged between components and the topology.

The Subsystem-level Reference model is represented in Figure 1.

Figure 1 – The MPAI-CAV – Architecture Reference Model

There are four subsystem-level reference models. Each subsystem is specified in term of:

  1. The functions the subsystem performs.
  2. The Reference model designed to be compatible with the AI Framework (MPAI-AIF) Technical Specification.
  3. The input/output data exchanged by the subsystem with other subsystems and the environment.
  4. The functions of each of the subsystem components, intended to be implemented as AI Modules.
  5. The input/output data exchanged by the component with other components.

In the following the functions and the reference models of the MPAI-CAV – Architecture will be given. The other three components can be found in the draft Technical Specification ( (htmlpdf)).

Human-CAV Interactions (HCI)

The HCI functions are:

  1. To authenticates humans, e.g., to let them into the CAV.
  2. To converses with humans interpreting utterances, e.g., to go to a destination, or during a conversation. HCI makes use of the MPAI-MMC “Personal Status” data type.
  3. To Converses with the Autonomous Motion Subsystem to implement human conversation and execute commands.
  4. To enables passengers to navigate the Full Environment Representation.
  5. Appears as a speaking avatar showing a Personal Status.

The HCI Reference Model is depicted in Figure 2.

Figure 2 – HCI Reference Model

The full HCI specification is available here.

Environment Sensing Subsystem (ESS)

The ESS functions are:

  1. To acquire Environment information using Subsystem’s RADAR, LiDAR, Cameras, Ultrasound, Offline Map, Audio, GNSS, …
  2. To receive Ego CAV’s position, orientation, and environment data (temperature, humidity, etc.) from Motion Actuation Subsystem.
  3. To produce Scene Descriptors for each sensor technology in a common format.
  4. To produce the Basic Environment Representation (BER) by integrating the sensor-specific Scene Descriptors during the travel.
  5. To hand over the BERs, including Alerts, to the Autonomous Motion Subsystem.

The ESS Reference Model is depicted in Figure 3.

Figure 3 – ESS Reference Model

The full ESS specification is available here.

Autonomous Motion Subsystem (AMS)

The AMS functions are:

  1. To compute human-requested Route(s).
  2. To receive current BER from Environment Sensing Subsystem.
  3. To communicate with other CAVs’ AMSs (e.g., to exchange subsets of BER and other data).
  4. To produce the Full Environment Representation by fusing its own BER with info from other CAVs in range.
  5. To send Commands to Motion Actuation Subsystem to take the CAV to the next Pose.
  6. To receive and analyse responses from MAS.

The AMS Reference Model is depicted in Figure 3.

Figure 4 – AMS Reference Model

The full AMS specification is available here.

Motion Actuation Subsystem

The MAS functions are:

  1. To transmit spatial/environmental information from sensors/mechanical subsystems to the Environment Sensing Subsystem.
  2. To receive Autonomous Motion Subsystem Commands.
  3. To translates Commands into specific Commands to its own mechanical subsystems, e.g., brakes, wheel directions, and wheel motors.
  4. To receive Responses from its mechanical subsystems.
  5. To Sends responses to Autonomous Motion Subsystem about execution of commands.

The MAS Reference Model is depicted in Figure 5.

Figure 5 – MAS Reference Model

The full MAS specification is available here.

The WD of Connected Autonomous Vehicle – Architecture is published with a request for Community Comments. The MPAI-CAV – Architecture Working Draft (htmlpdf) is published with a request for Community Comments. See also the video recordings (YTWimTV) and the slides of the presentation made on 06 September.  Anybody may make comment on the WD. Comments should reach the MPAI Secretariat by 2023/09/26T23:59 UTC. No specific format is required to make comments. MPAI plans on publishing MPAI-CAV – Architecture at the 36th General Assembly (29 September 2023).

The MPAI-CAV Architecture standard is the starting point for the next steps of the MPAI-CAV roadmap. The current specification does not include the Functional Requirements of the data exchanged between subsystems and components and this is exactly the activity that will start in October 2023.

Visit How to join to join MPAI.


What is the XR Venues – Live Theatrical Stage Performance Call for Technologies about?

XR Venues is an MPAI project addressing contexts enabled by Extended Reality (XR) – any combination of Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR) technologies – and enhanced by Artificial Intelligence (AI) technologies. The word “Venue” is used as a synonym for Real and Virtual Environments.

MPAI thinks that the Live Theatrical Stage Performance use case fits well with the current trend that sees theatrical stage performances such as Broadway theatres, musicals, dramas, operas, and other performing arts increasingly using video scrims, backdrops, and projection mapping to create digital sets rather than constructing physical stage sets, allowing the entire stage and theatre to become a digital virtual environment thus reducing the cost of mounting shows.

The use of immersion domes – especially LED volumes – can completely surround audiences with virtual environments that live performers can inhabit and interact with. In addition, Live Theatrical Stage Performance can extend into the metaverse as a digital twin. Elements of the Virtual Environment experience can be projected in the Real Environment and elements of the Real Environment experience can be rendered in the Virtual Environment (metaverse).

The purpose of the planned MPAI-XRV – Live Theatrical Stage Performance Technical Specification is to address AI Modules performing functions that facilitate live multisensory immersive performances which ordinarily require extensive on-site show control staff to operate. Use of the AI Modules organised in AI Workflows (see details here) enabled by the MPAI-XRV – LTSP Technical Specification will allow more direct, precise yet spontaneous show implementation and control to achieve the show director’s vision. It will also free staff from repetitive and technical tasks allowing them to amplify their artistic and creative skills.

Figure 1 provides the Reference Model of the Live Theatrical Stage Performance Use Case incorporating AI Modules (AIM’s). In this diagram, data extracted from the Real and Virtual Environments (on the left) are processed and injected into the same Real and Virtual Environments (on the right).

Data is collected from both the Real and Virtual Environments. This includes audio, video, volumetric or motion capture (mocap) data from stage performers, audio and video from participants, signals from control surfaces (e.g., audio, lighting, show control), and more. One or more AIMs extract features from participants (i.e., the audience) and performers which are output as Participant and Scene Descriptors. These Descriptors are further interpreted by Performance and Participant Status AIMs to determine the Cue Point in the show (according to the Script) and Participants Status (in general, an assessment of the audience’s reactions).

Figure 1 – Live theatrical stage performance architecture (AI Modules shown in green)

Likewise, data from the Show Control computer or control surface, consoles for audio, DJ, VJ, lighting and FX (typically commanded by operators) – if needed – are interpreted by the Operator Command Interpreter AIM and output as Interpreted Operator Control. The Action Generation AIM accepts Participant Status, Cue Point and Interpreted Operator Controls and uses them to direct action in both the Real and Virtual Environments via Scene and Action Descriptors. These general descriptors are converted into actionable commands (e.g., DMX, MIDI, USD) required by the Real and Virtual Environments – according to their Venue Specifications – to enable multisensory Experience Generation in both the Real and Virtual Environments. In this manner, the desired experience can automatically be adapted to a variety of specific real and virtual venue instances.

MPAI is seeking proposals of technologies that enable the implementation of standard components (AI Modules) to make real the vision described above. The deadline for submitting a response is November 20 at 23:59 UTC. See the published documents:

Those intending to submit a response should familiarise with the following documents:

Call for Technologies html,  pdf
Use Cases and Functional Requirements htmlpdf
Framework Licence htmlpdf
Template for responses html, docx

See the video recordings (YouTubeWimTV) and the slides from the presentation made on 12 September. Read What is the XR Venues – Live Theatrical Stage Performance Call for Technologies about?

 


An overview of MPAI Metaverse Model (MPAI-MMM) – Architecture

There is no common definition of the word metaverse. MPAI characterises the word Metaverse Instance (M-Instance) as a system that captures data from the real world (in the following, called Universe), processes it, and combines it with internally generated data to create virtual environments that users can interact with.

Examples of M-Instances are plentiful. So far, their developers have made technology decisions that best responded to their needs, often without considering the choices that other developers might have made for similar purposes. Recently, however, many have expressed concerns that “walled gardens” do not fully exploit the opportunities offered by current and expected technologies and have called for M-Instances to be “interoperable”.

MPAI has studied the issue of M-Instance interoperability. What can be called direct interoperability is the most desirable and effective but metaverse technologies are still rapidly evolving. Mediated interoperability is a solution that lets M-Instance implementers make their own technology decisions assigning to a conversion service the task of converting data from one format to another. The practical use of mediated interoperability is limited because the data semantics of different M-Instances may greatly vary. As indicated in Figure 1, dataA.1 of M-InstanceA may be converted to dataB.1 and dataB.2 but a part of dataA.1 may not be converted at all.

Figure 1 – Data conversion between M-Instances

If technology standardisation is too early and data conversion not a solution, a standard for the functional requirements of data can provide a form of interoperability. After publishing two Technical Reports on Functionalities and Functionality Profiles, MPAI is now publishing MPAI-Metaverse Model (MPAI-MMM) – Architecture Technical Specification. M-Instances can interoperate if they:

  1. Rely on the same Operation Model.
  2. Use:
    • The same Profile specified by MPAI-MMM – Architecture, and
    • Either the same Technologies, or
    • Independent Technologies while accessing appropriate Conversion Services.

The above numbered list is contained in the first normative chapter of the standard – Scope. The next chapter – Terms and Definitions – collecting a large number (about 100) of specified Terms is also normative.

The next chapter – Functional Requirements –is informative. It contains thoroughly revised and integrated Functionalities of the Technical Report and provides the basic elements on which the next (normative) chapter – M-Instance Operation – is based. The main elements of this chapter are:

  1. The definition of an M-Instance as a set of Processes performing Actions on Items at Locations and Times.
  2. Specifies the minimum metadata format of Processes, Actions, and Items (metadata are extensible to cope with specific cases).
  3. Specifies the protocol whereby a Process can request another Process – potentially in another M-Instance – to perform Action on Items.
  4. Specifies four types of Process: Device, User, Service, and App.
  5. Renders Users as Personae.

Figure 1 graphically represents the last two points: Devices are Processes connecting the real world with the M-Instance, Users are Processes representing and acting on behalf of a registered human, and Apps are Processes running on a Device. Personae are rendered Users.

Figure 2 – Universe, Metaverse, Human, Device, User, Persona, App

The following (normative) chapter identifies the Actions, Items, and Data Types provides their Functional Requirements.

The following (informative) chapter uses the MPAI Metaverse Use Case Description Language to describe nine Use Cases:

1.     Virtual Lecture

2.     Virtual Meeting

3.     Hybrid working

4.     eSports Tournament

5.     Virtual performance

6.     AR Tourist Guide

7.     Virtual Dance

8.     Virtual Car Showroom.

9.     Drive a Connected Autonomous Vehicle.

 

All relevant elements of the M-Instance Operation Model, and Actions, Items, and Data Types have been used. The nine Use Cases do not need more elements than those specified in the MPAI-MMM – Architecture standard.

The last (normative) chapter – Functional Profiles – revisits and extends the four profiles of different complexity identified as examples in the Functionality Profiles Technical Report. The Baseline Profile supports basic forms of lecture, meeting, and hang-out, the Finance Profile enables a User to post assets and make transactions, the Management Profile adds the management of registration and rights to the Baseline Profile, and the High Profile supports all currently identified Functionalities.

The MPAI-MMM – Architecture Working Draft (htmlpdf)  is published with a request for Community Comments. See video recordings (YouTubeWimTV) of the presentation made on 1st September 2023. Send to the MPAI Secretariat by 2023/09/21T23:59 UTC. MPAI will use the Comments received to develop the final draft planned to be published at the 36th General Assembly (29 September 2023).


An overview of Multimodal Conversation (MPAI-MMC) V2

The goal of the Multimodal Conversation (MPAI-MMC) standard is to provide technologies that enable a human-machine conversation that is more human-like, richer in content, and able to emulate human-human conversation in completeness and intensity.

By learning from human interaction, machines can improve their “conversational” capabilities in the two main phases of conversation: understanding the meaning of an element and generation of a pertinent response.

Multimodal Conversation Version 2 achieves this goal by providing, among other technologies, a new standard data type – Personal Status – that represents the “internal status” of a conversing human conveyed by text, speech, face, and gesture. The same Personal Status data type is used by the machine to represent its own internal status as if it were a human.

Currently, Personal Status is composed of three Personal Status Factors:

  1. Emotion, the coded representation of the internal state resulting from the interaction of a human or avatar with the Environment or subsets of it, such as “Angry”, “Sad”, “Determined”.
  2. Cognitive State, the coded representation of the internal state reflecting the way a human or avatar understands the Environment, such as “Confused”, “Dubious”, “Convinced”.
  3. Social Attitude, the coded representation of the internal state related to the way a human or avatar intends to position vis-à-vis the environment, e.g., “Respectful”, “Confrontational”, “Soothing”.

Each Factor is represented by a standard set of labels and associated semantics with 2 tables:

  1. Table 1: Label Set contains descriptive labels relevant to the Factor in a three-level format:
    1. The CATEGORIES column specifies the relevant categories using nouns (e.g., “ANGER”).
    2. The GENERAL ADJECTIVAL column gives adjectival labels for general or basic labels within a category (e.g., “angry”).
    3. The SPECIFIC ADJECTIVAL column gives more specific (sub-categorised) labels in the relevant category (e.g., “furious”).
  2. Table 2: Label Semantics provides the semantics for each label in the GENERAL ADJECTIVAL and SPECIFIC ADJECTIVAL columns of the Label Set Table. For example, for “angry” the semantic gloss is “emotion due to perception of physical or emotional damage or threat.”

The mission of the international, unaffiliated, non-profit Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) Standards Developing Organisation is to develop AI-enabled data coding standards. MPAI believes that its standards should enable humans to select machines whose internal operation they understand to some degree, rather than machines that are just “black boxes” resulting from unknown training with unknown data. Thus, an implemented MPAI standard breaks up monolithic AI applications, yielding a set of interacting components with identified data whose semantics is known, as far as possible. The AI Framework (MPAI-AIF) standards (html, pdf) specifies an environment where AI Workflows (AIW) composed of AI Modules (AIM) are executed in environments implemented as an AI Framework (AIF).

Figure 1 – Reference Model of AI Framework (AIF)

MPAI-MMC has defined two “Composite AI Modules (AIM)” that achieve the goal of representing the internal state of the conversing human and machine (or machine and machine). The first is called “Personal Status Extraction (PSE)” and is represented by Figure 2.

Figure 2 – Personal Status Extraction

PSE computes the Descriptors of each considered Modality – Text, Speech, Face, and Gesture. The Personal Status embedded in the Modality is obtained by interpreting the Modality Descriptors. The need of several MPAI Use Cases has suggested that Descriptors may be computed outside of the PSE and provided by another AIM, obviously with the same semantics. This is signalled by Input Selection. Personal Status Combination is an AIM that integrates the Personal Status of the four Modalities in a standard Personal Status Format.

The second Composite AIM – “Personal Status Display (PSD)” is used to convert the Text produced by the Machine with the associated Personal Status and is represented in Figure 3.

Figure 3 – Personal Status Display

Machine Text is passed as PSD output and synthesised as Speech using the Personal Status provided by PS-Speech. Face Descriptors are produced using Machine Speech and PS-Face. Body Descriptors are produced using Avatar Model, PS-Gesture, and Text. Avatar Descriptors, the combination of Face Descriptors and Body Descriptors are produced by the Avatar Description AIM. The ready-to-render Machine Avatar is produced by Avatar Synthesis.

Input Selection is used to indicate whether the PSD should produce Avatar Descriptors or ready-to-render Machine Avatar.

Let’s now see how the definition of the PSE and PSD Composite AIMs enables a compact representation of human-machine conversation use cases by considering the reference model of Conversation with Personal Status use case depicted in Figure 4.

Figure 4 – Conversation with Personal Status

The visual frontend (Visual Scene Description) describes the visual scene providing the Descriptors of the Face and the Body of the human and a digital representation of the objects in the scene. The Audio frontend (Audio Scene Description) separates the speech from other sound sources. Spatial Object Identification analyses the arm, hand, and fingers of the human to identify which of the objects the human refers to in a conversation. Audio-Visual Alignment assigns identifiers to audio, visual, and audio-visual objects. Speech Recognition, Language Understanding, and Dialogue Processing are the usual elements of the conversation chain. In MPAI-MMC, however, Dialogue Processing deals with additional information in the form of Object ID and Personal Status of the human. Therefore, Dialogue Processing is requested not only to produce Machine Text in response to the human, but also its own Personal Status. Both Text and Personal Status are provided to the Personal Status Display Composite AIM that provides the full multimodal response as Machine Text, Machine Speech, and Machine Avatar.

What does Multimodal Conversion specify?

  1. Technologies required to analyse the text and/or the speech and other non-verbal components exchanged in human-machine and machine-machine conversation.
  2. Use Cases that apply the technologies:
    • Conversation with Personal Status.
    • Conversation with Emotion.
    • Multimodal Question Answering.
    • Conversation About a Scene.
    • Human-CAV Interaction.
    • Virtual Secretary for Videoconference
    • Text and Speech Translation (one way, two ways, one to many)

Each Use Case normatively defines:

  1. The Functions of the AIW and (Composite) AIMs implementing the Use Case.
  2. The Connections between and among the AIMs.
  3. The Semantics and the Formats of the input and output data of the AIW and the AIMs.
  4. The JSON Metadata of the AIW.

Each AIM normatively defines:

  1. The Functions of the AIM.
  2. The Connections between and among the AIMs in case the AIM is Composite.
  3. The Semantics and the Formats of the input and output data of the AIMs.
  4. The JSON Metadata of the AIMs.

MPAI deals with all aspects of “AI-enabled Data Coding” in a unified way. Some of the data formats required by the MPAI-MMC Use Cases are specified by MPAI-MMC and some data formats by the standards produced by other MPAI groups. For instance, the Face and Body Descriptors are specified by the Avatar Representation and Animation standard and the “Spatial Object Identification” Composite AIM is specified by “Object and Scene Description” (MPAI-OSD).

The MPAI-MMC Version 2 Working Draft (html, pdf) is published with a request for Community Comments. Comments should be sent to the MPAI Secretariat by 2023/09/25T23:59 UTC. MPAI will use the Comments received to develop the final draft planned to be published at the 36th General Assembly (29 September 2023). An online presentation of the WD will be held on September 05 at 08 and 15 UTC. Register here for the 08 UTC and here for the 15 UTC presentations.


The MPAI standards portfolio

1          Introduction

Established in September 2020 with its mission defined by the MPAI Statutes as promotion of efficient data use by developing AI-enabled data coding standards and bridging the gap between MPAI standards and their practical use through Intellectual Property Rights Guidelines, MPAI has developed nine Technical Specifications. Purpose of this document is to provide a short overview of the nine Technical Specifications.

1       Introduction

2       AI Framework (MPAI-AIF)

2.1        Version 1

2.2        Version 2

3       Avatar Representation and Animation (MPAI-ARA

4       Context-based Audio Enhancement (MPAI-CAE)

4.1        Version 1

4.2        Version 2

5       Connected Autonomous Vehicle (MPAI-CAV)

6      Compression and understanding of industrial data (MPAI-CUI)

7       Governance of the MPAI Ecosystem (MPAI-GME)

8       Multimodal Conversation (MPAI-MMC)

8.1        Version 1

8.2        Version 2

9       MPAI Metaverse Model (MPAI-MMM) – Architecture

10          Neural Network Watermarking (MPAI-NNW)

2          AI Framework (MPAI-AIF)

MPAI believes that its standards should enable humans to select machines whose internal operation they understand to some degree, rather than machines that are “black boxes” resulting from unknown training with unknown data. Thus, an implemented MPAI standard breaks up monolithic AI applications, yielding a set of interacting components (AI Modules) with identified data. AIMs are combined in workflows (AIW), and exchange data with a known semantics to the extent possible, improving explainability of AI applications but also promoting a competitive market of components with standard interfaces, possibly with improved performance compared to other implementations.

Artificial Intelligence Framework (MPAI-AIF) is the standard that implements this vision by enabling creation and automation of mixed Artif­icial Intelligence – Machine Learning – Data Processing workflows.

2.1        Version 1

Figure 1 shows the MPAI-AIF V1 Reference Model.

Figure 1 – Reference model of the MPAI AI Framework (MPAI-AIF) V1

The MPAI-AIF Technical Specification V1 and Reference Software is available here.

2.2        Version 2

MPAI-AIF V1 assumed that the AI Framework was secure but did not provide support to developers wishing to execute an AI application in a secure environment. MPAI-AIF V2 responds to this requirement. As shown in Figure 1, the standard defines a Security Abstraction Layer (SAL). By accessing the SAL APIs, a developer can indeed create the required level of security with the desired functionalities.

Figure 2 – Reference model of the MPAI AI Framework (MPAI-AIF) V2

MPAI ha published a Working Draft of Version 2 (html, pdf) requesting Community Comments. Comments should be sent to the MPAI Secretariat by 2023/09/24T23:59 UTC. MPAI-AIF V2 is expected to be published on 29 September 2023.

3          Avatar Representation and Animation (MPAI-ARA)

In most cases the underlying assumption of computer-created objects called “digital humans”, i.e., digital objects that can be rendered to show a human appearance has been that creation, animation, and rendering is done in a closed environment. Such digital humans used to have little or no need for standards. However, in a communication and even more in a metaverse context, there are many cases where a digital human is not constrained within a closed environment. For instance, a client may send data to a remote client that should be able to unambiguously interpret and use the data to reproduce a digital human as intended by the transmitting client.

These new usage scenarios require forms of standardisation. Technical Specification: Avatar Representation and Animation (MPAI-ARA) is a first response to the need of a user wishing to enable their transmitting client to send data that a remote client can interpret to render a digital human, having the intended body movement and facial expression faithfully represented by the remote client.

Figure 2 is the system diagram of the Avatar-Based Videoconference Use Case enabled by MPAI-ARA.

Figure 2 – System diagram of ARA-ABV

Figure 3 is the Reference Model of the Transmitting Client.

Figure 3 – Reference Model of the ARA-ABV Transmitting Client

The MPAI-ARA Working Draft (htmlpdf) is published with a request for Community Comments stage and expected to be published on 29 September 2023.

4          Context-based Audio Enhancement (MPAI-CAE)

Context-based Audio Enhancement (MPAI-CAE) uses AI to improve the user experience for several audio-related applications including entertainment, communication, teleconferencing, gaming, post-production, restoration etc. in a variety of contexts such as in the home, in the car, on-the-go, in the studio etc. using context information to act on the input audio content.

4.1        Version 1

Figure 4 is the reference model of Unidirectional Speech Translation, a Use Case developed by Version 1.

Figure 4 – An MPAI-CAE Use Case: Emotion-Enhanced Speech

The MPAI-AIF Technical Specification V1 and Reference Software is available here.

4.2        Version 2

MPAI has developed the specification of the Audio Scene Description Composite AIM as part of the MPAI-CAE V2 standard (Figure 5 depicts the architecture of the CAE-ASD Composite AIM.

Figure 5 – Audio Scene Description Composite AIM

The MPAI-AIF Technical Specification V2 is available here.

5          Connected Autonomous Vehicle (MPAI-CAV)

Connected Autonomous Vehicle (CAV) is a project addressing the Connected Autonomous Vehicle (CAV) domain and the 5 main operating instances of a CAV:

  1. Human-CAV interaction (HCI) responds to humans’ commands and queries, senses human activities in the CAV passenger compartment and activates other subsystems as required by humans or as deemed necessary under the identified conditions.
  2. CAV-Environment interaction acquires information from the physical environment via a variety of sensors and creates a representation of the environment.
  3. Autonomous Motion Subsystem (AMS) uses different sources of information – ESS, other CAVs, Roadside Units, etc. – to improve the CAV’s understanding of the environment and instructs the CAV how to reach the intended destination.
  4. Motion Actuation Subsystem (MAS) is the subsystem that operates and actuates the motion instructions in the environment.

The interaction of the 4 subsystems is depicted in Figure 7.

Figure 7 – The MPAI-CAV subsystems

The CAV-HCI subsystem (Figure 8) is specified as an MPAI-MMC V2 Use Case.

Figure 8 – Reference Model of the Human-CAV Interaction Subsystem

The MPAI-CAV – Architecture Working Draft (htmlpdf) is published with a request for Community Comments and expected to be published on 29 September 2023.

6          Compression and understanding of industrial data (MPAI-CUI)

Compression and understanding of industrial data (MPAI-CUI) aims to enable AI-based filtering and extraction of key information to predict the performance of a company by applying Artificial Intellig­ence to governance, financial and risk data. This is depicted in Figure 6.

Figure 6 – The MPAI-CUI Company Performance Prediction Use Case

The set of specifications composing the MPAI-CUI standard is available here.

7          Governance of the MPAI Ecosystem (MPAI-GME)

Governance of the MPAI Ecosystem (MPAI-GME) is a foundational MPAI standard specifying the operation of the Ecosystem enabling:

  1. Implementers to develop components and solutions.
  2. Performance Assessors to assess the Performance of an Implementation.
  3. The MPAI Store to Test an Implementation for Conformance and post the Implementation to the MPAI Store website together with the results of Performance Assessment.
  4. End Users to download Implementations and report Experience Scores.

Figure 9 depicts the operation of the MPAI Ecosystem.

Figure 9 – Governance of the MPAI Ecosystem.

8          Multimodal Conversation (MPAI-MMC)

Multi-modal conversation (MPAI-MMC) aims to enable human-machine conversation that emul­ates human-human conversation in completeness and intensity by using AI. Its Use Cases are is an implementation of the MPAI “make available explainable AI applications” vision for human-machine conversation.

8.1        Version 1

Technical Specification: Multimodal Conversation (MPAI-MMC) V1 includes five Use Cases Conversation with emotion, Multimodal Question Answering (QA) and 3 Automatic Speech Translation.

Figure 10 depicts the Reference Model of the Conversation with Emotion Use Case.

Figure 10 – An MPAI-MMC V1 Use Case: Conversation with Emotion

The MPAI-MMC Technical Specification V1.2 is available here.

8.2        Version 2

Extending the role of emotion as introduced in Version 1.2 of the standard, MPAI-MMC V2 introduces Personal Status, i.e., the internal status of humans that a machine needs to estimate and that it artificially creates for itself with the goal of improving its conversation with the human or even with another machine. Personal Status is applied to several new Multi-modal conversation V2 (MPAI-MMC V2) Use Cases: Conversation About a Scene, Virtual Secretary for Videoconference, and Human-Connected Autonomous Vehicle Interaction.

Figure 11 is the reference model of the Conversation About a Scene (CAS) Use Case.

Figure 11 – An MPAI-MMC V2 Use Case: Conversation About a Scene

Figure 12 gives the Reference Model of a second use case: Virtual Secretary (used in the Avatar-Based Videoconference Use Case).

Figure 12 – Reference Model of Virtual Secretary for Videoconference

The MPAI-MMC Version 2 Working Draft (html, pdf) is published with a request for Community Comments and expected to be finally approved on 29 September 2023.

9          MPAI Metaverse Model – Architecture (MPAI-MMM)

MPAI characterises the metaverse as a system that captures data from the real world, processes it, and combines it with internally generated data to create virtual environments that users can interact with.

MPAI Metaverse Model (MMM) targets a series of Technical Reports and Specifications promoting Metaverse Interoperability. Two MPAI Technical Reports. on Metaverse Functionalities and Functionality Profiles, have laid down the groundwork. With the Technical Specification – MPAI Metaverse Model – Architecture, MPAI provides initial Interoperability tools by specifying the Functional Requirements of Processes, Items, Actions, and Data Types that allow two or more Metaverse Instances to Interoperate, possibly via a Conversion Service, if they implement the Technical Specification’s Operation Model and produce Data whose Format complies with the Specification’s Functional Requirements.

Figure 13 depicts one aspect of the Specification where a Process in a Metaverse Instance requests a Process in another Metaverse Instance to perform an Action by relying on the Instances’ Resolution Service.

Figure 13 – Resolution and Conversion Services

The MPAI-MMM – Architecture Working Draft (htmlpdf)  is published with a request for Community Comments level and expected to be finally approved on 29 September 2023.

10     Neural Network Watermarking (MPAI-NNW)

Neural Network Watermarking is a standard whose purpose is to enable watermarking technology providers to qualify their products by providing the means to measure, for a given size of the watermarking payload, the ability of:

  • The watermark inserter to inject a payload without deteriorating the NN performance.
  • The watermark detector to recognise the presence of the inserted watermark when applied to:
    • A watermarked network that has been modified (e.g., by transfer learning or pruning)
    • An inference of the modified model.
  • The watermark decoder to successfully retrieve the payload when applied to:
    • A watermarked network that has been modified (e.g., by transfer learning or pruning)
    • An inference of the modified model.
  • The watermark inserter to inject a payload at a measured computational cost in a given processing environment.
  • The watermark detector/decoder to detect/decode a payload from a watermarked model or from any of its inferences, at a measurable computational cost, e.g., execution time in a given processing environment.

Figure 14 depicts the configuration of one particular use of MPAI-NNW.

Figure 14 – Robustness evaluation of a Neural Network

The Neural Network Watermarking Technical Specification is available here.

 


MPAI issues three Calls for Technologies and publishes five standards for Community Comments

Geneva, Switzerland – 23 August 2023. Today, the international, non-profit, and unaffiliated Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) organisation developing AI-based data coding standards has concluded its 35th General Assembly (MPAI-35) approving the publication of three Calls for Technologies and five Technical Reports with request for Community Comments. The table gives links to the documents, dates of and registration links to online presentations and deadlines for submitting responses to Calls and comments on Technical Specifications.

Call for Technologies Link Presentation Deadline
AI for Health Data (AIH) X Sep 08 08 & 15 Oct 19 23:59
Object and Scene Description (OSD) X Sep 07 09 & 16 Sep 20 23:59
XR Venues – Live Theatrical Stage Performance (XRV) X Sep 12 07 & 17 Nov 20 23:59
Standard for Community Comments   Presentation Deadline
AI Framework (AIF) V2 X Sep 11 08 & 15 Sep 24 23:59
Avatar Representation and Animation (ARA) X Sep 07 08 & 15 Sep 27 23:59
Connected Autonomous Vehicles – Architecture (CAV) X Sep 06 08 & 15 Sep 26 23:59
Multimodal Conversation (MMC) V2 X Sep 05 08 & 15 Sep 25 23:59
MPAI Metaverse Model – Architecture (MMM) X Sep 01 08 & 15 Sep 21 23:59

Additional information about the purpose of the projects can be found here.
Anybody may respond to any of the three Calls for Technologies. However, non-members should join MPAI to participate in the development of the relevant standards.
Anybody can make comments on the Technical Specifications published with a request for Community Comments.
MPAI is continuing its work plan that includes the development of the following Technical Specifications:

  • AIF-DC, the group in charge of AI Framework (MPAI-AIF), is now working on the review of comments made on MPAI-AIF V2, developing the reference software and drafting the conformance testing.
  • Requirements (ARA), the group in charge of Avatar Representation and Animation (MPAI-ARA), is now working on the review of comments made on MPAI-AIF V2, developing the reference software and drafting the conformance testing.
  • MMC-DC, the group in charge of Multimodal Conversation (MPAI-MMC), is now working on the review of comments made on MPAI-AIF V2, developing the reference software and drafting the conformance testing.
  • Requirements (MMM), the group in charge of MPAI Metaverse Model (MPAI-MMM) – Architecture, is now working on the review of comments made on MPAI-AIF V2, developing the reference software and drafting the conformance testing.

The MPAI work plan also includes exploratory activities, some of which are close to becoming standard or technical report projects:

  • AI Health (MPAI-AIH). Targets an architecture where smartphones store users’ health data processed using AI and AI Models are updated using Federated Learning.
  • End-to-End Video Coding (MPAI-EEV). Extends the video coding frontiers using AI-based End-to-End Video coding.
  • AI-Enhanced Video Coding (MPAI-EVC). Improves existing video coding with AI tools for short-to-medium term applications.
  • Server-based Predictive Multiplayer Gaming (MPAI-SPG). Uses AI to train neural networks that help an online gaming server to compensate data losses and detects false data.
  • XR Venues (MPAI-XRV). Identifies common AI Modules used across various XR-enabled and AI-enhanced use cases where venues may be both real and virtual.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.
Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.


Do we need standards for Connected Autonomous Vehicles?

Enabling individuals or groups of people to move independently has been a major achievement that has changed human life for the better. Motor vehicles, however, have created a number of negative consequences such as, accidents causing damages, injuries, and deaths; congestion on the roads, millions of cars carrying a single person for a couple of ours and then staying, unused; air pollution, worsening of urban environments, etc.

Connected autonomous vehicles (CAV) have the potential to eliminate human error replacing it with a rate of machine errors orders of magnitude lower, optimise use of vehicles and infrastructure, give more time to human brains for rewarding activities, optimise traffic management, reduce congestion and pollution, help the elderly or disabled people to have a better life, and more.

Much has been happening since the first 1939 attempt at creating an autonomous vehicle. Today CAVs are technically feasible, and prototypes are driving on public roads and streets. The Society of Automotive Engineers in the USA has published a classification of autonomous vehicles based on levels.

Should we just wait for the industry to produce higher SAE-Level vehicles until one day we will only see CAVs around us? This is an option, but not necessarily the one that will let us reach the CAV holy grail in the most efficient and timely way.

Some 35 years ago, most public authorities, “owners” of their countries’ VHF and UHF bands, realised that digital television would allow them to keep their cherished terrestrial television service while getting a “digital dividend” in the form of VHF and UHF slots and re-assign them to other purposes. Especially in the United States, digital television was a national goal and steps were made to implement it. Some enlightened people understood the value of a global digital television standard (MPEG-2) and thing simply “happened”, not just for terrestrial, but also for ratellite and cable television, and packaged media as well.

Of course, cars are not television sets, but the game-changing role of standards can be the same. Standards can convert today’s niche market of CAVs (if we can call it a “market”) into a mass market. It can accelerate the availability of technology, promote competition, yield better and cheaper products, assuage consumer concerns, and provide tools for regulation.

Artificial Intelligence (AI) is the technology that can provide the solutions we need. MPAI can provide AI-based standards that are explainable.

MPAI intends to publish a standard called Connected Autonomous Vehicle (MPAI-CAV) – Architecture. This will enable component manufacturers to put their standard components on the market and car manufacturers to access an open global market of components with standard functions and interfaces that can be tested for conformance using standard procedures.

Register for one of the two online presentations on July/26 at 8 UTC and 15 UTC or read an overview of MPAI-CAV – Architecture.


MPAI issues Call for Technologies: Connected Autonomous Vehicle – Architecture

Geneva, Switzerland – 12 July 2023. Today, the international, non-profit, and unaffiliated Moving Picture, Audio and Data Coding by Artificial Intelligence (MPAI) organisation developing AI-based data coding standards has concluded its 34th General Assembly (MPAI-34) approving the Call for Technologies: Connected Autonomous Vehicle (MPAI-CAV) – Architecture. Two online presentations of the Call will be made on 26 July at 8 and 15 UTC. Responses are due by 15 August.

The goal of the MPAI-CAV standard is to promote the development of a CAV industry by specifying components that can be easily integrated into larger subsystems. To achieve this goal, MPAI intends to develop the MPAI-CAV standard as a series of standards each adding more details to enhance CAV component interoperability. The first issue, MPAI-CAV – Architecture, to be developed using the results of the Call, aims to partition CAVs into subsystems and to further partition those subsystems into components. Both subsystems and components are identified by their function and interfaces, i.e., data exchanged between subsystems and components.

Three documents are attached to the Call: the first is Use Cases and Functional Requirements. It includes an initial set of Functionalities that the Architecture should provide.

The second document is the Framework Licence designed to facilitate the timely access to IP that is essential to implement the planned MPAI-CAV – Architecture standard. Finally, the third document is a Template for responses that respondents to the Call may wish to use in their responses.

Anybody may respond to the Call. However, non-members should join MPAI to participate in the development of the MPAI-CAV – Architecture standard.

MPAI is continuing its work plan comprising the development of the following Technical Specifications:

  1. The AI Framework (MPAI-AIF) V2 Technical Specification will enable an implementer to establish a secure AIF environment to execute AI Workflows (AIW) composed of AI Modules (AIM).
  2. The Avatar Representation and Animation (MPAI-ARA) V1 Technical Specification will support creation and animation of interoperable human-like avatar models able to understand and express a Personal Status.
  3. The Multimodal Conversation (MPAI-MMC) V2 Technical Specification will generalise the notion of Emotion by adding Cognitive State and Social Attitude and specify a new data type called Personal Status.
  4. The MPAI Metaverse Model (MPAI-MMM) – Architecture V1 Technical Specification will specify the Operation Model and its components Actions, Items, and Data Types.

The MPAI work plan also includes exploratory activities, some of which are close to becoming standard or technical report projects:

  1. AI Health (MPAI-AIH). Targets an architecture where smartphones store users’ health data processed using AI and AI Models are updated using Federated Learning.
  2. End-to-End Video Coding (MPAI-EEV). Extends the video coding frontiers using AI-based End-to-End Video coding.
  3. AI-Enhanced Video Coding (MPAI-EVC). Improves existing video coding with AI tools for short-to-medium term applications.
  4. Server-based Predictive Multiplayer Gaming (MPAI-SPG). Uses AI to train neural networks that help an online gaming server to compensate data losses and detects false data.
  5. XR Venues (MPAI-XRV). Identifies common AI Modules used across various XR-enabled and AI-enhanced use cases where venues may be both real and virtual.

Legal entities and representatives of academic departments supporting the MPAI mission and able to contribute to the development of standards for the efficient use of data can become MPAI members.

Please visit the MPAI website, contact the MPAI secretariat for specific information, subscribe to the MPAI Newsletter and follow MPAI on social media: LinkedIn, Twitter, Facebook, Instagram, and YouTube.

 

 


An Introduction to the MPAI Metaverse Model Architecture – Part III

In parts I and II of this series of posts, we have highlighted:

  1. The basic elements that enable operation of an M-Instance, especially Processes, Items, Actions and Data Type. In particular, Processes can take the shape of a User (a representative of a human in an M-Instance), a Device (to enable the connection of an M-Instance with the real world, called Universe), and a Service.
  2. The functional requirements of an initial list of Actions that enable a Process to do useful things in an M-Instance.

We are now going to identify the functional requirements of an initial list of Items that enable a Process to do useful things in an M-Instance. For convenience, Items will be grouped in classes.

Remembers the online presentations at 8 and 15 UTC on 23 June where you can know more about the MPAI Metaverse Model Architecture and the Call for Technologies. Register here for the first and here for the second presentation.

Here are Items with a general applicability.

Functional requirements Item
An M-Instance is an abstract entity bearing an Identifier. An M-Instance may expose its Capabilities. M-Instance
An M-Environment is an abstract entity bearing an Identifier.

1.      An M-Environment is hosted by an M-Instance.

2.      An M-Environment may expose its Capabilities.

3.      The Capabilities of an M-Environment may extend the Capabilities of its hosting M-Instance.

M-Environment
An Item or a Process shall bear Identifiers in such a way that:

1.      An Identifier uniquely references an Item or Process.

2.      An Item can have more than one Identifier.

An Item may have a hierarchical structure, such as:

Item: M-InstanceID, M-EnvironmentID, M-LocationID, ItemID.

Process: M-InstanceID, M-EnvironmentID, ProcessID.

Identifier
With the Rights Item we can express the Actions that a Process can perform on Items, at M-/U-Locations, during a period, e.g.,

Action1 Item1 Location1 T11-T12
Action2 Item2 Location2 T21-T22
Rights
Program is Data (and Metadata) that can be executed.

A Program Item shall be executable in the M-Instance.

A Program Item may be subject to certification before being admitted to an M-Instance

Program
Contract is a special Program that can be activated (Executed) by an external entity, e.g., a User or another already activated Contract. The contract shall include:

1.      Offer: Rights.

2.      Acceptance: By both parties.

3.      Consideration: There may be a Transaction.

The terms of the Contract are enforced in the jurisdiction of the M-Instance.

Contract
An M-Instance/M-Environment may show its Capabilities, i.e., an Item describing the characteristics of an M-Instance/M-Environment, including:

1.      Currencies supported.

2.      Items supported with Data Formats.

3.      Data Types supported.

Capabilities

Here are Items related to the interaction between Processes.

Functional requirements Item
Processes may need to exchange application-level Messages. Message
A Process should be able to expose its Capabilities, i.e., an Item containing a description of its characteristics including:

1.        List of Actions that can be performed.

2.        List of Items supported with Data Formats.

3.        List of Data Types supported.

4.        The cost of performing an Action.

5.        Human represented (User)

6.        Apps on board (Device).

Capabilities
When a Process requests another Process to perform an Action on its ehalf, it issues a Request-Action, an Item including:

1.      Time the Request-Action was issued.

2.      The Source ProcessID.

3.      The Destination ProcessID.

4.      The Action requested.

5.      The ItemIDs relevant to the Action.

6.      The Location of the Items.

7.      The Location of the output Items produced by the Request-Action.

8.      The requested Rights on the output Items.

Request-Action
When a Process has received a Request-Action and succeeds in performing it, it provides a Response-Action, an Item containing:

1.      Time the Response-Action was issued.

2.      The Source ProcessID (Source refers to the Process that issued the request).

3.      The Destination ProcessID.

4.      The output Items produced by performing the Request-Action.

Response-Action
An M-Instance is an abstract entity bearing an Identifier. An M-Instance may expose its Capabilities. M-Instance
An M-Environment is an abstract entity bearing an Identifier.

4.      An M-Environment is hosted by an M-Instance.

5.      An M-Environment may expose its Capabilities.

6.      The Capabilities of an M-Environment may extend the Capabilities of its hosting M-Instance.

M-Environment

Here are Items related to the use of an M-Instance.

Functional requirements Item
Account in an Item that uniquely references a human who has Registered.

A User may have more than one Account with one or more M-Instances or M-Environments.

An Account shall include:

1.      The ID of the Registered human.

2.      An M-Instance-specific subset of the Registered human’s User Data.

3.      The Rights held by each Users in the M-Instance/M-Environment.

4.      The IDs of Devices, Apps, and Users, and Personae.

5.      The validity of:

5.1.   Rights.

5.2.   Account.

Account
Activity Data is an Item containing the record of the Actions made by a User at all M-Locations for a period. Therefore, Activity Data shall include a list of Activities and, for each activity:

1.      The M-LocationID the Activity Data refer to.

2.      The duration (t1-t2) the Activity Data refer to.

3.      The list of Action.

Activity Data
Personal Profile is an Item containing the Data about the human represented by a User. It may include:

1.      First Name

2.      Last Name

3.      Address

4.      Country

5.      Age

6.      Biometric data

7.     

Personal Profile
The Manager of an M-Instance sets Rules, an Item expressing the terms and conditions under which Processes operate in the M-Instance. The Rules may express:

1.      The ability of a User to perform Actions on Items for which it has Rights.

2.      The inability of a User to perform Actions on Items for which it has no Rights.

3.      The duty of a User to perform Actions on Items.

4.      The ability of a User to make Transactions on the Rights of Items.

Rules
Social Graph is a representation of a User’s network of connections with Items and Processes representing the following:

1.      The types and the connections with Items and their M-Locations.

2.      The types and the connections with Devices (frequency of use, etc.).

3.      The types and the connections with Services (frequency of use, etc.).

4.      The types and the connections with Users, groups of Users in terms of:

4.1.   Time

4.2.   M-Locations.

4.3.   Declared purpose.

Social Graph
User Data is an Item that collects all the Data related to a human and their Users:

1.      Rights held by the human’s Users in the M-Instance.

2.      The Personal Profile of the human.

3.      The Personae that the human’s Users impersonate.

4.      The Activity Data of the human’s Users.

5.      The Social Graphs of the human’s Users.

User Data should have a representation that allow easy identification, extraction, and sharing of subsets of a User Data.

User Data

Here are Items with a financial impact.

Functional requirements Item
An Item that may be the object of a Transaction is called Asset. An Asset may be:

1.        MM-Embedded at an M-Location.

2.        Posted to a Service.

An Asset shall:

1.      Preserve the Data Formats of the Item that has spawned it.

2.      Include the date it was created.

Asset
It is useful to consider the Ledger associated with a specific Asset. This Item includes the list of all Transactions executed:

1.      On an Asset.

2.      Starting from the first Transaction and including the last.

3.      The Marketplace on which a Transaction was performed.

Ledger
The Provenance Item shall include the list of all Transactions executed:

4.      On an Asset.

5.      Starting from the first Transaction and including the last.

The Marketplace on which a Transaction was performed.Provenance.

Provenance
Transaction is Item representing the changed state of:

1.        The Rights on an Asset held by a seller User and a buyer User.

2.        The Accounts of the Users and of the Service facilitating/enabling the Transaction (Optional).

The Transaction shall represent:

1.      The Time the Transaction is performed.

2.      The Value moving into the Wallet of User 1 (seller).

3.      The Value moved from the Wallet of User2 (buyer).

4.      The Value moved into the Wallet of User 3 (service) – optional.

5.      The Time the Values were moved.

6.      The Rights to Act owned by User1 after Time.

7.      The Rights to Act owned by User2 after Time.

Transaction
Value is expressed by an Amount and the Currency related to the Amount. It

shall have a representation that enables the expression of the Amount and the Currency used to represent the Amount.

Value
A Wallet is a container of Currency units. A Wallet shall enable the representation of:

1.      Each Currency’s Amounts contained in the Wallet for each Currency.

2.      The Transactions performed.

Wallet

Here are Items specifically used to access a group of Services.

Functional requirements Item
To Authenticate an Entity (an Item that can be perceived), a special Item called AuthenticateIn is produced. This contain:

1.      The (ID of the) Entity Authenticated.

2.      (Optionally) information related to the way AuthenticateOut is rendered.

The Entity to be Authenticated can be:

1.      Speech produced by a User.

2.      The visual appearance of a User, etc.

Information on the rendering of InterpretOut is provided by:

1.      Media type (text, speech, image, etc.) used for rendering.

2.      Spatial Attitude of the Object rendering AuthenticateOut.

AuthenticateIn
AuthenticateOut is the Item containing the result of the Service Acting on the Request-Authenticate Item and information about its rendering. It is rendered as requested in AuthenticateIn. AuthenticateOut
To Discover Items, an Item called DiscoverIn is produced that contains:

1.      A description of the Items to be Discovered.

2.      Information related to the rendering of DiscoverOut.

Items candidate for Discovery may be described by:

1.      Verbal/text description.

2.      Similar Items.

3.      Belonging to specific M-Instances/M-Environments/M-Locations.

4.      Belonging to specific sections of Activity Data.

Information on DiscoverOut Rendering may be provided by:

1.      Media type used for rendering.

2.      Spatial Attitude of the Object rendering DiscoverOut.

DiscoverIn
DiscoverOut is the Item containing the result of the Service Acting on the Request-Discover Item and information about its rendering. It is rendered as requested in DiscoverIn. DiscoverOut
To obtain information on an Item, a User produces InformIn, an Item containing:

1.      A description of the Item about which information is requested.

2.      Information related to the rendering of InformOut.

InformIn may refer to:

1.      Item Metadata

2.      Any other information that a Service may have on the Item.

Information on rendering of InformOut may be provided by:

1.      Media type used for rendering.

2.      Spatial Attitude of the InformOut rendered Object.

InformIn
InformOut is the Item containing the result of the Service Acting on the Request-Inform Item and information about its rendering. It is rendered as requested in InformIn. InformOut
To obtain interpretation of an Item, a User produces InterpretIn, An Item containing:

1.      The ID or the Item to be Interpreted.

2.      Information related to the rendering of InterpretOut.

Items candidate for interpretation may be identified by: Item or ItemID.

Information on InterpretOut Rendering may be provided by:

1.      Media type used for rendering.

2.      Spatial Attitude of InterpretOut rendered Object.

InterpretIn
InterpretOut is the Item containing the result of the Service Acting on the Request-Interpret Item and information about its rendering. It is rendered as requested in InterpretIn. InterpretIn

Here are Items producing a perceptible experience.

Functional requirements Item
An Entity is an Item that can be perceived. MPAI introduces the following perceptible Items: Object, Model, Scene, Event, and Experience. Entity
It is useful to introduce Event, an Entity that includes selected Entities at an M-Location and their Animations during a period. Therefore, an Event shall include:

1.      M-LocationID.

2.      Start Time and End Time.

3.      List of Entities, their Animations, and Interactions.

Event
It is also useful to introduce the Entity Experience, comprising selected Entities of an Event and User Interactions with the Entities of the Event. Therefore, an Experience shall include:

1.      Start Time and End Time

2.      EventID

3.      List of selected Entities, their Animations, and User Interactions.

Experience
Object is an Entity representing an object including:

1.      The type(s) of Media (Audio-Visual-Haptic) composing the Model.

2.      The Data representation.

3.      The Data Format used.

Object
Model is an Object representing an object with its features ready to be MM-Animated or UM-Animated. Model
Persona is a Model representing a human. Persona
Scene is a composition of Objects with the following features:

1.      May be hierarchical.

2.      May be MM-Embedded at a specified M-Location.

3.      Represent Objects:

3.1.   With a Spatial Attitude.

3.2.   Animated by a stream or by an autonomous agent.

Scene
A Stream is an Item made by a continuous flow of Data with the following features:

1.      May be scalable in space and time.

2.      May be used to:

2.1.   Animate a Model.

2.2.   Represent a Digitised Object in an M-Instance.

Stream
Interaction is an Item containing the Request-Action issued by a User on an Entity at an M-Locations and the corresponding Time. Interaction
Map is the basic Item of an AR application. It is an Item containing a structure establishing a correspondence between U-Locations with M-Locations. Therefore, A Map shall include:

1.      The M-Instance the Map refers to.

2.      For each U-Location having one correspondence with an M-Location:

2.1.   The ID of the M-Location corresponding to the U-LocationID.

2.2.   Metadata related to the U-LocationID.

2.3.   Metadata related to the M-LocationID.

Map

Here are Item with a spatial impact.

Functional requirements Item
M-Location is an Identifiable delimited spatial portion of an M-Instance, e.g., the place occupied by representation of a human. An M-Location:

1.      Shall define the space of the M-Instance belonging to the M-Location.

2.      May enable the creation of sub-spaces defining sub-M-Locations

M-Location
U-Location is an Identifiable delimited spatial portion of the Universe, e.g., the place occupied by the human. A U-Location shall:

1.      Define the space in the Universe belonging to the U-Location.

2.      Enable the definition of sub-spaces (sub-U-Locations) comprised in the U-Location.

The enforcement of Rights to a U-Location is not intended to be part of the MPAI-MMM Architecture.

U-Location

Of course, more Items can be identified but those introduced above have been tested to cover a significantly large number of use cases in a variety of application domains.


An Introduction to the MPAI Metaverse Model Architecture – Part II

In part I of this series of posts, we have highlighted the basic elements that enable operation of an M-Instance, especially Processes, Items, Actions and Data Type. In particular, Processes can take the shape of a User (a representative of a human in an M-Instance), a Device (to enable the connection of an M-Instance with the real world, called Universe), and a Service.

We are now going to identify the functional requirements of an initial list of Actions that enable a Process to do useful things in an M-Instance.

Functional requirements Actions
To Register with an M-Instance or M-Environment. This Action may only be performed by a human or a legal entity, not by a User Register (human)
To transmit a Request-Action to a Process. The Request-Action should contain: the Action, the Items involved in the Action, the location where the necessary input Items are located, the location where the produced output Items are located, and the Rights that the requesting Process wishes to hold to be able to perform Actions on the Items produced. Request (Request-Action)
To transmit a Response-Action to a Process. The Response-Action should contain the output Items or an error message. Respond (Response-Action)
To enable a User to increase or diminish the Rights of a Process, e.g., because new Rights have been acquired or because a User has not complied with the Rules, we need the Action Change (Rights of a Process)
To confirm that the speech or the face of a human or an object imported into an M-Instance is from a specific human or U-Location (place in the real world), we need the Action. Note: The User requesting Authentication may also request Rights to use the information received, e.g., to publish it. Authenticate (Item).
To disable access to certain Item no longer accessible by all Processes (assuming that Rights have not been irrevocably granted to a Process) we need the Action Note: The Item may be made accessible again depending on the Rights of the User Hiding the Item. Hide (Item).
To create an Item out of Data and Metadata. For instance, a Device may capture Media as Data subject to certain Rights for use in an M-Instance. Create converts them to an Item usable in the M-Instance because an M-Instance can only Act on Items. Identify (Item)
To create a new Item by modifying an original Item with new or partially new Data and Metadata. For instance, a User with Rights on an Item may wish clone and then modify the components of an existing Item. Modify (Item)
To author an Item by calling a Service and providing it with Data and Metadata. Note: An M-Instance can provide a Service, internal or external to the M-Instance, that Users can call to create Items for use in the M-Instance. Author (Item)
To find Items by giving a description of the Items. Comments: An M-Instance can provide a Service that Users can call to find Items or Processes they need. Alternatively, the M-Instance may allow a User to Call an external Service to find Items of interest also outside of the M-Instance. Discover (Item)
To inform about an Item. A User may wish to know more about an Item, starting from its Metadata but potentially including other information the a Service has collected on the Item. Inform (Item)
To interpret an Item. For instance, a User may need the translation of an utterance produced by an avatar, recognise the face of an avatar, have its own message expressed in sign language into a speech segment. Interpret (Item)
To display an Asset. For instance, a User may wish to manifest its intention to surrender (part of) its Rights on an Asset. This can be done by placing the Asset at an M-Location that other Users can see or by posting it to a marketplace. Post (Asset)
To make a Transaction of an object. A User may like to surrender (part of) the Rights to an Asset to another User, possibly recognising the facilitation role of a Service in the Transaction. At the end of the Transaction the parties making the Transaction have different Rights on the Asset and the status of their Wallets may have changed. Transact (Asset)
To place an Entity (an Item that can be perceived) at an M-Location in such a way that other Users may not perceive it. This may be useful when the User needs to add more Entities to the M-Location without showing the preparations. MM-Add (Entity)
To make an Entity perceptible that was not until that moment, e.g., because the User did not want to show the Entity is not at a given time. MM-Enable (Entity)
To stop making an Entity perceptible, e.g., because the User does not want to show the setting of an event when the event is over. MM-Disable (Entity)
To transmit an Item to a Process. This is done, e.g., when a Device sends Data and Metadata coming from the Universe to a Service that Identifies the Item created from Data and Metadata, when a Uses captures an Entity at an M-Location (i.e., it asks the Service to MM-Send the Entity) or when a User transmits an Entity to a Device for rendering in the Universe. MM-Send (Item or Data and Metadata)
To activate a Contract. i.e., a Program and its Metadata stored on a Device and activated by an external entity, e.g., a User, or another activated Contract. Of course, Contracts may be Executed by an underlying Blockchain. Execute (Contract)
To animate a Model, i.e., an Object in the M-Instance representing an object at a U-Location with its features ready to be animated using a Process that is an autonomous agent. MM-Animate (Item)
To animate a Model using a Process that receives a Stream from a U-Location and animates the Model. The Process may be provided by the M-Instance, the human, or a third-party. UM-Animate
To present Media (i.e., Data and Metadata representing perceptible information) available at a Device to a U-Location as an Entity with an associated Spatial Attitude (i.e., Position and Orientation). For instance, a User may request that a Device present the Media is has received as an Entity from the M-Instance via an MM-Send Action. MU-Actuate (Media)
To present an Entity that is at an M-Location to a U-Location as an Entity with an associated Spatial Attitude. This operation is performed in two steps: MM-Send the Entity to a Device and MU-Actuate the Media from the Device. MU-Render (Entity)
To capture a scene at a U-Location as Media. A User may ask a Device to capture a scene at a U-Location as Media. UM-Capture (scene)
To transmit Data and Metadata to a Process. A User may ask a Device to transmit the Data corresponding to the Media and Device Metadata. UM-Send (Data and Metadata)
To present a scene that is at a U-Location to an M-Location as an Entity with an associated Spatial Attitude. This operation is performed in three steps: the Device captures the scene that is at the U-Location as Media (UM-Capture), then it UM-Sends the Data and Device Metadata to a Service that Identifies the Entity and MM-Embeds it at the M-Location. UM-Render (scene)
To store an Item at an Address, i.e., to an Item to a Device or to store an Item at an Address. MU-Send (Item)
To place a Model at an M-Location, animate it with a Stream, and present the animated Model at a U-Location with an associated Spatial Attitude. In other terms, the round trip real-virtual-real is established. Track (Model)
To verify that a Process has Rights to make an Action on an Item, to preserve the integrity of M-Instance operation. Validate (Process)
To convert an Item of a Request-Action or Response-Action to another Data Format. As for other Services, Convert can be a Service offered by the M-Instance or available outside of the M-Instance. Convert (Item)
To transmit a Request-Action to a Resolution Service to enable a ProcessA in M-InstanceA to communicate with ProcessB in a different M-InstanceB. As for other Services, Resolution can be a Service offered by the M-Instance or available outside of the M-Instance. Request (Request-Action)
To transmit a Response-Action to a Resolution Service that has sent a Request-Action. Respond (Response-Action)

Of course, more Actions can be identified but those introduced above have been tested to cover a significantly large number of use cases in a variety of application domains.

To know more about the MPAI Metaverse Model Architecture and the Call for Technologies, join the online presentations at 8 and 15 UTC on 23 June. Register here for the first and here for the second presentation.