Highlights

Public presentation: XRV-LTP

The presentation introduces the scope of the Live Theatrical Performance standard, the real and virtual environments in which any such performance takes place, and the AIMs that interpret what happens on stage at a particular time and then generate the next actions per the script.

Both presentations are open to non-members. The article that follows gives a first introduction to the standard, and another article describes how an AIM written by an Implementer reaches a Controller and becomes part of a running Module – the workflow that every MPAI implementation, including the one followed by XRV-LTP.

Register to attend the public presentation held on 17 November, 18:00 UTC.

XRV-LTP: staging a performance across real and virtual environments

What the standard is for

A live theatrical performance is a sequence of moments. Lights change, a scene is revealed, a performer enters, a sound begins, and spectators react. Today each of them is handled by a different operator or a different system, and the connection between them lives in a script and in the operators’ heads. XRV-LTP – Live Theatrical Performance specifies how to describe a performance in machine-readable language so that AI Modules can monitor it, interpret what happens in the venue and act on that per the script – in a physical theatre, in a virtual world, or across both at the same time.

The standard belongs to the family of Extended Reality Venues (MPAI-XRV): venues where real and virtual worlds meet. In XRV-LTP the Real Environment is the physical venue with its performers, spectators, lights, sound, and stage machinery; the Virtual Environment is a digital copy or extension of it in which avatars, scenery and effects exist; and the two can be connected so an event in one has consequences in the other.

The Script and the Cue List

A performance is described by a Script. The Script contains a Cue List: an ordered set of Cues. A Cue has three parts:

  • the Conditions that must hold for the Cue to fire – a performer in a place, a line spoken, a time reached, a spectator’s reaction;
  • the Actions to carry out, drawn from the Action Table – light, sound, video, scenery, stage machinery, avatar behaviour;
  • the next Cue, so the performance can follow different paths according to what happens.

A Cue can be a Real Cue (acting on the physical venue), a Virtual Cue (acting on the virtual one) or a Mixed Cue (acting on both). The Venue Specifications reveal the configuration of the specific venue – the type and positioning of lights, loudspeakers, displays etc. – so the same Script can easily be staged in multiple venues.

Figure 2 – The Reference Model of XRV-LTP

The AIMs

The architecture allows for reciprocal monitoring and experience generation within and between real and virtual environments. The AIMs include:

  • Environment Description builds a description of the venue(s) – real, virtual or both – from what it senses as guided by the venue specifications.
  • Performance Status Interpretation, Participant Status Interpretation and Operator Status Interpretation interpret what the performers, the participants (the audience) and the operators are doing relevant to the current cue and produce an interpreted status.
  • Action Generation compares the interpreted status with the current Cue Conditions. When Cue Conditions are met, the associated Action ID is handed off to the Experience Generation AIMs and jumps to the next Cue associated with the Cue Conditions.
  • RE Experience Generation and VE Experience Generation turn the Actions into data and commands specific to Real and Virtual Environments (per their Venue Specification) for execution.

As with every MPAI standard, the AIMs are specified by their inputs, outputs and functions, not by the technology inside them. A theatre may implement the interpretation of the performers with a neural network and the interpretation of the audience with a simple sensor; another may do the opposite. The AIMs interoperate because the data they exchange – the Script, the Cue List, the Action Table, the status descriptions – are clearly specified.

Who benefits

  • Theatres and venues can describe a performance once and easily stage it in different venues. Operators may monitor and assist the performance or allow autonomous operation.
  • Technology providers can offer their best AIM – a better interpretation of gestures, a better generation of lighting – and know that it will work in any XRV-LTP system.
  • Audiences can attend a real or virtual environment in person or virtually via the Virtual Environment which can be a digital twin of the performance.

How to take part

XRV-DC welcomes comments on the specification and implementations of its AIMs. The specification is on the MPAI web site (mpai.community), under the MPAI-XRV page.

One page for every MPAI standard

MPAI has twenty-five standards and standard projects, and each now has a one-page introduction at its own address: mpai.community hosts the full web pages for each standard, and the one-pager is at <acronym>.mpai.community – for example aif.mpai.community, cae.mpai.community or xrv.mpai.community. Each page is meant to be read in a few minutes by someone who has never heard of the standard.

What a one-pager contains

  • What the standard does. The problem the standard addresses and what it specifies, in the words of the people who wrote it.
  • Use cases and functions. What can be built with it: the use cases a standard supports, the data types it defines, the functions it provides.
  • Who benefits. Technology providers, application developers and users, as for every MPAI standard.
  • Where to go next. Links to the Technical Specification, and to the Reference Software, the Conformance Testing, and the Performance Assessment where they exist.

Why they were made

MPAI standards are written to be implemented, and Implementers need an entry point shorter than a specification. A one-pager is that entry point: for an engineer who wants to know whether a standard matters to their product, for a journalist or a policy maker who needs the idea without the syntax, for a member who must explain a standard to a colleague. The pages also show how the standards relate: most are built on the AI Framework, common data types are defined in OSD, PAF, TFA, and others, and many use cases draw on AI Modules from several standards.

The one-pagers at a glance

The table lists them in alphabetical order. The acronym is a link to the one-pager. The kind describes the status: a standard, a standard project still in development, or a Technical Report. Each line is a summary of the page, not a substitute for it.

Standard Name Kind In one sentence
MPAI-AIF Artificial Intelligence Framework V3.0 Standard The environment in which AI applications, built as workflows of AI Modules with standard interfaces, are created, deployed and executed – locally or distributed, with trusted services.
MPAI-AIH AI for Health – Health Secure Platform Standard Interoperable health data services: people collect, process and share their health data and keep control of who uses it, for what and under which conditions.
MPAI-AIP Artificial Intelligence Passport Project A standard description of an AI system – identity, function, data semantics, origin, performance and compliance – so that humans and machines can discover, interpret and trust it.
MPAI-AIT Artificial Intelligence Tools Project A repository that turns the standards into working software: components, specifications and the other services of the AI Tools project launched at MPAI-71.
MPAI-CAE Context-based Audio Enhancement Standard AI Modules to improve, preserve and enrich audio and speech: from the preservation of analogue recordings to better audioconferences and expressive speech.
MPAI-CAV Connected Autonomous Vehicle – Technologies Standard The Connected Autonomous Operation of a vehicle as standard components: interaction with humans, understanding the environment, decision making and vehicle control.
MPAI-CUI Compression and Understanding of Industrial Data Standard Company Performance Prediction: AI-based assessment of governance, risk of discontinuity and of failure of a company over a chosen horizon.
MPAI-EEV AI-based End-to-End Video Coding Project Video compression rethought as a single, end-to-end optimised neural system, not as a conventional codec with AI tools added.
MPAI-EVC AI-Enhanced Video Coding Standard Up-sampling Filter for Video Applications: AI-trained super-resolution filters that restore detail after resolution reduction, with control of complexity.
MPAI-GME Governance of the MPAI Ecosystem Standard What MPAI publishes for each standard – Technical Specification, Reference Software, Conformance Testing, Performance Assessment – and who does what with it.
MPAI-GSA Integrative Genomic/Sensor Analysis Project Requirements of the interfaces through which machine and statistical learning accesses processed and unprocessed genomic and sensor data.
MPAI-HMC Human and Machine Communication V2.1 Standard Communicating Entities in Context: humans, digital humans and machines conversing across different languages, cultures and forms.
MPAI-MAS MPAI as a Service V1.0 Project Remote access to AI processing: a Remote Client Application asks an AI Framework instance, over a standard HTTPS interface, to execute AI Modules.
MPAI-MMC Multimodal Conversation V2.5 Standard Conversation systems that combine text, speech, vision and behavioural signals: answering multimodal questions, talking about a scene, conversing with emotion.
MPAI-MMM MPAI Metaverse Model – Technologies Standard A framework for interoperable metaverse ecosystems: Users, Items and Processes acting in Metaverse Instances and in the physical world.
MPAI-NNW Neural Network Watermarking – Traceability Standard Methods to evaluate a traceable neural network, whether watermarked or fingerprinted: robustness, imperceptibility and reliability.
MPAI-OSD Object and Scene Description V1.5 Standard A shared language for objects, scenes, space and time – real and virtual – reused across MPAI specifications.
MPAI-PAF Portable Avatar Format V1.6 Standard Digital humans represented with everything needed to render them as the sender intended: appearance, speech, face and body movement, context.
MPAI-PGM Pursuing Goals in the metaverse Project Starting with the Autonomous User Architecture: a User that perceives, understands, decides and acts in a Metaverse Instance, under the responsibility of a human.
MPAI-PRF AI Module Profiles Standard A label that tells exactly which inputs, outputs and functions an instance of an AI Module supports, readable by machines and by people.
MPAI-PTF Process Instance Trust Framework V1.0 Project Trust among AI Modules and Processes in distributed, zero-trust environments: identity, credentials, attestation, policies and protocols.
MPAI-SPG Server-based Predictive Multiplayer Gaming Technical Report Mitigation of Data Loss Effects: the game server predicts the Game State when data from some players is late or missing.
MPAI-TFA Data Types, Formats and Attributes V1.5 Standard Qualifiers – Sub-Types, Formats and Attributes – that tell an AI Module what the data it receives is, without guessing.
MPAI-WMG AIW and AIM Implementation Guidelines V1.0 Technical Report Guidance for those who implement the AI Workflows and AI Modules of six MPAI Technical Specifications.
MPAI-XRV Extended Reality Venues Project Real and virtual environments working together, with use cases from live theatre to eSports; the first specification is Live Theatrical Performance (XRV-LTP).

 

If you find a page that is unclear, out of date, or silent on something you need, tell the MPAI Secretariat: a one-pager is only useful if a newcomer can understand it.

From the Implementer to the Controller: how a Module comes to life

The AI Framework, Modules and AIMs in brief

The AI Framework (MPAI-AIF) is the MPAI standard that says how an MPAI application is built from parts, so that the parts can come from different Implementers and still work together. Three notions are enough to follow this article.

  • AIM (AI Module). The unit that performs, for example, speech recognition, emotion detect, audio object editing, and interpretation of what a performer does. An AIM is specified by its functional role and by the Data Types and Port Numbers of its input and output ports, not by the technology inside, so anyone can implement it with the technology they prefer and replace it with a better or cheaper one. An AIM can itself be made of AIMs (a Composite AIM), as CAE-ASM is.
  • A set of AIMs connected by a Topology – which output port feeds which input port – to perform a larger function. A Module is an AI application.
  • The component that runs a Module. It finds and obtains the AIMs, checks that they are what their Implementers declared, instantiates them, connects them, gives each the Storage and the Access data it is entitled to, and records what it does.
  • User Agent. The part of the application that the person meets: it asks the Controller to start the Module, it orchestrates its steps, and receives its results.

The functions of the AI Framework are therefore three: to describe an application as AIMs and a Topology, to obtain and verify the AIMs, and to execute them under a Controller that applies the MPAI rules of trust and security.

One rule: nothing is trusted just because of where it runs

In the AI Framework a Module is a set of AIMs connected by a Topology and managed by a Controller. An AIM can be written by anyone, in any programming language, and run by the Controller. For this to work, a Controller that has never met an Implementer must be able to find the AIM, obtain it, check that it is what the Implementer said it is, and start it – and a user must be able to rely on that. The workflow below is the one specified by MPAI-AIF V3.0, and it is the same for the simplest AIM as well as for a Composite AIM such as CAE-ASM.

Figure 3 – From the Implementer to a running Module

The Implementer’s side

  1. The AIM and what it uses. The Implementer writes the software of the AIM. If it needs a model, a large database or a program, it does not carry them inside the AIM: it points to the public place where each is published.
  2. The AIM Instance Metadata. The Implementer describes the AIM in its Metadata: the ports, by Data Type and Port Number; the Implementations, one for each platform, with their fingerprints; and the AccessSources, the data the AIM needs from outside – a model, a database – each identified by its address (URI) and by its SHA-256, with the terms of use of its owner and the place where it is stored. An AIM Licence says under what conditions the AIM may be used.
  3. The Metadata check. The Metadata is checked against the AIM Metadata that the standard specifies for that AIM.
  4. The MPAI Store. The Implementer submits the signed Metadata to the MPAI Store. The Store holds only the Metadata – a few kilobytes – not the Implementations, which stay where the Implementer keeps them. It validates the submission, publishes a new version, and records the fingerprint of each Implementation. Nothing is published that fails validation.

The Controller’s side

  1. Finding the AIMs. The Controller needs the AIMs of the Module it wants to run. It looks first in its own AIM repository, where it keeps what it has already obtained. If it cannot find a suitable Module or any of its AIMs, it asks the Store, which returns the approved Metadata of the Module/AIM and its versions.
  2. Resolving the Topology. The Controller connects the AIMs by the Data Type and the Number for their Ports.
  3. Obtaining what is needed. The Controller obtains the Implementation for its platform from the location in the Metadata, and the models and data of the AccessSources from their addresses. It need not download what it already holds: it compares the SHA-256 and reuses the copy it has. What is downloaded is checked against its SHA-256 before it is used.
  4. Verification. The Controller measures the fingerprint of each Implementation it has and asks the Store: the answer is Same, Different or NotRecorded. A Module/AIM whose hash is not the declared one is refused. An AIM that does not verify is not started: the Module is reported as not trusted.
  5. Instantiation. The Controller instantiates each AIM, which may be run by the Controller or by another one, and sets up the Channels between the AIMs and the Grants that say which AIM may do what. It binds the AIM to the Storage and to the Access data it declared, under the rules of each data instance: who may read it, for how long, and with what trace in the log.
  6. Execution. The User Agent – the part that the person uses – starts the Module through the Controller API and receives its results. The Module can run once on request or continuously.

What the workflow guarantees

  • For the Implementer: the AIM is found and used through its AIM Instance Metadata, regardless of its programming language; any needed AI Models are obtained from their owners, not redistributed by the MPAI Store; its terms of use travel with it.
  • For the User: every Implementation and every AI Model is the one that was approved or declared, and every decision of the Controller is recorded in a signed, hash-chained Trace.
  • For MPAI and its Members: a standard, such as CAE-ASM or XRV-LTP, becomes running software through the same path, so implementations from different Implementers can be composed together in one Module.

Taking part

The AI Framework and the standards built on it are open to anyone who implements them or builds on them. MPAI welcomes those who want to review the specifications, try the Reference Software, implement AIMs, or take part in the development of MPAI standards.

You can find the MPAI software at https://github.com/mpai-community/mpai-sw

 

Meetings in the November 2026 cycle

Group name 02-06 Nov 09-13 Nov 16-20 Nov 23-27 Nov Time (UTC)
AI Framework 2 9 16  23 16
AI-based End-to-End Video Coding 11 15
AI-Enhanced Video Coding 11 13:30
Artificial Intelligence for Health 4  15
AI Passport 18 13:30
23 10
Communication 5  21 14
Connected Autonomous Vehicle 5 12 19  26 15
Context-based Audio enhancement 3 10 17 (*)  24 17
Industry and Standards 6  20 17
MPAI Metaverse Model 5 12 19  26 16
Multimodal Conversation 3 10 17 24 15
Neural Network Watermarking 3 10 17  24 16
Portable Avatar Format 6 13 20 27 9
XR Venues 3 10 17 (*) 24 18
General Assembly (MPAI-74)  25 16

 

(*) Public Presentation