Go to XRV-LTP AI Modules

Function Ref. Model I/O Data SubAIMs JSON MData Profiles Ref. Software Conformance Performance

1. Functions

The Live Theatrical Performance (XRV-LTP) AIM receives Venue-specific Specifications and data from:

  1. The Real and Virtual live-performance Environment
  2. The live performance Script
  3. Historical and current Participant data
  4. Venue-specific command, data, and associations between Script and Venue.

As the Live Theatrical Performance proceeds, the XRV-LTP AIM controls the audience experience according to the Script based on the Actions of the Performance (Performers and related objects), Operators, and Participants.

2. Reference Model

Figure 1 depicts the Reference Model of the Live Theatrical Performance (XRV-LTP) AIM.

Figure 1 – Reference Model of Live Theatrical Performance (XRV-LTP) AIM

3. Input/Output Data

Table 2 specifies the Input and Output Data.

Table 2 – I/O Data of MPAI-XRV – Live Theatrical Performance

Input Description
RE Data In Real-time input data such as App Data, Audio/VJ/DJ, Audio-Visual, Biometric Data, Controller, LiDAR, Lighting/FX, MoCap, Sensor Data, Show Control, Skeleton/Mesh, and Volumetric data.
RE Venue Specification An input to the Environment Description AIM and the Real Experience Generation AIM defining protocols, data formats, and command structures for the specific Real Environment Venue and also includes number, type, and placement of lighting fixtures, special effects, sound and video reproduction resources.
VE Data In App Data, Audio/VJ/DJ, Audio-Visual, Biometric Data, Controller, Lighting/FX, MoCap, Sensor Data, Show Control, Skeleton/Mesh, and Volumetric data.
VE Venue Specification An input to the Virtual Experience Generation AIM defining protocols, data formats, and command structures for the specific Virtual Environment Venue and also includes all actionable elements relevant to the Script including number, type, and placement of lights, effects, avatars, objects, animation scripts, and sound and video reproduction resources.
Participant Data Non-identifying information about each Participant loaded into the system and modified in real time.
Script A machine-readable functional description of the entire show performance including actions to be performed in the Real and Virtual Environment based on the Performance, Operator, and Participant behaviours.
Action Table Commands required to achieve a desired Action in the Real and Virtual Environments per the Script.
Script-Venue Associations Metadata that associates measurable elements of the Script (e.g., characters, objects, participants, operator consoles) with data sources and sinks specific to the Real and Virtual Environments, as defined by the Qualifiers in the Venue Specifications.
Output Description
VE Data Out Parameters controlling 3D geometry, shading, lighting, materials, cameras, physics, and all A/V experiential elements, including audio, video, and capture cameras/microphones in the Virtual Envirnment. The actual format used is specified by the Virtual Environment Venue Specifications.
RE Data Out Parameters controlling all A/V experiential elements including lighting, rigging, FX, audio, video, and cameras/microphones. The actual format used is specified by the Real Environment Venue Specification.
Participant Data Non-identifying information about each Participant updated in real time as the Performance evolves.

4. SubAIMs

4.1 Reference Model

Figure 2 depicts the Reference Model of the Live Theatrical Performance (XRV-LTP) Composite AIM.

Figure 2 – Reference Model of Live Theatrical Performance (XRV-LTP) Composite AIM

4.2 Operation

  1. A range of sensors/controllers collect Data from both the Real and Virtual Environments including audio, video, volumetric or motion capture (MoCap), and avatar data from the Performance, Operators, and Participants.
  2. The Environment Description AIM
    1. Extracts features from Performance, Operators, and Participants,
    2. Conditions the data and provides them as
      1. Performance Descriptors (including description of behaviour of the Performance (Performers and related objects) in the Real and Virtual Environment
      2. Participant Descriptors in the Real and Virtual Environment (describing the audience’s behaviour)
      3. Operator Descriptors in the Real and Virtual Environment (Data from the Show Control computer or control surface, consoles for audio, DJ, VJ, lighting and FX, typically commanded by operators)
      4. Show Data (which includes real-time data streams such as MoCap and Volumetric which are used for the the Real and Virtual Environment Experience Generation).
  3. The Performance, Participant, and Operator Status Interpretation AIMs determine the components of the Performance, Participant, and Operator Descriptors in the Real and Virtual Environments that are relevant to the current Cue Point provided by the Action Generation AIM.
  4. Action Descriptors Generation uses Real and Virtual Environment Performance Status, Participant Status, and Operator Status to direct actions in both the Real and Virtual Environments via RE Action IDs and VE Action IDs, and provides the current Cue Point to the three Interpretation AIMs.
  5. VE Experience Generation and RE Experience Generation use Show Descriptors from Environment Description AIM and in association with RE Action IDs and VE Action IDs to craft actionable commands (via the Action Table) as required by the Real and Virtual Environments – according to their Venue Specifications – to generate multisensory experiences in both the Real and Virtual Environments.

4.3 Functions of AI Modules

Table 2 specifies the Functions of the AI Modules.

Table 2 – Functions of AI Modules

AI Module
Environment Description Process RE Data In and VE Data In and converts them into Performance Descriptors, Participant Descriptors, and Operator Descriptors using the RE Venue Specification and VE Venue Specification.
Performance Status Interpretation Interprets the Performance Descriptors to produce the Performance Status, consisting of status elements relevant to the current Cue Point conditions which are used by Action Descriptor Generation to identify and execute Cues per the Script.
Participant Status Interpretation Interprets the Participant Descriptors to produce the Participant Status, consisting of status elements relevant to the current Cue Point conditions which are used by Action Descriptor Generation to identify and execute Cues per the Script.
Operator Command Interpretation Interprets the Operator Descriptors to produce the Operator Status, consisting of status elements relevant to the current Cue Point conditions which are used by Action  Generation to identify and execute Cues per the Script.
Action Generation Uses Performance Status, Participant Status, and Operator Status to determine the next Cue Point and output Action IDs that reference the Actions necessary to create the complete experience in accordance with the Script.
VE Experience Generation Receives Action IDs from the Action Generation AIM and Show Descriptors from the Environment Description AIM and produces VE Data Out to the Virtual Environment (referencing the Action Table) in accordance with the VE Venue Specification.
RE Experience Generation Receives Action IDs from the Action Generation AIM and Show Descriptors from the Environment Description AIM and produces RE Data Out to the Real Environment (referencing the Action Table) in accordance with the RE Venue Specification.

4.4 I/O Data of AI Modules

Table 2 specifies the Function of the AI Modules.

Table 2 – Functions of AI Modules

AI Module Receives Produces
Environment Description VE Data In
VE Venue Specification
RE Data In
RE Venue Specification
Show Data		 Performance Descriptors
Participant Descriptors
Operator Descriptors
Performance Status Interpretation Performance Descriptors
Cue Point Status		 Performance Status
Participant Status Interpretation Participant Descriptors
Cue Point Status		 Participant Status
Operator Command Interpretation Operator Descriptors
Cue Point Status		 Operator Status
Action Generation Performance Status
Participant Status
Operator Status		 Cue Point Status
RE Action Descriptors
VE Action Descriptors
VE Experience Generation Show Data
VE Action Descriptors
VE Venue Specification		 VE Commands
VE Data Out
RE Experience Generation Show Data
RE Action Descriptors
RE Venue Specification		 RE Commands
RE Data Out

4.5 AIMs and JSON Metadata

Table 4 provides the links to the AIW and AIM specifications and to the JSON syntaxes. AIMs/1 indicates that the column contains Composite AIMs and AIMs indicates that the column contains their Basic AIMs.

Table 4 – AIW, AIMs, and JSON Metadata

AIM AIMs Name JSON
XRV-LTP Live Theatrical Performance X
XRV-END Environment Description X
XRV-PFI Performance Status Interpretation X
XRV-PTI Participant Status Interpretation X
XRV-OCI Operator Command Interpretation X
XRV-ADG Action Descriptor Generation X
XRV-VEG VE Experience Generation X
XRV-REG RE Experience Generation X

5 JSON Metadata

https://schemas.mpai.community/AIH1/V1.0/AIMs/LiveTheatricalPerformance

6 Profiles

No Profiles

7 Reference Software

8 Conformance Testing

9 Performance Assessment