| 1. Functions | 2. Reference Model | 3. Input/Output Data |
| 4. Functions of AI Modules | 5. Input/output Data of AI Modules | 6. AIW, AIMs, and JSON Metadata |
1. Functions
Data from both the Real and Virtual Environments of a Live Theatrical Performance specified by XRV-LTP (see Figure) include audio, video, volumetric or motion capture (mocap), and avatar data from performers, participants, operators, signals from control surfaces and more as defined by the Real Venue (RE)/Virtual Venue (VE) specifications. This Input Data is processed and converted into actionable commands required by the Real (the “stage”) and Virtual (the “metaverse”) Environments – according to their respective Real and Virtual Venue Specifications to enable multisensory experiences (as defined by the Script) in both the Real and Virtual Environments.
2. Reference Model
Figure 1 depicts the Reference Architecture of Live Theatrical Stage Performance.
Figure 1 – Reference Model of Live Theatrical Stage Performance (XRV-LTP) AIW
This is the flow of operation of the XRV-LTP AIW Reference Model:
- A range of sensors/controllers collect Data from both the Real and Virtual Environments including audio, video, volumetric or motion capture (mocap) and avatar data from performers, signals from control surfaces and more.
- Environment Description extracts features from performers and objects, participants, and operators, conditions the data and provides them as Performance Descriptors (including description of behaviour of performers and objects on the stage or in the metaverse), Participant Descriptors (describing the audience’s behaviour), and Operator Descriptors (Data from the Show Control computer or control surface, consoles for audio, DJ, VJ, lighting and FX, typically commanded by operators) and Show Data (which includes real-time data streams such as MoCap and Volumentric which are used for RE and VE Experience Generation).
- The Performance, Participant, and Operator Status Interpretation AIMs determine the components of the Performance, Participant, and Operator Descriptors that are relevant to the current Cue Point provided by the Action Generation AIM.
- Action Descriptors Generation uses Performance Status, Participant Status, and Operator Status to direct actions in both the Real and Virtual Environments via RE and VE Action Descriptors, and provides the current Cue Point to the three Interpretation AIMs.
- VE Experience Generation and RE Experience Generation uses Show Data from Environment Descriptors and converts RE and VE Action Descriptors into actionable commands required by the Real and Virtual Environments – according to their Venue Specifications – to enable multisensory experience generation in both the Real and Virtual Environments.
3. Input/Output Data
Table 2 specifies the Input and Output Data.
Table 2 – I/O Data of MPAI-XRV – Live Theatrical Stage Performance
| Input | Description |
| RE Data In | Input data such as App Data, Audio/VJ/DJ, Audio-Visual, Biometric Data, Controller, Lidar, Lighting/FX, MoCap, Sensor Data, Show Control, Skeleton/Mesh, and Volumetric data. |
| RE Venue Specification | An input to the Environment Description AIM and the Real Experience Generation AIM defining protocols, data formats, and command structures for the specific Real Environment Venue and also includes number, type, and placement of lighting fixtures, special effects, sound and video reproduction resources. |
| VE Data In | App Data, Audio/VJ/DJ, Audio-Visual, Biometric Data, Controller, Lidar, Lighting/FX, MoCap, Sensor Data, Show Control, Skeleton/Mesh, and Volumetric data. |
| VE Venue Specification | An input to the Virtual Experience Generation AIM defining protocols, data formats, and command structures for the specific Virtual Environment Venue and also includes all actionable elements relevant to the Script including number, type, and placement of lights, effects, avatars, objects, animation scripts, and sound and video reproduction resources. |
| Output | Description |
| VE Data Out | Parameters controlling 3D geometry, shading, lighting, materials, cameras, physics, and all A/V experiential elements, including audio, video, and capture cameras/microphones. The actual format used is specified by the Virtual Environment Venue Specification. |
| VE Commands | Commands controlling 3D geometry, shading, lighting, materials, cameras, physics, and all A/V experiential elements, including audio, video, and capture cameras/microphones. Relevant commands are specified by the Virtual Environment Venue Specification. |
| RE Data Out | Parameters controlling all A/V experiential elements including lighting, rigging, FX, audio, video, and cameras/microphones. The actual format used is specified by the Real Environment Venue Specification. |
| RE Commands | Commands controlling for all A/V experiential elements including lighting, rigging, FX, audio, video, and cameras/microphones. The actual format used is specified by the Real Environment Venue Specification. |
4. Functions of AI Modules
Table 2 specifies the Function of the AI Modules.
Table 2 – Functions of AI Modules
| AI Module | |
| Environment Description | Process RE Data In and VE Data In and converts them into Performance Descriptors, Participant Descriptors, and Operator Descriptors using the RE Venue Specification and VE Venue Specification. |
| Performance Status Interpretation | Interprets the Performance Descriptors to produce the Performance Status, consisting of status elements relevant to the current Cue Point conditions which are used by Action Descriptor Generation to identify the next Cue Point of the Script. |
| Participant Status Interpretation | Interprets the Participant Descriptors to produce the Participant Status, consisting of status elements relevant to the current Cue Point conditions which are used by Action Descriptor Generation to identify the next Cue Point of the Script. |
| Operator Command Interpretation | Interprets the Operator Descriptors to produce the Operator Status, consisting of status elements relevant to the current Cue Point conditions which are used by Action Descriptor Generation to identify the next Cue Point of the Script. |
| Action Descriptor Generation | Uses Performance Status, Participant Status, and Operator Status to determine the next Cue Point and output Action Descriptors that describe the Actions necessary to create the complete experience in accordance with the Script. |
| VE Experience Generation | Processes Action Descriptors and produces VE Commands and VE Data Out to the Virtual Environment in accordance with the VE Venue Specification. |
| RE Experience Generation | Processes Action Descriptors and produces RE Commands and RE Data Out to the Real Environment in accordance with the RE Venue Specification. |
5. Input/output Data of AI Modules
Table 2 specifies the Function of the AI Modules.
Table 2 – Functions of AI Modules
6. AIW, AIMs, and JSON Metadata
Table 4 provides the links to the AIW and AIM specifications and to the JSON syntaxes. AIMs/1 indicates that the column contains Composite AIMs and AIMs indicates that the column contains their Basic AIMs.
Table 4 – AIW, AIMs, and JSON Metadata
| AIM | AIMs | Name | JSON |
| XRV-LTP | Live Theatrical Performance | X | |
| XRV-END | Environment Description | X | |
| XRV-PFI | Performance Status Interpretation | X | |
| XRV-PTI | Participant Status Interpretation | X | |
| XRV-OCI | Operator Command Interpretation | X | |
| XRV-ADG | Action Descriptor Generation | X | |
| XRV-VEG | VE Experience Generation | X | |
| XRV-REG | RE Experience Generation | X |
