XR Venues (MPAI-XRV) is a project addressing a multiplicity of use cases enabled by Extended Reality (XR) – Extended Reality (XR), the combination of Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR) – technologies and enhanced by Artificial Intelligence (AI) technologies. Venue is used as a synonym for Real and Virtual Environments.

Among the different use cases analysed there are: Live Theatrical Performance,  Collaborative Immersive Laboratory, eSports Tournament, and Experiential retail/shopping. The development of Technical Specification: XR Venues (MPAI-XRV) – Live Theatrical Performance (XRV-LTP) has been stimulated by theatrical stage performances such as Broadway theatres, musicals, dramas, operas, and other performing arts that increasingly use video scrims, backdrops, and projection mapping to create digital sets and animated backdrops rather than constructing physical stage elements. Digital stages reduce the cost of mounting shows and enable automation. Immersion domes – especially LED volumes – promise to surround audiences with virtual environments that the live performers can inhabit and interact with.

The setting up and operation of Live Theatrical Performances (LTP) is costly, requires the support of extended staff, and is generally tied to a specific venue making it difficult to reproduce LTP events at different venues.

Application of Artificial Intelligence technologies to suitable points of LTP event workflows makes them less costly in terms of equipment and personnel involved and more flexible and receptive of the show director intention. Use of XR technologies offer further opportunities to extend the stage into the metaverse as a digital twin according to the model of Figure 1 where the first three yellow blocks – Data Capture, Feature Extraction, and Feature Interpretation process data from the Real Environment (i.e., stage and surrounding space). Their interpretation allows the Action block to generate commands that generate and apply a Virtual Experience to the Virtual Environment (in the following also called Metaverse) but also can modify the Real Environment as well. The blue blocks perform operations the same starting from data captured from the Virtual Environment.

Figure 1 – Real (yellow) and Virtual Environment (blue) Interactions.

In this case, elements of the Virtual Environment experience can be projected in the Real Environment and elements of the Real Environment experience can be rendered in the Virtual Environment.

Technical Specification: XR Venues (MPAI-XRV) – Live Theatrical Performance (XRV-LTP) provides the technologies to implement the above vision thus enabling:

  1. Rapid mounting of a show into a variety of real and virtual venues.
  2. Orchestration of the complex lighting, video, audio, and stage set cues that must adapt to the pace of live performers without extensive staff.
  3. Large shows to tour to smaller venues that otherwise could not support complex productions.
  4. Live performances spanning both Virtual and Real Environments, including in-person or remote participants and performers while offering enhanced participant interactivity.
  5. A more direct connection between the artist and participants by consolidating many complex experiential modalities into a simple user interface.
  6. A more direct, precise yet spontaneous show implementation and control to achieve the show director’s vision freeing staff from repetitive and technical tasks while allowing them to amplify their artistic and creative skills.
  7. Access by artists to a large amount of data from opted-in individuals which can be incorporated into the visual and musical performance. Each show can thus be unique for each audience.

Important note:

The MPAI-XRV – Live Theatrical Stage Performance use case involves the collection of large amounts of potentially sensitive Participant Data. This document does not address the processes that oversee the collection and processing of Participant Data. Rather, MPAI-XRV – Live Theatrical Stage Performance assumes that whatever processing is carried out, it conforms with the necessary and ethical/legal constraints, e.g., with the consent of the right holders of the data[1],[2]. Implementers must take great care in data security, assuring the correct possibility to opt-in/opt-out offered and proper use of the data. Also, care must be taken in training and testing AI Models to assure conformance with local laws and regulations and to prevent offensive or unintended experiences.

In all Chapters and Sections, Terms beginning with a capital letter are defined in Table 1 if they are specific to this Technical Specification and in Table 2 if they are common to all MPAI Technical Specifications. All Chapters, Sections, and Annexes are Normative unless they are labelled as Informative.

[1] https://openvoicenetwork.org/documents/ovn_ethical_guidlines_voice_experiences.pdf

[2] https://ec.europa.eu/futurium/en/ai-alliance-consultation.1.html