1     Functions  2     Reference Model 3     I/O Data
4     Functions of AI Modules 5     I/O Data of AI Modules 6     AIW, AIMs, and JSON Metadata

1      Functions of Virtual Meeting Secretary

In Virtial Meeting applications, such as in the Avatar Based Videoconference, i.e., a videoconference where avatars participate realistically impersonating the human participants, the Virtual Secretary:

  1. Listens to the Speech of each avatar.
  2. Monitors their Personal Status.
  3. Drafts a Summary using the avatars’ Personal Status and Text obtained from Automatic Speech Recognition or directly via Text input in the meeting’s common language.
  4. The Summary can be made available to participants in two different ways:
    • Transferred to an external application so that participants can edit the Summary.
    • Displayed to avatars where:
      • Avatars make Speech or Text comments (e.g., offline via chat).
      • The Virtual Secretary edits the Summary by understanding Text, Speech, and the avatars’ Personal Statuses.

2      Reference Architecture of Virtual Meeting Secretary

Figure 1 specifies the architecture of the Virtual Secretary AIW. It is assumed that Meaning represents both meaning of Input Text and meaning of Refined Text.

 

Figure 1 – Reference Model of the Virtual Meeting Secretary (MMC-VMS) AIW

The Virtual Secretary processes one avatar at a time according to the following workflow:

  1. Portable Avatar Demultiplexing produces Input Text, Input Speech, Avatar ID, Body Descriptors, and Face Descriptors.
  2. Automatic Speech Recognition extracts Text from Avatar Speech.
  3. Natural Language Understanding:
    • Receives Recognised Text.
    • Produces Refined Text (of Recognised Text) and Meaning.
  4. Personal Status Extraction:
    • Receives Meaning, Speech, and Body and Face Descriptors.
    • Produces the Personal Status of the avatar it is interacting with.
  5. Summary Creation Module:
    • Receives Refined Text, Personal Status, and Meaning
    • Produces Summary using Personal Status and Text in the meeting’s common language.
    • Receives Edited Summary from Entity Dialogue Processing.
  6. Entity Dialogue Processing:
    • Sends Summary to external application.
    • Sends Edited Summary produced from Refined Text (from Speech), Avatar’s Text (from chat), Meaning, and Summary back to Summary Creation Module.
    • Produces VMS Text and VMS Personal Status.
  7. Personal Status Display produces VMS Portable Avatar containing VMS Avatar Model, VMS Text, VMS Speech, and VMS Avatar Descriptors.

3      I/O Data of Virtual Meeting Secretary

Table 1 gives the input/output data of Virtual Meeting Secretary.

Table 1 – I/O data of Virtual Meeting Secretary

Input data From Description
Portable Avatar Server Participants’ Portable Avatars as re-multiplexed by Server
Output data To Descriptions
VMS Portable Avatar Server VMS Portable Avatar to Server
Summary Server Summary of avatars’ interventions

4      Functions of AI Modules of Virtual Meeting Secretary

Table 2 gives the functions of Virtual Meeting Secretary AIMs.

Table 2 – Functions of Virtual Meeting Secretary AI Modules

AIM Functions
Portable Avatar Demultiplexing 1.     Receives Portable Avatar.
2.     Provides the Data required by Virtual Secretary’s AIMs.
Automatic Speech Recognition 1.     Receives Speech.
2.     Provides Recognised Text.
Natural Language Understanding 1.     Refines Recognised Text.
2.     Extracts Meaning.
Personal Status Extraction 1.     Receives Meaning, Input Speech, Body Descriptors, Face Descriptors.
2.     Extracts Personal Status.
Summary Creation Module 1.     Receives Meaning, Refined Text, Avatar ID, Input Personal Status of Avatar ID, and Edited Summary (from Entity Dialogue Processing.)
2.     Produces and refines Summary using Edited Summary.
Entity Dialogue Processing 1.     Receives Input Text, Refined Text, Meaning, Summary, Input Personal Status.
2.     Produces Text, Virtual Secretary Personal Status, and Edited Summary.
Personal Status Display 1.     Receives Virtual Secretary’s Avatar Model, Personal Status, and Text.
2.     Shows Virtual Secretary as Virtual Secretary Portable Avatar.

5      I/O Data of AI Modules of Virtual Meeting Secretary

Table 3 gives the AI Modules of the Virtual Secretary depicted in Figure 4.

Table 3 – AI Modules of Virtual Secretary

AIM Receives Produces
Portable Avatar Demultiplexing Portable Avatar 1.     Input Text
2.     Input Speech
3.     AvatarID
4.    Body Descriptors
5.    Face Descriptors
Automatic Speech Recognition Speech Recognised Text
Natural Language Understanding Recognised Text 1.  Refined Text
2.  Meaning
Personal Status Extraction 1.    Meaning
2.    Speech
3.    Face Descriptors
4.    Body Descriptors		 Personal Status
Summary Creation Module 1.    Meaning
2.   Refined Text
3.   Edited Summary		 Summary
Entity Dialogue Processing 1.    Refined Text
2.    Portable Avatar
3.    Meaning
4.    Summary		 1.     VMS Portable Avatar
2.     VMS Text
3.     Edited Summary
Personal Status Display 1.     VMS Text
2.    VMS Avatar Model
2.     VMS Personal Status		 VMS Portable Avatar

6      Virtual Meeting Secretary AIMs and JSON Metadata

Table 4 – AIMs and JSON Metadata

Note: AIM1/s are Composite AIMs, AIM/2s are Basic AIMs.

AIW AIMs/1 AIMs/2 Name JSON
MMC-VMS Virtual Meeting Secretary X
PAF-PDX Portable Avatar Multiplexing X
MMC-ASR Automatic Speech Recognition X
MMC-NLU Natural Language Understanding X
MMC-PSE Personal Status Extraction X
MMC-ETD Entity Text Description X
MMC-ESD Entity Speech Description X
PAF-EFD Entity Face Description X
PAF-EBD Entity Body Description X
MMC-PTI PS-Text Interpretation X
MMC-PSI PS-Speech Interpretation X
PAF-PFI PS-Face Interpretation X
PAF-PGI PS-Gesture Interpretationn X
MMC-PMX Personal Status Multiplexing X
MMC-SCM Summary Creation Module X
MMC-EDP Entity Dialogue Processing X
PAF-PSD Personal Status Display X
MMC-TTS Text-To-Speech X
PAF-EFD Entity Face Description X
PAF-EBD Entity Body Description X
PAF-PMX Portable Avatar Multiplexing X