1     Functions 2     Reference Model 3     I/O Data
4     Functions of AI Modules 5     I/O Data of AI Modules 6     AIW, AIMs, and JSON Metadata

1      Functions of Virtual Meeting Secretary

In Avatar Based Videoconference [3], i.e., a videoconference where avatars participate realistically impersonating the human participants, the Virtual Secretary:

  1. Listens to the Speech of each avatar.
  2. Monitors their Personal Status.
  3. Drafts a Summary using the avatars’ Personal Status and Text obtained from Automatic Speech Recognition or directly via Text input in the meeting’s common language handled in two different ways:
    • Transferred to an external application so that participants can edit the Summary.
    • Displayed to avatars:
      • Avatars make Speech comments or Text comments (e.g., offline via chat).
      • The Virtual Secretary edits the Summary interpreting Text, and the avatars’ Personal Statuses.

2      Reference Architecture of Virtual Meeting Secretary

Figure 1 specifies the architecture of the Virtual Secretary AIW. It is assumed that Meaning represents both meaning of Input Text and meaning of Refined Text.

Figure 1 – Reference Model of the Virtual Meeting Secretary Use Case

The Virtual Secretary processes one avatar at a time according to the following workflow:

  1. Portable Avatar Demultiplexing produces Input Text, Input Speech, Avatar ID, Body Descriptors, and Face Descriptors.
  2. Automatic Speech Recognition extracts Text from Avatar Speech.
  3. Natural Language Understanding:
    • Receives Recognised Text.
    • Produces Refined Text (of Recognised Text) and Meaning.
  4. Personal Status Extraction:
    • Receives Meaning, Speech, and Body and Face Descriptors.
    • Produces the Personal Status of the avatar it is interacting with.
  5. Summary Creation Module:
    • Receives Refined Text, Personal Status, and Meaning
    • Produces Summary using Personal Status and Text in the meeting’s common language.
    • Receives Edited Summary from Entity Dialogue Processing.
  6. Entity Dialogue Processing:
    • Sends Summary to external application.
    • Sends Edited Summary produced from Refined Text (from Speech), Avatar’s Text (from chat), Meaning, and Summary back to Summary Creation Module.
    • Produces VS Text and VS Personal Status.
  7. Personal Status Display produces VS Portable Avatar containing VS Avatar Model, VS Text, VS Speech, and VS Avatar Descriptors.

3      I/O Data of Virtual Meeting Secretary

Table 1 gives the input/output data of Virtual Meeting Secretary.

Table 1 – I/O data of Virtual Meeting Secretary

Input data From Description
Portable Avatar Server Portable Avatars as re-multiplexed by Server
Output data To Descriptions
VS Portable Avatar Server VS Portable Avatar to Server
Summary Server Summary of avatars’ interventions

4      Functions of AI Modules of Virtual Meeting Secretary

Table 2 gives the functions of Virtual Meeting Secretary AIMs.

Table 2 – Functions of Virtual Meeting Secretary AI Modules

AIM Functions
Portable Avatar Demultiplexing 1.     Receives Portable Avatar.
2.     Provides the Data required by Virtual Secretary’s AIMs.
Automatic Speech Recognition 1.     Receives Speech.
2.     Provides Recognised Text.
Natural Language Understanding 1.     Refines Recognised Text.
2.     Extracts Meaning.
Personal Status Extraction 1.     Receives Meaning, Input Speech, Body Descriptors, Face Descriptors.
2.     Extracts Personal Status.
Summary Creation Module 1.     Receives Meaning, Refined Text, Avatar ID, Input Personal Status of Avatar ID, and Edited Summary (from Entity Dialogue Processing.)
2.     Produces and refines Summary using Edited Summary.
Entity Dialogue Processing 1.     Receives Input Text, Refined Text, Meaning, Summary, Input Personal Status.
2.     Produces Text, Virtual Secretary Personal Status, and Edited Summary.
Personal Status Display 1.     Receives Virtual Secretary’s Avatar Model, Personal Status, and Text.
2.     Shows Virtual Secretary as Virtual Secretary Portable Avatar.

5      I/O Data of AI Modules of Virtual Meeting Secretary

Table 3 gives the AI Modules of the Virtual Secretary depicted in Figure 4.

Table 3 – AI Modules of Virtual Secretary

AIM Receives Produces
Portable Avatar Demultiplexing Portable Avatar 1.     Input Text
2.     Input Speech
3.     AvatarID
4.    Body Descriptors
5.    Face Descriptors
Automatic Speech Recognition Speech Recognised Text
Natural Language Understanding Recognised Text 1.     Refined Text
2.    Meaning
Personal Status Extraction 1.    Meaning
2.    Speech
3.    Face Descriptors
4.    Body Descriptors		 Personal Status
Summary Creation Module 1.    Meaning
2.     Refined Text
3.     Edited Summary		 Summary
Entity Dialogue Processing 1.     Refined Text
2.    Portable Avatar
3.    Meaning
4.    Summary		 1.     VS Portable Avatar
2.     VS Text
3.     Edited Summary
Personal Status Display 1.     VS Text
2.     VS Personal Status		 VS Portable Avatar

6     AIWs, AIMs, and JSON Metadata of Virtual Meeting Secretary

Table 4 – AIMs and JSON Metadata

AIMs Name  JSON
MMC-VMS Virtual Meeting Secretary X
– PAF-PDX Portable Avatar Demultiplexing X
– MMC-ASR Automatic Speech Recognition X
– MMC-NLU Natural Language Understanding X
– MMC-PSE Personal Status Extraction X
– MMC-ITD Input Text Description X
– MMC-ISD Input Speech Description X
– PAF-IFD Input Face Description X
– PAF-IBD Input Body Description X
– MMC-PTI PS-Text Interpretation X
– MMC-PSI PS-Speech Interpretation X
– PAF-PFI PS-Face Interpretation X
– PAF-PGI PS-Gesture Interpretation X
– MMC-PMX Personal Status Multiplexing X
– MMC-SCM Summary Creation Module X
– MMC-EDP Entity Dialogue Processing X
– PAF-PSD Personal Status Display X
– MMC-TTS Text-To-Speech X
– PAF-IFD Input Face Description X
– PAF-IBD Input Body Description X
– PAF-PMX Portable Avatar Multiplexing X