1 Introduction
Testing the Conformance of MMC-CWE, MMC-MQA, and MMC-UST requires datasets to test Data, AIMs, and AIWs. The Data Formats belong to one of Text, Audio, Video, and JSON and should have the characteristics of Table 5:
Table 5 – Data Types for Conformance Testing of MMC-CWE, MMC-MQA, and MMC-UST
| Data Type | Characteristics |
| Text | The texts files are composed of Unicode characters. |
| Speech file | The speech files are conforming .wav files. |
| Video file | The video files are conforming MP4 files [6]. |
| JSON data | The Emotion, Meaning, and Intention files are conforming JSON files. |
Conformance Testing may be carried out using visual and auditory inspection of a human. Appropriate software may replace human a Conformance Tester.
Conformance Testing Datasets are publicly available upon registration.
2 Conversation with Emotion
2.1 Text
2.1.1 Coherent scenarios
| Happy | 1. Today was a wonderful day. I spent quality time with my parents, and the restaurant was excellent as well. I look forward to seeing them again! |
| 2. I’m so excited about Christmas. This year, my girlfriend and I are going to celebrate the holiday together. We’ll decorate our room, and it’ll be so much fun. | |
| 3. Today I watched a movie called ‘The Pianist.’ Not only was it touching, but also very absorbing. Now I feel very happy thanks to the memorable experience. | |
| 4. The weather is awesome these days. It is not too cold, not too hot, and the sun shines beautifully. I look forward to the picnic that is scheduled this weekend. | |
| 5. Nowadays my business is running very smoothly. There are no unexpected issues arising, and my employees are working very diligently. I am very relieved. | |
| Angry | 1. Today my coworkers treated me really badly. They blamed me for the things that were neither my responsibility nor the result of my actions. This is so unfair. |
| 2. I am angry with my sister. She not only does not finish her chores, but forces me to do the chores for her. This is not a new occasion, but this time I can’t, | |
| stand it. | |
| 3. Yesterday I had an argument with a friend of mine. He always wants me to listen to him very carefully and provide advice, but when I’m in need of the help of the same sort, he doesn’t fulfill his duty at all. I’m furious about this. | |
| 4. These days consumer price is skyrocketing. However, the government and political parties are busy blaming the external variables, not trying hard to solve. | |
| the problem that ordinary citizens are facing. Why is there no one trying to be responsible? | |
| 5. Because of my superior in my workplace, I am doing monotonous tasks all day long these days. I have to look at thousands of boring images and classify them each day, which drives me crazy. I cannot but blame my superior. | |
| Neutral | 1. Seoul is the capital city of the Republic of Korea. It is a city of almost ten million residents. According to “The Global Livability Index” Seoul is ranked the fourth most livable city in Asia as of 2023. |
| 2. There is a famous proverb, “Honesty is the best policy.” In essence, it suggests that honesty is the most effective and beneficial approach in various aspects of life. | |
| 3. There is a famous saying, “Don’t judge a book by its cover.” This advises people not to form an opinion or make assumptions about someone or something based solely on its outward appearance. | |
| 4. Global warming refers to the long-term increase in Earth’s average surface temperature due to human activities, primarily the emission of greenhouse gases. Greenhouse gases trap heat in the Earth’s atmosphere, leading to the warming effect. | |
| 5. Inflation is a general increase of the prices of goods and services in an economy. This is usually measured using the consumer price index (CPI). |
2.1.2 Incoherent scenarios
| Text | Meaning | Speech | Face | Sentences |
| Happy | Happy | Angry | Angry | I’m headed to a yoga class now, and then I have a cozy evening planned with a good book. Life is good, for sure. |
| Happy | Happy | Neutral | Neutral | With a big scoop of ice cream in hand, I laughed and played in the park, feeling super happy as the sun shone brightly overhead. |
| Angry | Angry | Happy | Happy | Witnessing my neighbor being rude and disrespectful to an old stranger asking for directions, I couldn’t be sane, because that old man was my father. |
| Neutral | Neutral | Happy | Happy | A political party is an organization that coordinates candidates to compete in a particular country’s elections. It is common for the members of a party to hold similar ideas about politics. |
| Neutral | Neutral | Angry | Angry | According to Max Weber, a state is a compulsory political organization with a centralized government that maintains a monopoly of the legitimate use of force within a certain territory. |
2.2 Audio and Video Files
2.2.1 Neutral
MPAI emotions neutral 1 audio.240309.1041.wav
MPAI emotions neutral 1 video.240309.1041.mp4
MPAI emotions neutral 1.240309.1041.mp4
MPAI emotions neutral 2 audio.240309.1041.wav
MPAI emotions neutral 2 video.240309.1041.mp4
MPAI emotions neutral 2.240309.1041.mp4
MPAI emotions neutral 3 audio.240309.1041.wav
MPAI emotions neutral 3 video.240309.1041.mp4
MPAI emotions neutral 3.240309.1041.mp4
MPAI emotions neutral 4 audio.240309.1041.wav
MPAI emotions neutral 4 video.240309.1041.mp4
MPAI emotions neutral 4.240309.1041.mp4
MPAI emotions neutral 5 audio.240309.1041.wav
MPAI emotions neutral 5 video.240309.1041.mp4
MPAI emotions neutral 5.240309.1041.mp4
2.2.2 Angry
MPAI emotions angry 5.240309.1041.mp4
MPAI emotions angry 5 video.240309.1041.mp4
MPAI emotions angry 5 audio.240309.1041.wav
MPAI emotions angry 4.240309.1041.mp4
MPAI emotions angry 4 audio.240309.1041.wav
MPAI emotions angry 3.240309.1041.mp4
MPAI emotions angry 3 video.240309.1041.mp4
MPAI emotions angry 3 audio.240309.1041.wav
MPAI emotions angry 2.240309.1041.mp4
MPAI emotions angry 2 video.240309.1041.mp4
MPAI emotions angry 2 audio.240309.1041.wav
MPAI emotions angry 1.240309.1041.mp4
MPAI emotions angry 1 video.240309.1041.mp4
MPAI emotions angry 1 audio.240309.1041.wav
2.2.3 Happy
MPAI emotions happy 1 audio.240309.1041.wav
MPAI emotions happy 1 video.240309.1041.mp4
MPAI emotions happy 1.240309.1041.mp4
MPAI emotions happy 2 audio.240309.1041.wav
MPAI emotions happy 2 video.240309.1041.mp4
MPAI emotions happy 2.240309.1041.mp4
MPAI emotions happy 3 audio.240309.1041.wav
MPAI emotions happy 3 video.240309.1041.mp4
MPAI emotions happy 3.240309.1041.mp4
MPAI emotions happy 4 audio.240309.1041.wav
MPAI emotions happy 4 video.240309.1041.mp4
MPAI emotions happy 4.240309.1041.mp4
MPAI emotions happy 5 audio.240309.1041.wav
MPAI emotions happy 5 video.240309.1041.mp4
MPAI emotions happy 5.240309.1041.mp4
2.2.4 Incoherent
MPAI emotions angry text happy voice.240309.1041.mp4
MPAI emotions angry text happy voice audio.240309.1041.wav
MPAI emotions angry text happy voice video.240309.1041.mp4
MPAI emotions happy text angry voice.240309.1041.mp4
MPAI emotions happy text angry voice audio.240309.1041.wav
MPAI emotions happy text angry voice video.240309.1041.mp4
MPAI emotions happy text neutral voice.240309.1041.mp4
MPAI emotions happy text neutral voice audio.240309.1041.wav
MPAI emotions happy text neutral voice video.240309.1041.mp4
MPAI emotions neutral text angry voice.240311.0915.mp4
MPAI emotions neutral text angry voice audio.240311.0915.wav
MPAI emotions neutral text angry voice video.240311.0915.mp4
MPAI emotions neutral text happy voice audio.240309.1041.wav
MPAI emotions neutral text happy voice video.240309.1041.mp4
2.3 Emotion JSON Files
The JSON files below represent Happy, Angry, and Neutral Emotions.
{
“EmotionType”:{
“emotionDegree”:”high”,
“emotionName”:”happy”,
“emotionSetName”:”MPAI Basic Emotion Set”
}
}
{
“EmotionType”:{
“emotionDegree”:”high”,
“emotionName”:”happy”,
“emotionSetName”:”MPAI Basic Emotion Set”
}
}
{
“EmotionType”:{
“emotionDegree”:”high”,
“emotionName”:”happy”,
“emotionSetName”:”MPAI Basic Emotion Set”
}
}
2.4 Meaning JSON Files
Sentence 1: Today was a wonderful day! I spent quality time with my parents, and the McDonald restaurant was excellent, too. I’m looking forward to seeing them again!
{
“meaning”: {
“POS_tagging”: {
“POS_tagging_set”: “CST’s Part-Of-Speech tagger (Brill, with adaptations)”,
“POS_tagging_result”: “Today/RB was/VBD a/DT wonderful/JJ day/NN !/. I/PRP spent/VBD quality/NN time/NN with/IN my/PRP$ parents/NNS ,/, and/CC the/DT McDonald/NNP restaurant/NN was/VBD excellent/JJ ,/, too/RB ./. I’m/NNP looking/VBG forward/RB to/TO seeing/VBG them/PRP again/RB !/.”
},
“NE_tagging”: {
“NE_tagging_set”: “CST’s named entity recogniser”,
“NE_tagging_result”: ” [Today,misc,uncertain] was a wonderful day ! I spent quality time with my parents, and the [McDonald,person,likely] restaurant was excellent , too . I’m looking forward to seeing them again!”
},
“dependency_tagging”: {
“dependency_tagging_set”: “CG-dependency, https://edu.visl.dk/visl/en/parsing/automatic/dependency.php “,
“dependency_tagging_result”: “<ß>\nToday [today] <*> <atemp> ADV @ADVL> #1->2\nwas [be] <mv> V IMPF 1/3S @FS-STA #2->0\na [a] <indef> ART S @>N #3->5\nwonderful [wonderful] ADJ POS @>N #4->5\nday [day] <dur> <per> <idf> <nhead> N S NOM @<SUBJ #5->2\n! [!] PU @PU #6->0\n</s>\n<ß>\nI [I] <*> PERS 1S NOM @SUBJ> #1->2\nspent [spend] <cjt-head> <mv> V IMPF @FS-STA #2->0\nquality [quality] <f-q> <f-phys> <comp1> <first> <idf> <comp1> <ncomp> N S NOM @>N #3->4\ntime [time] <ac-cat> <temp> <per> <num+> <second> <comp2> <idf> <nhead> N S NOM @<ACC #4->2\nwith [with] PRP @<ADVL #5->2\nmy [I] <poss> <refl> <det> PERS 1S GEN @>N #6->7\nparents [parent] <Hfam> <def> <nhead> N P NOM @P< #7->5\n, [,] PU @PU #8->0\nand [and] <clb?> <co-fin> KC @CO #9->2\nthe [the] <def> ART S/P @>N #10->12\nMcDonald [McDonald] <*> <Proper> <first> <ncomp> N S NOM @>N #11->12\nrestaurant [restaurant] <inst> <second> <def> <nhead> N S NOM @SUBJ> #12->13\nwas [be] <cjt> <mv> V IMPF 1/3S @FS-STA #13->2\nexcellent [excellent] <Q:good> ADJ POS @<SC #14->13\n, [,] PU @PU #15->0\ntoo [too] ADV @<ADVL #16->13\n. [.] PU @PU #17->0\n</s>\n<ß>\nI-m [I-m] <*> <unit> <ac-sign> <heur> <idf> <nhead> N S NOM @SUBJ> #1->2\nlooking [look] <mv> V PCP1 @ICL-ADVL #2->0\nforward [forward] <adir> <advl-close> ADV @<ADVL #3->2\nto [to] <advl-close> PRP @<ADVL #4->2\nseeing [see] <vq> <v.contact> <vtk+ADJ> <mv> V PCP1 @ICL-P< #5->4\nthem [they] PERS 3P ACC @<ACC #6->5\nagain [again] <atemp> ADV @<ADVL #7->5\n! [!] PU @PU #8->0\n</ß>”
},
“SRL_tagging”: {
“SRL_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“SRL_tagging_result”: “Today/ARG1 was/PRED (a wonderful day)/ARG2 ! I/ARG0 spent/PRED (quality time)/ARG1 (with my parents)/ARG2, and (the McDonald restaurant)/ARG1 was/PRED excellent/ARG2, too/ARGM-ADV. I/ARG0’m looking/PRED forward/ARGM-DIR (to seeing them again)/ARG1!”
}
}
}
Sentence 2: I’m really excited about Christmas! This year, my girlfriend and I are gonna celebrate the holiday together. We’re gonna decorate our room, and it’ll be so much fun!
{
“meaning”: {
“POS_tagging”: {
“POS_tagging_set”: “CST’s Part-Of-Speech tagger (Brill, with adaptations)”,
“POS_tagging_result”: ” I’m/NNP really/RB excited/VBD about/IN Christmas/NNP !/.\nThis/DT year/NN ,/, my/PRP$ girlfriend/NN and/CC I/PRP are/VBP gon/VBG na/TO celebrate/VB the/DT holiday/NN together/RB ./. We’re/NNP gon/VBG na/TO decorate/VB our/PRP$ room/NN ,/, and/CC it’ll/NN be/VB so/RB much/JJ fun/NN !/.”
},
“NE_tagging”: {
“NE_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“NE_tagging_result”: ” I’m really excited about Christmas/DATE ! This year, my girlfriend and I are gonna celebrate the holiday together. We’re gonna decorate our room, and it’ll be so much fun! ”
},
“dependency_tagging”: {
“dependency_tagging_set”: “CG-dependency, https://edu.visl.dk/visl/en/parsing/automatic/dependency.php “,
” dependency_tagging_result”: “\n<ß>\nI-m [I-m] <*> <unit> <ac-sign> <heur> <idf> <nhead> N S NOM @NPHR #1->0\nreally [really] <ly> <ameta> <ADJ:real+ly> ADV @>A #2->3\nexcited [excited] <np-close> ADJ POS @N< #3->1\nabout [about] <pp-temp> PRP @A< #4->3\nChristmas [Christmas] <*> <temp> <per> <nhead> N S NOM @P< #5->4\n! [!] PU @PU #6->0\n</s>\n<ß>\nThis [this] <*> <dem> DET S @>N #1->2\nyear [year] <per> <dur> <def> <nhead> N S NOM @ADVL> #2->10\n, [,] PU @PU #3->0\nmy [I] <poss> <det> PERS 1S GEN @>N #4->5\ngirlfriend [girlfriend] <cjt-head> <Hfam> <def> <nhead> N S NOM @SUBJ> #5->8\nand [and] <co-subj> KC @CO #6->5\nI [I] <cjt> <*> PERS 1S NOM @SUBJ> #7->5\nare [be] <vch> <aux> V PR -1/3S @FS-STA #8->0\ngonna [going=to] <complex> <aux> V PCP1 @ICL-AUX< #9->8\ncelebrate [celebrate] <mv> V INF @ICL-AUX< #10->9\nthe [the] <def> ART S/P @>N #11->12\nholiday [holiday] <temp> <per> <def> <nhead> N S NOM @<ACC #12->10\ntogether [together] ADV @<ADVL #13->10\n. [.] PU @PU #14->0\n</s>\n<ß>\nWe-re [We-re] <*> <Hmyth> <rem> <heur> <idf> <nhead> N S NOM @SUBJ> #1->3\ngonna [going=to] <cjt-head> <complex> <aux> V PCP1 @FS-STA #2->0\ndecorate [decorate] <v.contact> <mv> V INF @ICL-AUX< #3->2\nour [we] <poss> <det> PERS GEN 1P @>N #4->5\nroom [room] <Lh> <am> <def> <nhead> N S NOM @<ACC #5->3\n, [,] PU @PU #6->0\nand [and] <clb?> KC @CO #7->5\nit-ll [it-ll] <heur> <idf> <nhead> N S NOM @SUBJ> #8->9\nbe [be] <cjt> <mv> V SUBJ @FS-STA #9->2\nso [so] <aquant> ADV @>A #10->11\nmuch [much] <quant> DET ABS S @>N #11->12\nfun [fun] <sem-c> <percep-f> <idf> <nhead> N S NOM @<SC #12->9\n! [!] PU @PU #13->0\n</ß>”
},
“SRL_tagging”: {
“SRL_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“SRL_tagging_result”: ” I/ARG1 ’m/PRED (really excited about Christmas)/ARG2! This year, (my girlfriend and I) /ARG0 are gonna celebrate/PRED (the holiday)/ARG1 together/ARGM-MNR. We/ARG0 ’re gonna decorate/PRED (our room)/ARG1, and it/ARG1 ’ll/ARGM-MOD be/PRED (so much fun)/ARG2 !”
}
}
}
Sentence 3: Today I watched a movie called ‘The Pianist.’ It was not only touching, but really absorbing, too. Now I’m feeling really happy, thanks to this memorable experience.
{
“meaning”: {
“POS_tagging”: {
“POS_tagging_set”: “CST’s Part-Of-Speech tagger (Brill, with adaptations)”,
“POS_tagging_result”: ” Today/RB I/PRP watched/VBD a/DT movie/NN called/VBN ‘/” The/DT Pianist/NNP ./. ‘/POS It/PRP was/VBD not/RB only/RB touching/VBG ,/, but/CC really/RB absorbing/VBG ,/, too/RB ./.\nNow/RB I’m/NNP feeling/NN really/RB happy/JJ ,/, thanks/NNS to/TO this/DT memorable/JJ experience/NN ./.”
},
“NE_tagging”: {
“NE_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“NE_tagging_result”: ” Today I watched a movie called ‘The Pianist.’/WORK_OF_ART It was not only touching, but really absorbing, too. Now I’m feeling really happy, thanks to this memorable experience.”
},
“dependency_tagging”: {
“dependency_tagging_set”: “CG-dependency, https://edu.visl.dk/visl/en/parsing/automatic/dependency.php “,
” dependency_tagging_result”: “\n<ß>\nToday [today] <*> <atemp> ADV @ADVL> #1->3\nI [I] <*> PERS 1S NOM @SUBJ> #2->3\nwatched [watch] <DL:bio> <mv> V IMPF @FS-STA #3->0\na [a] <indef> ART S @>N #4->5\nmovie [movie] <sem-w> <DL:bio> <idf> <nhead> N S NOM @<ACC #5->3\ncalled [call] <vtk+N> <vtk+ADJ> <vtk+N> <vtk+PROP> <vq> <v.contact> <DL:bio> <mv> <np-close> V PCP2 PAS @ICL-N< #6->5\n-The [-The] <heur> <DL:bio> <idf> <nhead> N S NOM @<SC #7->6\nPianist [Pianist] <*> <Proper> <DL:bio> <nhead> N S NOM @<OC #8->6\n. [.] PU @PU #9->0\n<ß>\n- [-] PU @PU #1->0\n</ß>\n</s>\n<ß>\nIt [it] <*> PERS NEU 3S NOM @SUBJ> #1->2\nwas [be] <DL:bio> <mv> V IMPF 1/3S @FS-STA #2->0\nnot [not] ADV @>A #3->4\nonly [only] <ly> <ADJ:on+ly> <advl-close> ADV @<ADVL #4->2\ntouching [touching] <DL:bio> ADJ POS @<SC #5->2\n, [,] PU @PU #6->0\nbut [but] KC @CO #7->5\nreally [really] <ly> <ameta> <ADJ:real+ly> ADV @ADVL> #8->9\nabsorbing [absorb] <v.contact> <DL:bio> <mv> V PCP1 @ICL-N<PRED #9->1\n, [,] PU @PU #10->0\ntoo [too] <advl-close> ADV @<ADVL #11->9\n. [.] PU @PU #12->0\n</s>\n<ß>\nNow [now] <*> <atemp> ADV @ADVL #1->0\nI-m [I-m] <*> <unit> <ac-sign> <DL:bio> <heur> <nhead> N S NOM @NPHR #2->1\nfeeling [feel] <v.contact> <v-cog> <DL:bio> <mv> <np-close> V PCP1 @ICL-N<PRED #3->2\nreally [really] <ly> <ameta> <ADJ:real+ly> ADV @>A #4->5\nhappy [happy] <jpsych> <DL:bio> ADJ POS @<SC #5->3\n, [,] PU @PU #6->0\nthanks to [thanks=to] <insertion> <complex> PRP @<ADVL #7->3\nthis [this] <dem> DET S @>N #8->10\nmemorable [memorable] <DL:bio> ADJ POS @>N #9->10\nexperience [experience] <f-psych> <percep-f> <DL:bio> <def> <nhead> N S NOM @P< #10->7\n. [.] PU @PU #11->0\n</ß>”
},
“SRL_tagging”: {
“SRL_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“SRL_tagging_result”: “Today/ARG-TMP I/ARG0 watched/PRED a movie/ARG1 called/PRED ‘The Pianist.’ It/ARG1 was/PRED (not only touching, but really absorbing, too)/ARG2. Now/ARG-TMP I/ARG0 ’m feeling/PRED (really happy)/ARG1, thanks to this memorable experience.”
}
}
}
3 Multimodal Question Answering
3.1 Text Files
Q1: What is the tool in the picture?
Q2: What is the nickname of the person in the picture?
Q3: What is the job of the person on the left hand-side in the picture
Q4: What is the family name of the person in the centre of the picture?
Q5: What is the name of the square in the picture?
3.2 Audio Files
Q1.wav
Q2.wav
Q3.wav
Q4.wav
Q5.wav
3.3 Images
| Images for Q1 | Q1-1.jpg Q1-2.jpg Q1-3.jpg |
| Image for Q2 | Q2-Joseph Gordon Levitt.jpg |
| Image for Q3 | Q3-1.jpg Q3-2.jpg |
| images for Q4 | Q4-1.jpg Q4-1.jpg Q4-3.jpg |
| 1 image for Q5 | Q5-1.jpg |
3.4 Meaning JSON Files
Sentence 1: What is the tool in the picture?
{
“meaning”: {
“POS_tagging”: {
“POS_tagging_set”: “CST’s Part-Of-Speech tagger (Brill, with adaptations), https://cst.dk/online/pos_tagger/uk/”,
“POS_tagging_result”: “What/WP is/VBZ the/DT tool/NN in/IN the/DT picture/NN ?/.”
},
“NE_tagging”: {
“NE_tagging_set”: “CST’s named entity recogniser, https://cst.dk/online/navnegenkenderCSTNER/uk/”,
“NE_tagging_result”: ” [What,misc,uncertain] is the tool in the picture ?”
},
“dependency_tagging”: {
“dependency_tagging_set”: “CG-dependency, https://edu.visl.dk/visl/en/parsing/automatic/dependency.php “,
” dependency_tagging_result”: “<ß>\nWhat [what] <clb> <*> <interr> INDP S/P @SC> #1->2\nis [be] <mv> V PR 3S @FS-QUE #2->0\nthe [the] <def> ART S/P @>N #3->4\ntool [tool] <tool> <def> <nhead> N S NOM @<SUBJ #4->2\nin [in] <advl-fs> PRP @<ADVL #5->2\nthe [the] <def> ART S/P @>N #6->7\npicture [picture] <pict> <repr> <def> <nhead> N S NOM @P< #7->5\n? [?] PU @PU #8->0\n</ß>”
},
“SRL_tagging”: {
“SRL_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“SRL_tagging_result”: ” What/ARG2 is/PRED (the tool in the picture)/ARG1 ?”
}
}
}
Sentence 2: What is the nickname of the person in the picture?
What/WP is/VBZ the/DT nickname/NN of/IN the/DT person/NN in/IN the/DT picture/NN ?/.
{
“meaning”: {
“POS_tagging”: {
“POS_tagging_set”: “CST’s Part-Of-Speech tagger (Brill, with adaptations)”,
“POS_tagging_result”: ” What/WP is/VBZ the/DT nickname/NN of/IN the/DT person/NN in/IN the/DT picture/NN ?/.”
},
“NE_tagging”: {
“NE_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/ner.html “,
“NE_tagging_result”: “”
},
“dependency_tagging”: {
“dependency_tagging_set”: “CG-dependency, https://edu.visl.dk/visl/en/parsing/automatic/dependency.php “,
” dependency_tagging_result”: ” <ß>\nWhat [what] <clb> <*> <interr> INDP S/P @SC> #1->2\nis [be] <mv> V PR 3S @FS-QUE #2->0\nthe [the] <def> ART S/P @>N #3->4\nnickname [nickname] <ac-cat> <def> <nhead> N S NOM @<SUBJ #4->2\nof [of] <np-close> PRP @N< #5->4\nthe [the] <def> ART S/P @>N #6->7\nperson [person] <H> <def> <nhead> N S NOM @P< #7->5\nin [in] <advl-fs> PRP @<ADVL #8->2\nthe [the] <def> ART S/P @>N #9->10\npicture [picture] <pict> <repr> <def> <nhead> N S NOM @P< #10->8\n? [?] PU @PU #11->0\n</ß>”
},
“SRL_tagging”: {
“SRL_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“SRL_tagging_result”: ” What/ARG2 is/PRED (the nickname of the person in the picture)/ARG1 ?”
}
}
}
Sentence 3: What is the job of the person on the left hand-side in the picture?
{
“meaning”: {
“POS_tagging”: {
“POS_tagging_set”: “CST’s Part-Of-Speech tagger (Brill, with adaptations)”,
“POS_tagging_result”: ” What/WP is/VBZ the/DT job/NN of/IN the/DT person/NN on/IN the/DT left/VBN hand-side/JJ in/IN the/DT picture/NN ?/.”
},
“NE_tagging”: {
“NE_tagging_set”: ” https://cst.dk/online/navnegenkenderCSTNER/uk/”,
“NE_tagging_result”: ” [What,misc,uncertain] is the job of the person on the left hand-side in the picture ?”
},
“dependency_tagging”: {
“dependency_tagging_set”: “CG-dependency, https://edu.visl.dk/visl/en/parsing/automatic/dependency.php “,
” dependency_tagging_result”: ” <ß>\nWhat [what] <clb> <*> <interr> INDP S/P @SC> #1->2\nis [be] <mv> V PR 3S @FS-QUE #2->0\nthe [the] <def> ART S/P @>N #3->4\njob [job] <pos-soc> <sem-c> <def> <nhead> N S NOM @<SUBJ #4->2\nof [of] <np-close> PRP @N< #5->4\nthe [the] <def> ART S/P @>N #6->7\nperson [person] <H> <def> <nhead> N S NOM @P< #7->5\non [on] <advl-fs> PRP @<ADVL #8->2\nthe [the] <def> ART S/P @>N #9->11\nleft [left] ADJ POS @>N #10->11\nhand-side [hand-side] <Lsurf> <HH> <geom> <heur> <def> <nhead> N S NOM @P< [hand-side] <heur> <def> N S NOM @P< #11->8\nin [in] <advl-fs> PRP @<ADVL #12->2\nthe [the] <def> ART S/P @>N #13->14\npicture [picture] <pict> <repr> <def> <nhead> N S NOM @P< #14->12\n? [?] PU @PU #15->0\n</ß>”
},
“SRL_tagging”: {
“SRL_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“SRL_tagging_result”: ” What/ARG2 is/PRED (the job of the person on the left hand-side in the picture)/ARG1 ?”
}
}
}
Sentence 4: What is the family name of the person in the centre of the picture?
{
“meaning”: {
“POS_tagging”: {
“POS_tagging_set”: “CST’s Part-Of-Speech tagger (Brill, with adaptations)”,
“POS_tagging_result”: ” What/WP is/VBZ the/DT family/NN name/NN of/IN the/DT person/NN in/IN the/DT centre/NN of/IN the/DT picture/NN ?/.”
},
“NE_tagging”: {
“NE_tagging_set”: ” https://cst.dk/online/navnegenkenderCSTNER/uk/”,
“NE_tagging_result”: ” [What,misc,uncertain] is the family name of the person in the centre of the picture ?”
},
“dependency_tagging”: {
“dependency_tagging_set”: “CG-dependency, https://edu.visl.dk/visl/en/parsing/automatic/dependency.php “,
” dependency_tagging_result”: ” <ß>\nWhat [what] <clb> <*> <interr> INDP S/P @SC> #1->2\nis [be] <mv> V PR 3S @FS-QUE #2->0\nthe [the] <def> ART S/P @>N #3->5\nfamily [family] <HH> <comp1> <comp1> <ncomp> N S NOM @>N #4->5\nname [name] <ac-cat> <comp2> <def> <nhead> N S NOM @<SUBJ #5->2\nof [of] <np-close> PRP @N< #6->5\nthe [the] <def> ART S/P @>N #7->8\nperson [person] <H> <def> <nhead> N S NOM @P< #8->6\nin [in] <advl-fs> PRP @<ADVL #9->2\nthe [the] <def> ART S/P @>N #10->11\ncentre [centre] <Labs> <inst> <def> <nhead> N S NOM @P< #11->9\nof [of] <np-close> PRP @N< #12->11\nthe [the] <def> ART S/P @>N #13->14\npicture [picture] <pict> <repr> <def> <nhead> N S NOM @P< #14->12\n? [?] PU @PU #15->0\n</ß>”
},
“SRL_tagging”: {
“SRL_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“SRL_tagging_result”: ” What/ARG2 is/PRED (the family name of the person in the centre of the picture)/ARG1 ?”
}
}
}
Sentence 5: What is the name of the square in the picture?
{
“meaning”: {
“POS_tagging”: {
“POS_tagging_set”: “CST’s Part-Of-Speech tagger (Brill, with adaptations)”,
“POS_tagging_result”: ” What/WP is/VBZ the/DT name/NN of/IN the/DT square/NN of/IN the/DT picture/NN ?/.”
},
“NE_tagging”: {
“NE_tagging_set”: ” https://cst.dk/online/navnegenkenderCSTNER/uk/”,
“NE_tagging_result”: “[What,misc,uncertain] is the name of the square of the picture ?”
},
“dependency_tagging”: {
“dependency_tagging_set”: “CG-dependency, https://edu.visl.dk/visl/en/parsing/automatic/dependency.php “,
” dependency_tagging_result”: ” \n<ß>\nWhat [what] <clb> <*> <interr> INDP S/P @SC> #1->2\nis [be] <mv> V PR 3S @FS-QUE #2->0\nthe [the] <def> ART S/P @>N #3->4\nname [name] <ac-cat> <sem-c> <def> <nhead> N S NOM @<SUBJ #4->2\nof [of] <np-close> PRP @N< #5->4\nthe [the] <def> ART S/P @>N #6->7\nsquare [square] <Lh> <geom> <def> <nhead> N S NOM @P< #7->5\nof [in] <np-close> PRP @N< #8->7\nthe [the] <def> ART S/P @>N #9->10\npicture [picture] <pict> <repr> <def> <nhead> N S NOM @P< #10->8\n? [?] PU @PU #11->0\n</ß>”
},
“SRL_tagging”: {
“SRL_tagging_set”: “HanLP, https://hanlp.hankcs.com/en/demos/srl.html”,
“SRL_tagging_result”: ” What/ARG2 is/PRED (the name of the square of the picture)/ARG1 ?”
}
}
}
3.5 Intention JSON Files
Q1: What is the tool in the picture?
{
“Intention”: {
“qtopic”: “tool”,
“qfocus”: “What”,
“qLAT”: “tool”,
“qSAT”: “ETC”,
“qdomain”: “everyday life”
}
}
Q2: What is the nickname of the person in the picture?
{
“Intention”: {
“qtopic”: “person”,
“qfocus”: “What”,
“qLAT”: “nickname”,
“qSAT”: “PS_NAME”,
“qdomain”: “famous people”
}
}
Q3: What is the job of the person on the left hand-side in the picture
{
“Intention”: {
“qtopic”: “person”,
“qfocus”: “What”,
“qLAT”: “job”,
“qSAT”: “CV_OCCUPATION”,
“qdomain”: “famous people”
}
}
Q4: What is the family name of the person in the centre of the picture?
{
“Intention”: {
“qtopic”: “person”,
“qfocus”: “What”,
“qLAT”: “family name”,
“qSAT”: “PS_NAME”,
“qdomain”: “famous people”
}
}
Q5: What is the name of the square in the picture?
{
“Intention”: {
“qtopic”: “square”,
“qfocus”: “What”,
“qLAT”: “square”,
“qSAT”: “LC_TOUR”,
“qdomain”: “traveling”
}
}
4 Unidirectional Speech Translation
4.1 Text Files
The Text Files of Conversation with Emotion are used.
4.2 Audio Files
The Audio Files of Conversation with Emotion are used.
