Table 5 Comparison of the different QbE STD evaluations
Evaluation | Language/s | Type of speech | # queries dev./test | Primary metrics |
|---|---|---|---|---|
MediaEval 2011 | English, Hindi, Gujarati, and Telugu | Tel. | 64/36 | ATWV |
MediaEval 2012 | 2011 + isiNdebele,Siswati, Tshivenda, and Xitsonga | Tel. | 164/136 | ATWV |
MediaEval 2013 | ALB, BAS, CZE, NN-ENG, ISIX, ISIZ, ROM, SEP, and SET | Tel. and mic. | > 600/ > 600 | ATWV |
MediaEval 2014 | ALB, BAS, CZE,NN-ENG, ROM, and SLO | Tel. and mic. | 560/555 | C nxe |
NTCIR-11 2014 | Japanese | mic. workshop | 63/203 | F-measure |
NTCIR-12 2016 | Japanese | mic. workshop | 120/1620 | F-measure ATWV MAP |
ALBAYZIN 2012 | Spanish | mic. workshop | 60/60 | ATWV |
ALBAYZIN 2014 | Spanish | mic. workshop | 94/99 | ATWV |
ALBAYZIN 2016 | Spanish | mic. workshop+parliament | 102/106+95 | ATWV |
ALBAYZIN 2018 | Spanish | mic. workshop+BNews+conv. | 102 + 103/106 + 108 + 91 | ATWV |