Skip to main content

Table 4 Comparison of the different QbE STD evaluations: Albanian (ALB), Basque (BAS), Czech (CZE), non-native English (NN-ENG), Isixhosa (ISIX), Isizulu (ISIZ), Romanian (ROM), Sepedi (SEP), Setswana (SET), and Slovak (SLO)

From: ALBAYZIN Query-by-example Spoken Term Detection 2016 evaluation

Evaluation Language/s Type of speech No. of queries dev./test Primary metrics
MediaEval 2011 English, Hindi, Gujarati, and Telugu Tel. 64/36 ATWV
MediaEval 2012 2011 + isiNdebele, Siswati, Tshivenda, and Xitsonga Tel. 164/136 ATWV
MediaEval 2013 ALB, BAS, CZE, NN-ENG, ISIX, ISIZ, ROM, SEP, and SET Tel. and mic. > 600/ > 600 ATWV
MediaEval 2014 ALB, BAS, CZE,NN-ENG, ROM, and SLO Tel. and mic. 560/555 C nxe
NTCIR-11 2014 Japanese Mic. workshop 63/203 F-measure
NTCIR-12 2016 Japanese Mic. workshop 120/1620 F-measure, ATWV, MAP
ALBAYZIN 2012 Spanish Mic. workshop 60/60 ATWV
ALBAYZIN 2014 Spanish Mic. workshop 94/99 ATWV
ALBAYZIN 2016 Spanish Mic. workshop + parliament 102/106+95 ATWV
  1. Tel. telephone, mic. microphone, dev. development, ATWV actual term weighted value, C nxe normalized cross entropy cost, MAP mean average precision