Skip to main content

Table 5 Comparison of the different QbE STD evaluations

From: Search on speech from spoken queries: the Multi-domain International ALBAYZIN 2018 Query-by-Example Spoken Term Detection Evaluation

Evaluation Language/s Type of speech # queries dev./test Primary metrics
MediaEval 2011 English, Hindi, Gujarati, and Telugu Tel. 64/36 ATWV
MediaEval 2012 2011 + isiNdebele,Siswati, Tshivenda, and Xitsonga Tel. 164/136 ATWV
MediaEval 2013 ALB, BAS, CZE, NN-ENG, ISIX, ISIZ, ROM, SEP, and SET Tel. and mic. > 600/ > 600 ATWV
MediaEval 2014 ALB, BAS, CZE,NN-ENG, ROM, and SLO Tel. and mic. 560/555 C nxe
NTCIR-11 2014 Japanese mic. workshop 63/203 F-measure
NTCIR-12 2016 Japanese mic. workshop 120/1620 F-measure ATWV MAP
ALBAYZIN 2012 Spanish mic. workshop 60/60 ATWV
ALBAYZIN 2014 Spanish mic. workshop 94/99 ATWV
ALBAYZIN 2016 Spanish mic. workshop+parliament 102/106+95 ATWV
ALBAYZIN 2018 Spanish mic. workshop+BNews+conv. 102 + 103/106 + 108 + 91 ATWV
  1. Tel. telephone, mic. microphone, BNews broadcast news, conv. conversational, dev. development, ATWV actual term-weighted value, Cnxe normalized cross entropy cost, MAP mean average precision, ALB Albanian, BAS Basque, CZE Czech, NN-ENG non-native English, ISIX Isixhosa, ISIZ Isizulu, ROM Romanian, SEP Sepedi, SET Setswana, SLO Slovak