Homogenous ensemble phonotactic language recognition based on SVM supervector reconstruction

Wei-Wei Liu; Wei-Qiang Zhang; Michael T Johnson; Jia Liu

doi:10.1186/s13636-014-0042-5

Impact Factor 1.579

EURASIP Journal on Audio, Speech, and Music Processing Cover Image

Research

Open Access

Homogenous ensemble phonotactic language recognition based on SVM supervector reconstruction

Wei-Wei Liu^{1, 2},
Wei-Qiang Zhang¹Email author,
Michael T Johnson³ and
Jia Liu¹

EURASIP Journal on Audio, Speech, and Music Processing20142014:42

https://doi.org/10.1186/s13636-014-0042-5

Received: 14 May 2014

Accepted: 28 October 2014

Published: 24 December 2014

Abstract

Currently, acoustic spoken language recognition (SLR) and phonotactic SLR systems are widely used language recognition systems. To achieve better performance, researchers combine multiple subsystems with the results often much better than a single SLR system. Phonotactic SLR subsystems may vary in the acoustic features vectors or include multiple language-specific phone recognizers and different acoustic models. These methods achieve good performance but usually compute at high computational cost. In this paper, a new diversification for phonotactic language recognition systems is proposed using vector space models by support vector machine (SVM) supervector reconstruction (SSR). In this architecture, the subsystems share the same feature extraction, decoding, and N-gram counting preprocessing steps, but model in a different vector space by using the SSR algorithm without significant additional computation. We term this a homogeneous ensemble phonotactic language recognition (HEPLR) system. The system integrates three different SVM supervector reconstruction algorithms, including relative SVM supervector reconstruction, functional SVM supervector reconstruction, and perturbing SVM supervector reconstruction. All of the algorithms are incorporated using a linear discriminant analysis-maximum mutual information (LDA-MMI) backend for improving language recognition evaluation (LRE) accuracy. Evaluated on the National Institute of Standards and Technology (NIST) LRE 2009 task, the proposed HEPLR system achieves better performance than a baseline phone recognition-vector space modeling (PR-VSM) system with minimal extra computational cost. The performance of the HEPLR system yields 1.39%, 3.63%, and 14.79% equal error rate (EER), representing 6.06%, 10.15%, and 10.53% relative improvements over the baseline system, respectively, for the 30-, 10-, and 3-s test conditions.

Keywords

Phonotactic language recognition Support vector machine (SVM) supervector reconstruction Phone recognition-vector space modeling (PR-VSM)

1 Introduction

Spoken language recognition (SLR) refers to the task of automatic determination of language identity. It is estimated that there are about 6,000 spoken languages in the world [1]. An increasing number of multilingual speech processing applications require spoken language recognition as a frontend, with the result that SLR continues to grow in importance. Spoken language recognition is an enabling technology for a wide range of intelligence and security applications for information distillation, such as spoken document retrieval, multilingual speech recognition, and spoken language translation [2].

Language cues can be categorized according to their level of knowledge abstraction as acoustic (spectrum, phone inventory), prosodic (duration, pitch, intonation), phonotactic (sequence of sounds), lexical (vocabulary, morphology), and syntax (phrases, grammar) [3],[4]. Language recognition systems are usually identified by the features they employ, e.g., acoustic systems, phonotactic systems, prosodic systems, and lexical systems. Currently, acoustic language recognition (LR) systems [5] and phonotactic LR systems [3] are both widely used.

Generally, the performance of SLR systems can be improved in two ways: (1) longitudinally, through the development of new techniques to perform the SLR tasks more precisely, e.g., i-vector [6]-[8], JFA [9], discriminative training [10] methods, N-gram modeling methods [3], and support vector machines (SVMs) [11]; (2) transversely, by adding variety to the SLR subsystems, which extracts and integrates more information from the utterances. State-of-the-art language recognition systems fuse multiple subsystems in parallel via a post-processing backend [12]. In the National Institute of Standards and Technology (NIST) language recognition evaluation (LRE) tasks, teams from all over the world compete to build the best SLR system and have shown that better results can be obtained by combining more subsystems, creating larger and larger SLR systems. In NIST LRE 2011, all submitted language recognition systems were stacked ensembles of at least five language recognition subsystems [13]-[15]. Much effort goes into trying different variations of subsystems. Generally, the phonotactic LR subsystems can be varied in three ways: (a) extracting various acoustic features to provide feature diversification, for example, Mel-Frequency Cepstral Coefficients (MFCC) [16], Perceptual Linear Predictive (PLP) [17], and Temporal Patterns Neural Network (TRAPs/NN) [18]; (b) training phone recognizers on multiple language-specific speech data to provide phonetic diversification [3], e.g., the Russian, Hungarian, Czech, and English phone recognizers developed by Brno University of Technology (BUT) [18] or universal phone recognizer (UPR) [19]; and (c) training phone recognizers on the same language-specific speech data but using different acoustic models to provide acoustic diversification [20], such as the Artificial Neural Network-Hidden Markov Model (ANN-HMM) [21], Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) [22], and Deep Neural Network-Hidden Markov Model (DNN-HMM) [23]. Certainly, in phonotactic language recognition systems, the subsystems must undergo different process of feature extracting, decoding, N-gram counting, and vector space modeling, which means an added computational cost of N times than single subsystem, where N is the number of the subsystems.

This paper demonstrates an architecture to provide a new diversification for phonotactic language recognition systems. The underlying motivation of these algorithms is to provide richer language identifying information without significant additional computation. The subsystems are verified using SVM supervector reconstruction (SSR) algorithms to provide vector space modeling diversification. In this architecture, the subsystems share the same preprocessing of feature extracting, decoding, and expected counting, but models in different vector space, so we call it homogeneous ensemble phonotactic language recognition (HEPLR) system. The HEPLR subsystems increase the variety of the SVM supervector, decrease the computational cost, and improve the SLR accuracy. There are many SVM supervector reconstruction algorithms such as recurrent neuron network (RNN) SVM supervector reconstruction [24]. In this paper, we present three SVM supervector reconstruction algorithms including relative SVM reconstruction [25], functional SVM reconstruction, and perturbative SVM reconstruction.

The remainder of the paper is organized as follows. Section 3 presents the lattice-based phonotactic language recognition system used as a baseline in this paper. Section 3 describes the proposed approaches, includes relative, functional, and perturbative SVM supervector reconstruction. Section 3 demonstrates the architecture for the homogeneous ensemble phonotactic language recognition system. The experimental setup to evaluate our proposed method is described in Section 3. Results obtained in language recognition experiments on the NIST LRE 2009 database are presented and discussed in Section 3. Finally, conclusions and future work are outlined in Section 3.

2 Baseline phonotactic SLR system

In this work, the phone recognition-vector space modeling (PR-VSM) [26] phonotactic language recognition system is employed as a baseline system. The motivation behind phonotactic language recognition approach is the belief that a spoken language can be characterized by special probabilities of its lexical-phonological constraints. An N-gram vector space model (VSM) is built for the language recognition task using phone transcriptions, which is a stochastic model describing the probabilities of phoneme strings. In the PR-VSM system, each N-gram VSM of produces a likelihood score by SVM classifier to a given utterance. The languages used for training the phone recognizers need not be the same with any of those recognized.

The traditional PR-VSM language recognition system works by mapping the input utterances from data space into a high-dimensional feature space : Φ:→ and then building linear machines in the feature space to find a maximal margin separation. The vectors built in the high-dimensional feature space are SVM supervectors, which consist of N-gram counts of features representing the phonotactics of an input speech wave sample.

In PR-VSM systems, an utterance x can be mapped to the high-dimensional feature space as follows:

Φ : x \to φ (x),

(1)

where φ(x) is the SVM supervector computed as

φ (x) = [p (d_{1} | ℓ_{x}), p (d_{2} | ℓ_{x}), \dots, p (d_{F} | ℓ_{x})],

(2)

where ℓ_x is the lattice produced from data x by a phone recognizer, d_i is the N-gram phoneme string [27]d_i=s_i…s_i+n−1 (n=N), and F is the dimension of the SVM supervector. p(d_i|ℓ_x) is the probability of the phoneme sequence d_i in the lattice, which is computed as

p (d_{i} | ℓ_{x}) = \frac{c (d_{i} | ℓ_{x})}{\sum_{i} c (d_{i} | ℓ_{x})},

where c(d_i|ℓ_x) denotes the N-gram occurrence of d_i given the lattice ℓ_x. This is calculated over all possible hypotheses in the lattice as follows [27]:

\begin{matrix} c (s_{i} \dots s_{i + N - 1} | ℓ) = \sum_{S \in ℓ} p (S | ℓ) c (s_{i} \dots s_{i + N - 1} | S) \\ = \sum_{s_{i} \dots s_{i + N - 1} \in ℓ} [α (s_{i}) β (s_{i + N - 1}) \prod_{j = i}^{i + N - 1} ξ (s_{j})], \end{matrix}

where p(S|ℓ) denotes the probability of the sequence S in the lattice ℓ, α(s_i) and β(s_i+N−1) are the forward probability of the starting node and the backward probability of the ending node in the N-gram s_i…s_i+N−1, respectively. ξ(s_j) is the posterior probability of the edge phoneme s_j.

The SVM supervector φ(x) is sent to the SVM classifier and a decision is made based on the most likely hypothesis score. In a PR-VSM system, the decision is made based on the SVM output score using

f (φ (x)) = \sum_{l} α_{l} K_{TFLLR} (φ (x), φ (x_{l})) + d,

(3)

where φ(x_l) are support vectors obtained from training set using the Mercer condition. The term frequency log-likelihood ratio (TFLLR) kernel K_TFLLR is computed as [28]:

K_{TFLLR} (φ (x_{i}), φ (x_{j})) = \sum_{q = 1}^{F} \frac{p (d_{q} | ℓ_{x_{i}})}{\sqrt{p (d_{q} | ℓ_{all})}} \frac{p (d_{q} | ℓ_{x_{j}})}{\sqrt{p (d_{q} | ℓ_{all})}},

(4)

where p(d_i|ℓ_all) is calculated from the observed probability of d_i across all lattices. In this paper, the training stage is always carried out with a one-versus-rest strategy between the positive set (the samples in the target language) and negative set (all other samples).

3 SVM supervector reconstruction algorithms

The motivation behind SVM supervector reconstruction is to provide vector space modeling diversification to improve the performance of the overall language recognition system. In the language recognition system employed in this paper, we focus on how a change in the input to the SVM affects the output.

Given an SVM supervector φ(x), we define a function ϕ_SSR which operates on φ(x):

Φ_{SSR} : x \to ϕ_{SSR} (φ (x)) .

(5)

We are interested in understanding how ϕ_SSR(φ(x)) affects the behavior of the output scores of the SVM. The goal is to define the relationship between φ(x) and ϕ_SSR(φ(x)) to enhance the variety of the supervector input.

Selecting SVM supervector reconstruction methods is an open question, so here we propose some typical methods to the implementation. In this section, three SVM supervector reconstruction methods are proposed: relative SVM supervector reconstruction, functional SVM supervector reconstruction, and perturbative SVM supervector reconstruction. Relative SVM supervector reconstruction has been presented in [25], while functional and perturbative reconstructions are new methods. Relative reconstruction is a linear reconstruction, while functional and perturbative reconstructions are non-linear ones.

3.1 Relative SVM supervector reconstruction

The relative SVM supervector method uses a relative feature approach. Relative features in contrast to absolute features, which represent directly calculable information, are defined by the relationship between an utterance and a set of selected datum utterances. We have presented the concept of relative features in [25].

Calculating relative features requires a relationship measurement, such as distance on similarity. By selecting a proper relationship measurement, the identifiable characteristics can be strengthened and nuisance attributes of the utterance can be discarded. Unlike absolute features, relative features make utterances more convenient to classify by showing the relationship between the utterances and the datum database directly.

Here, we introduce a relative SVM supervector reconstruction defined using the similarity between the utterance SVM supervectors. The widely used kernel methods offer efficient similarity measurements between two SVM supervectors. In this paper, the empirical kernel [29] is introduced into language recognition and a relativized SVM supervector developed. Kernel methods have been used for face recognition [30] and handwritten digit recognition [31] and achieved higher robustness to noise [30]. Using the SVM supervectors that are already built into a language recognition system, we can easily compose a new relativized SVM supervector with only a small increase in computation.

The architecture of the relative SVM supervector reconstruction subsystem is shown in Figure 1. To construct the SVM supervector relativization map, a database s =[s₁,s₂,…s_m] containing m utterances is used as the datum mark of similarity. The datum database is stochastically selected from some corpus, whose language need not be the same with the target language. s is mapped into vector space:

s \to φ (s) = [φ (s_{1}), φ (s_{2}), \dots, φ (s_{m})] .

(6)

Figure 1
**Architecture of relative SVM supervector reconstruction subsystem.**

The vector relativizational (VR) kernel between two supervectors φ(x_i) and φ(x_j) is

\begin{matrix} K_{VR} (φ (x_{i}), φ (x_{j})) = < φ (x_{i}), φ (x_{j}) > \\ = \sum_{q = 1}^{F} \frac{p (d_{q} | ℓ_{x_{i}})}{\sqrt{p (d_{q} | ℓ_{S})}} \frac{p (d_{q} | ℓ_{x_{j}})}{\sqrt{p (d_{q} | ℓ_{S})}} . \end{matrix}

(7)

The VR kernel is similar to the TFLLR kernel, but normalized by the observed probability across all lattices of the datum dataset p(d_i|ℓ_s). This kernel reflects the degree of similarity between two supervectors.

The utterance x is mapped from the input data space to a relativized m-dimensional Euclidean space ^m: Φ_REL:→^m as follows:

\begin{matrix} Φ_{REL} : x \to φ_{REL} (x) \\ = K_{VR} (φ (x), φ (s)) = < φ (x), φ (s) > \\ = [K_{VR} (φ (x), φ (s_{1})), \dots, K_{VR} (φ (x), φ (s_{m}))] . \end{matrix}

In general, K_VR(φ(x),φ(S )) defines a space in which each dimension corresponds to the similarities to a prototype. Thus, K_VR(·,φ(S )) can be viewed as a mapping onto an m-dimensional relativized vector space.

The SVM output score is computed as

f^{'} (φ_{REL} (x)) = \sum_{l^{'}} α_{l^{'}} K^{'} (φ_{REL} (x), φ_{REL} (x_{l^{'}})) + d^{'},

(8)

where

φ_{REL} (x_{l^{'}})

are support vectors obtained from the training set using the Mercer condition. Selecting a radial basis function (RBF) kernel, K RBF′ is computed as

\begin{matrix} K_{RBF}^{'} (φ_{REL} (x_{i}), φ_{REL} (x_{j})) = exp (- \frac{| φ_{REL} (x_{i}) - φ_{REL} (x_{j}) |^{2}}{D_{RF}}), \end{matrix}

(9)

where D_RF is the dimension of the relativized SVM supervector. Selecting a TFLLR kernel, K TFLLR′ is computed as

\begin{matrix} K_{TFLLR}^{'} (φ_{REL} (x_{i}), φ_{REL} (x_{j})) \\ = \sum_{q = 1}^{m} \frac{K_{VR} (φ (x_{i}), φ (s_{q})) K_{VR} (φ (x_{j}), φ (s_{q}))}{K_{VR} (φ (x_{all}), φ (s_{q}))} . \end{matrix}

(10)

3.2 Functional SVM supervector reconstruction

In actual test conditions, the training and test data are variable in speakers, background noise, and channel conditions. To achieve higher robustness to variable test conditions, the widely used kernel methods offer efficient similarity measurements between two SVM supervectors in PR-VSM system [30]. The geometrical structure of the SVM vector space is completely determined by the kernel, so the selection of the kernel has a crucial impact on the performance of the language recognition systems. The functional SVM supervector reconstruction method defines a mixture between the functional and the original kernels, which can offer the robust discriminative information of the data and get robust language model.

But how to select a proper function is an open problem. There are many functions that can be used to the reconstruction, while not every function is available for the reconstruction that can reduce the equal error rate (EER). What we need to do is to find out what kind of functions can be used in feature reconstruction. The functions need to satisfy the following conditions: (1) monotonic and (2) can make the identifiable characteristics strengthened and nuisance attributes of the utterance discarded. The proposed functional SVM supervector reconstruction method does not rely on prior knowledge to select the functional to reconstruct the supervector. A development database is used for cross validation to select the function. So, here, three functions selected to be used in this paper include

(a)

$φ_{FUN} (p (d_{i} | ℓ_{x})) = sin (p (d_{i} | ℓ_{x})) + cos (p (d_{i} | ℓ_{x})),$

(11)
(b)

$φ_{FUN} (p (d_{i} | ℓ_{x})) = p (d_{i} | ℓ_{x}) + (p (d_{i} | ℓ_{x}))^{2},$

(12)
(c)

$φ_{FUN} (p (d_{i} | ℓ_{x})) = p (d_{i} | ℓ_{x}) - {(p (d_{i} | ℓ_{x}))}^{2} + {(p (d_{i} | ℓ_{x}))}^{3} .$

(13)

The utterance x is mapped onto a functionalized vector space:

\begin{matrix} Φ_{FUN} : x \to φ_{FUN} (x) \\ = [φ_{FUN} (p (d_{1} | ℓ_{x})), φ_{FUN} (p (d_{2} | ℓ_{x})), \dots, φ_{FUN} (p (d_{F} | ℓ_{x}))] . \end{matrix}

(14)

The three functions are all monotonic in the range of amplitude of the SVM supervector. Selecting a TFLLR kernel, K TFLLR′ is computed as

\begin{matrix} K_{TFLLR}^{'} (φ_{FUN} (x_{i}), φ_{FUN} (x_{j})) \\ = \sum_{q = 1}^{F} \frac{φ_{FUN} (p (d_{q} | ℓ_{x_{i}})) φ_{FUN} (p (d_{q} | ℓ_{x_{j}}))}{φ_{FUN} (p (d_{q} | ℓ_{all}))} . \end{matrix}

(15)

The SVM output score is computed as

f^{'} (φ_{FUN} (x)) = \sum_{l^{'}} α_{l^{'}} K^{'} (φ_{FUN} (x), φ_{FUN} (x_{l^{'}})) + d^{'} .

(16)

3.3 Perturbational SVM supervector reconstruction

For spoken language recognition, the first and most essential step is to tokenize the running speech into sound units or lattices using a phone recognizer. The phoneme error rate is around 40% to 60% [32] when tokenizing an utterance. The decoding errors are deletion, insertion, and substitution errors, which are expressed as some discrete ‘noise’ when mapped to the high-dimensional SVM supervector space (shown in Figure 2). So, here, we introduce a perturbational denoising method for the SVM supervector. Given a supervector φ(x) and some perturbation operator on φ(x), we are interested in understanding how a small perturbation added to the supervector affects the behavior of the SVM [33]. This relationship can be represented using a mapping onto a perturbational vector space.

Figure 2
**Effections of decoding errors on SVM supervector.**

There are three purposes of proposing perturbational SVM supervector reconstruction method: first, adding perturbational noise to reduce the impact of noise in the SVM supervector introduced by the decoding errors; second, generating a more robust language model to provide input variety to the SVM classifier; and third, highlighting the most discriminative information of the SVM supervector and drowning the non-discriminative information into the perturbation (shown in Figure 3).

Figure 3
**SVM supervector of an utterance (a) before and (b) after perturbation.**

To accomplish the above goals, the type and strength of the perturbation must be selected carefully. How to define a proper perturbation is an open problem. There are a wide variety of perturbations,which can be categorized into multiple ways, including (1) global perturbation and local perturbation, (2) stochastic perturbation and constant perturbation according to the amplitude, (3) absolute perturbation and relative perturbation according to the relationship between the SVM supervector and the perturbation, and (4) addictive perturbation and multiplicative perturbation.

For feature supervectors in vector space, the perturbations are always discrete, maybe random in a certain range or change with the amplitude of the expected value of the supervector. So, we consider both deterministic perturbation $δ = w^{*} E_{p (d | ℓ_{x})}$ and stochastic perturbation $δ = w^{*} uniform [0, E_{p (d | ℓ_{x})}]$ , where $E_{p (d | ℓ_{x})}$ is the mean of the SVM supervector, $uniform [0, E_{p (d | ℓ_{x})}]$ is the uniform distribution between 0 and $E_{p (d | ℓ_{x})}$ , and w^∗ is the perturbation weight. More details of the perturbation methods are discussed below.

3.3.1 Perturbational approach 1 (deterministic additive perturbation)

δ = w^{*} E_{p (d | ℓ_{x})}, φ_{PER} (p (d_{i} | ℓ_{x})) = p (d_{i} | ℓ_{x}) + δ .

(17)

This kind of perturbation represents the assumption that the expected count of every phoneme sequence is perturbed by an equivalent additive amount.

3.3.2 Perturbational approach 2 (stochastic additive perturbation)

δ = w^{*} random [0, E_{p (d | ℓ_{x})}], φ_{PER} (p (d_{i} | ℓ_{x})) = p (d_{i} | ℓ_{x}) + δ .

(18)

This perturbation represents the assumption that the expected count of every phoneme sequence is perturbed by an amount proportional to the frequency of the phoneme sequences.

3.3.3 Perturbational approach 3 (deterministic multiplicative perturbation)

δ = w^{*} E_{p (d | ℓ_{x})}, φ_{PER} (p (d_{i} | ℓ_{x})) = p (d_{i} | ℓ_{x}) δ .

(19)

This kind of perturbation represents the assumption that the expected count of every phoneme sequence perturbed by an equivalent multiple amount.

3.3.4 Perturbational approach 4 (stochastic multiplicative perturbation)

δ = w^{*} random [0, E_{p (d | ℓ_{x})}], φ_{PER} (p (d_{i} | ℓ_{x})) = p (d_{i} | ℓ_{x}) δ .

(20)

This perturbation represents the assumption that the expected count of every phoneme sequence is perturbed by a proportional to the frequency of the phoneme sequences.

From above, it can be seen that methods 1 and 2 implement absolute perturbation, and methods 3 and 4 implement relative perturbation. All are global perturbation algorithms, operating across the entire vector space. We can also investigate local perturbation using these same approaches. Local perturbation is more flexible and realistic for the noises would have effect on part of the expected counting. The proposed methods also do not rely on prior knowledge to put noising into the supervector; we use development database for cross validation to select a better perturbation.

The utterance x is mapped onto a perturbational vector space:

\begin{matrix} Φ_{PER} : x \to φ_{PER} (x) \\ = [φ_{PER} (p (d_{1} | ℓ_{x})), φ_{PER} (p (d_{2} | ℓ_{x})), \dots, φ_{PER} (p (d_{F} | ℓ_{x}))] . \end{matrix}

(21)

where φ_PER(x) is a perturbation of φ(x). Selecting a TFLLR kernel, K TFLLR′ is computed as

\begin{matrix} K_{TFLLR}^{'} (φ_{PER} (x_{i}), φ_{PER} (x_{j})) \\ = \sum_{q = 1}^{F} \frac{φ_{PER} (p (d_{q} | ℓ_{x_{i}})) φ_{PER} (p (d_{q} | ℓ_{x_{j}}))}{φ_{PER} (p (d_{q} | ℓ_{all}))} . \end{matrix}

(22)

The SVM output score is computed as

f^{'} (φ_{PER} (x)) = \sum_{l^{'}} α_{l^{'}} K^{'} (φ_{PER} (x), φ_{PER} (x_{l^{'}})) + d^{'} .

(23)

4 Homogeneous ensemble language recognition system

The architecture of the HEPLR system is shown in Figure 4. All the SVM supervectors are reconstructed into the corresponding vector space and fused at the vector level for training and testing. In this paper, we use the classical method of vector fusion, which is to group several sets of reconstructed SVM supervectors into a large composite supervector [34]. Suppose

φ_{{REL}_{N_{1_{1}}}} (x)

, …,

φ_{{REL}_{N_{1_{m}^{'}}}} (x)

,

φ_{{FUN}_{N_{2_{1}}}} (x)

, …,

φ_{{FUN}_{N_{2_{m}^{'}}}} (x)

, and

φ_{{PER}_{N_{3_{1}}}} (x)

, …,

φ_{{PER}_{N_{3_{m}^{'}}}} (x)

are the reconstructed SVM supervectors for an input utterance x. The concatenated SVM supervectors can be represented as φ_REL(x), φ_FUN(x), and φ_PER(x), respectively. Denoting each SVM supervector as d_m-dimensional, the concatenated SVM supervectors are (d₁+…+d m′)-dimensional. The concatenated SVM supervectors are defined by

φ_{REL} (x) = [w_{N_{1_{1}}} φ_{{REL}_{N_{1_{1}}}} (x), \dots, w_{N_{1_{m}^{'}}} φ_{{REL}_{N_{1_{m}^{'}}}} (x)],

(24)

Figure 4
**Architecture of the HEPLR system.**

φ_{FUN} (x) = [w_{N_{2_{1}}} φ_{{FUN}_{N_{2_{1}}}} (x), \dots, w_{N_{2_{m}^{'}}} φ_{{FUN}_{N_{2_{m}^{'}}}} (x)],

(25)

φ_{PER} (x) = [w_{N_{3_{1}}} φ_{{PER}_{N_{3_{1}}}} (x), \dots, w_{N_{3_{m}^{'}}} φ_{{PER}_{N_{3_{m}^{'}}}} (x)],

(26)

where

w_{N_{j_{i}}} = {min}_{\forall i} (E_{j_{n}}) / E_{j_{i}}

(j=1,2,3) with

E_{j_{i}}

the priori knowledge of the EER performance of the development data of the subsystem. The logistic regression optimized weighting (LROW) method is used to optimize the reconstructed SVM supervector weighting coefficients. Since not all the SVM supervector reconstruction subsystems are effective when fused, we also extend the work by formulating quantitative measures to select the subsystems for fusion. The output score of the SVM classifier is computed as follows:

f^{*} (φ^{*} (x)) = \sum_{l^{*}} α_{l^{*}} K^{*} (φ^{*} (x), φ^{*} (x_{l^{*}})) + d^{*},

(27)

where the reconstruction methods are represented by means of ‘*’, and φ^∗(x_l∗) are support vectors obtained from the reconstructed SVM supervectors using the Mercer condition.

As mentioned previously, in the HEPLR language recognition system, the training stage is carried out between the positive set and negative set with one-versus-rest strategy.

The linear discriminant analysis-maximum mutual information (LDA-MMI) method is used to maximize the posterior probabilities of all the belief score vectors [35], using objective function [36]:

F_{MMI} (λ) = \sum_{\forall i} log \frac{p (x_{i} |) P (g (i))}{\sum_{\forall j} p (x_{i} | λ_{j}) P (j)},

(28)

where g(i) indicates the class label of x_i and P(j) denotes the prior probability of class j. Vector fusion is implemented directly as

x = [w_{1}^{'} f (φ_{REL} (x)), w_{2}^{'} f (φ_{FUN} (x)), w_{3}^{'} f (φ_{PER} (x))],

(29)

The probability density function p(x |λ) is a Gaussian Mixture Model defined on the N-dimensional vector x :

p (x | λ) = \sum_{\forall m^{'}} ω_{m^{'}}^{'} N (x; μ_{m^{'}}, Σ_{m^{'}}),

(30)

The proposed homogeneous ensemble language recognition system has three advantages. First, SVM supervector reconstruction provides vector space modeling diversification for richer language identification information. Second, in the HEPLR system, the subsystems share the same preprocessing steps for feature extraction, decoding, and expected counting, which minimizes additional computational cost. Third, fusing the reconstructed SVM supervector with the original supervector at the vector level means that more information can be retained than that given by score fusion.

5 Experimental setup

5.1 Baseline language recognition system setup

The TRAPs/NN phonotactic language recognizers developed by the BUT [37] based on phone lattices, N-gram counts, and SVM scoring are used as baseline systems. An energy-based voice activity detector that splits and removes long-duration non-speech segments from the signals is applied initially. Following this, the BUT decoders for Czech (CZ), Hungarian (HU), and Russian (RU) are applied to compute phone posteriori probabilities, as used in NIST LRE tasks by many groups [38],[39]. The phone inventory is 43 for Czech, 59 for Hungarian, and 50 for Russian. Posteriori probabilities are put into the HVite decoder produced by HTK to produce phone lattices, which encode multiple hypotheses with acoustic likelihoods. The N-gram counts are produced by lattice-tool from SRILM (SRI International, Menlo Park, CA, USA) [40]. The LIBLINEAR tool [41] for multiclass SVMs with linear kernels is applied to give SVM scores. Finally, the LDA-MMI algorithm [42] is used for score calibration and fusion.

5.2 Training, development, and test datasets

Evaluation is carried out on the NIST LRE 2009 tasks. This data includes 41793 utterances including 30-, 10-, and 3-s nominal duration, closed condition. The NIST LRE 2009 core task recognition of is to recognize 23 languages, including Amharic, Bosnian, Cantonese, Creole, Croatian, Dari, American English, Indian English, Farsi, French, Georgian, Hausa, Hindi, Korean, Mandarin, Pashto, Portuguese, Russian, Spanish, Turkish, Ukrainian, Urdu, and Vietnamese. The evaluation involves radio broadcasts and conversational telephone speech channel conditions.

The training data comes from different sources including CallHome, CallFriend, OGI, OHSU, VOA, and the development corpora for the 2003, 2005, and 2007 NIST LRE evaluations.

About 25,000 utterances are selected randomly from VOA and 2003, 2005, and 2007 NIST LRE datasets used as development data.

5.3 Evaluation measures

In this work, the performance of language recognition systems is compared using: (1) EER and (2) average cost performance Cavg defined by NIST [43], which are obtained by one-versus-rest tragedy.

6 Experimental results and discussion

We demonstrate the effectiveness of our approaches on NIST LRE 2009 tasks under 30-, 10-, and 3-s conditions. Results are shown in Tables 1, 2, 3, 4, 5, and 6 and Figures 1, 2, 3, 4, 5, 6, 7, and 8 in the following sections. The EER and Cavg performance of individual subsystems and fusions is also shown in the tables below for reference.

Figure 5
**Performance of relative SVM supervector reconstruction subsystem versus dimension.** NIST LRE 2009, 30-s, HU frontend (EER and Cavg in percent). **(a)** TFLLR kernel. **(b)** RBF kernel. **(c)** TFLLR and RBF kernel.

Figure 6
**Performance of functional SVM supervector reconstruction subsystems.** NIST LRE 2009, 30-s, HU frontend. ‘+’ indicated fusion (EER and Cavg in percent).

Figure 7
**Performance of perturbational SVM supervector reconstruction subsystems.** NIST LRE 2009, 30-s, HU frontend (EER and Cavg in percent). **(a)** Approach 1. **(b)** Approach 2. **(c)** Approach 3. **(d)** Approach 4.