Speaker adaptation in the maximum a posteriori framework based on the probabilistic 2-mode analysis of training models

Abstract

In this article, we describe a speaker adaptation method based on the probabilistic 2-mode analysis of training models. Probabilistic 2-mode analysis is a probabilistic extension of multilinear analysis. We apply probabilistic 2-mode analysis to speaker adaptation by representing each of the hidden Markov model mean vectors of training speakers as a matrix, and derive the speaker adaptation equation in the maximum a posteriori (MAP) framework. The adaptation equation becomes similar to the speaker adaptation equation using the MAP linear regression adaptation. In the experiments, the adapted models based on probabilistic 2-mode analysis showed performance improvement over the adapted models based on Tucker decomposition, which is a representative multilinear decomposition technique, for small amounts of adaptation data while maintaining good performance for large amounts of adaptation data.

1 Introduction

In automatic speech recognition (ASR) systems using hidden Markov models (HMMs)[1], mismatches between the training and testing conditions lead to performance degradation. One of such mismatches results from speaker variation. Thus, speaker adaptation techniques[2] are employed to transform a well-trained canonical model (e.g., speaker-independent (SI) HMM) to the target speaker. Speaker adaptation requires fewer adaptation data than needed to build a speaker-dependent (SD) model. Among speaker adaptation techniques, eigenvoice (EV)[3] expresses the model of a new speaker as a linear combination of basis vectors, which are built from the principal component analysis (PCA) of the HMM mean vectors of training speakers.

In a similar approach, speaker adaptation based on tensor analysis using Tucker decomposition[4] was investigated in[5], where bases were constructed from the multilinear decomposition of a tensor that consisted of the HMM mean vectors of training speakers. In the approach, all the training models were collectively arranged in a third-order tensor (3-D array):

$$\begin{array}{l}{\mathcal{ℳ}}^{R\times D\times S}\end{array}$$

(1)

where the first, second, and third modes (dimensions) were for the mixture component, dimension of the mean vector, and training speaker. In[5], Tucker decomposition was used to build bases and in the experiments, speaker adaptation using Tucker decomposition showed better performance than eigenvoice and maximum likelihood linear regression (MLLR) adaptation[6]. The improvement seemed to be attributable to the increased number of adaptation parameters and compact bases. Also noticed in[5] was that the increased number of adaptation parameters did not guarantee a good performance when the amount of adaptation data was small (the determination of the proper number of adaptation parameters for given adaptation data is a model-order selection problem). Extending the tensor-based approach, in[7], the fourth mode for noise was added (so,$\mathcal{ℳ}$ became a 4-D array) so that the training models of various speakers and noise conditions were decomposed.

In this article, we describe a speaker adaptation method using probabilistic 2-mode analysis, which is an application of probabilistic tensor analysis (PTA)[8] to the second-order tensor (i.e., matrix); PTA is an application of probabilistic PCA (PPCA)[9] to tensor objects. Using probabilistic 2-mode analysis, we derive bases from training models in a probabilistic framework, and formulate the speaker adaptation equation in the maximum a posteriori (MAP) framework[10]. The speaker adaptation equation based on the probabilistic approach becomes similar to MAP linear regression (MAPLR) adaptation[11] as shown below. The experiments showed that the proposed method further improved the performance of the speaker adaptation based on Tucker decomposition for small amounts of adaptation data.

The rest of this article is organized as follows. Section 2.1 explains some tensor algebra and tensor decomposition. Section 2.3 explains the probabilistic 2-mode analysis of a set of mean vectors of training HMMs. In Section 2.5, the estimation of the prior distribution of the adaptation parameter is described. Section 2.6 describes the speaker adaptation in the MAP framework using the bases and the prior. Section 2.2 describes the speaker adaptation using Tucker decomposition, which is compared with the probabilistic 2-mode analysis-based method. We explain the experiments in Section 3 and conclude the article in Section 4. Some of the notations used in this article are summarized in Table1.

Table 1 Notations used in the article

2 Methods

2.1 Multilinear decomposition

Following the convention of multilinear algebra, we denote vectors, matrices, and tensors by lowercase boldface letters (e.g., M), uppercase boldface letters (e.g., M), and calligraphic letters (e.g.,$\mathcal{ℳ}$), respectively, in this article.

A tensor is a multidimensional array, and an N-dimensional array is called the N th-order tensor (or N-way array). The order of a tensor is the number of indices for addressing the tensor; so the order of${\mathcal{ℳ}}^{I_1\times I_2\times \cdots \times I_N}$ is N. Scalar, vector, and matrix are zeroth-, first-, and second-order tensors, respectively. There are three indices for addressing the array in a third-order tensor as depicted in Figure1.

Figure 1

A third-order tensor.

Tensor algebra is performed in terms of matrix and vector representations of tensors; the mode-n flattening (matricization) of tensor$\mathcal{ℳ}$, which is denoted as M_(n), is obtained by reordering the elements as follows:

$$\begin{array}{l}{\mathbf{M}}_{\left(n\right)}\in {ℝ}^{I_n\times (I_1\times \cdots \times I_{(n-1)}\times I_{(n+1)}\times \cdots \times I_N)}.\end{array}$$

(2)

That is, all the column vectors along the mode n are arranged into a matrix. For example, a third-order tensor${\mathcal{ℳ}}^{I\times J\times K}$ can be flattened into an I×(J K), J×(K I), or K×(I J) matrix as depicted in Figure2; for a${\mathcal{ℳ}}^{2\times 2\times 2}$ tensor:

$$\begin{array}{l}\mathcal{ℳ}=\left\{\mathbf{M}(:,:,1)\mathbf{M}(:,:,2)\right\}\\ \mathbf{M}(:,:,1)=\left[\begin{array}{ll}{m}_{111}& m_{121}\\ m_{211}& m_{221}\end{array}\right],\mathbf{M}(:,:,2)=\left[\begin{array}{ll}{m}_{112}& m_{122}\\ m_{212}& m_{222}\end{array}\right],\end{array}$$

(3)

Figure 2

Mode- n flattening of a third-order tensor.

the mode-n flattening is given as:

The operation of the mode-n flattening will be denoted as mat _n (·), i.e.,${\text{mat}}_n\left(\mathcal{ℳ}\right)={\mathbf{M}}_{\left(n\right)}$.

Multiplication of a tensor and a matrix is performed by n-mode product; the n-mode product of a tensor$\mathcal{W}$ with a matrix U is denoted as

$$\begin{array}{l}\mathcal{ℳ}=\mathcal{W}{\times }_n\mathbf{U}\end{array}$$

(5)

and is carried out by matrix multiplication in terms of flattened matrices:

$$\begin{array}{l}{\mathbf{M}}_{\left(n\right)}=\mathbf{U}{\mathbf{W}}_{\left(n\right)}\end{array}$$

(6)

or elementwise

$$\begin{array}{l}\left(\right.\mathcal{W}{\times }_n\mathbf{U}{}_{i_1\dots i_{n-1}j{i}_{n+1}\dots i_N}=\sum _{i_n=1}^{I_n}{w}_{i_1{i}_2\dots i_N}{u}_{j{i}_n}\end{array}$$

(7)

where w and u denote the elements of$\mathcal{W}$ and U, respectively. If$\mathcal{W}\in {ℝ}^{I_1\times I_2\times \cdots \times I_N}$ and${\mathbf{U}}^T\in {ℝ}^{K_n\times I_n}$, then the dimension of$\mathcal{W}{\times }_n{\mathbf{U}}^T$ becomes I₁×I₂×⋯×I_n−1×K _n ×I_n+1×⋯×I _N .

As an extension of singular value decomposition (SVD) to tensor objects, Tucker decomposition decomposes a tensor as follows[4]:

$$\begin{array}{l}{\mathcal{ℳ}}^{I_1\times I_2\times \cdots \times I_N}\simeq {\mathcal{W}}^{K_1\times K_2\times \cdots \times K_N}{\prod }_{n=1}^N{\times }_n{\mathbf{U}}_n\end{array}$$

(8)

where${\mathbf{U}}_n\in {ℝ}^{I_n\times K_n}$, K _n ≤I _n (n=1,…,N). The core tensor$\mathcal{W}$ and mode matrices U _n ’s correspond to the matrices of singular values and orthonormal basis vectors in matrix SVD, respectively. An example of Tucker decomposition of a third-order tensor is illustrated in Figure3.

Figure 3

Tucker decomposition of a third-order tensor.

The core tensor$\mathcal{W}$ and mode matrices U _n ’s in Tucker decomposition can be computed such that they minimize

$$\begin{array}{l}\text{Error}=\parallel \mathcal{ℳ}-\mathcal{W}{\prod }_{n=1}^N{\times }_n{\mathbf{U}}_n{\parallel }^2\end{array}$$

(9)

where the norm of a tensor is defined as$\parallel \mathcal{X}\parallel =\sqrt{\sum _{i_1=1}^{I_1}\sum _{i_2=1}^{I_2}\dots \sum _{i_N=1}^{I_N}{x}_{i_1{i}_2\dots i_N}^2}$. A representative technique for Tucker decomposition is the alternating least squares (ALS)[12]; the basic idea is to compute each mode matrix U _n alternatingly with other mode matrices fixed. For more details on Tucker decomposition, refer to[4]. In the following section, we explain probabilistic 2-mode analysis in the context of speaker adaptation.

2.2 Speaker adaptation using Tucker decomposition

The probabilistic 2-mode analysis based method is a probabilistic extension of the Tucker decomposition based method. Thus, we compare the probabilistic approach with the Tucker decomposition based method in the experiments. In this section, we explain the speaker adaptation based on the Tucker decomposition of training models in[5]. In this article, speaker adaptation is performed by updating the mean vectors of the output distribution of an HMM. The HMM mean vectors of each training speaker are arranged in an R×D matrix:

$$\begin{array}{l}{\mathbf{M}}_s={\left[{\mu }_{s;1}\dots {\mu }_{s;r}\dots {\mu }_{s;R}\right]}^T,s=1,\dots ,\mathrm{S.}\end{array}$$

(10)

Here, μ_s;r denotes the mean vector corresponding to mixture r of the s th training speaker model.

All the centered HMM mean vectors of training speakers,${\left\{{\mathbf{M}}_s-\overline{\mathbf{M}}\right\}}_{s=1}^S$ where$\overline{\mathbf{M}}=1/S\sum _s{\mathbf{M}}_s$, are collectively expressed as a third-order tensor$\widetilde{\mathcal{ℳ}}$, and we decompose the training tensor by Tucker decomposition as follows:

$$\begin{array}{ll}{\widetilde{\mathcal{ℳ}}}^{R\times D\times S}& \simeq {\mathcal{G}}^{K_R\times K_D\times K_S}{\times }_1{\mathbf{U}}_{\text{mixture}}{\times }_2{\mathbf{U}}_{\text{dim}}{\times }_3{\mathbf{U}}_{\text{speaker}}\\ =\left(\right.{\mathcal{G}}^{K_R\times K_D\times K_S}{\times }_3{\mathbf{U}}_{\text{speaker}}\left)\right.{\times }_1{\mathbf{U}}_{\text{mixture}}{\times }_2{\mathbf{U}}_{\text{dim}}.\end{array}$$

(11)

In the above equation,${\mathbf{U}}_{\text{mixture}}\in {ℝ}^{R\times K_R}$,${\mathbf{U}}_{\text{dim}}\in {ℝ}^{D\times K_D}$, and${\mathbf{U}}_{\text{speaker}}\in {ℝ}^{S\times K_S}$ are basis matrices for the mixture component, dimension of the mean vector, and training speaker, respectively (K _R ≤R−1, K _D ≤D−1, and K _S ≤S−1); the core tensor$\mathcal{G}$ is common across the mixture component, dimension of the mean vector, and training speaker. In Equation (11), the s th row vector of U_speaker, which is denoted as u_speaker;s, corresponds to the speaker weight of the s th speaker, thus the low-rank approximation of the s th speaker model is given by

$$\begin{array}{l}{\mathbf{M}}_s\simeq \left(\right.{\mathcal{G}}^{K_R\times K_D\times K_S}{\times }_3{\mathbf{u}}_{\text{speaker};s}\left)\right.{\times }_1{\mathbf{U}}_{\text{mixture}}{\times }_2{\mathbf{U}}_{\text{dim}}+\overline{\mathbf{M}}.\end{array}$$

(12)

If we define the augmented speaker weight${\mathbf{W}}_s^{K_R\times K_D}\equiv {\mathcal{G}}^{K_R\times K_D\times K_S}{\times }_3{\mathbf{u}}_{\text{speaker};s}$, Equation (12) becomes

$$\begin{array}{ll}{\mathbf{M}}_s& \simeq {\mathbf{W}}_s{\times }_1{\mathbf{U}}_{\text{mixture}}{\times }_2{\mathbf{U}}_{\text{dim}}+\overline{\mathbf{M}}\\ ={\mathbf{U}}_{\text{mixture}}{\mathbf{W}}_s{\mathbf{U}}_{\text{dim}}^T+\overline{\mathbf{M}}.\end{array}$$

(13)

Thus, we express the model of a new speaker as

$$\begin{array}{l}{\mathbf{M}}_{\text{new}}={\mathbf{U}}_{\text{mixture}}{\mathbf{W}}_{\text{new}}{\mathbf{U}}_{\text{dim}}^T+\overline{\mathbf{M}}.\end{array}$$

(14)

For the given adaptation data O={o₁,…,o _T }, we derive the equation for finding the speaker weight in a maximum likelihood (ML) criterion:

$$\begin{array}{l}\sum _t\sum _r{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}\underset{\equiv {\mathbf{W}}_{\text{new,aug}}^T}{\underbrace{{\mathbf{U}}_{\text{dim}}{\mathbf{W}}_{\text{new}}^T}}{\mathbf{u}}_{\text{mixture};r}^T{\mathbf{u}}_{\text{mixture};r}\\ =\sum _t\sum _r{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}\left(\right.{\mathbf{o}}_t-{\overline{\mathbf{m}}}_r^T\left)\right.{\mathbf{u}}_{\text{mixture};r}\end{array}$$

(15)

where γ _r (t) denotes the occupation probability of being at mixture r at t given O, C _r the covariance matrix of the r th Gaussian component of an SI HMM (in this article, a diagonal covariance matrix is used); u_mixture;r and${\overline{\mathbf{m}}}_r$ denote the r th row vectors of U_mixture and$\overline{\mathbf{M}}$, respectively. In the above equation, W_{new, aug} can be computed using a technique similar to MLLR adaptation and the weight of the new speaker is obtained by

$$\begin{array}{l}{\widehat{\mathbf{W}}}_{\text{new}}={\mathbf{W}}_{\mathrm{new},\mathit{\text{aug}}}{\mathbf{U}}_{\text{dim}}\end{array}$$

(16)

which is plugged into Equation (14) to produce the model updated for the new speaker.

2.3 Probabilistic 2-mode analysis

The advantage of probabilistic 2-mode analysis over Tucker decomposition is similar to that of PPCA over standard PCA; probabilistic 2-mode analysis can deal with missing entries in the data tensor (although this is not the case in our experiments). In the modeling perspective, probabilistic 2-mode analysis assumes a distribution of latent variables, thus it is suitable for a MAP framework.

In this section, the ensemble of training models is expressed as

$$\begin{array}{l}M={\left\{{\mathbf{M}}_s\right\}}_{s=1}^S.\end{array}$$

(17)

Assuming the HMM mean vectors of training speakers are drawn from the matrix-variate normal distribution[13], we derive the adaptation equation based on the probabilistic 2-mode analysis of training models. We use probabilistic 2-mode analysis, the second-order case of PTA[8], to decompose the training models expressed in matrix form. The latent tensor model is expressed as

$$\begin{array}{l}\mathcal{ℳ}=\mathcal{W}\prod _{n=1}^N{\times }_n{\mathbf{U}}_n+{\mathcal{ℳ}}_{\text{mean}}+\mathcal{E}\end{array}$$

(18)

where$\mathcal{W}$ denotes the latent tensor, U _n ’s the factor loading matrices,${\mathcal{ℳ}}_{\text{mean}}$ the mean, and$\mathcal{E}$ is the error/noise process. The 2-mode case of the latent tensor model is given by

$$\begin{array}{l}\mathbf{M}=\mathbf{W}{\times }_1{\mathbf{U}}_1{\times }_2{\mathbf{U}}_2+{\mathbf{M}}_{\text{mean}}+\mathbf{E}\end{array}$$

(19)

which becomes, for the training models {M₁,…,M _S },

$$\begin{array}{ll}{\mathbf{M}}_s& ={\mathbf{W}}_s{\times }_1{\mathbf{U}}_1{\times }_2{\mathbf{U}}_2+{\mathbf{M}}_{\text{mean}}+{\mathbf{E}}_s\\ ={\mathbf{U}}_{\text{mixture}}{\mathbf{W}}_s{\mathbf{U}}_{\text{dim}}^T+{\mathbf{M}}_{\text{mean}}+{\mathbf{E}}_s\end{array}$$

(20)

where${\mathbf{W}}_s\in {ℝ}^{K_R\times K_D}$ denotes the latent matrix,${\mathbf{U}}_{\text{mixture}}\in {ℝ}^{R\times K_R}$ and${\mathbf{U}}_{\text{dim}}\in {ℝ}^{D\times K_D}$ the factor loading matrices (K _R ≤R−1 and K _D ≤D−1), M_mean the mean, and E _s the error/noise process. (Mode matrices and dimensions are defined as follows: U₁=U_mixture, U₂=U_dim, I₁=R, I₂=D, K₁=K _R , and K₂=K _D .) The distribution of W _s is assumed to be a matrix-variate normal, i.e.,${\mathbf{W}}_s\sim \mathcal{N}({\mathbf{0}}_{K_R\times K_D},{\mathbf{I}}_{K_R}\otimes {\mathbf{I}}_{K_D})$ where ⊗ denotes the Kronecker product, and independent of E _s whose elements follow$\mathcal{N}(0,{\sigma }^2)$. Figure4 shows the graphical model representing the probabilistic 2-mode model.

Figure 4

Graphical model representation of the probabilistic 2-mode model.

In Equation (20), it is computationally intractable to calculate U _n ’s simultaneously. So, the following decoupled predictive density is defined:

$$\begin{array}{l}p\left(\mathbf{M}|{\mathbf{M}}_{\text{mean}},{\left\{{\mathbf{U}}_n\right\}}_{n=1}^N,{\sigma }^2\right)\propto \prod _{n=1}^{N}p\left(\right.\mathbf{M}{\bar{\times }}_n{\mathbf{U}}_n^T|{\bar{\mathbf{t}}}_n,{\sigma }_n^2\left)\right.\end{array}$$

(21)

where${\bar{\mathbf{t}}}_n\in {ℝ}^{I_n\times 1}$ and${\sigma }_n^2$ denote the mean vector and noise variance, respectively, for mode n;$\mathbf{M}{\bar{\times }}_n{\mathbf{U}}_n^T\equiv \mathbf{M}{\times }_1{\mathbf{U}}_1^T\dots {\times }_{n-1}{\mathbf{U}}_{n-1}^T{\times }_{n+1}{\mathbf{U}}_{n+1}^T\dots {\times }_N{\mathbf{U}}_N^T$, i.e., the product of M with all the mode matrices except mode n, which is called the contracted n-mode product[14]. That is, the n th probabilistic function is defined as the projected M by all U _j ’s expect U _n . Given observed data M, the decoupled posterior probabilistic function is defined as

$$\begin{array}{l}p\left(\right.{\mathbf{M}}_{\text{mean}},{\left\{{\mathbf{U}}_n\right\}}_{n=1}^N,{\sigma }^2|M\left)\right.\propto \prod _{n=1}^{N}p\left(\right.{\bar{\mathbf{t}}}_n,{\mathbf{U}}_n,{\sigma }_n^2,M,{\left\{{\mathbf{U}}_j\right\}}_{j=1,j\ne n}^N\left)\right..\end{array}$$

(22)

By Bayes’ theorem, the n th posterior distribution can be expressed in terms of the decoupled likelihood function and the decoupled prior distribution:

$$\begin{array}{ll}p\left(\right.& {\bar{\mathbf{t}}}_n,{\mathbf{U}}_n,{\sigma }_n^2,M,{\left\{{\mathbf{U}}_j\right\}}_{j=1,j\ne n}^N\left)\right.\propto \\ p\left(\right.M,{\left\{{\mathbf{U}}_j\right\}}_{j=1,j\ne n}^N|{\bar{\mathbf{t}}}_n,{\mathbf{U}}_n,{\sigma }_n^2\left)\right.p\left(\right.{\bar{\mathbf{t}}}_n,{\mathbf{U}}_n,{\sigma }_n^2\left)\right..\end{array}$$

(23)

Therefore, the decoupled predictive density is given by

$$\begin{array}{ll}p\left(\mathbf{M}|M\right)\propto & p\left(\mathbf{M}|{\mathbf{M}}_{\text{new}},{\left\{{\mathbf{U}}_n\right\}}_{n=1}^N,{\sigma }^2\right)\\ \times p\left(\right.{\mathbf{M}}_{\text{mean}},{\left\{{\mathbf{U}}_n\right\}}_{n=1}^N,{\sigma }^2|M\left)\right.\\ =\prod _{n=1}^N& p\left(\mathbf{M}{\bar{\times }}_n{\mathbf{U}}_n^T|{\bar{\mathbf{t}}}_n,{\sigma }_n^2\right)\\ \times p\left(M,{\left\{{\mathbf{U}}_j\right\}}_{j=1,j\ne n}^N|{\bar{\mathbf{t}}}_n,{\mathbf{U}}_n,{\sigma }_n^2\right).\end{array}$$

(24)

$\left(\right.p\left(\right.{\bar{\mathbf{t}}}_n,{\mathbf{U}}_n,{\sigma }_n^2\left)\right.$ is dropped out for a fixed U _n ). This is the 2-mode case of the PTA in[8]. In our case, Equation (24) is given by

$$\begin{array}{l}p\left(\mathbf{M}|M\right)\propto p\left({\mathbf{U}}_{\text{dim}}^T{\mathbf{M}}^T|{\bar{\mathbf{t}}}_{\text{mixture}},{\sigma }_{\text{mixture}}^2\right)\\ \times p\left(\right.M,{\mathbf{U}}_{\text{dim}}|{\bar{\mathbf{t}}}_{\text{mixture}},{\mathbf{U}}_{\text{mixture}},{\sigma }_{\text{mixture}}^2\left)\right.\\ \times p\left(\right.{\mathbf{U}}_{\text{mixture}}^T\mathbf{M}|{\bar{\mathbf{t}}}_{\text{dim}},{\sigma }_{\text{dim}}^2\left)\right.\\ \times p\left(\right.M,{\mathbf{U}}_{\text{mixture}}|{\bar{\mathbf{t}}}_{\text{dim}},{\mathbf{U}}_{\text{dim}},{\sigma }_{\text{dim}}^2\left)\right..\end{array}$$

(25)

Now, U _n ’s are obtained by maximizing the following posterior distribution:

$$\begin{array}{l}p\left(\right.{\left\{{\mathbf{U}}_n\right\}}_{n=1}^N|M\left)\right.\approx \prod _{n=1}^{N}p\left(\right.{\mathbf{U}}_n|M{\bar{\times }}_n{\mathbf{U}}_n^T\left)\right.\end{array}$$

(26)

where$p\left(\right.{\mathbf{U}}_n|M{\bar{\times }}_n{\mathbf{U}}_n^T\left)\right.\equiv \prod _{s=1}^S{\mathbf{M}}_s{\bar{\times }}_n{\mathbf{U}}_n^T$. The expectation-maximization (EM) algorithm[15] is applied to compute U _n ’s. The application of the EM algorithm to construct probabilistic 2-mode model is explained in the next section.

2.4 Construction of probabilistic 2-mode model for speaker adaptation

In Equation (20), for the given training models, the maximum likelihood (ML) estimate of M_mean is given as$\overline{\mathbf{M}}=(1/S)\sum _s{\mathbf{M}}_s$ and$\{{\mathbf{U}}_n,{\sigma }_n^2\}$ can be estimated as follows. First, let us define the followings: Let${\mathbf{t}}_{n;j}\in {ℝ}^{I_n\times 1}$ be the j th column vector of

$$\begin{array}{l}{\mathbf{T}}_{\left(n\right)}={\text{mat}}_n\left(\right.\mathbf{M}{\bar{\times }}_n{\mathbf{U}}_n^T\left)\right.\end{array}$$

(27)

for$1\le j\le {Ī}_{n}S$ (${Ī}_n=\prod _{j=1,j\ne n}^N{I}_j$) and${\mathbf{x}}_{n;j}\in {ℝ}^{K_n\times 1}$ be the j th column vector of

$$\begin{array}{l}{\mathbf{X}}_{\left(n\right)}={\text{mat}}_n\left(\right.\mathbf{M}\prod _{n=1}^N{\times }_n{\mathbf{U}}_n^T\left)\right..\end{array}$$

(28)

Let us suppose${\mathbf{t}}_n|{\mathbf{x}}_n\sim \mathcal{N}({\mathbf{U}}_n{\mathbf{x}}_n+{\bar{\mathbf{t}}}_n,{\sigma }_n^2{\mathbf{I}}_{I_n})$ and${\mathbf{x}}_n\sim \mathcal{N}({\mathbf{0}}_{K_n\times 1},{\mathbf{I}}_{K_n})$. Then, by integrating out x _n ,${\mathbf{t}}_n\sim \mathcal{N}({\bar{\mathbf{t}}}_n,{\mathbf{G}}_n)$ where${\bar{\mathbf{t}}}_n=1/\left({Ī}_{n}S\right)\sum _{j=1}^{{Ī}_{n}S}{\mathbf{t}}_{n;j}$ and${\mathbf{G}}_n={\mathbf{U}}_n{\mathbf{U}}_n^T+{\sigma }_n^2{\mathbf{I}}_{I_n}$. Consequently,

$$\begin{array}{l}{\mathbf{x}}_n|{\mathbf{t}}_n\sim \mathcal{N}\left(\right.{\mathbf{H}}_n^{-1}{\mathbf{U}}_n^T({\mathbf{t}}_n-{\bar{\mathbf{t}}}_n),{\sigma }_n^2{\mathbf{H}}_n^{-1}\left)\right.\end{array}$$

(29)

where${\mathbf{H}}_n={\mathbf{U}}_n^T{\mathbf{U}}_n+{\sigma }_n^2{\mathbf{I}}_{K_n}$. The right-hand side of Equation (26) becomes

$$\begin{array}{l}logp\left(\right.{\mathbf{U}}_n|M{\bar{\times }}_n{\mathbf{U}}_n^T\left)\right.\propto -\frac{{Ī}_{n}S}{2}\left(\right.log|{\mathbf{G}}_n|+\text{tr}\left[\right.{\mathbf{G}}_n^{-1}{\mathbf{S}}_n\left]\right.\left)\right.\end{array}$$

(30)

where${\mathbf{S}}_n=1/({Ī}_{n}S-1)\sum _{j=1}^{{Ī}_{n}S}\left(\right.{\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n\left)\right.\left(\right.{\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n{}^T$ and tr[ · ] denotes the trace of a matrix. Summing up for all the modes, we obtain the following log-likelihood function of the posterior distribution:

$$\begin{array}{ll}L=& \sum _{n}logp\left(\right.{\mathbf{U}}_n|M{\bar{\times }}_n{\mathbf{U}}_n\left)\right.\propto \\ -\sum _n\left\{\right.\frac{{Ī}_{n}S}{2}\left(\right.log\left|{\mathbf{G}}_n\right|+\text{tr}\left[\right.{\mathbf{G}}_n^{-1}{\mathbf{S}}_n\left]\right.\left)\right.\left\}\right..\end{array}$$

(31)

The graphical model representation of the decoupled probabilistic model is shown in Figure5.

Figure 5

Graphical model representation of the decoupled probabilistic model.

We seek to find U _n ’s that maximize the log-likelihood function alternatingly. Mode matrices U₁ and U₂ are initialized with the results from the Tucker decomposition which minimizes the reconstruction error:

$$\begin{array}{l}\text{Error}=\sum _s∥{\mathbf{M}}_s-\left(\right.{\mathbf{W}}_s{\times }_1{\mathbf{U}}_1{\times }_2{\mathbf{U}}_2+\overline{\mathbf{M}}\left)\right.{}^2.\end{array}$$

(32)

With the initial U₁ and U₂, the following procedure is performed for each mode (n=1,2).

Each training model is projected into mode matrices except mode n and expressed in a mode-n matrix:

$$\begin{array}{l}{\mathbf{T}}_{s,\left(n\right)}={\text{mat}}_n\left(\right.{\mathbf{M}}_s{\bar{\times }}_n{\mathbf{U}}_n^T\left)\right..\end{array}$$

(33)

All the column vectors of$\left\{\right.{\mathbf{T}}_{s,\left(n\right)}{}_{s=1}^S$ constitute the training data set:

$$\begin{array}{l}\left\{{\mathbf{t}}_{n;j}\right\},1\le j\le {Ī}_n\mathrm{S.}\end{array}$$

(34)

Then, with an initial estimate of${\sigma }_n^2$ (e.g., 0.005 was used in the experiments), the EM algorithm is iterated as follows until U _n and${\sigma }_n^2$ converge.

E-step: From Equation (31), the expectation of the log-likelihood function of complete data w.r.t.$p\left(\right.{\mathbf{x}}_{n;j}|{\mathbf{t}}_{n;j},{\bar{\mathbf{t}}}_n,{\mathbf{U}}_n,{\sigma }_n^2\left)\right.$ is given as

$$\begin{array}{ll}〈L_{\mathrm{c}}〉& =\sum _n\sum _{s}E\left[\right.logp({\mathbf{M}}_s,{\mathbf{W}}_s|{\left\{\mathbf{U}\right\}}_{j=1,j\ne n}^N)\left]\right.\\ =\sum _n\sum _{j=1}^{I_{n}S}E\left[\right.logp({\mathbf{t}}_{n;j},{\mathbf{x}}_{n;j}|{\left\{\mathbf{U}\right\}}_{j=1,j\ne n}^N)\left]\right.\end{array}$$

(35)

where

$$\begin{array}{ll}logp({\mathbf{t}}_{n;j},{\mathbf{x}}_{n;j}|{\left\{\mathbf{U}\right\}}_{j=1,j\ne n}^N)& =logp\left({\mathbf{x}}_{n;j}\right)+logp\left({\mathbf{t}}_{n;j}\right|{\mathbf{x}}_{n;j})\\ \propto -\parallel {\mathbf{x}}_{n;j}{\parallel }^2& -\frac{I_n}{2}log\left(\underset{n}{\overset{2}{\sigma }}\right)\\ -\frac{1}{{\sigma }_n^2}\parallel {\mathbf{t}}_{n;j}-{\mathbf{U}}_n{\mathbf{x}}_{n;j}-{\bar{\mathbf{t}}}_n{\parallel }^2.\end{array}$$

(36)

So,

$$\begin{array}{ll}〈L_{\mathrm{c}}〉\propto & -\sum _n\sum _{j=1}^{{Ī}_{n}S}\left\{\right.\text{tr}\left[\right.〈{\mathbf{x}}_{n;j}{\mathbf{x}}_{n;j}^T〉\left]\right.\\ +\frac{1}{{\sigma }_n^2}{({\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n)}^T({\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n)\\ +\frac{I_n}{2}log\left(\underset{n}{\overset{2}{\sigma }}\right)+\frac{1}{{\sigma }_n^2}\text{tr}\left[\right.{\mathbf{U}}_n^T{\mathbf{U}}_n〈{\mathbf{x}}_{n;j}{\mathbf{x}}_{n;j}^T〉\left]\right.\\ -\frac{2}{{\sigma }_n^2}{〈{\mathbf{x}}_{n;j}〉}^T{\mathbf{U}}_n^T({\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n)\left\}\right.\end{array}$$

(37)

with the sufficient statistics are given as follows from Equation (29):

$$\begin{array}{l}〈{\mathbf{x}}_{n;j}〉={\mathbf{H}}_n^{-1}{\mathbf{U}}_n^T\left(\right.{\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n\left)\right.\\ 〈{\mathbf{x}}_{n;j}{\mathbf{x}}_{n;j}^T〉={\sigma }_n^2{\mathbf{H}}_n^{-1}+〈{\mathbf{x}}_{n;j}〉{〈{\mathbf{x}}_{n;j}〉}^T.\end{array}$$

(38)

M-step: Model parameters are updated by maximizing 〈L_c〉 w.r.t. U _n and${\sigma }_n^2$. Setting${\partial }_{{\mathbf{U}}_n}〈L_{\mathrm{c}}〉=0$ produces

$$\begin{array}{l}{\mathbf{U}}_n=\left[\right.\sum _{j=1}^{{Ī}_{n}S}\left(\right.{\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n\left)\right.{〈{\mathbf{x}}_{n;j}〉}^T\left]\right.\left[\right.\sum _{j=1}^{{Ī}_{n}S}〈{\mathbf{x}}_{n;j}{\mathbf{x}}_{n;j}^T〉{}^{-1}.\end{array}$$

(39)

Next, setting${\partial }_{{\sigma }_n^2}〈L_{\mathrm{c}}〉=0$ produces

$$\begin{array}{ll}{\sigma }_n^2=\frac{1}{I_n{Ī}_{n}S}\sum _{j=1}^{{Ī}_{n}S}\left\{\right.\parallel {\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n{\parallel }^2& -2{〈{\mathbf{x}}_{n;j}〉}^T{\mathbf{U}}_n^T\left(\right.{\mathbf{t}}_{n;j}-{\bar{\mathbf{t}}}_n\left)\right.\\ +\text{tr}\left[\right.〈{\mathbf{x}}_{n;j}{\mathbf{x}}_{n;j}^T〉{\mathbf{U}}_n^T{\mathbf{U}}_n\left]\right.\left\}\right..\end{array}$$

(40)

Essentially, the procedure applies PPCA to the data set {t_n;j} for each mode.

2.5 Estimation of prior distribution

Given model parameters$\{\overline{\mathbf{M}},{\mathbf{U}}_n,{\sigma }_n^2\}$, the weight matrix for the training speaker model M _s is obtained by

$$\begin{array}{ll}{\mathbf{W}}_s& =\left(\right.{\mathbf{M}}_s-\overline{\mathbf{M}}\left)\right.\prod _{n=1}^2{\times }_n\left(\right.{\mathbf{H}}_n^{-1}{\mathbf{U}}_n^T\left)\right.\\ =\left(\right.{\mathbf{H}}_1^{-1}{\mathbf{U}}_{\text{mixture}}^T\left)\right.\left(\right.{\mathbf{M}}_s-\overline{\mathbf{M}}\left)\right.\left(\right.{\mathbf{H}}_2^{-1}{\mathbf{U}}_{\text{dim}}^T{}^T.\end{array}$$

(41)

From the set of weight matrices${\left\{{\mathbf{W}}_s\right\}}_{s=1}^S$, the distribution of the weight is estimated. In deriving the adaptation equation in the MAP framework, the parameters for the prior distribution can be obtained in closed-form solutions if p(W) follows a conjugate distribution. Hence, we assume the prior distribution of the weight to be a matrix-variate normal:

$$\begin{array}{ll}p\left(\mathbf{W}\right)& \propto \frac{1}{\left|\mathbf{\Sigma }{|}^{K_D/2}\right|\mathbf{\Psi }{|}^{K_R/2}}\\ exp\{-\frac{1}{2}\text{tr}[{(\mathbf{W}-{\mathbf{W}}_{\text{mean}})}^T{\mathbf{\Sigma }}^{-1}(\mathbf{W}-{\mathbf{W}}_{\text{mean}}){\Psi }^{-1}\left]\right\}.\end{array}$$

(42)

Furthermore, the hyperparameters of p(W) can easily be estimated in an ML criterion if Ψ is known[16]. So, Ψ is assumed to be the identity matrix[17], and the hyperparameters are estimated as:

$$\begin{array}{l}{\hat{\mathbf{W}}}_{\text{mean}}=\frac{1}{S}\sum _s{\mathbf{W}}_s={\mathbf{0}}_{K_R\times K_D}\\ \hat{\Sigma }=\frac{1}{S-1}\sum _s{\mathbf{W}}_s{\mathbf{W}}_s^T.\end{array}$$

(43)

2.6 Speaker adaptation in the MAP framework

Based on Equation (20), we express the model of a new speaker as

$$\begin{array}{l}{\mathbf{M}}_{\text{new}}={\mathbf{U}}_{\text{mixture}}{\mathbf{W}}_{\text{new}}{\mathbf{U}}_{\text{dim}}^T+\overline{\mathbf{M}}.\end{array}$$

(44)

For the given adaptation data O={o₁,…,o _T }, we estimate the adaptation parameter in a MAP criterion:

$$\begin{array}{ll}{\Lambda }_{\text{MAP}}& =\underset{\Lambda }{arg\; max}p\left(\Lambda \right|\mathbf{O})\\ \propto \underset{\Lambda }{arg\; max}p\left(\mathbf{O}\right|\Lambda \left)p\right(\Lambda )\\ \propto \underset{\Lambda }{arg\; max}logp\left(\mathbf{O}\right|\Lambda )+logp(\Lambda )\end{array}$$

(45)

where Λ={W_new} denotes the model parameter.

Using the EM algorithm, we obtain the following auxiliary Q-function to be optimized (discarding the terms that are independent of the model parameter):

$$\begin{array}{ll}Q(\Lambda ,\widehat{\Lambda })=& -\frac{1}{2}\sum _t\sum _r{\gamma }_r\left(t\right)\text{tr}\left[\right.{({\mathbf{o}}_t-{\mathbf{m}}_{\text{new};r}^T)}^T{\mathbf{C}}_r^{-1}({\mathbf{o}}_t-{\mathbf{m}}_{\text{new};r}^T)\left]\right.\\ -\frac{1}{2}\text{tr}\left[\right.{\left({\mathbf{W}}_{\text{new}}{\mathbf{U}}_{\text{dim}}\right)}^T{\hat{\Sigma }}^{-1}\left({\mathbf{W}}_{\text{new}}{\mathbf{U}}_{\text{dim}}\right)\left]\right.\end{array}$$

(46)

where Λ and$\widehat{\Lambda }$ denote the current and updated model parameters, respectively, and${\mathbf{m}}_{\text{new};r}={\mathbf{u}}_{\text{mixture};r}{\mathbf{W}}_{\text{new}}{\mathbf{U}}_{\text{dim}}^T+{\overline{\mathbf{m}}}_r$. In finding the speaker weight, we compute W_{new, aug}≡W_newU_dim, from which W_new is obtained. Solving in this way, we can use the row-by-row technique in MLLR adaptation[6]. Setting${\partial }_{{\mathbf{W}}_{\text{new}}}Q(\Lambda ,\widehat{\Lambda })=0$ yields the following equation:

$$\begin{array}{l}\sum _t\sum _r{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}\underset{\equiv {\mathbf{W}}_{\text{new, aug}}^T}{\underbrace{{\mathbf{U}}_{\text{dim}}{\mathbf{W}}_{\text{new}}^T}}{\mathbf{u}}_{\text{mixture};r}^T{\mathbf{u}}_{\text{mixture};r}+\underset{\equiv {\mathbf{W}}_{\text{new, aug}}^T}{\underbrace{{\mathbf{U}}_{\text{dim}}{\mathbf{W}}_{\text{new}}^T}}{\hat{\Sigma }}^{-1}\\ =\sum _t\sum _r{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}\left(\right.{\mathbf{o}}_t-{\overline{\mathbf{m}}}_r^T\left)\right.{\mathbf{u}}_{\text{mixture};r}.\end{array}$$

(47)

The above equation can be solved for W_{new, aug} in a similar way to MLLR adaptation in[6]: we define the followings:

$$\begin{array}{l}{\mathbf{V}}_r=\sum _t{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}\\ {\mathbf{D}}_r={\mathbf{u}}_{\text{mixture};r}^T{\mathbf{u}}_{\text{mixture};r}\\ {\mathbf{G}}_{\left(i\right)}=\sum _r{v}_r(i,i){\mathbf{D}}_r\\ \mathbf{Z}=\sum _t\sum _r{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}\left(\right.{\mathbf{o}}_t-{\overline{\mathbf{m}}}_r^T\left)\right.{\mathbf{u}}_{\text{mixture};r}\\ {\Sigma }_{\left(i\right)}=\frac{1}{S-1}\sum _s{\mathbf{w}}_{s;i}{\mathbf{w}}_{s;i}^T\end{array}$$

(48)

where v _r (i,i) denotes the (i,i) element of V _r and w_s;i the i th column vector of W_{s, aug}≡W _s U_dim. Then, the speaker weight can be computed:

$$\begin{array}{l}{\mathbf{w}}_{\text{new, aug},\left(i\right)}^T=\left[\right.{\mathbf{G}}_{\left(i\right)}+{\Sigma }_{\left(i\right)}^{-1}{}^{-1}{\mathbf{z}}_{\left(i\right)}^T,i=1,\dots ,D\end{array}$$

(49)

where w_{new, aug, (i)} denotes the i th row of W_{new, aug} and z_(i) the i th row vector of Z. The method becomes similar to MAPLR adaptation in[11]. Finally, the speaker weight is obtained as

$$\begin{array}{l}{\widehat{\mathbf{W}}}_{\text{new}}={\mathbf{W}}_{\text{new, aug}}{\mathbf{U}}_{\text{dim}}^{+}\end{array}$$

(50)

where [ · ]⁺ denotes the pseudoinverse of a matrix. The weight is plugged into Equation (44) to produce the model adapted to the new speaker.

2.7 Speaker adaptation techniques compared in the experiments

In this section, we briefly review the speaker adaptation techniques compared with the probabilistic 2-mode analysis based method: eigenvoice adaptation[3], MLLR adaptation[6], and MAPLR adaptation[11].

In eigenvoice adaptation, the collection of HMM mean vectors of speaker s is arranged in an (R D)×1 vector:

$$\begin{array}{l}{\mu }_s=\left[\begin{array}{l}{\mu }_{s;1}\\ {\mu }_{s;2}\\ ⋮\\ {\mu }_{s;R}\end{array}\right].\end{array}$$

(51)

Then, the set of S supervectors, {μ₁,…,μ _S }, is decomposed by PCA to produce the adaptation model

$$\begin{array}{l}{\mu }_{\text{new}}=\Phi {\mathbf{w}}_{\text{new}}+\bar{\mu }\end{array}$$

(52)

where Φ=[ϕ₁…ϕ _K ], the basis matrix consisting of the K dominant eigenvectors from PCA, and$\bar{\mu }=1/S\sum _s{\mu }_s$. The K×1 weight vector can be obtained by maximizing the likelihood of the adaptation data, which is given by

$$\begin{array}{l}{\widehat{\mathbf{w}}}_{\text{new}}={\left[\sum _t,\sum _r,{\gamma }_r,\left(,t,\right),,{\Phi }_r^T,{\mathbf{C}}_r^{-1},{\Phi }_r\right]}^{-1}\\ \times \left[\sum _t\sum _r{\gamma }_r\right(t){\Phi }_r^T{\mathbf{C}}_r^{-1}\left(\right.{\mathbf{o}}_t-{\bar{\mu }}_r\left)\right.]\end{array}$$

(53)

where Φ _r and${\bar{\mu }}_r$ denote the D×K submatrix and D×1 subvector corresponding to the r th mixture of Φ and$\bar{\mu }$, respectively.

In MLLR adaptation, the updated model for a new speaker is obtained by linearly transforming the SI model (assuming a global regression matrix):

$$\begin{array}{l}{\mu }_{\mathit{\text{new}},r}={\mathbf{W}}_{\text{new}}{\xi }_r,{\xi }_r=\left[\begin{array}{l}\omega \\ {\mu }_{\text{SI,}r}\end{array}\right]\end{array}$$

(54)

where μ_{SI, r} denotes the mean vector of the SI HMM corresponding to mixture r and ω is the bias offset term: ω=1 to include the term and ω=0 otherwise (ω=1 in our experiments). The D×(D+1) transformation matrix can be obtained in an ML criterion, which yields the following equation:

$$\begin{array}{l}\sum _t\sum _r{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}{\mathbf{o}}_t{\xi }_r^T\\ =\sum _t\sum _r{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}{\mathbf{W}}_{\text{new}}{\xi }_r{\xi }_r^T.\end{array}$$

(55)

The above equation can be solved for W_new:

$$\begin{array}{l}{\widehat{\mathbf{w}}}_{\text{new,}\left(i\right)}^T={\mathbf{G}}_{\left(i\right)}^{-1}{\mathbf{z}}_{\left(i\right)}^T,i=1,\dots ,D\end{array}$$

(56)

where${\widehat{\mathbf{w}}}_{\text{new,}\left(i\right)}$ and z_(i) denote the i th row vectors of${\widehat{\mathbf{W}}}_{\text{new}}$ and Z, respectively; G_(i) and Z are defined as:

$$\begin{array}{l}{\mathbf{V}}_r=\sum _t{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}\\ {\mathbf{D}}_r={\xi }_r{\xi }_r^T\\ {\mathbf{G}}_{\left(i\right)}=\sum _r{v}_r(i,i){\mathbf{D}}_r\\ \mathbf{Z}=\sum _t\sum _r{\gamma }_r\left(t\right){\mathbf{C}}_r^{-1}{\mathbf{o}}_t{\xi }_r^T\end{array}$$

(57)

where v _r (i,i) denotes the (i,i) element of V _r .

In MAPLR adaptation, the prior for the transformation matrix is used in the MLLR framework. The parameters for the prior are obtained from the MLLR transformation matrices of training speakers {W₁,…,W _S }:

$$\begin{array}{l}{\bar{\mathbf{w}}}_{\left(i\right)}=\frac{1}{S}\sum _s{\mathbf{w}}_{s,\left(i\right)}\\ {\Sigma }_{\left(i\right)}=\frac{1}{S-1}\sum _s\left(\right.{\mathbf{w}}_{s,\left(i\right)}-{\bar{\mathbf{w}}}_{\left(i\right)}{}^T\left(\right.{\mathbf{w}}_{s,\left(i\right)}-{\bar{\mathbf{w}}}_{\left(i\right)}\left)\right.\end{array}$$

(58)

where w_{s, (i)} denotes the i th row vector of W _s . Then, the transformation matrix for a new speaker is obtained in a MAP criterion. Deriving the equation in the same way as above, we can obtain the following:

$$\begin{array}{l}{\widehat{\mathbf{w}}}_{\text{new,}\left(i\right)}^T=\left[\right.{\mathbf{G}}_{\left(i\right)}+{\Sigma }_{\left(i\right)}^{-1}{}^{-1}\left[\right.{\mathbf{z}}_{\left(i\right)}+{\bar{\mathbf{w}}}_{\left(i\right)}{\Sigma }_{\left(i\right)}^{-1}{}^T.\end{array}$$

(59)

3 Experiments

We carried out the large-vocabulary continuous-speech recognition (LVCSR) experiments using the Wall Street Journal corpus WSJ0[18]. In building the SI model, we used 12754 utterances of 101 speakers from the corpus. As the acoustic feature vector, we used the 39-dimensional vector consisting of 13-dimensional mel-frequency cepstral coefficients (MFCCs) including the 0th cepstral coefficient, their derivative coefficients, and their acceleration coefficients. The feature vector was extracted with the 20-ms Hamming window with the frame sliding of 10 ms. Using the HMM toolkit (HTK)[19], we built a tied-state triphone model (word-internal triphones) with 3472 tied states and 8-mixture Gaussian.

To build training models for constructing bases, we transformed the SI model by MLLR adaptation[6] using 32 regression classes followed by maximum a posteriori (MAP) adaptation[10]. We used the 101 adapted models to build the Tucker decomposition and probabilistic tensor based models as well as eigenvoice.

For adaptation and recognition test, we used Nov’92 5K non-verbalized adaptation and test sets. The number of testing speakers was 8; adaptation set was used for adaptation and testing set of 330 sentences was used for recognition test (the number of testing utterances per speaker was about 40). The length of an adaptation sentence was about 6 s and the adaptation was performed in supervised mode. In recognition test, we used WSJ 5K non-verbalized 5k closed-vocabulary set and WSJ standard 5K non-verbalized closed bigram.

The word recognition accuracy of the SI model is 91.54%. Table2 shows the results of the Tucker decomposition and probabilistic 2-mode based methods (K _S =100 in the Tucker decomposition based model). In the table, the probabilistic 2-mode based method shows improved performance over the Tucker decomposition based method for small amounts of adaptation data, which can be evidently seen in Figure6 for the Tucker decomposition and probabilistic 2-mode based models with (K _R =20,K _D =35). The results of MAPLR[11] are also shown in the figure. The use of MAP framework contributes to improved performance for small amounts of adaptation data. The number of free parameters of each method is given as follows: 20 · 35 for the Tucker 3-mode and probabilistic 2-mode based models, and 39 · 40 for MAPLR adaptation. In Figure7, the Tucker decomposition based method is compared with MLLR and eigenvoice adaptation techniques. The figure shows that the Tucker decomposition based method outperforms MLLR and eigenvoice adaptation techniques for adaptation sentences > 1. It can be inferred from the figure that eigenvoice adaptation will outperform the Tucker decomposition based method or MLLR for sparse adaptation data. The p-values from the matched-pair t-test are shown in Table3; although the values are not always small, the performance improvement of the probabilistic 2-mode based method seems meaningful. Additionally, Figure8 shows the performance of the probabilistic 2-mode based model with (K _R =20,K _D =35), MLLR adaptation with a full regression matrix, and MAPLR adaptation for adaptation data of about 6–240 s; for adaptation sentences ≥ 10 (about 60 s), the probabilistic 2-mode based model shows the comparable performance with MLLR adaptation and MAPLR adaptation. In Figure8, the p-values are given as: p<0.01 for 1–5 adaptation sentences between the probabilistic 2-mode based model and MLLR adaptation, p<0.05 for 2–5 adaptation sentences between the probabilistic 2-mode based model and MAPLR adaptation. The number of free parameters of each method is summarized in Table4.

Figure 6

Word recognition accuracy of the probabilistic 2-mode based model, Tucker 3-mode based model, and MAPLR adaptation.

Figure 7

Word recognition accuracy of the Tucker 3-mode based model, MLLR, and eigenvoice adaptation.

Table 2 Word recognition accuracy (%) of the Tucker 3-mode and probabilistic 2-mode based methods

Table 3 p -values from the matched-pair t -test

Figure 8

Word recognition accuracy of the probabilistic 2-mode based model, MLLR, and MAPLR adaptation.

Table 4 Number of free parameters of adaptation techniques

We think that the performance improvement of the proposed method over MLLR or MAPLR adaptation comes from the use of basis vectors and speaker weight of large dimension. Additionally, we think that the performance improvement of the probabilistic 2-mode based method in the MAP framework over the Tucker decomposition based method in the ML framework for small amounts of adaptation data (e.g., 1 adaptation sentence) is due to its constraint on the weight. If the amount of adaptation data is small (e.g., 1 adaptation sentence), the weight cannot be reliably estimated in the ML framework where the weight is estimated using only adaptation data without constraint, as done in the Tucker decomposition based method. The results confirm that constraint on the weight in the MAP framework can produce better model when the amount of adaptation data is small.

The selection of appropriate dimensions of model parameters (e.g., K _R and K _D ) in the probabilistic 2-mode analysis depends on the training models and also available adaptation data. The selection of model parameters affects the performance of the system, but how to choose the optimum model parameters is not obvious, which needs a further study.

4 Conclusions

In this article, we applied probabilistic tensor analysis to the adaptation of HMM mean vectors to a new speaker. The training models consisted of the mean vectors of HMMs expressed in matrix form and the training set was decomposed by probabilistic 2-mode analysis. The prior distribution of the adaptation parameter was estimated from the training models. Then, the speaker adaptation equation was derived in the MAP framework. Compared with the speaker adaptation method based on Tucker 3-mode decomposition in the ML framework, the proposed method further improved the performance for small amounts of adaptation data.