Speech enhancement with an acoustic vector sensor: an effective adaptive beamforming and post-filtering approach

Abstract

Speech enhancement has an increasing demand in mobile communications and faces a great challenge in a real ambient noisy environment. This paper develops an effective spatial-frequency domain speech enhancement method with a single acoustic vector sensor (AVS) in conjunction with minimum variance distortionless response (MVDR) spatial filtering and Wiener post-filtering (WPF) techniques. In remote speech applications, the MVDR spatial filtering is effective in suppressing the strong spatial interferences and the Wiener post-filtering is considered as a popular and powerful estimator to further suppress the residual noise if the power spectral density (PSD) of target speech can be estimated properly. With the favorable directional response of the AVS together with the trigonometric relations of the steering vectors, the closed-form estimation of the signal PSDs is derived and the frequency response of the optimal Wiener post-filter is determined accordingly. Extensive computer simulations and a real experiment in an anechoic chamber condition have been carried out to evaluate the performance of the proposed algorithm. Simulation results show that the proposed method offers good ability to suppress the spatial interference while maintaining comparable log spectral deviation and perceptual evaluation of speech quality performance compared with the conventional methods with several objective measures. Moreover, a single AVS solution is particularly attractive for hands-free speech applications due to its compact size.

1 Introduction

As the presence of background noise significantly deteriorates the quality and intelligibility of speech, enhancement of speech signals has been an important and challenging problem and various methods have been proposed in the literature to tackle this problem. Spectral subtraction, Wiener filtering, and their variations [1] are commonly used for suppressing additive noise, but they are not able to effectively suppress spatial interference. In order to eliminate spatial interferences, beamforming techniques applied to microphone array recordings can be employed [2–9]. Among these, the minimum variance distortionless response (MVDR) beamformer known as the Capon beamformer and their equivalent generalized sidelobe cancellers (GSC) work successfully in remote speech enhancement applications [2]. However, the performance of MVDR-type methods is proportional to the number of array sensors used, thus limiting their application. Moreover, the MVDR beamformer is not effective at suppressing additive noise, leaving residual noise in its output. As a result, the well-known Wiener post-filtering solution normally can be employed to further reduce the residual noise from the output of the beamformer [7]. Recently, speech enhancement using the acoustic vector sensor (AVS) array has received research attention due to the merit of spatial co-location of microphones and signal time alignment [5, 10–12]. Compared with the traditional microphone array, the compact structure (occupying a volume of approximately 1 cm³) makes the AVS much more attractive in portable speech enhancement applications. Research showed that the AVS array beamformer with the MVDR method [5, 10] successfully suppresses spatial interferences but fails to effectively suppress background noise. The integrated MVDR and Wiener post-filtering method using AVS array [12] offers good performance in terms of suppression of spatial interferences and background additive noise, but it requires more than two AVS units as well as the good voice activity detection (VAD) technique.

In this paper, we focus on developing a speech enhancement solution capable of effectively suppressing spatial interferences and additive noise at a less computational cost using only one AVS unit. More specifically, by exploring the unique spatial co-location property (the signal arrives at the sensors at the same time) and the trigonometric relations of the steering vectors of the AVS, a single AVS-based speech enhancement system is proposed. The norm-constrained MVDR method is employed to form the spatial filter, while the optimal Wiener post-filter is designed by using a novel closed-form power spectral density (PSD) estimation method. The proposed solution does not depend on the VAD technique (for noise estimation) and hence has advantages of small size, less computation cost, and the ability to suppress both spatial interferences and background noise.

The paper is organized as follows. The data model of an AVS and the frequency domain MVDR (FMV) with a single AVS are presented in Section 2. The detailed derivation of the closed-form estimation of the signal PSDs for an optimal Wiener post-filtering (WPF) using the AVS structure is given in Section 3. The proposed norm-constrained FMV-effective Wiener post-filtering (NCFMV-EWPF) algorithm for speech enhancement is presented in Section 4. Simulation results are presented in Section 5. Section 6 concludes our work.

2 Problem formulation

2.1 Data model for an AVS unit

An AVS unit generally consists of four co-located constituent sensors, including one omnidirectional sensor (denoted as the o-sensor) and three orthogonally oriented directional sensors depicted as the u-sensor, v-sensor, and w-sensor, respectively. As an example, Figure 1 shows a data capture system with an AVS unit. In this paper, focusing on deriving the algorithm and making the derivation clear, let us assume that there is one target speech s(t) at (θ _s , ϕ _s ) = (90°, ϕ _s ) and one interference signal s _i (t) at (θ _i , ϕ _i ) = (90°, ϕ _i ) impinging on this AVS unit, where ϕ _s , ϕ _i ∈ [0°, 360°) are the azimuth angles. Since s(t) and s _i (t) arrive in the horizontal plane, we only need the u-sensor, v-sensor, and o-sensor to capture signals from the AVS unit. The angle difference between s(t) and s _i (t) is defined as

$$\mathrm{\Delta \varphi }={\mathit{\varphi }}_{\mathit{s}}-{\mathit{\varphi }}_{\mathit{i}}$$

(1)

Figure 1

Illustration of one AVS data capturing system[13, 14].

The corresponding steering vectors are given by

$$\mathbf{v}\left({\mathit{\varphi }}_{\mathit{s}}\right)={\left[{\mathit{u}}_{\mathit{s}},{\mathit{v}}_{\mathit{s}},1\right]}^{\mathit{T}}={\left[cos{\mathit{\varphi }}_{\mathit{s}},sin{\mathit{\varphi }}_{\mathit{s}},1\right]}^{\mathit{T}}$$

(2)

$$\mathbf{v}\left({\mathit{\varphi }}_{\mathit{i}}\right)={\left[{\mathit{u}}_{\mathit{i}},{\mathit{v}}_{\mathit{i}},1\right]}^{\mathit{T}}={\left[cos{\mathit{\varphi }}_{\mathit{i}},sin{\mathit{\varphi }}_{\mathit{i}},1\right]}^{\mathit{T}}$$

(3)

where [.]^T denotes the vector/matrix transposition.

In the cases that room reverberation does not exist, the received data of the AVS can be modeled as [13]

$${\mathbf{x}}_{\mathrm{avs}}\left(\mathit{t}\right)=\mathbf{v}\left({\mathit{\varphi }}_{\mathit{s}}\right)\mathit{s}\left(\mathit{t}\right)+\mathbf{v}\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{s}}_{\mathit{i}}\left(\mathit{t}\right)+{\mathbf{n}}_{\mathrm{avs}}\left(\mathit{t}\right)$$

(4)

where n_avs(t) is assumed as the additive white Gaussian noise at the AVS unit. Specifically, we have the following definitions:

$${\mathbf{x}}_{\mathrm{avs}}\left(\mathit{t}\right)={\left[{\mathit{x}}_{\mathit{u}}\left(\mathit{t}\right),{\mathit{x}}_{\mathit{v}}\left(\mathit{t}\right),{\mathit{x}}_{\mathit{o}}\left(\mathit{t}\right)\right]}^{\mathit{T}}$$

(5)

$${\mathbf{n}}_{\mathrm{avs}}\left(\mathit{t}\right)={\left[{\mathit{n}}_{\mathit{u}}\left(\mathit{t}\right),{\mathit{n}}_{\mathit{v}}\left(\mathit{t}\right),{\mathit{n}}_{\mathit{o}}\left(\mathit{t}\right)\right]}^{\mathit{T}}$$

(6)

where x _u (t), x _v (t), and x _o (t) are the received data of the u-, v-, and o-sensor, respectively, and n _u (t), n _v (t), and n _o (t) are the captured noise at the u-, v-, and o-sensor, respectively. The task of speech enhancement with an AVS is to estimate s(t) from x _avs (t).

In this study, without loss of generality, we follow the commonly used assumptions [4]: (1) s(t) and s _i (t) are mutually uncorrelated; (2) n _u (t), n _v (t), and n _o (t) are mutually uncorrelated.

2.2 FMV with a single AVS

The MVDR beamformer is considered as one of the most popular beamforming methods for suppressing spatial interferences in remote speech applications. In this subsection, we present the formulation of the frequency domain MVDR beamformer (FMV) with two sensors (u-sensor and v-sensor) of the AVS unit. From (2) to (4), the data received by the u-sensor and the v-sensor can be modeled as [14]

$$\mathbf{x}\left(\mathit{t}\right)={\left[{\mathit{x}}_{\mathit{u}}\left(\mathit{t}\right),{\mathit{x}}_{\mathit{v}}\left(\mathit{t}\right)\right]}^{\mathit{T}}=\mathbf{a}\left({\mathit{\varphi }}_{\mathit{s}}\right)\mathit{s}\left(\mathit{t}\right)+\mathbf{a}\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{s}}_{\mathit{i}}\left(\mathit{t}\right)+\mathbf{n}\left(\mathit{t}\right)$$

(7)

where

$$\mathbf{a}\left({\mathit{\varphi }}_{\mathit{s}}\right)={\left[{\mathit{u}}_{\mathit{s}},{\mathit{v}}_{\mathit{s}}\right]}^{\mathit{T}}={\left[cos{\mathit{\varphi }}_{\mathit{s}},sin{\mathit{\varphi }}_{\mathit{s}}\right]}^{\mathit{T}}$$

(8)

and

$$\mathbf{a}\left({\mathit{\varphi }}_{\mathit{i}}\right)={\left[{\mathit{u}}_{\mathit{i}},{\mathit{v}}_{\mathit{i}}\right]}^{\mathit{T}}={\left[cos{\mathit{\varphi }}_{\mathit{i}},sin{\mathit{\varphi }}_{\mathit{i}}\right]}^{\mathit{T}}$$

(9)

The frequency domain formulation of the data model of (7) is given by

$$\mathbf{X}\left(\mathit{f}\right)=\mathbf{a}\left({\mathit{\varphi }}_{\mathit{s}}\right)\mathit{S}\left(\mathit{f}\right)+\mathbf{a}\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{S}}_{\mathit{i}}\left(\mathit{f}\right)+\mathbf{N}\left(\mathit{f}\right)$$

(10)

where X(f) = [X _u (f), X _v (f)]^T and N(f) = [N _u (f), N _v (f)]^T. The beamforming is then performed by applying a complex weight to the captured signals, and the output of the FMV can be denoted as

$$\begin{array}{ll}\mathit{Y}\left(\mathit{f}\right)={\mathbf{w}}^{\mathit{H}}\left(\mathit{f}\right)\mathbf{X}\left(\mathit{f}\right)& ={\mathbf{w}}^{\mathit{H}}\left(\mathit{f}\right)\mathbf{a}\left({\mathit{\varphi }}_{\mathit{s}}\right)\mathit{S}\left(\mathit{f}\right)\\ +{\mathbf{w}}^{\mathit{H}}\left(\mathit{f}\right)\left(\mathbf{a}\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{S}}_{\mathit{i}}\left(\mathit{f}\right)+\mathbf{N}\left(\mathit{f}\right)\right)\end{array}$$

(11)

where (.)^H denotes the Hermitian transposition. w^H(f) = [w _u (f), w _v (f)] is the weight vector of the FMV. Let us define

$$\mathit{g}\left({\mathit{\varphi }}_{\mathit{s}},\mathit{f}\right)={\mathbf{w}}^{\mathit{H}}\left(\mathit{f}\right)\mathbf{a}\left({\mathit{\varphi }}_{\mathit{s}}\right)$$

(12)

$$\mathit{g}\left({\mathit{\varphi }}_{\mathit{i}},\mathit{f}\right)={\mathbf{w}}^{\mathit{H}}\left(\mathit{f}\right)\mathbf{a}\left({\mathit{\varphi }}_{\mathit{i}}\right)$$

(13)

Obviously, g(ϕ _s , f) and g(ϕ _i , f) can be viewed as the spatial response gains of the FMV to the target spatial signal S(f) and the spatial interference signal S _i (f), respectively. Substituting (12) and (13) into (11), we can get

$$\begin{array}{ll}\mathit{Y}\left(\mathit{f}\right)={\mathbf{w}}^{\mathit{H}}\left(\mathit{f}\right)\mathbf{X}\left(\mathit{f}\right)& =\mathit{g}\left({\mathit{\varphi }}_{\mathit{s}},\mathit{f}\right)\mathit{S}\left(\mathit{f}\right)\\ +\mathit{g}\left({\mathit{\varphi }}_{\mathit{i}},\mathit{f}\right){\mathit{S}}_{\mathit{i}}\left(\mathit{f}\right)+{\mathbf{w}}^{\mathit{H}}\left(\mathit{f}\right)\mathbf{N}\left(\mathit{f}\right)\end{array}$$

(14)

The basic idea of designing the optimal FMV is to maintain g(ϕ _s , f) = 1 for S(f) while minimizing the output signal power (P _YY  = E[Y(f)Y*(f)]) of the FMV to suppress other undesired sources. Hence, the optimal weight vector of the FMV can be obtained by solving the constrained optimization cost function [2]:

$$\begin{array}{ll}{\mathbf{w}}_{\mathrm{FMV}}\left(\mathit{f}\right)& =\underset{\mathit{w}}{argmin}{\mathit{P}}_{\mathit{YY}}\\ \mathrm{subject}\mathrm{to}\mathit{g}\left({\mathit{\varphi }}_{\mathit{s}},\mathit{f}\right)=1,\mathrm{and}{\mathit{P}}_{\mathit{YY}}={\mathbf{w}}^{\mathit{H}}\left(\mathit{f}\right){\mathbf{R}}_{\mathbf{x}}\left(\mathit{f}\right)\mathbf{w}\left(\mathit{f}\right)\end{array}$$

(15)

where R _x (f) = E[X(f)X^H(f)] is the autocorrelation matrix of the received data of the FMV. The optimal solution of (15) is given as [2]

$${\mathbf{w}}_{\mathrm{FMV}}\left(\mathit{f}\right)\mathbf{=}\frac{{\mathbf{R}}_{\mathbf{x}}^{-1}\left(\mathit{f}\right)\mathbf{a}\left({\mathit{\varphi }}_{\mathit{s}}\right)}{{\mathbf{a}}^{\mathit{T}}\left({\mathit{\varphi }}_{\mathit{s}}\right){\mathbf{R}}_{\mathbf{x}}^{-1}\left(\mathit{f}\right)\mathbf{a}\left({\mathit{\varphi }}_{\mathit{s}}\right)}$$

(16)

Equation 16 is the standard form of the FMV. It is clear that when a(ϕ _s ) is fixed (speech target is static), w_FMV(f) depends on the estimate of ${\mathbf{R}}_{\mathbf{x}}^{-1}\left(\mathit{f}\right)$. There are several methods that have been proposed to estimate R _x (f) [1], and the diagonal loading technique is one of the robust algorithms aiming at avoiding the non-singularity in (16), which leads to a norm-constrained FMV (NCFMV) as shown in (17) [3]:

$${\mathbf{w}}_{\mathit{NC}}\left(\mathit{f}\right)=\frac{\left({\mathbf{R}}_{\mathbf{x}}\left(\mathit{f}\right)+\mathit{\gamma }\mathbf{I}\left){}^{-1}\mathbf{a}\right({\mathit{\varphi }}_{\mathit{s}}\right)}{{\mathbf{a}}^{\mathit{T}}\left({\mathit{\varphi }}_{\mathit{s}}\right){\left({\mathbf{R}}_{\mathbf{x}}\left(\mathit{f}\right)+\mathit{\gamma }\mathbf{I}\right)}^{-1}\mathbf{a}\left({\mathit{\varphi }}_{\mathit{s}}\right)+\mathit{\sigma }}$$

(17)

where I is an identity matrix, γ is the positive loading factor, and σ is a small positive number to avoid the denominator becoming zero. It is expected that the NCFMV will greatly suppress the spatial unwanted signals. Obviously, the output of the NCFMV can be derived as follows with (17), (12), (13), and some simple manipulations:

$$\begin{array}{ll}\mathit{Y}\left(\mathit{f}\right)& ={\mathbf{w}}_{\mathit{NC}}^{\mathit{H}}\left(\mathit{f}\right)\mathbf{X}\left(\mathit{f}\right)=\mathit{S}\left(\mathit{f}\right)+{\mathbf{w}}_{\mathit{NC}}^{\mathit{H}}\left(\mathit{f}\right)\mathbf{a}\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{S}}_{\mathit{i}}\left(\mathit{f}\right)\\ +{\mathbf{w}}_{\mathit{NC}}^{\mathit{H}}\left(\mathit{f}\right){\mathbf{N}}_{\mathit{un}}\left(\mathit{f}\right)=\mathit{S}\left(\mathit{f}\right)+\mathit{g}({\mathit{\varphi }}_{\mathit{i}},\mathit{f}){\mathit{S}}_{\mathit{i}}\left(\mathit{f}\right)\\ +{\mathbf{w}}_{\mathit{NC}}^{\mathit{H}}\left(\mathit{f}\right){\mathbf{N}}_{\mathit{un}}\left(\mathit{f}\right)\end{array}$$

(18)

2.3 The estimation of the power spectral density

As discussed above, the NCFMV is only effective in suppressing the spatial interferences. In this section, a new solution has been proposed by incorporating the well-known Wiener post-filter (WPF) to further suppress the residual noise in beamformer output Y(f) in (18).

Basically, according to the formulation of the Wiener filter in the frequency domain, to estimate S(f) from Y(f), the frequency response of the Wiener filter is given by [6, 8]

$${\mathit{W}}_{\mathit{pf}}\left(\mathit{f}\right)={\mathit{\psi }}_{\mathit{YS}}\left(\mathit{f}\right)/{\mathit{\psi }}_{\mathit{YY}}\left(\mathit{f}\right)\approx {\mathit{\psi }}_{\mathit{SS}}\left(\mathit{f}\right)/{\mathit{\psi }}_{\mathit{YY}}\left(\mathit{f}\right)$$

(19)

where ψ _YS (f) is the cross-power spectrum density (CSD) of S(f) and Y(f) and ψ _YY (f) is the power spectral density (PSD) of Y(f). Generally, Y(f) are considered uncorrelated to interferences, and we can approximately get the second equation in (19) via (18). From (19), it is clear that a good estimate of ψ _SS (f) and ψ _YY (f) from X(f) and Y(f) are very crucial to the performance of the WPF. There are some PSD estimation algorithms that have been proposed under different spatial-frequency joint estimation schemes. For single-channel application as an example, the voice activity detection (VAD) method is usually applied to get the noise and speech segments, and then the spectrum subtraction algorithm can be used to remove noise components before estimating ψ _SS (f). Moreover, for microphone array post-filtering schemes, ψ _SS (f) can be estimated from the available multichannel signals, which are assumed to be within an incoherent noise environment [6].

Motivated by the unique properties of the AVS, where multichannel signals are available (u-, v, and o-sensor signals) and there exists a trigonometric relationship between the steering vectors a(ϕ _s ) and a(ϕ _i ) of the AVS, in this paper, we will derive a closed-form solution to estimate ψ _SS (f) and ψ _YY (f) to form an optimal WPF. The system diagram proposed is shown in Figure 2.

Figure 2

Our proposed speech enhancement system using single AVS.

3 The formulation of the Wiener post-filter

3.1 Derivation of the estimate of CSD and PSD

For presentation clarity, let us define the notation of the cross-power spectral density (CSD) between α(f) and β(f) as

$${\mathit{\psi }}_{\mathit{\alpha \beta }}\left(\mathit{f}\right)=\mathit{E}\left[\mathit{\alpha }\left(\mathit{f}\right){\mathit{\beta }}^{*}\left(\mathit{f}\right)\right]$$

(20)

From (10), we have

$${\mathit{X}}_{\mathit{u}}\left(\mathit{f}\right)=cos\left({\mathit{\varphi }}_{\mathit{s}}\right)\mathit{S}\left(\mathit{f}\right)+cos\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{S}}_{\mathit{i}}\left(\mathit{f}\right)+{\mathit{N}}_{\mathit{u}}\left(\mathit{f}\right)$$

(21)

$${\mathit{X}}_{\mathit{v}}\left(\mathit{f}\right)=sin\left({\mathit{\varphi }}_{\mathit{s}}\right)\mathit{S}\left(\mathit{f}\right)+sin\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{S}}_{\mathit{i}}\left(\mathit{f}\right)+{\mathit{N}}_{\mathit{v}}\left(\mathit{f}\right)$$

(22)

For presentation simplicity, the frequency index f will be dropped in the following derivation. Ideally, the additive noises of u-, v-, and o-sensors have the same power, and then we have

$${\mathit{\psi }}_{\mathit{NN}}=\mathit{E}\left[{\mathit{N}}_{\mathit{u}}{\mathit{N}}_{\mathit{u}}^{*}\right]=\mathit{E}\left[{\mathit{N}}_{\mathit{v}}{\mathit{N}}_{\mathit{v}}^{*}\right]=\mathit{E}\left[{\mathit{N}}_{\mathit{o}}{\mathit{N}}_{\mathit{o}}^{*}\right]$$

(23)

It is noted that the assumption of equal power for all channels used in (23) is not true for the real signals recorded by the AVS unit, but these can be calibrated in practice [15]. With (18), (21), (22), and (23), the CSD and PSD of signals can be derived following the definition given in (20):

$$\begin{array}{ll}{\mathit{\psi }}_{\mathit{uu}}& =\mathit{E}\left[{\mathit{X}}_{\mathit{u}}{\mathit{X}}_{\mathit{u}}^{*}\right]={cos}^2\left({\mathit{\varphi }}_{\mathit{s}}\right)\mathit{E}\left[\mathit{S}{\mathit{S}}^{*}\right]+{cos}^2\left({\mathit{\varphi }}_{\mathit{i}}\right)\mathit{E}\left[{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}^{*}\right]\\ +\mathit{E}\left[{\mathit{N}}_{\mathit{u}}{\mathit{N}}_{\mathit{u}}^{*}\right]={cos}^2\left({\mathit{\varphi }}_{\mathit{s}}\right){\mathit{\psi }}_{\mathit{SS}}+{cos}^2\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}+{\mathit{\psi }}_{\mathit{NN}}\end{array}$$

(24)

$$\begin{array}{ll}{{\mathit{\psi }}_{\mathit{v}}}_{\mathit{v}}& =\mathit{E}\left[{\mathit{X}}_{\mathit{v}}{\mathit{X}}_{\mathit{v}}^{*}\right]={sin}^2\left({\mathit{\varphi }}_{\mathit{s}}\right)\mathit{E}\left[\mathit{S}{\mathit{S}}^{*}\right]+{sin}^2\left({\mathit{\varphi }}_{\mathit{i}}\right)\mathit{E}\left[{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}^{*}\right]\\ +\mathit{E}\left[{\mathit{N}}_{\mathit{v}}{\mathit{N}}_{\mathit{v}}^{*}\right]={sin}^2\left({\mathit{\varphi }}_{\mathit{s}}\right){\mathit{\psi }}_{\mathit{SS}}+{cos}^2\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}+{\mathit{\psi }}_{\mathit{NN}}\end{array}$$

(25)

$${\mathit{\psi }}_{\mathit{oo}}=\mathit{E}\left[{\mathit{X}}_{\mathit{o}}{\mathit{X}}_{\mathit{o}}^{*}\right]={\mathit{\psi }}_{\mathit{SS}}+{\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}+{\mathit{\psi }}_{\mathit{NN}}$$

(26)

$${\mathit{\psi }}_{\mathit{YY}}=\mathit{E}\left[\mathit{Y}{\mathit{Y}}^{*}\right]={\mathit{\psi }}_{\mathit{SS}}+{\mathit{g}}^2\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}+\left|\right|{\mathit{w}}_{\mathit{NC}}\left|\right|{}^2{\mathit{\psi }}_{\mathit{NN}}$$

(27)

$${\mathit{\psi }}_{\mathit{u}+\mathit{v}}={\mathit{\psi }}_{\mathit{uu}}+{\mathit{\psi }}_{\mathit{vv}}={\mathit{\psi }}_{\mathit{SS}}+{\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}+2{\mathit{\psi }}_{\mathit{NN}}$$

(28)

$${\mathit{\psi }}_{\mathit{Yo}}=\mathit{E}\left[\mathit{Y}{\mathit{X}}_{\mathit{o}}^{*}\right]={\mathit{\psi }}_{\mathit{SS}}+{\mathit{g}}^{*}\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}$$

(29)

$${\mathit{\psi }}_{\mathit{oY}}=\mathit{E}\left[{\mathit{X}}_{\mathit{o}}{\mathit{Y}}^{*}\right]={\mathit{\psi }}_{\mathit{SS}}+\mathit{g}\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}$$

(30)

From (24) to (30), it is clear that there are seven equations with four unknown variables ψ _NN , g(ϕ _i ), ${\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}$, and ${\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}$. Hence, using (28) and (26), the PSD of noise can be derived as

$${\mathit{\psi }}_{\mathit{NN}}={\mathit{\psi }}_{\mathit{u}+\mathit{v}}-{\mathit{\psi }}_{\mathit{oo}}$$

(31)

Similarly, the gain response of the NCFMV on the interference S _i can be given by

$$\mathit{g}\left({\mathit{\varphi }}_{\mathit{i}}\right)=\left({\mathit{\psi }}_{\mathit{oY}}-{\mathit{\psi }}_{\mathit{YY}}+\left|\right|{\mathbf{w}}_{\mathit{NC}}\left|\right|{}^2{\mathit{\psi }}_{\mathit{NN}}\right)/\left({\mathit{\psi }}_{\mathit{oo}}-{\mathit{\psi }}_{\mathit{Yo}}-{\mathit{\psi }}_{\mathit{NN}}\right)$$

(32)

Moreover, the PSD of the interference S _i and the target speech S can be derived, respectively, as follows:

$${\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}=\left({\mathit{\psi }}_{\mathit{oo}}-{\mathit{\psi }}_{\mathit{oY}}-{\mathit{\psi }}_{\mathit{NN}}\right)/\left(1-\mathit{g}\left({\mathit{\varphi }}_{\mathit{i}}\right)\right)$$

(33)

$${\mathit{\psi }}_{\mathit{SS}}={\mathit{\psi }}_{\mathit{oY}}-\mathit{g}\left({\mathit{\varphi }}_{\mathit{i}}\right){\mathit{\psi }}_{{\mathit{S}}_{\mathit{i}}{\mathit{S}}_{\mathit{i}}}$$

(34)

3.2 The proposed EWPF method and some discussions

Till now, we have mathematically derived the closed-form expressions of the ψ _SS in (34), ψ _YY in (27), and W _pf in (19). Since Y, X _u , X _v , and X _o can be measured, the estimates of ψ _SS and ψ _YY can be determined accordingly. Hence, (33), (34), (27), and (19) describe the basic form of our proposed effective Wiener post-filtering algorithm for further enhancing the speech with an AVS (here, we term it as EWPF for short). In the following context, we will have some discussions on our proposed EWPF method.

Firstly, to implement the EWPF, the cross-power spectral density ψ _αβ (f) needs to be estimated. It is well known that the recursive update formula is a popular approach:

$${\widehat{\mathit{\psi }}}_{\mathit{\alpha \beta }}\left(\mathit{f},\mathit{l}\right)=\mathit{\lambda }{\widehat{\mathit{\psi }}}_{\mathit{\alpha \beta }}\left(\mathit{f},\mathit{l}-1\right)+\left(1-\mathit{\lambda }\right)\mathit{\alpha }\left(\mathit{f},\mathit{l}\right){\mathit{\beta }}^{*}\left(\mathit{f},\mathit{l}\right)$$

(35)

where l is the frame index and λ ∈ (0, 1] is the forgetting factor.

Secondly, it is noted that when Δϕ defined in (1) is close to or equal to 0, the denominator in (32) goes to 0. To avoid this situation, one small positive factor σ _r should be added to the denominator of (32) and we get

$$\widehat{\mathit{g}}\left({\mathit{\varphi }}_{\mathit{i}}\right)=\left({\widehat{\mathit{\psi }}}_{\mathit{oY}}-{\widehat{\mathit{\psi }}}_{\mathit{YY}}+\left|\right|{\mathbf{w}}_{\mathit{NC}}∥{}^2{\widehat{\mathit{\psi }}}_{\mathit{NN}}\right)/\left({\widehat{\mathit{\psi }}}_{\mathit{oo}}-{\widehat{\mathit{\psi }}}_{\mathit{Yo}}-{\widehat{\mathit{\psi }}}_{\mathit{NN}}+{\mathit{\sigma }}_{\mathit{r}}\right)$$

(36)

Thirdly, analyzing the properties of g(ϕ _i ), we observe the following: (1) If the target source s(t) is considered as short-time spatially stationary (approximately true for speech applications), w _NC in (17) can be updated every L _u frames for reducing computational complexity. Therefore, from the definition of (13), the gain g(ϕ _i ) will remain unchanged within L _u frames. However, ${\widehat{\mathit{\psi }}}_{\mathit{\alpha \beta }}\left(\mathit{f},\mathit{l}\right)$ is estimated frame by frame via (35); therefore, a more accurate estimation of g(ϕ _i ) can be achieved by averaging over L _u frames. (2) From (36), it is clear that the small denominator will lead to a large variation of g(ϕ _i ), reflecting incorrect estimates since the NCFMV is designed to suppress rather than to amplify the interference. Hence, it is reasonable to apply a clipping function f _c (x, b) (see (43)) to remove the outliers in the estimate of $\widehat{\mathit{g}}\left({\mathit{\varphi }}_{\mathit{i}}\right)$.

4 The proposed NCFMV-EWPF algorithm

Similar to the existing remote speech enhancement applications, our proposed algorithm is implemented in the frequency domain by segmenting the received signal into frames and then the short-time Fourier transforms (STFTs) are applied. Specifically, to determine w _NC in (17), the estimate of the R _x is given by [10]

$${\widehat{\mathbf{R}}}_{\mathit{x}}\left(\mathit{k}\right)=\frac{1}{\mathit{F}}\left[\begin{array}{cc}\hfill {\mathit{C}}_{\mathit{d}}\mathit{A}\left(\mathit{k}\right)\hfill & \hfill \mathit{B}\left(\mathit{k}\right)\hfill \\ \hfill \mathit{C}\left(\mathit{k}\right)\hfill & \hfill {\mathit{C}}_{\mathit{d}}\mathit{D}\left(\mathit{k}\right)\hfill \end{array}\right]$$

(37)

where k is the frequency bin index and k = 1,2,…, K. C _d is a constant slightly greater than the one that helps avoid matrix singularity. F is the frame number used for estimating R _x (k), and in our study it is set as F = 2L _u . Let us define X _u (k,l) and X _v (k,l) as the k th component of the spectrum of the l th frame of x _u (n) and x _v (n), respectively, and we have

$$\mathit{A}\left(\mathit{k}\right)={\displaystyle \sum _{\mathit{i}=0}^{\mathit{F}-1}{\mathit{X}}_{\mathit{u}}^{*}\left(\mathit{k},\mathit{l}-\mathit{i}\right){\mathit{X}}_{\mathit{u}}\left(\mathit{k},\mathit{l}-\mathit{i}\right)}$$

(38)

$$\mathit{B}\left(\mathit{k}\right)={\displaystyle \sum _{\mathit{i}=0}^{\mathit{F}-1}{\mathit{X}}_{\mathit{u}}^{*}\left(\mathit{k}\mathit{,}\mathit{l}-\mathit{i}\right){\mathit{X}}_{\mathit{v}}\left(\mathit{k}\mathit{,}\mathit{l}-\mathit{i}\right)}$$

(39)

$$\mathit{C}\left(\mathit{k}\right)={\displaystyle \sum _{\mathit{i}=0}^{\mathit{F}-1}{\mathit{X}}_{\mathit{v}}^{*}\left(\mathit{k}\mathit{,}\mathit{l}-\mathit{i}\right){\mathit{X}}_{\mathit{u}}\left(\mathit{k}\mathit{,}\mathit{l}-\mathit{i}\right)}$$

(40)

$$\mathit{D}\left(\mathit{k}\right)={\displaystyle \sum _{\mathit{i}=0}^{\mathit{F}-1}{\mathit{X}}_{\mathit{v}}^{*}\left(\mathit{k}\mathit{,}\mathit{l}-\mathit{i}\right){\mathit{X}}_{\mathit{v}}\left(\mathit{k}\mathit{,}\mathit{l}-\mathit{i}\right)}$$

(41)

From (37) to (41), we can see the autocorrelation matrix R _x (k) is estimated by using the F most recent fast Fourier transforms (FFTs). Therefore, the robust estimation of W _pf (k) in (19) asks for the robust estimation of g(ϕ _i ,k). According to the discussions in Section 3.2, we adopt the following estimation:

$$\widehat{\mathit{g}}\left({\mathit{\varphi }}_{\mathit{i}},\mathit{k}\right)=\frac{1}{{\mathit{L}}_{\mathit{u}}}{\displaystyle \sum _{\mathit{l}={\mathit{L}}_1}^{{\mathit{L}}_2}{\mathit{f}}_{\mathit{c}}\left(\frac{{\widehat{\mathit{\psi }}}_{\mathit{oY}}\left(\mathit{k},\mathit{l}\right)-{\widehat{\mathit{\psi }}}_{\mathit{YY}}\left(\mathit{k},\mathit{l}\right)+\left|\right|{\mathbf{w}}_{\mathit{NC}}\left(\mathit{k},\mathit{l}\right)\left|\right|{}^2{\widehat{\mathit{\psi }}}_{\mathit{NN}}\left(\mathit{k},\mathit{l}\right)}{{\widehat{\mathit{\psi }}}_{\mathit{oo}}\left(\mathit{k},\mathit{l}\right)-{\widehat{\mathit{\psi }}}_{\mathit{Yo}}\left(\mathit{k},\mathit{l}\right)-{\widehat{\mathit{\psi }}}_{\mathit{NN}}\left(\mathit{k},\mathit{l}\right)+{\mathit{\sigma }}_{\mathit{r}}},\mathit{b}\right)}$$

(42)

where L₁ = fix((l − 1)/L _u )L _u  + 1, L₂ = fix((l − 1)/L _u )L _u  + L _u , fix(.) is the floor operation, b is a predefined threshold, and f _c (x,b) is the clipping function and defined as

$${\mathit{f}}_{\mathit{c}}\left(\mathit{x},\mathit{b}\right)=\mathit{x}\mathrm{when}0<\mathit{x}\le \mathit{b}\mathrm{else}{\mathit{f}}_{\mathit{c}}\left(\mathit{x},\mathit{b}\right)=0$$

(43)

For presentation completeness, the proposed NCFMV-EWPF algorithm is summarized in Algorithm 1.

5 Simulation study

The performance evaluation of our proposed NCFMV-EWPF algorithm has been carried out in this section. The commonly used performance measurement metrics have been adopted, which include the following:

• 1. Output signal to interference plus noise ratio (SINR) defined as [7]

  $$\mathrm{SINR}=10log\left({∥{\mathit{z}}_{\mathit{s}}\left(\mathit{t}\right)∥}^2/{∥{\mathit{x}}_{\mathit{o}}\left(\mathit{t}\right)-{\mathit{z}}_{\mathit{s}}\left(\mathit{t}\right)∥}^2\right)$$

  (44)

where z _s (t) is the enhanced speech of the system and x _o (t) is the received signal of the o-sensor. Moreover, a segmental output SINR is calculated on a frame-by-frame basis and then averaged over the total number frames to get more accurate prediction of perceptual speech quality [7].

• 2. Log spectral deviation (LSD), which is used to measure the speech distortion and defined as [16]

  $$\mathrm{LSD}=∥ln\left({\mathit{\psi }}_{\mathit{ss}}\left(\mathit{f}\right)/{\mathit{\psi }}_{\mathit{zz}}\left(\mathit{f}\right)\right)∥$$

  (45)

where ψ _ss (f) is the PSD of the target speech and ψ _zz (f) is the PSD of the enhanced speech. It is clear that the smaller LSD indicates the less speech distortion. Similar to the calculation of SINR, the segmental LSD is computed.

• 3. Perceptual evaluation of speech quality (PESQ) [17]: To evaluate the performance of the speech enhancement algorithms, ITU-PESQ software [17] is utilized.

In addition, we also compared the performance of the Zelinski post-filter (ZPF) [4], NCFMV [5], and NCFMV-ZPF [6] algorithms under the same conditions to our proposed algorithm. The setup of the single AVS unit is shown in Figure 1.

In computer simulation studies, for each trial, a male speech lasting about 5 s acts as the target speech s(t) and babble speech taken from the Noisex-92 database [18] acts as the interference speech s _i (t). One set of the typical waveforms used in our simulation studies is shown in Figure 3.

Figure 3

Waveforms. s(t) is the target speech, s _i (t) is the interference speech, n(t) is the additive noise, and z(t) is the enhanced speech signal.

5.1 Experiments on simulated data

5.1.1 Experiment 1: the SINR performance under different noise conditions

In this experiment, we have carried out nine trials (numbered as trial 1 to trial 9) to evaluate the performance of the algorithms under different spatial and additive noise conditions [9]. The experimental settings are as follows: The sampling rate is set to be 16 kHz and a 512-point FFT is used. The target speaker is located at (90°, 45°) and the interference speaker is set at (90°, 0°). For the proposed NCFMV-EWPF algorithm, parameters are set as λ = 0.6, σ _r  = 10⁻³, L _u  = 4, γ = σ = 10⁻⁵, C _d  = 1.1, and b = 6, which produced the best experimental results under this specific setup. For comparison algorithms, the parameter settings are set as the same as those in the relevant papers. The experimental results are listed in Table 1.

Table 1 SINR-out for different algorithms (dB)

As shown in Table 1, the best performance for different conditions is addressed in italics. The proposed NCFMV-EWPF algorithm outperforms other algorithms in terms of SINR-out in trials 1 to 4 and trials 7 to 9, and gives comparable performance in trial 5 and inferior performance in trial 6. It is noted that, in trials 4 to 6, there is no spatial interference considered (i.e., s _i (t) = 0). The performance for trial 5 indicates that the proposed NCFMV-EWPF is not as effective as the ZPF in suppressing the additive noise with higher SNR (SNR > 10 dB) when spatial interference is not present. Therefore, these experimental results demonstrate the superior capability of the proposed NCFMV-EWPF in suppressing the spatial and adverse additive interferences. For visualization purposes, the results in Table 1 have also been plotted in Figure 4, where the x-axis represents the SINR of the signal captured by the AVS (termed as SINR-input) and the y-axis represents the SINR of the enhanced speech (termed as SINR-out).

Figure 4

The performance of SINR-out versus SINR-input. (a) n_avs(t) = 0 and s _i (t) ≠ 0. (b) n_avs(t) ≠ 0 and s _i (t) = 0. (c) n_avs(t) ≠ 0 and s _i (t) ≠ 0.

5.1.2 Experiment 2: the impact of the angle between the target and interference speakers

This experiment evaluates the impact of the angle between the target and interference speakers (Δϕ = ϕ _s  − ϕ _i ) on the performance of the NCFMV-EWPF algorithm. The results of the SINR-out versus Δϕ are shown in Figure 5, where the same experimental settings as those used for trial 7 in experiment 1 were adopted except the target speech location ϕ _s varied from (90°,0°) to (90°,360°) with 45° increments. From Figure 5, it is clear to see that when Δϕ → 0° (the target speaker moves closer to the interference speaker), for both algorithms, the SINR-out drops significantly and almost goes to 0. This means the speech enhancement is very much limited under this condition. However, when Δϕ > 0°, the SINR-out gradually increases. It is quite encouraging to see that the SINR-out of our proposed NCFMV-EWPF algorithm is superior to that of the NCFMV algorithm for all angles. Moreover, the SINR-out of our proposed NCFMV-EWPF algorithm maintains about 15 dB when Δϕ ≥ 45°.

Figure 5

The SINR-out versus Δ ϕ (NCFMV[5]).

5.1.3 Experiment 3: SINR, LSD, and PESQ performance

In this experiment, we adopted three performance metrics (SINR, LSD, and PESQ) to evaluate the performance of the algorithms. The same experimental settings of those used in experiment 1 were employed, where the SINR-input is set as 0 dB, the target speaker is located at (90°, 45°), and the interference speaker is at (90°,0°) (Δϕ = 45°). The experimental results are given in Table 2. It can be seen that the overall performance of our proposed NCFMV-EWPF algorithm is superior to that of other comparison algorithms. The LSD and PESQ performance of the NCFMV-EWPF algorithm is comparable to that of the NCFMV-ZPF [6] algorithm. It is encouraging to see that the proposed NCFMV-EWPF algorithm is able to effectively suppress the interference and additive noise while maintaining good speech quality and less distortion.

Table 2 Performance comparison

5.2 Experiments on recorded data in an anechoic chamber

5.2.1 Experiment 4: the SINR-out performance with different speakers

In this experiment, we conducted the speech enhancement by using the recorded data from Ritz's lab [19]. The experimental setup is shown in Figure 6. The AVS has been built by two Knowles NR-3158 pressure gradient sensors (u-sensor and v-sensor) and one Knowles EK-3132 sensor (o-sensor) (Knowles Electronics Inc., Itasca, IL, USA). Recordings were made of 10 different speech sentences from the IEEE speech corpus [20] in an anechoic chamber and background noise only from computer servers and air conditioning. The anechoic chamber is similar to the noise field: n_avs(t) = 0 and s _i (t) ≠ 0. The sampling rate was 48 kHz and then down-sampled to 16 kHz for speech enhancement. The speakers were placed in front of the AVS at a distance of 1 m. Target speech was located at a fixed position (90°, 45°), while interference speech was located at (90°, 90°). Ten trials were carried out using the 10 different target speeches.

Figure 6

The illustration of the experimental setup in an anechoic chamber in Ritz's lab[19]. The leftmost image is the AVS used, s(t) is the target speech (source 1) at 45°, and s _i (t) is the interference speech (source 2) at 90°.

The experimental results are shown in Figure 7. The x-axis represents the number of trials, and the y-axis represents the SINR of the enhanced speech (in dB). It is clear to see that the proposed NCFMV-EWPF algorithm provides superior SINR-out performance for all trails when the SINR-input of the recorded data is at about −5 dB. The experimental results with the real recorded data further validate the effectiveness of the proposed NCFMV-EWPF in suppressing the strong competing speech.

Figure 7

The SINR-out performance versus trial number (NCFMV[5]).

5.2.2 Experiment 5: the impact of the angle between the target and interference speakers

Similar to experiment 2, this experiment evaluates the impact of the angle between the target and interference speakers (Δϕ = |ϕ _s  − ϕ _i |) on the performance of the NCFMV-EWPF algorithm. The results of the SINR-out versus Δϕ are shown in Figure 8, where the experimental setup is the same as that of experiment 4 except that the angle of the target speaker (ϕ _s ) varies from (90°,90°) to (90°,0°) with 15° decrement.

Figure 8

The SINR-out versus Δ ϕ (NCFMV[5]).

From Figure 8, it is clear to see that the performance of the proposed NCFMV-EWPF algorithm is superior to that of the NCFMV algorithm for all Δϕ values. Compared to the results shown in Figure 5 using the simulated data, similar conclusions can be drawn for the proposed NCFMV-EWPF algorithm. More specifically, with the recorded data, when Δϕ > 15°, the proposed NCFMV-EWPF algorithm can effectively enhance the target speech.

5.2.3 Experiment 6: PESQ performance versus Δϕ

In this experiment, we only adopted one performance metrics (PESQ) to evaluate the performance of the algorithms. The same experimental settings as those used in experiment 5 were employed, where the angle of the interference speaker (ϕ _i ) was fixed at (90°,90°) and the angle of the target speaker (ϕ _s ) varied from (90°,90°) to (90°,0°) with 15° decrement. The experimental results are given in Figure 9. It can be seen that the overall performance of PESQ for our proposed NCFMV-EWPF algorithm is superior to that of the comparison algorithm for all angle differences. This experiment also demonstrates the ability of the proposed NCFMV-EWPF algorithm in effectively suppressing the interference and additive noise while maintaining good speech quality and less distortion when Δϕ > 15°.

Figure 9

The PESQ performance versus Δ ϕ (NCFMV[5]).

6 Conclusions

In this paper, a novel speech enhancement algorithm named as NCFMV-EWPF has been derived with a single AVS unit by an efficient closed-form estimation of the power spectral densities of signals. The results of computer simulation show that the proposed NCFMV-EWPF algorithm outperforms the existing ZPF, NCFMV, and NCFMV-ZPF algorithms, in terms of suppressing the competing speaker and noise field. The results of real experiments show that compared with the NCFMV algorithms, the proposed NCFMV-EWPF algorithm can effectively suppress the competing speech and additive noise while maintaining good speech quality and less distortion. In addition, it is noted that the NCFMV-EWPF algorithm does not require the VAD technique, which not only reduces the computational complexity but also provides more robust performance in a noisy environment, such as the higher output SINR, less speech distortion, and better speech intelligibility. It is expected that this novel approach developed in this paper is a suitable solution for implementation within hands-free speech recording systems.