Pitch- and Formant-Based Order Adaptation of the Fractional Fourier Transform and Its Application to Speech Recognition

Traditional speech processing methods generally treat speech as short-time stationary, that is, process speech in 20~30-milliseconds frames. In practice, however, intonation and coarticulation introduce combined spectrotemporal fluctuations to speech even for the typical frame sizes used in the front-end analysis. Modeling speech signals as frequency modulation signals therefore might accord better with speech characteristics from both production and perception views.

From the speech production view, traditional linear source-filter theory lacks the ability to explain the fine structure of speech in a pitch period. In the 1980s, Teager experimentally discovered that vortices could be the secondary source to excite the channel and produce the speech signal. Therefore, speech should be composed of the plane-wave-based linear part and the vortices-based nonlinear part [1]. According to such theory, Maragos et al. proposed an AM-FM modulation model for speech analysis, synthesis and coding. The AM-FM model represents the speech signal as the sum of formant resonance signals each of which contains amplitude and frequency modulation [2]. From the perception view, neurophysiological studies show that the auditory system of mammals is sensitive to FM-modulated (chirpy) sounds. Experiments in ferrets showed that the receptive fields found in primary auditory cortex have, as their counterparts in the visual cortex, Gabor-like shapes and respond to modulations in the time-frequency domain [3]. This fact underpins the notion of the high sensitivity of the human hearing system to nonstationary acoustic events with changing pitch (police and ambulance siren). In acoustic signal processing this effect is called auditory attention [4]. Recently, a number of works related to AM-FM modeling of speech as well as its applications to speech analysis and recognition recently have been reported [5–13].

A simple but very effective analysis tool is the spectrogram based on the short-time Fourier transform (STFT), which considers signals as short-time stationary signals. For sound signals, especially human speech signals, it gained very good results and thus has been very widely used, but a compromise of the window length has always to be made to satisfy the requirements of time and frequency resolution. To solve this problem, many time-frequency analysis methods have been introduced, such as the wavelet transform, the Wigner-Ville distribution, the Radon-Wigner transform, and the Fractional Fourier transform.

Fractional Fourier transform, as a new time-frequency analysis tool, is attracting more and more attention in signal processing literature. In 1980, Namias first introduced the mathematical definition of the FrFT [14]. Later Almeida analyzed the relationship between the FrFT and the Wigner-Ville Distribution (WVD) and interpreted it as a rotation operator in the time-frequency plane [15]. Since FrFT can be considered as a decomposition of the signal in terms of chirps, FrFT is especially suitable for the processing of chirp-like signals [16]. Several approaches to modeling speech or audio signals as chirp-like signals have been studied [17–19]. In [20], chirped autocorrelations and the fractional Fourier transform are used to estimate the features which can characterize a measured marine-mammal vocalization. In [21], sinewave analysis and synthesis is done based on the Fan-Chirp transform [22]. Because the Fan-Chirp transform can provide a set of FM-sinewave basis functions consistent with harmonic FM, the developed sinewave analysis/synthesis system can obtain more accurate sinewave frequencies and phases, thus creating more accurate frequency tracks than that derived from the short-time Fourier transform, especially for high-frequency regions of large-bandwidth analysis. The segmental signal-to-noise ratio with synthesis was also improved with that technique. There are also some papers on chirp-sensitive artificial auditory cortical model [23, 24]. For example, [23] uses a combination of several (at least three) Harmonic-Chirp transform instances which project the time-frequency energy on different views. The mechanism shows biological parallels such as intrinsic chirp sensitivity and response to the logarithm of the stimulus energy and was validated with several mammal sounds including human speech. Research on the application of FrFT or similar transforms to speech signal processing mainly focuses on speech analysis [23, 25–28], pitch estimation [4, 29], speech enhancement [30, 31], speech recognition [32], speaker recognition [33], and speech separation [34]. These methods basically can give higher time-frequency resolution than the traditional FFT-based method, a more accurate pitch estimate, and have shown to be beneficial for speech enhancement, speech recognition, speaker recognition, and monaural speech separation.

When applying the FrFT, the determination of the optimal FrFT transform order is a crucial issue. The order is a free parameter that is directly related with the chirp rate, that is, the temporal derivative of the instantaneous frequency of the FrFT basis function. There is still no effective way to calculate the order optimally, that is, in a way that the chirp rates of the basis functions and of the signal match. The step search method [16] is simple, but a compromise has to be made between the computational complexity and the accuracy. The method based on the location of minimum second-order moments of the signal's FrFT also has its limitations [35, 36]. The Wigner-Hough transform based method in [37] needs to calculate the integrations along all the lines in the time-frequency plane; so the computation time is rather extensive. In [16], a quasi-Newton method is used to simplify the peak detection in the fractional Fourier domains. In [38], the order is estimated by calculating the ambiguity function of the signal. This method decreases the computation time because it detects the chirp rate by only integrating along all the lines which pass through the origin.

All those existing order estimation methods were not proposed for speech signals; so they do not consider or take advantage of the special characteristics of speech. In this work, we show that the representation of the time-varying properties of speech may benefit from using the values of pitch and formants to set the order of the FrFT. Different order adaptation methods based on pitch and formants are investigated by using the FFT- and FrFT- based spectrograms of an artificially generated vowel. In order to compare the performance of these order adaptation methods, tone classification experiments are conducted on a small set of Mandarin vowels, where the classes correspond to the four basic types of tones. The discrimination ability is measured using acoustic features based on the combination of MFCC and FrFT for the different order adaptation methods. Finally, these methods are further assessed using speech recognition experiments which are conducted on intervocalic English consonants.

The rest of the paper is organized as follows. In Section 2, the AM-FM model of speech is described, and the motivation of the proposed method is given. In Section 3, the definition and some basic properties of the FrFT are briefly introduced. In Section 4, different order adaptation methods are described and illustrated using FFT- and FrFT- based spectrograms of an artificially generated vowel. In Section 5, a tonal vowel discrimination test is designed, and the results are given and analyzed. Section 6 presents the ASR experimental results and discussion. Conclusions and suggestions for future work are given in Section 7.

A speech production model generally contains three components: an excitation source, a vocal tract model and a radiation model. In speech processing, pitch is traditionally considered as constant within a frame, so for voiced speech, the excitation signal is produced by a periodic pulse generator. Practically, in particular for tonal languages, the pitch value is changing even within a frame. Considering the fluctuation of pitch and the harmonic structure, voiced speech can be modeled as an AM-FM signal. The AM-FM model proposed in [2] represents the speech signal as the sum of several formant resonance signals, each of which is an AM-FM signal. Herewith, we use an expression which tries to model the speech as a sum of the AM-FM harmonics:

(1)

where is the time-varying amplitude signal, is the fundamental (angular) frequency or pitch, is the initial phase, is the index of the harmonics, and is the frequency modulation function. Making the reasonable simplification that the frequency is changing linearly within the frame, that is,

(2)

where k is the chirp rate of the pitch (referred to as pitch rate in the rest of the paper), and its unit is . We can obtain:

(3)

The chirp rate of the nth harmonic is the second derivative of the phase function:

(4)

which means that the chirp rate of the th harmonic is n times the pitch rate.

The FrFT of signal is represented as [16]

(5)

where p is a real number which is called the order of the FrFT, is the transform angle, denotes the FrFT operator, and is the kernel of the FrFT:

(6)

The kernel has the following properties:

(7)

Hence, the inverse FrFT is

(8)

Equation (8) indicates that the signal can be interpreted as a decomposition to a basis formed by the orthonormal Linear Frequency Modulated (LFM) functions (6) in the domain. This means that an LFM signal with a chirp rate corresponding to the transform order can be transformed into an impulse in a certain fractional domain. For instance, it can be seen from the kernel function form in (6) that for the th harmonic with chirp rate (see (3) and (4)), when the transform angle satisfies the equation: , this harmonic can ideally be transformed into an impulse. Therefore, the FrFT has excellent localization performance for LFM signals.

Three types of order adaptation methods based on the pitch and formants have been investigated and will be demonstrated in this section by applying them to an artificially-generated vowel [i:] with time-varying pitch. The excitation of the vowel is a pulse train with linearly decreasing frequency from 450 Hz to 100 Hz, and the formants of the vowel are 384 Hz, 2800 Hz, and 3440 Hz, which are extracted from a real female vowel. The sampling rate is 8000 Hz, and the total duration of the signal is 1.5 secondes. Short-time analysis with FFT and FrFT was done with a Hamming window of length 640 samples, and a window shift of 20 samples(long duration windows were used to better visualize the methods. For the discrimination and recognition experiments, however, window lengths typical for speech processing were used).

4.1. Multiple of Pitch Rate

Since the chirp rates for different harmonics are different, the FrFT is emphasizing the th harmonic when setting the transform order according to times the pitch rate . The transform angle is then determined by

(9)

Take as an example. Figures 1 and 2 show the FFT-based and FrFT-based spectrograms of the vowel with and without inclusion of formants, respectively.

Figure 1 shows that the th harmonic and its neighbors will be emphasized by the FrFT analysis; that is, the line representing the th harmonic becomes thinner than in the FFT-based spectrogram. From Figures 1 and 2, it can also be seen that the representation of those harmonics whose chirp rates are not close to times the pitch rate will be smeared. This also holds true for the formants, because the frequency variations of the formants are generally smaller than those of the harmonics; that is, the chirp rates of the formants are generally much smaller than times the pitch rate when gets large, for example, .

4.2. Pitch and Formants

The subband energies that are usually employed to compute the speech recognition features, for example, in the widely used mel-frequency cepstral coefficients (MFCCs), are a representation of the envelope, that is, the formant structure, of voiced speech. Therefore, the aim of the mel-scale subband integration (and, additionally, the truncation of the sequence of cepstral coefficients in the MFCC representation) is to make the harmonic structure disappear in order to have a pitch-free envelope representation. Nevertheless, the FFT-based spectral harmonics are an intermediate step in the computation of the envelope, so a more precise representation of the harmonics in relevant regions of the spectral envelope may help to get more accurate formant estimates and also more discriminative speech features. This is the motivation for the order adaptation method based on pitch and formants that is introduced in the following.

As in (9), the transform angle is determined by times of the pitch rate :

(10)

M will be computed from the frequency of a formant and the pitch frequency as

(11)

Here, is different for different analysis frames if either pitch or formant frequency or both vary with time.

Figure 3 shows the FrFT- based spectrograms of the artificial vowel using the pitch as well as the first (app. 400 Hz Figure 3(a)) and the second formant (app. 2.8 kHz Figure 3(b)). We can see from Figure 3 that the spectral lines of harmonics are thinnest when going through the corresponding formants that were selected for the order determination. As the formants are smeared to certain extent, it needs to be investigated whether the better representation of the harmonics in the vicinity of the formants outweighs the smearing of the formants.

4.3. Multiorder Multiplication

Since different optimal orders are needed for different harmonics, we can calculate the FrFT with the orders corresponding to times of the pitch rate and multiply them together. Multiplication of the FrFT spectrograms is a somewhat heuristic approach and can be regarded as being similar to a logical "and" operation. By this, the transform with the order best suited for tracking a specific feature will "win" in the final representation. This method can obtain a compromise among several harmonics, that is, the harmonics selected in the formant regions will be enhanced and the smearing of the formants will be limited. Figure 4 shows the FrFT spectrogram using this method for a multiplication of FrFT orders from 1 to 10.

Finally, multiorder multiplication was applied to the three FrFT spectrograms that target the first three formants according to the technique described in Section 4.2. The resulting multiplied FrFT spectrogram is shown in Figure 5. In this case, formant smearing is limited, while still enhancing the harmonics going through the formant resonances.

In tonal languages as Mandarin, the time evolution of pitch inside a syllable (the tone) is relevant for the meaning. Consequently, there are relatively fast changes of pitch which are usual and informative. In Mandarin, there are four basic lexical tones and a neutral tone. The number of tonal syllables is about 1,300, and it is reduced to about 410 when tone discriminations are discarded [39]. Fundamental frequency or pitch is the major acoustic feature to distinguish the four basic tones.

Since the proposed FrFT order adaptation methods may show a more accurate representation of the time-varying characteristics of the harmonics than the Fourier transform, we tested their performance in a tonal vowel discrimination experiment.

5.1. Experiment Design

We recorded the five Mandarin vowels [a], [i](yi), [u](wu), [e], and [o](wo) with four tones: the flat tone (tone 1), the rising tone (tone 2), the falling and rising tone (tone 3), and the falling tone (tone 4). Each tone of each vowel from a female voice is recorded five times. The utterances are sampled at 8 kHz, with a 16-bit quantization. We use 16-dimensional standard MFCC features as the baseline. The features based on the FrFT are computed with the same processing used for the MFCCs, but substituting the Fourier transform by the FrFT (we will refer to them as FrFT-MFCC) [40]. The performance of FrFT-MFCC using different order adaptation methods is compared with the baseline. Speech signals are analyzed using a frame length of 25 milliseconds and a frame shift of 10 milliseconds.

Because the recorded utterances have variable lengths, we use Dynamic Time Warping (DTW) to calculate the distances between all the utterances for the individual vowels. Thus, five distance matrices are obtained (4 tones, 5 times). The discriminative ability of features can be analyzed using the Fisher score, which is defined as the ratio between the between-class variance and the within-class variance. Here, we take the distances calculated by DTW to compute a similar score (that will also be referred to as Fisher Score):

(12)

represents the token of a vowel with tone and are the total numbers of the between-class and within-class tokens, respectively. represents the Euclidean Distance after pattern matching using DTW. By this analysis, the discriminability across different tones of the same vowel is assessed. The discrimination among different vowels is also assessed here for comparison.

The specific novelty of this work is that we have proposed several order adaptation methods for FrFT in speech signal analysis and recognition which are based on the pitch and the formants (or just envelope peaks) of voiced speech. These order selection methods are specifically proposed according to the characteristics of speech, and their merits are indicated by FFT and FrFT based spectrograms of an artificially-generated vowel. The order selection methods are adopted in the calculation of the FrFT-MFCC features, and then used in a tonal vowel discrimination test and in a speech recognition experiment. It is shown that FrFT-MFCC features can greatly increase the discrimination of tones compared to FFT-based MFCC features and that the discrimination of Mandarin vowels and English intervocalic consonants is slightly increased.

It is well known that the FFT-based MFCC, and almost all other features conventionally used for speech recognition, discard pitch information and can not track the fast-changing events in speech signals. It can be assumed that this lack of short-term temporal information may lead to problems for the recognition of quickly changing events such as plosives. Moreover, formant transitions, a key aspect in the perception of speech, are also only covered indirectly by the MFCCs [3]. The assumption that the FrFT might better track temporal features is verified by the tonal vowel discrimination test and the speech recognition experiments, which show that considering the change of pitch and harmonics is not always harmful in the discriminability of speech features. However, it was also shown that the information on gross spectral peaks (formants) might be increasingly smeared when using high-resolution FrFT analysis.

On the other hand, the FrFT is a kind of linear transform and thus does not have the problem of cross-term interference known from other high-resolution transforms such as the Wigner-Ville distribution. Nevertheless, speech signals show a very complex spectrotemporal dynamics, and thus cannot be simply decomposed into several independent components using the FrFT or similar methods. When the analysis emphasizes one component in a certain fraction domain, it will bring dispersion effect to some other components, so a compromise has to be made when determining the transform order. In the proposed order selection methods, it seems that an optimal value always exists to achieve the compromise between tracking the dynamic characteristics of the speech harmonics and avoiding severe smearing of the formants.

The major difference between the proposed methods based on the FrFT and the related Fan-Chirp transform is that the Fan-chirp transform considers all harmonics at the same time, whereas our approach selects only a subset, for example, those who are close to the formants or a specific set of harmonic numbers. In both cases formants get smeared similarly. As to whether the proposed methods can lead to fundamentally different results from the Fan-Chirp transform is unclear, and this needs more experiments and detailed analysis. However, it could be beneficial to combine our multiorder methods with the Fan-Chirp transform.

From the experimental results, it seems that FrFT-MFCC might be better for vowels if using multiorder methods. However, smearing the formants in a single order method does not seem to be very harmful. Considering the effectiveness of the FFT analysis on formant determination and that of the FrFT analysis on emphasizing harmonics, one possible further approach might be to combine FFT- and FrFT-based MFCCs to get an improved representation of speech features for speech analysis and recognition.

Another interesting conclusion is that the proposed methods are not only useful for tonal language processing, but may also be useful for nontonal languages. It is reasonable that tonal languages can benefit from the FrFT-MFCC features with the proposed order adaptation methods, because different tone patterns have different pitch evolving trace, and the proposed method can track these dynamic characteristics of speech. For the toneless languages, the quickly changing events in speech can also benefit from this merit, leading to a somewhat improved speech recognition rate. Actually, the presented FrFT-MFCC features use the initial pitch estimates for all signal segments including consonants that do not have a prominent pitch. This fact might suggest that consonant recognition could degrade, because essentially invalid initial pitch estimates are used for some of the consonants. However, contrary to the expectation, in our test with English intervocalic consonants a slight, although not statistically significant, performance increase was measured. Overall, FrFT-MFCC seems to outperform the FFT-MFCC-baseline, because consonant recognition is not decreased and tone and vowel discriminability are increased.

The proposed order adaptation methods are based on initial short-time pitch, pitch rate and formant estimates which are derived directly from the signal. It's well-known that using the perfect a-priori knowledge of the pitch can improve the performance of tone discrimination and speech recognition [45]. However, the proposed methods can achieve better performance even with nonperfect pitch and formant estimation, which is shown by the fact that, although the improvement of the consonant recognition is not so statistically significant, it at least does not decrease despite using essentially invalid initial pitch estimates. This also indicates that the FrFT-MFCC features might degrade more gracefully with increasing noise level than the initial estimates of pitch and formants that are used to derive them. Furthermore, FrFT-MFCC is shown to improve tone and vowel discrimination compared to FFT-MFCC even though pitch harmonics are blended by the mel-frequency filterbank and formants are partially smeared as a consequence of the FrFT analysis. This forms one of the major outcomes of the study and indicates that the benefit of the FrFT-MFCC does not just result from the fact that initial pitch estimates are used. FrFT-MFCC features seem to provide a better combination of pitch and formant-related information than FFT-MFCC and this might be beneficial for ASR.

The clear disadvantage of the order selection methods determined by pitch and formants is that they rely greatly on the accuracy of pitch and formants determination, which is a tough problem in noisy environments. However, all model-based methods that consider AM-FM models have the problem that parameter adaptation is deteriorated by noise. In this paper, we wanted to show the potential of the proposed FrFT-based analysis method and demonstrate its benefit at high signal-to-noise ratios (SNRs). Since the results are encouraging, it is worth to look for noise-robust methods of deriving pitch and formants, and then to investigate the proposed methods in noisy conditions. This goes beyond the scope of this study and will be part of future work.