Deep neural networks for automatic speech processing: a survey from large corpora to limited data

Abstract

Most state-of-the-art speech systems use deep neural networks (DNNs). These systems require a large amount of data to be learned. Hence, training state-of-the-art frameworks on under-resourced speech challenges are difficult tasks. As an example, a challenge could be the limited amount of data to model impaired speech. Furthermore, acquiring more data and/or expertise is time-consuming and expensive. In this paper, we focus on the following speech processing tasks: automatic speech recognition, speaker identification, and emotion recognition. To assess the problem of limited data, we firstly investigate state-of-the-art automatic speech recognition systems, as this is the hardest task (due to the wide variability in each language). Next, we provide an overview of techniques and tasks requiring fewer data. In the last section, we investigate few-shot techniques by interpreting under-resourced speech as a few-shot problem. In that sense, we propose an overview of few-shot techniques and the possibility of using such techniques for the speech problems addressed in this survey. It is true that the reviewed techniques are not well adapted for large datasets. Nevertheless, some promising results from the literature encourage the usage of such techniques for speech processing.

1 Introduction

Automatic speech processing systems have improved dramatically over the past few years, especially automatic speech recognition (ASR) systems. This is also the case for other speech processing tasks such as speaker identification or emotion classification. This success was made possible by the large amount of annotated data available combined with the extensive use of deep learning techniques and the capacity of modern graphics processing units. Some modelings are already deployed for everyday use, such as personal assistants in smartphones and connected speakers. Nevertheless, challenges remain for automatic speech processing systems. They lack the robustness to deal with extensive vocabularies in a real-world environment: this includes noises, distance from the speaker, paucity of robustness to speech variations, reverberations, and other alterations [1]. Some challenges, such as CHiME [2], provide data to let the community try to handle some of these problems. Ways are sought to improve the generalization of modern models by avoiding the inclusion of additional annotated data for each possible environment.

State-of-the-art (SOTA) techniques for most speech tasks require large datasets. Indeed, with modern DNN speech processing systems, having more data usually implies better performance. The TED-LIUM 3 (from [3], with 452 h) provides more than twice the data of the TED-LIUM 2 dataset. The authors thus obtain better results by training their model on TED-LIUM 3 than training their model using TED-LIUM 2 data. This improvement in performance for ASR systems is also observed with the LibriSpeech dataset, from [4]. V. Panayotov et al. obtained better results on the Wall Street Journal (WSJ) test set by training a model using the LibriSpeech dataset (1000 h) than with the WSJ training set (82 h) [4].

This phenomenon, that having more data leads to better performance, is also observable in speaker recognition with the VoxCeleb 2 dataset compared to the VoxCeleb dataset [5]: the authors increased the number of sentences from 100,000 to one million and increased the number of individuals from 1251 to 6112 compared to the previous version of VoxCeleb. They thus obtained better performance than by training their model with the previous VoxCeleb dataset.

We summarized state-of-the-art approaches for automatic speech recognition in Tables 1 and 2, for emotion recognition in Table 3, and for speaker recognition in Table 4. From these tables, we can see that having more data does not always induce better results. Nevertheless, using more speech as pretraining of an unsupervised model (such as the libri-light [6] 60k hours) and bigger models (model having more parameters) helps to obtain some state-of-the-art results on unimpaired speech. Hence, scaling either the number of parameters or the amount of data can be useful when possible.

Table 1 SOTA results over test-clean set from LibriSpeech and quantity of data used. Some self-supervised results are provided from [7]

Table 2 SOTA results over test-other set from LibriSpeech and quantity of data used. Some self-supervised results come from [7] experiments

Table 3 SOTA results over IEMOCAP using 4 emotions (happiness, neutral, anger, and sadness) and quantity of data used. Self-supervised results come from [7] experiments

Table 4 SOTA results over VoxCeleb1 and quantity of data used. Self-supervised results come from [7] experiments

With under-resourced languages (such as [18]) and/or some tasks (pathology detection with speech signals), we lack large datasets [19]. By under-resourced, we mean limited digital resources (limited acoustic and text corpora) and/or a lack of linguistic expertise. For a more precise definition and details of the problem, see [20]. Some non-conventional speech tasks such as disease detection (such as Parkinsons, gravity of head and neck cancer and others) using audio are examples of under-resourced tasks [19]. Training deep neural network models in such contexts is a challenge for these under-resourced speech datasets. This is especially the case for tasks involving a large vocabulary. M. Moore et al. showed that recent ASR systems are not well adapted for impaired speech [21], and M. B. Mustafa et al. showed the difficulties in adapting such models with limited amounts of data [22].

Few-shot learning consists of training a model using k-shot (where shot means an example per class), where k≥1 and k is a low number. Training an ASR system on a new language, adapting an ASR system on pathological speech, or performing speaker identification (with impaired voice) with few examples are still complicated tasks [19, 21]. We think that few-shot techniques may be useful for tackling these problems.

This survey will be focused on how to train deep neural network (DNN) models with low resources for speech data with non-overlapping mono signals. Therefore, we will first review SOTA ASR techniques that use a large amount of data (Section 2). Then, we will review techniques and speech tasks (speaker identification, emotion recognition) requiring fewer data than SOTA techniques (Section 3). We will also look into pathological speech processing for ASR using adaptation techniques (Section 3.2). Finally, we will review few-shot techniques for audio (Section 4) which is the focus of this survey.

2 Automatic speech processing

In this section, we will review SOTA ASR, speaker identification, and emotion recognition systems using multi-models and end-to-end models. Here, we are focused on mono speech sequences x=[x₁,x₂,…,x_n] where x_i can be speech features or audio samples. ASR systems consist in matching x into a sequence of words y=[y₁,y₂,…,y_u] (where u≤n), while speaker and emotion recognition systems map the sequence x into a single outcome y representing a class. The reviewed systems were evaluated using word error rate (WER) as a measure for ASR systems and the accuracy of recognition for speaker identification and emotion recognition.

2.1 Multi-models

A multi-model approach consists in solving a problem using multiple models. These models are designed to solve sub-tasks (related to the problem) and the targeted task. The minimum configuration is with two models (let us say f and g) to solve a given task. Classically for the ASR task, we can first train an acoustic model (a phoneme classifier or equivalent sound unit) then train on top of it a language model that output the desired sequence of words. Hence, we have:

$$\hat{\mathbf{y}} = f(g(\mathbf{x})),$$

(1)

with f being the language model predicting the output sequence $\hat {\mathbf {y}}$ (which can be different from the real sequence y) and g being the acoustic model. Note that for emotion and speaker recognition, the output of f is $\hat {y}$ instead of $\hat {\mathbf {y}}$ and is not necessarily a language model. Both can be trained separately or conjointly. Usually, hybrid models are used as acoustic models.

Hybrid models consist in using probabilistic models with deterministic ones. Probabilistic models involve randomness using random variables combined with trained parameters. Hence, every prediction is slightly different on a given example x. Gaussian mixture models (GMMs) are an example of such models. Deterministic models do not involve randomness and every prediction is the same, given an input x. DNNs are an example of such models. A popular and efficient hybrid model is the DNN-hidden Markov model (DNN-HMM). DNN-HMM consists in replacing the GMMs that estimate the probability density functions by DNNs. The DNNs can be trained as phoneme classifiers. They form the acoustic model. This acoustic model is combined with a language model (LM) that maps the phonemes into a sequence of words. Lüscher et al. used DNN-HMMs combined with a LM to obtain SOTA on the LibriSpeech test-other set (official augmented test set) [11]. This model processes mel frequency cepstral coefficients (MFCCs), which are computed on the audio signals. Their best LM approach consisted in the use of transformer from [23]. Transformers are autoregressive models (depending on the previous outputs of the models) using soft attention mechanisms. Soft attention consists in determining a glimpse g, which is a selection of characteristics from the input x that help to filter non-useful information. Their best hybrid model got a WER of 5.7% for the test-other set and 2.7% for test-clean set.

Hybrid model is also used in emotion recognition and obtains SOTA on IEMOCAP using context-dependent domain adversarial neural network (DAAN) [16]. The IEMOCAP database [24] here was modified to obtain in a four-class emotion problem. Those emotions are angry, happy, neutral, and sad.

DAAN consists in using a lexical model (pretrained over one billion words) and audio features (such as MFCC and energy) that represent 6373 features for each input frame. They fuse these inputs using attention. Then, multiple GRU layers are used. Note that their approach use two tasks to be learned (emotion recognition and domain recognition). Such approach requires multiple labels, which IEMOCAP database provides. Doing so, they achieved an accuracy of 82.7% over all the emotions.

2.2 End-to-end systems

In end-to-end approaches, the goal is to determine a model f that can do the mapping:

$$\hat{\mathbf{y}} = f(\mathbf{x}).$$

(2)

It will be trained to directly map x to $\hat {\mathbf {y}}$ (which can be different from the real sequence y) with a single learnable function. Only supervised methods can function end-to-end to solve the speech tasks we are focused on.

In ASR systems, Kim et al. got SOTA on LibriSpeech test-clean official set [12]. Compared to [11], they used vocal tract length perturbation as the input of their end-to-end model. The model is based on the encoder-decoder architecture using stacked long short-term memory (LSTM) for the encoder and LSTM combined with soft attention for the decoder [12]. They obtained a WER of 2.44% on test-clean set and 8.29% on test-other set. Those results are close to [11] (best hybrid model results) and show that end-to-end approaches are competitive compared to multi-model approaches.

In emotion recognition, Lian et al. got SOTA results with a the same modified version of the IEMOCAP database [24] as other methods mentioned for [16]. They used an end-to-end multi-task system with only supervised tasks: gender identification and emotion identification [15]. The resulting model achieved an overall accuracy for the emotion task (which is the main target) of 81.6% and an average accuracy of each emotion category of 82.8%. Using such an approach allowed them to achieve balanced results from unbalanced data. Furthermore, their results are similar to hybrid SOTA approach from [16]. Nevertheless, using only supervised tasks requires multiple ground truths for the targeted dataset.

In speaker identification, autoSpeech [17] architecture is SOTA over the VoxCeleb1 dataset [25]. Their approach consists in an algorithm that automatically search the best convolutional neural network architecture to solve the task. With their approach, they obtained an accuracy of 87.66%.

3 Techniques requiring fewer data

Some techniques require fewer data than the techniques of the previous section. In this section, we will enumerate the principal ways to leverage (to our best knowledge) the lack of large datasets such as impaired speech. We will not discuss semi-supervised techniques that use a large amount of unsupervised data.

3.1 Data augmentation

The first way to leverage the lack of data is to artificially augment the number of samples. To do so, the classic approach consists in adding noise or deformation, as in [26]. The authors obtain near SOTA results on LibriSpeech (1000 hours from [4]) with an end-to-end model. Nevertheless, they obtain SOTA results on SwitchBoard (300 h from [27]) with a WER of 6.8% on Switchboard and 14.1% on the CallHome portion using shallow fusion and their data augmentation. But these are handcrafted augmentations, and some of them require additional audio (such as adding noise).

Some other approaches use generative models to have new samples, as in [28, 29]. Chatziagapi et al. used conditional generative adversarial networks (GANs) to generate new samples [28]. With conditional GANs, we can control the mode of the generated samples [30]. By doing so, they balanced their initial dataset and obtained better results. Jiao et al. used deep convolutional GANs to generate dysarthric speech and improved their results [29].

3.2 Domain transposition

Another way to leverage the lack of data is to use data domain transposition. The idea consists of mapping speech features from one domain (such as spectrogram containing speech and noises) to another domain to reduce the data complexity (such as spectrogram with only speech). In machine learning, data complexity is defined by the size of the model needed to replicate the data, the size of the shortest encoding possible (that let the reconstruction of the initial data), and the error rate of the best model possible given a task [31]. Here are some recent examples on speech:

• Wang et al. used GAN to dereverberate speech signals [32]. In their work, the generator is used as a mapping function for converting reverberated signals into dereverberated speech signals.

• Chen et al. performed vocal conversion using GAN with a controller mapping impaired speech to a representation space z [33]. z is the input of the generator that is used as a mapping function to have unimpaired speech signals.

• Zhao et al. used Cycle GAN (framework designed for domain transfer) as an audio enhancer [34]. Their resulting model is SOTA on the CHiME-4 dataset.

3.3 Models requiring fewer parameters

Having fewer data means that overfitting can occur if neural network models require too many parameters. This is why some experimental techniques tried models requiring fewer parameters. Here, we highlight some recent techniques that we find interesting:

• The use of SincNet layers, from [35], to replace classic 1D convolutions over raw audio. Here, instead of requiring window_size parameters (with window_size being the window size of the 1D convolution) per filter, we only need two parameters per filter for every window size. These two parameters indirectly represent the values of the bandwidth at high and low energy.

• The use of LightGRU (LiGRU), from [36], based on the gated recurrent unit (GRU) framework. LiGRU is a simplification of the GRU framework given some assumptions concerning the speech signal. They removed the reset gate of the GRU and used the ReLU activation function (combined with Batch Normalization [37]) instead of the tanh activation function.

• The use of quaternion neural networks, from [38], for speech processing. The quaternion formulation allows 4 dimensions to be fused into one inducing a drastic reduction of required parameters in their experiments (almost 4 times fewer).

3.4 Multi-task approach

Multi-task models can be viewed as an extension of the Encoder-Decoder architecture where you have a decoder per task with a shared encoder (as in Fig. 1). These tasks are then trained conjointly with classic feed-forward algorithms. The goal of multi-task learning is to obtain an encoder that outputs sufficient information for each task. It can thus potentially improve the performance of each task compared to mono-task architectures. It is a way to have a more representative encoder given the same amount of data.

Fig. 1

Illustration of multi-task architecture. The output of the encoder is given to each decoder to produce the prediction for each t_i task

Pascual et al. used a combination of self-supervised tasks to tackle the lack of ground truth and used the resulting encoder for transfer learning [39]. They recently improved this work in [8] where they use more tasks, a recurrent unit on top of the encoder and denoising mechanisms using multiple data augmentation techniques on their system.

3.5 Transfer learning

Transfer learning techniques consist in using a pre-trained model and using it as a feature extractor or using its parameters as initializers (that can be fine-tuned) to solve a related problem/task.

Contrastive predictive coding (CPC from [40]) is a framework for training self-supervised audio representation using a 2-level architecture combined with a self-supervised loss. The authors achieved better results by transferring the obtained parameters onto a speaker identification task and a phoneme classification task (on the LibriSpeech dataset) than with the use of MFCC features.

Some binary tasks of the multi-task model from Pascual et al. use predictive coding like in CPC. The latter developed an unsupervised multi-task model to obtain better encoders for transfer learning [39]. They applied it on multiple tasks and obtained acceptable results on speaker identification (using VTCK [41]), emotion recognition (using INTERFACE [42]), and ASR (using TIMIT [43]).

Nowadays, CPC approaches such as Wav2Vec [44] (last version Wav2Vec2.0 [9]) and HuBERT [10] represent SOTA self-supervised techniques. Such techniques are able to pretrain models over 60k hours of unlabeled speech such as libri-light [6]. Wav2Vec and Wav2Vec2.0 consist in learning representations from the waveform using the CPC framework. Wav2Vec2.0 combines convolutional layers to process the waveform and obtain a low latent representation, which is fed to transformer layers to obtain a contextual representation. HuBERT is using a BERT encoder (which is a transformer) inside the CPC framework and penalizes the loss function using the BERT loss [45]. These approaches have been compared in [7] benchmarks (among others). For speaker identification, the best model obtained an accuracy of 90.33% over the VoxCeleb test set [46]. Then, for emotion recognition, the best model obtained an accuracy of 67.62% over the IEMOCAP [47] test set. And for ASR, the best model obtained a word error rate of 2.94% over librispeech [4] test-clean. A more detailed overview (with a complete benchmark across several speech tasks) of self-supervised approaches is available in [7]. Note that this benchmark does not include pathological speech data or related tasks.

The benefits of pre-trained networks for transfer learning decrease as the target task diverges from the original task of the pre-trained network [48]. To tackle this, Van den Oord et al. attempted to have generic tasks with their unsupervised approach, and they obtained promising results [40]. The benefits of transfer learning also decrease when the dissimilarity between the datasets increases [48]. This problem can discourage the use of transfer learning for some pathological speech. However, dysarthric and accented speech seem similar to speech in the LibriSpeech dataset, according to [49]. Shor et al. successfully used transfer learning to improve their results with a 36.7-h dataset.

Nevertheless, Mustapha et al. showed that the acoustic characteristics of unimpaired and impaired speech are very different [22]. Where few data are available, fine-tuning generic representations from unimpaired speech to impaired speech can be critical. Furthermore, learning from scratch could be a hard task. This is why looking into few-shot techniques could be helpful as a replacement or combined with generic pre-training.

4 Few-shot learning and speech

In the previous sections, we reviewed models that require a large amount of data to learn directly a task or to train more generic representations (to allow tackling some tasks with fewer data). But there is not always enough data to train a model from generic representations (by using them or adapting them); this is the case for pathological speech [19]. Google is trying to acquire more data of that nature¹. But acquiring such data can be expensive and time consuming. Mustafa et al. recommend the use of adaptive techniques to tackle the limited amount of data problem in such case [22]. We think few-shot techniques can be another solution to this problem. Nevertheless, some non-common tasks such as pathological speech or dialect identification with few examples are still hard to train with SOTA techniques based on large speech datasets. This is why we investigate the following few-shot techniques and see the adaptations required for using them on speech datasets.

4.1 Few-shot notations

Let us consider a distribution P from which we draw independent identically distributed (iid) episodes ($\mathcal {E}$ or datasets). $\mathcal {E}$ is composed of a support set $\mathcal {S}$, unlabeled data $\bar {\mathbf {x}}$ and a query set $\mathcal {Q}$. The support set corresponds to the supervised samples to which the model has access:

$$\mathcal{S} = \{(x_{1}, y_{1}), \ldots (x_{s}, y_{s}) \},$$

(3)

with x_i being samples and y_i being the corresponding labels, such as y_i∈{1,2,…,K} and K being the number of classes appearing in P. The query set is composed of samples to classify $\hat {\mathbf {x}}$ with $\hat {\mathbf {y}}$ being the corresponding ground truth.

To summarize, episodes drawn from P have the following form:

$$\begin{aligned} \mathcal{E} = \{ & \mathcal{S} = \{(x_{1}, y_{1}), \ldots, (x_{s}, y_{s}) \},\\ & \bar{\mathbf{x}} = (\bar{x}_{1}, \ldots, \bar{x}_{r}),\\ & \mathcal{Q} = \{(\hat{x}_{1}, \hat{y}_{1}), \ldots, (\hat{x}_{t}, \hat{y}_{t})\} \end{aligned}$$

(4)

with s, r, and t fixed values that respectively represent the number of supervised samples for the support set, the number of unsupervised samples, and the number of supervised samples for the query set.

In this survey, we will focus on few-shot learning techniques where r=0, t≥1 and s=kn, with n being the number of times each label appears for the support set and k the number of classes selected from P, such as k≤K. Hence, we have an n-shot with k-ways (or classes) for each episode. One-shot learning is just a special case of few-shot learning where n=1. In some few-shot frameworks, we only sample one episode from P and it represents our task.

4.2 Few-shot learning techniques

In this section, we will review frameworks that impacted the few-shot learning field in image processing, frameworks with a formulation that seems suitable for speech processing, and frameworks already successfully used by the speech community.

4.2.1 Siamese technique

Siamese neural networks are designed to be used per episode [50]. They involve measuring the distance between two samples and judging whether or not they are similar. Hence, a siamese network uses the samples from the support set $\mathcal {S}$ as references for each class. It is then trained using all the combinations of samples from $\mathcal {S} \bigcup \mathcal {Q}$ which provides much more training than having only s+t samples in classical feedforward frameworks. Siamese Networks take two samples (x₁ and x₂) as input and compute a distance between them, as follows:

$$\phi(x_{1}, x_{2}) = \sigma(\sum\boldsymbol{\alpha}|Enc(x_{1}) - Enc(x_{2})|),$$

(5)

with Enc being a DNN encoder that represents the signal input, σ being the sigmoid function, α learnable parameters that weight the importance of each component of the encoder, and x₁ and x₂ sampled from either the support set or the queries set.

To define the class of a new sample from $\mathcal {Q}$ or any new data, we must compute the distance between each reference from $\mathcal {S}$ and the new sample. An example of comparison between a reference and a new example is shown in Fig. 2. The class of the reference with the lowest distance then becomes the prediction of the model. To train such a model, [50] used this loss function:

$$\mathcal{L} = \mathbb{E}_{y(x_{i}) = y(\tilde{x}_{j})} \log(\phi(x_{i}, \tilde{x}_{j})) + \mathbb{E}_{y(x_{i}) \neq y(\tilde{x}_{j})} \log(1 - \phi(x_{i}, \tilde{x}_{j}))$$

with $\tilde {\mathbf {x}} = [x_{1}, \ldots, x_{s}, \hat {x}_{1}, \ldots, \hat {x}_{t}]$ from $\mathcal {S}$ and $\mathcal {Q}$. y(x) is a function that returns the label corresponding to the example x. The last layer of ϕ should be a softmax function.

Fig. 2

Example of comparison between a reference (x_i) and a new example ($\hat {x}_{j}$) from the query set, where Enc is the same network applied to both x_i and $\hat {x}_{j}$. The model outputs the distance between the classes x_i and $\hat {x}_{j}$

Eloff et al. used a modified version of this framework for multimodal learning with the modalities being speech and image signals [51], but to our knowledge, there is no study yet concerning just speech processing. The speech signals used consist of 11-digit numbers (zero to nine and the ‘‘oh’’ used in phone numbers) with the corresponding 10 images (‘‘oh’’ and zero give the same images). The problem is to associate speech signals with the corresponding images. In their experiment, the model shows some invariance to speakers (accuracy of 70.12% ±0.68) using only a one-shot configuration, which is a promising result.

Siamese neural networks are not very suitable when the number of classes K or the number of shots q become too high. It increases the number of references to be compared and the computation time to forward the model. The primary problem concerns training the model. Once the model has been trained, we can reduce this effect by pre-calculating all encodings of the examples of the support set. This also dramatically increases the number of combinations for the training phase, which can be viewed as a positive point. This framework does not seem appropriate for end-to-end ASR with large vocabularies, such as in English (around 470,000 words), though it may be sufficient for languages such as Esperanto (around 16,780 words). The other way to use such a framework in ASR systems is to use it in hybrid models as an acoustic model, where we can train it on every phoneme (for example 44 phonemes/sounds in English) or more refined sound units.

The siamese framework seems interesting for tasks such as speaker identification, as a new speaker can be added without retraining the model (supposing the model had generalized) or changing the architecture of the model. We only have to add at least one sample of the new speaker to the references. Furthermore, the siamese formulation seems well adapted for speaker verification. We only need to replace the pair (x,speaker_id) by the pair $(\mathbf {x}, \mathcal {S}_{top5})$, where $\mathcal {S}_{top5}$ is a support set composed of signals from the top 5 predictions of the identification sub-task.

Nevertheless, this framework will be of limited use if the number of speakers to identify become too high. Even so, it is possible to use such techniques in an end-to-end ASR system when the vocabulary is limited, such as in the experiment described in [51]. Also, this framework was used in emotion recognition [52]. In their experiments, they used their approach over the IEMOCAP [47] using a 3-way task (which is different from all other papers reviewed in this work that use 4 classes). Nevertheless, they managed to obtain an unweighted average recall of 67.4% using a 10-shot configuration, which is an encouraging result.

4.2.2 Matching network

The matching networks system described in [53] is a few-shot framework designed to be trained with a set of multiple episodes (with typically 5-ways to 25-ways), which consists of a single model φ. This model evaluates new examples given the support set $\mathcal {S}$ as in the siamese framework:

$$\varphi(\hat{x}, \mathcal{S}) :\rightarrow \hat{y}.$$

(6)

In matching learning, φ is as follows:

$$\varphi(\hat{x}, \mathcal{S}) = \sum\limits_{(x_{i}, y_{i}) \in \mathcal{S}} a(\hat{x}, x_{i}) y_{i},$$

(7)

with, a being the attention kernel.

In [53], this attention kernel is as follows:

$$a(\hat{x}, x_{i}) = \text{softmax}(c(f(\hat{x}), g(x_{i}))),$$

(8)

where c is the cosine distance, and f and g are embedding functions.

Vinyals et al. used a recurrent architecture to modulate the representation of f using the support set S [53]. The goal is to have f following the same type of representation as g. To do this, the g function is as follows:

$$g(x_{i}) = \overrightarrow{h_{i}} + \overleftarrow{h_{i}} + g'(x_{i}),$$

(9)

where $\overrightarrow {h_{i}}$ and $\overleftarrow {h_{i}}$ represent a bi-LSTM output over g^′(x_i), which is a DNN.

The f function is as follows:

$$f(\hat{x}) = attLSTM(f'(\hat{x}), g(S), m),$$

(10)

with attLSTM being an LSTM requiring a fixed number of recurrences (here m) and g(S) representing the application of g to each x_i from the S set. f^′ is a DNN with the same architecture as g^′, but not necessarily sharing the parameter values.

Training this framework therefore consists in the maximization of the log likelihood of φ given the parameters of g and f.

Figure 3 illustrates forward time of the matching network model. For forward time on new samples, $g(\mathcal {S})$ can be pre-calculated to gain computation time. Nevertheless, matching networks have the same disadvantages as siamese networks when q and/or K become too high. Furthermore, adding new classes to a trained matching network model is not as easy as for siamese network models. As this requires retraining the matching network model to add an element to the support set. Despite these disadvantages, matching learning showed better results than the siamese framework on image datasets [53]. This is why it should be investigated in speech processing to see if it is still the case.

Fig. 3

Illustration of the matching network model to predict the class of a new example $\hat {x}_{i}$

4.2.3 Prototypical networks

Prototypical networks [54] are designed to work with multiple episodes. In the prototypical framework, the model φ makes its predictions given the support set $\mathcal {S}$ of an episode such as the previously seen frameworks. This framework uses training episodes as mini-batches to obtain the final model. This model is formulated as follows:

$$\varphi(\hat{x}, S) = {softmax}_{k}(-d(f(\hat{x}), \mathbf{c}_{k})),$$

(11)

where c_k is the prototype of the class k, d being a Bregman divergence (for their useful properties in optimization, see [54] for more details), which also has the following property: Rⁿ×Rⁿ→[0,+ inf[.

Snell et al. used the Euclidean distance for d instead of the cosine distance used in meta learning and matching learning papers [54]. As a result, they obtain better results in their experiments. Next, they go further by reducing the Euclidean distance to a linear function.

In the prototypical framework, there is only one prototype for each class k as illustrated in Fig. 4. It is computed as follows:

$$\mathbf{c}_{k} = \frac{1}{|\mathcal{S}_{k}|} \sum\limits_{(x_{i}, y_{i}) \in \mathcal{S}_{k}}f(x_{i}),$$

(12)

Fig. 4

Illustration of the prototypical network model to predict class of a new example $\hat {x}_{i}$

with f being a mapping function such as $\mathbb {R}^{D} \rightarrow \mathbb {R}^{M}$ and $\mathcal {S}_{k}$ being the samples with k of the support set.

Prototypical networks require only one comparison per class and not q per class for q-shot learning as in siamese and matching learning networks. That is why this framework is less subject to the high computation problem for prediction of new samples, as it is only influenced by high K. It will certainly be insufficient for end-to-end ASR systems on the English language due to the large vocabulary issues described in Section 4.2.1, but it is a step towards it.

In speaker recognition, prototypical networks were used over a portion of Voxceleb1 [25] by [55]. They obtained under 20-ways and using 5-shot an accuracy of 72.77 which are promising results.

4.2.4 Meta-learning

Meta-learning systems [56] are designed to be trained over multiple episodes (also called datasets). In this framework, a trainee model ($\mathcal {T}$) with parameters $\theta ^{\mathcal {T}}$ trained from the start of every episode, usually has a classic DNN architecture. The support set and the query set in the episodes are considered as the training set and the test set for the trainee model.

Along with this trainee model, a second model is trained: the meta model ($\mathcal {M}$) with $\theta ^{\mathcal {M}}$ parameters. This meta model is the key of meta learning, it consists in monitoring the trainee model by updating $\theta ^{\mathcal {T}}$ parameters. To train this meta model, Ravi et al. suggests sampling iid episodes from P to form the meta-dataset ($\mathcal {D}$) [56]. This meta-dataset is composed of a training set ($\mathcal {D}_{train}$), a validation set ($\mathcal {D}_{valid}$), and a testing set ($\mathcal {D}_{test}$).

While the trainee model is training on an episode $\mathcal {E}_{j}$, the meta model is used to update its parameters:

$$\theta^{\mathcal{T}_{j}}_{t} = \mathcal{M}(\theta^{\mathcal{T}_{j}}_{t-1}, \mathcal{L}^{\mathcal{T}_{j}}, \nabla_{\theta^{\mathcal{T}_{j}}_{t-1}}\mathcal{L}^{\mathcal{T}_{j}}),$$

(13)

with $\mathcal {L}^{\mathcal {T}_{j}}$ being the loss function of the trainee model learned with the episode $\mathcal {E}_{j}$ and $\theta _{t-1}^{\mathcal {T}_{j}}$ being the parameters of the trainee model at step t−1. Also, $\mathcal {M}$ has to guess initial weights of the trainee models at step t=0 ($\theta ^{\mathcal {T}_{j}}_{0}$).

The learning curve (loss) of the trainee model with $\mathcal {E}_{j}$ is viewed in [56] as a sequence that can be the input of the meta model $\mathcal {M}$. For simplicity, we will use the notation of $\mathcal {T}$ instead of $\mathcal {T}_{j}$ for the next few paragraphs. Figure 5 illustrates the learning steps of the trainee using the meta model.

Fig. 5

Illustration of Meta-Learning for training with episode $\mathcal {E}_{j}$ at step t. Here the Meta model $\mathcal {M}$ processes the different training steps of the trainee $\mathcal {T}$ as a sequence

4.2.4.1 Trainee parameters update

Ravi and Larochelle identify the learning process of $\mathcal {T}$ using classic feedforward update on the episode E_j to be similar to the c_t update gate of the LSTM framework [56]. In the meta learning framework, c_t is used as the $\theta ^{\mathcal {T}}_{t}$ estimator, as follows:

$$\theta_{t}^{\mathcal{T}} = f_{t} \odot \theta_{t-1}^{\mathcal{T}} + i_{t} \odot \tilde{\theta_{t}^{\mathcal{T}}},$$

(14)

with $\tilde {\theta _{t}^{\mathcal {T}}} = -\alpha _{t} \nabla _{\theta _{t-1}^{\mathcal {T}}} \mathcal {L}_{t}^{\mathcal {T}}$ being the update term of the parameters $\theta _{t-1}^{\mathcal {T}}$, f_t being the forget gate, and i_t the update gate.

4.2.4.2 Parameters of the meta model

Both i_t and f_t are part of the meta learner. In the meta-learning framework, the update gate is formulated as follows:

$$i_{t} = \sigma(\mathbf{W}_{I}.[\nabla_{\theta_{t-1}^{\mathcal{T}}}\mathcal{L}_{t}^{\mathcal{T}}, \mathcal{L}_{t}^{\mathcal{T}}, \theta_{t-1}^{\mathcal{T}}, i_{t-1}] + \mathbf{b}_{I}),$$

(15)

with W_I and b_I being parameters of $\mathcal {M}$. The update gate is used to control the update term in equation 14, like the learning rate in the classic feedforward approach.

Next, the forget gate in the meta-learning framework is formulated as follows:

$$f_{t} = \sigma(\mathbf{W}_{F}.[\nabla_{\theta_{t-1}^{\mathcal{T}}}\mathcal{L}_{t}^{\mathcal{T}}, \mathcal{L}_{t}^{\mathcal{T}}, \theta_{t-1}^{\mathcal{T}}, f_{t-1}] + \mathbf{b}_{F}),$$

(16)

with W_F and b_F parameters of $\mathcal {M}$.

This gate is here to decide whether the training of the trainee should restart or not. This can be useful to avoid the problem of a sub-optimal local minimum. Note that this gate is not present in classic feedforward approaches (where this gate is equal to one).

The trainee model ($\mathcal {T}$) of this framework can be any kind of model, such as a siamese neural network. It can therefore have the advantages of this framework. It can also avoid the disadvantages of the siamese neural network, as it can use any other framework (usually classic DNN). This framework is interesting to training efficient models for speech processing (in terms of learning speed) when we have multiple ASR tasks with different vocabularies. For example, suppose we have the following kinds of speech episodes: dialing numbers, commands to a robot A, and commands to a robot B. The model can initialize good filters for the first layers (as this still involves speech processing). Another example could be training acoustic models for multiple languages (with each episode corresponding to a language).

4.2.5 Graph neural network

Graph neural networks (GNNs) are used by Garcia and Bruna to introduce their few-shot framework [57]. This framework is designed to be used with multiple episodes, they called tasks. In this framework, one model is used over a complete graph G: G=(V,E) and every node corresponds to an example. For few-shot learning, a GNN consists in applying graph convolution layers over the graph G.

Initial vertices to guess the ground truth of a query $\tilde {x_{i}}$ from the query set $\mathcal {Q}$ are constructed as follows:

$$\begin{aligned} V^{(0)} = (& (Enc(x_{1}), h(y_{1})), \ldots, (Enc(x_{s}), h(y_{s})), \\ & (Enc(\bar{x_{1}}), u), \ldots, (Enc(\bar{x_{r}}), u)\\ & (Enc(\tilde{x_{i}}), u)) \end{aligned},$$

(17)

where Enc is an embedding extraction function (a neural network or any classic feature extraction technique), h the one-hot encoding function, and u=K⁻¹1_K a uniform distribution for examples with unknown labels (the unsupervised ones from $\bar {\mathbf {x}}$ and/or from the query set $\mathcal {Q}$).

The vertices at each layer l (with 0 being the initial vertices) will henceforth be denoted:

$$V^{(l)} = (v_{1}, \ldots, v_{n}),$$

(18)

where n=s+r+1 and $V^{(l)} \in \mathbb {R}^{n*d_{l}}$.

Every layer (with an illustration of a layer in Fig. 6) in a GNN is computed as follows:

$$V^{(l+1)} = Gc(V^{(l)}, A^{(l)}),$$

(19)

Fig. 6

Illustration of the input of the first layer (here a graph convolution) of a GNN. Here, we have three samples (represented by vertices v_i, v_j and v_k) in the support set and one query (represented by the vertex v_u)

with A^(l) being the adjacency operators constructed from V^(l) and Gc being the graph convolution.

4.2.5.1 Construction of the adjacency operators

The adjacency operator uses a set:

$$A^{(l)} = \{\tilde{A}^{(l)}, \mathbf{1}\},$$

(20)

with $\tilde {A}^{(l)}$ being the adjacency matrix of V^(l).

For every (i,j)∈E (remember that we have complete graphs), we compute the values of the adjacency matrix as follows:

$$\tilde{A}^{(l)}_{i, j} = \phi(v_{i}^{(l)}, v_{j}^{(l)}),$$

(21)

where:

$$\phi(v_{i}^{(l)}, v_{j}^{(l)}) = f(|v_{i}^{(l)} - v_{j}^{(l)}|),$$

(22)

with f being a multi-layer perceptron with its parameters denoted θ_f. $\tilde {A}^{(l)}$ is then normalized using the softmax function over each line.

4.2.5.2 Graph convolution

The graph convolution requires the construction of the adjacency operators set and is computed as follows:

$$Gc(V^{(l)}, A^{(l)}) = \rho(\sum\limits_{B \in A} BV^{(l)} \theta^{(k)}_{B, l}),$$

(23)

with B being an adjacency operator from A, $\theta _{B, l}^{(k)} \in \mathbb {R}^{d_{l-1}, d_{l}}$ learnable parameters and ρ being a pointwise linearity (usually leaky ReLU).

4.2.5.3 Training the model

The output of the resulting GNN model is a mapping of the vertices to a K-simplex that gives the probability of $\tilde {x_{i}}$ being in class k. V. Garcia and J. Bruna used the cross-entropy to train the model using all other samples in the query set $\mathcal {Q}$ [57]. Hence, the GNN few-shot framework consists in learning θ_f and θ_1,l…θ_card(A),l parameters with all episodes.

4.2.5.4 Few-shot GNN on audio

This framework was used by [58] on 5-way audio classification problems. The 5-way episodes are randomly selected from the initial dataset: AudioSet [59] for creating the 5-ways training episodes and TV program (from [60]) data to create the 5-ways test episodes.

Zhang et al. compare the use of per class (or intra-class) attention and global attention, which gave the best results [58]. They applied it for each layer. Their experiments were performed for 1-shot, 5-shots, and 10-shots with the respective accuracy of 69.4% ± 0.66, 78.3% ± 0.46, and 83.6% ± 0.98. Such results are an encouragement for the use of few-shot learning for speech signals. Nevertheless, this framework does not allow the use of multiple classes and shots per episode, which increase the number of nodes and thus the computations in forward time. Hence, it is not suitable for large vocabulary problems.

5 Preliminary results on phoneme recognition

Following this review of few-shot techniques, we implemented our first approach of a few-shot solution for phoneme recognition on TIMIT. We did not try word recognition due to the large vocabulary issues described in Section 4.2.1. To select our architecture, we reused the five first layers of the PyTorch-Kaldi model using MFCC [61]. We use this architecture as an encoder for the siamese framework and the prototypical framework, as they require the same type of architecture. To evaluate our results, we used the phone accuracy (where PER=1−phone accuracy) to match the usual metric used in few-shot learning. Note that this is only a first attempt; more complete experiments (including all frameworks, speaker recognition, and emotion recognition) will be done in our future work. Nevertheless, our initial results may help others to avoid some difficulties.

In our experiments, we encountered difficulties in getting these models to learn (the problems being similar for the siamese and prototypical networks). In our initial experiments, both architectures converged to a decrease in the cost function, but this resulted in a decrease in the phone accuracy score on the test set (which is the unwanted behavior). To solve this problem, we had to make the following changes:

• 1 When learning the model, we presented as many positive pairs as negative ones for each batch (specific to siamese networks)

• 2 We made sure to balance the classes present for each batch (siamese and prototypical networks).

• 3 We balanced the numbers of male and female speakers for the reference examples (siamese and prototypical networks).

• 4 We observed that reducing the number of parameters of the chosen architecture improved the results. The new architecture used is in Table 5. Thus, we go from an architecture of about 23 million parameters to about 5 million parameters. Therefore, we assume that these methods do not allow learning from scratch models with a large number of parameters.

  Table 5 Architecture used for siamese and prototypical networks

• 5 The choice of examples from different dialect regions² allows an increase in the scores but is not always possible depending on the available data. It represents an improvement of 5% in accuracy.

Thus, for the recognition of the 39 English phonemes of TIMIT, we obtained at best an accuracy of 32.58% for the siamese network and an accuracy of 41.38% for the prototypical network. These initial results are encouraging, especially considering that we used 1.1% of the data from the TIMIT training set (when we used 40 samples for training the model). To compare these two methods, we have made two compilation tables (Tables 6 and 7) where we used the accuracy as a metric. It should be noted that for the siamese network architecture, the computation time of our implementation was higher (about one day for an experiment) than our implementation of the prototypical networks (about 6h for an experiment). Even if our implementation can be improved, this difference is due to the nature of the frameworks. In siamese networks, each reference is compared to a new sample, while in prototypical only, the prototypes are compared to new samples. Note that our implementation can be improved to diminish the computational time for both techniques, but the difference between the two frameworks will remain. This difference in computational time also explains why the number of trials we attempted is less important for siamese networks. Considering these initial results, we consider that the prototypical architecture is more interesting since the computation times are lower and the results are better. Moreover, we notice that the prototypical network seems more stable if the number of supervised examples increases (compared to the siamese network).

Table 6 Results on the 39 phonemes of TIMIT with the siamese network

Table 7 Results on the 39 phonemes of TIMIT with the prototypical network

Nevertheless, these results are just preliminary ones. We will include data augmentations in our next experiments, hoping it will increase the accuracy, as all reviewed few-shot techniques use data-augmentation.

6 Summary and future directions

In this survey, we investigated few-shot techniques for speech usage. In order to do so, we started with state-of-the-art speech processing systems. These systems require a large amount of data and are not suited for under-resourced speech problems. We also looked into techniques requiring fewer data using data augmentation, domain transposition, models requiring fewer parameters, the multi-task approach, and transfer learning. Nevertheless, these techniques are not always sufficient in a data-limited context, especially for pathological speech [19]. Next, we studied few-shot techniques and how well the different frameworks are adapted for classical speech tasks.

The main drawback of the reviewed techniques is the amount of computation required for large datasets (such as LibriSpeech from [4]) compared to SOTA models we reviewed in Section 2. Nevertheless, we considered some recent works already using few-shot techniques on speech with promising results. Such techniques seem useful for classical speech tasks on impaired speakers. Moreover, we think it can be useful for unconventional speech tasks such as measuring the intelligibility of a person (with impaired or unimpaired speakers) to help the re-education process (by identifying the problems faster). Acquiring a large amount of data is time consuming and laborious for some patients (with severe pathologies). We believe that few-shot techniques may help the community to tackle this problem.

Our initial results over the TIMIT dataset indicate that such an approach is adaptable to phoneme recognition. Indeed, we managed to obtain an accuracy of 41% using only 40 supervised samples per phoneme with the prototypical network over MFCC features. Furthermore, this result was obtained without requiring data augmentation, which should improve this first result (as all reviewed few-shot techniques use data augmentation). To better see the potential of such techniques, we will continue our work by establishing a benchmark for phoneme recognition, speaker recognition tasks and emotion recognition. Afterwards, we plan to use this technique with the best results on this benchmark as a base for teaching the concept of intelligibility.