N-dimensional N-microphone sound source localization

Source localization has been one of the fundamental problems in sonar [1], radar [2], teleconferencing or videoconferencing [3], mobile phone location [4], navigation and global positioning systems (GPS) [5], localization of earthquake epicenters and underground explosions [6], microphone arrays [7], robots [8], microseismic events in mines [9], sensor networks [10, 11], tactile interaction in novel tangible human-computer interfaces [12], speaker tracking [13], surveillance [14], and sound source tracking [15]. Our goal is real-time sound source localization in outdoor environments, which necessitates a few points to be considered. For localizing such sound sources, a far-field assumption is usual. Furthermore, our experiments confirm that placing localization system in suitable higher heights often reduces the reverberation degree, especially for flying objects. Also, many such sound source signals are wideband signals. Moreover, outdoor high-accuracy sound source localization in different climates needs highly sensitive and high-performance microphones which are very expensive. Therefore, reduction of the number of microphones is very important, which in turn leads to reduced localization accuracies using conventional methods. Here, we intend to introduce a real-time accurate wideband sound source localization system in low degree reverberation far-field outdoor cases using fewer microphones.

The structure of this paper is as follows. After a literature review, we explain HRTF-, ILD-, and TDE-based methods and discuss TDE-based phase transform (PHAT). In Section 4, we explain sound source angle of arrival and location calculations using ILD and PHAT. Section 5 covers the introduction of TDE-ILD-based method to two-dimensional (2D) half-plane sound source localization using only two microphones. Section 6 includes simulation of this method for 2D cases, where according to simulation results and due to the use of ILD, we introduce source counting. In Section 7, we propose, and in Section 8, we implement our TDE-ILD-HRTF-based method for 2D whole-plane and three-dimensional (3D) entire-space sound source localization. Section 9 includes the implementations. Finally conclusions will be made in Section 10.

Passive sound source localization methods, in general, can be divided into direction of arrival (DOA) [16], time difference of arrival (TDOA) or TDE or interaural time difference (ITD)- [17–20], ILD- [21–24], and HRTF-based methods [25–30]. DOA-based beamforming and sub-space methods typically need a large number of microphones for estimation of narrowband source locations in far-field cases and wideband source locations in near-field cases. Also, they have higher processing needs in comparison to other methods. Many localization methods for near-field cases have been proposed in the literature, such as maximum likelihood (ML), covariance approximation, multiple signal estimation (MUSIC), and estimation of signal parameters via rotational invariance techniques (ESPRIT) [16]. However, these methods are not applicable to the localization of wideband signal sources in far-field cases with small number of microphones. On the other hand, ILD-based methods are mostly applicable to the case of a single dominant sound source (high signal-to-noise ratio (SNR)) [21–24]. TDE-based methods with high sampling rates are commonly used for 2D and 3D high-accuracy wideband near-field and far-field sound source localization. In the case of ILD or TDE-based methods, minimum number of microphones required is three for 2D positioning and four for the 3D case [17–20, 31–39]. Finally, HRTF-based methods are applicable only to the case of calculating arrival angle in azimuth or elevation [30].

TDE- and ILD-based outdoor far-field accurate wideband sound source localization in different climates needs highly sensitive and high-performance microphones which are very expensive. In the last decade, some papers were published which introduce 2D sound source localization methods using just two microphones in indoor cases using TDE- and ILD-based methods simultaneously. However, it is noticeable that using ILD-based methods requires only one dominant source to be active, and it is known that, by using a linear array in the proposed TDE-ILD-based method, two mirror points will be produced simultaneously (half-plane localization) [40]. In this paper, we apply this method in outdoor (low-degree reverberation) cases for a dominant sound source. We also propose a novel method to have 2D whole-plane (without producing two mirror points) and 3D entire-space dominant sound source localization using TDE-, ILD-, and HRTF-based methods simultaneously (TDE-ILD-HRTF method). Based on the proposed method, a special reflector for the implemented simple microphone arrangement is designed, and source counting method is used to find that only one dominant sound source is active in the localization area.

In TDE and ILD localization approaches, calculations are carried out in two stages: estimation of time delay or intensity level differences and location calculation. Correlation is most widely used for time delay estimation [17–20, 31–39]. The most important issue in this approach is high-accuracy time delay estimation between microphone pairs. Meanwhile, the most important issue in ILD-based approach is high accuracy level difference measurement between microphone pairs [21–24]. Also, numerous results were published in the last decades for the second stage, i.e., location calculation. Equation complexities and large processing times are the important obstacles faced at this stage. In this paper, we propose a simple microphone arrangement that solves both these problems simultaneously.

In the above-mentioned first stage, the classic methods of source localization from time delay estimates by detecting radio waves were Loran and Decca [31]. However, generalized cross-correlation (GCC) using a ML estimator, proposed by Knapp and Carter [32], is the most widely used method for TDE. Later, a number of techniques were proposed to improve GCC in the presence of noise [3, 33–36]. As GCC is based on an ideal signal propagation model, it is believed to have a fundamental weakness of inability to cope well with reverberant environments. Some improvements were obtained by cepstral prefiltering by Stephenne and Champagne [37]. Even though more sophisticated techniques exist, they tend to be computationally expensive and are thus not well suited for real-time applications. Later, PHAT was proposed by Omologo and Svaizer [38]. More recently, a new PHAT-based method was proposed for high-accuracy robust speaker localization, known as steered response pattern-phase or power-phase transform (SRP-PHAT) [39]. Its disadvantage is higher processing time in comparison to PHAT, as it requires a search of a large number of candidate locations. According to the fact that in the application of this research, the number of candidate locations is much higher due to direction and distance estimation; this disadvantage does not allow us to use it in real-time applications.

In the last decade, according to the fact that the received signal energy is inversely proportional to the squared distance between the source and the receiving sensor, there has been some interest in using the received signal level at different sensors for source localization. Due to the spatial separation of the sensors, the source signal will arrive at the sensors with different levels so that the level differences can be utilized for source localization. Sheng and Hu [41] followed by Blatt and Hero [42] have proposed different algorithms for locating sources using a sensor network based on energy measurements. Birchfield and Gangishetty [22] applied ILD to sound source localization. While these works used only ILD-based methods to locate a source, Cui et al. [40] tried 2D sound source localization by a pair of microphones using TDE- and ILD-based methods simultaneously. When the source signal is captured at the sensors, both time delay and signal level information are used for source localization. This technique is applicable for 2D localization with two sensors only. Also, due to the use of a linear array, it generates two mirror points simultaneously (half-plane localization). Ho and Sun [24] addressed a more common scenario of 3D localization using more than four sensors to improve the source location accuracy.

Human hearing system allows finding sound sources direction of arrival in 3D with just two ears. Pinnas, shoulders, and head diffract the incoming sound waves [43]. These propagation effects collectively are termed the HRTF. Batteau reported that the external ears play an important role in estimating the elevation angle of arrival [44]. Roffler and Butler [45], Oldfield and Parker [46], and Hofman et al. [47] have tried to find experimental evidence for this claim by using a Plexiglas headband to flatten the pinna against the head [45]. Based on HRTF measuring, Brown and Duda have made an extensive experimental study and provided empirical formulas for the multipath delays produced by pinna [48]. Although more sophisticated HRTF models have been proposed [49], the Brown-Duda model has the advantage that it provides an analytical relationship between the multipath delays and the azimuth and elevation angles. Recently, Sen and Nehorai considered the Brown-Duda HRTF model as an example to model the frequency-dependent head-shadow effects and the multipath delays close to the sensors for analyzing a 3D direction finding system with only two sensors inspired by the human auditory system [43]. However, they did not consider white noise gain error or spatial aliasing error in their model. They computed the asymptotic frequency domain Cramer-Rao bound (CRB) on the error of the 3D direction estimate for zero-mean wide-sense stationary Gaussian source signals. It should be noted that HRTF-based works are just able to estimate the azimuth and elevation angles [50]. In the last decades, some papers were published which tried to apply HRTF along with TDE for azimuth and elevation angle of arrival estimation [30]. According to this ability, we apply HRTF in our TDE-ILD-based localization system for solving the ambiguity in the generation of two mirror location points. We named it TDE-ILD-HRTF method.

Given a set of TDEs and ILDs from a small set of microphones using PHAT- and ILD-based methods, respectively, the second stage of a two-stage algorithm determines the best point for the source location. The measurement equations are non-linear. The most straightforward way is to perform an exhaustive search in the solution space. However, this is computationally expensive and inefficient. If the sensor array is known to be linear, the position measurement equations are simplified. Carter focused on a simple beamforming technique [1]. However, it requires a search in the range and bearing space. Also, beamforming methods need many more microphones for high-accuracy source localization. The linearization solution based on Taylor-series expansion by Foy [51] involves iterative processing, typically incurs high computational complexity, and for convergence, requires a tolerable initial estimate of the position. Hahn proposed an approach [20] that assumes a distant source. Abel and Smith proposed an explicit solution that can achieve the Cramer-Rao Lower Bound (CRLB) in the small error region [52]. The situation is more complex when sensors are distributed arbitrarily. In this case, emitter position is determined from the intersection of a set of hyperbolic curves defined by the TDOA estimates using non-Euclidean geometry [53, 54]. Finding the solution is not easy as the equations are non-linear. Schmidt has proposed a formulation [18] in which the source location is found as the focus of a conic passing through three sensors. This method can be extended to an optimal closed-form localization technique [55]. Delosme [56] proposed a gradient method for search in a localization procedure leading to computation of optimal source locations from noisy TDOA's. Fang [57] gave an exact solution when the number of TDOA measurements is equal to the number of unknowns (coordinates of transmitter). This solution, however, cannot make use of extra measurements, available when there are extra sensors, to improve position accuracy. The more general situation with extra measurements was considered by Friedlander [58], Schau and Robinson [59], and Smith and Abel [55]. These methods are not optimum in the least-squares sense and perform worse in comparison to the Taylor-series method.

Although closed-form solutions have been developed, their estimators are not optimum. The divide and conquer (DAC) method [60] from Abel can achieve optimum performance, but it requires that the Fisher information is sufficiently large. To obtain a precise position estimate at reasonable noise levels, the Taylor-series method [51] is commonly employed. It is an iterative method that starts with an initial guess and improves the estimate at each step by determining the local linear least-squares (LS) solution. An initial guess close to the true solution is needed to avoid local minima. Selection of such a starting point is not simple in practice. Moreover, convergence of the iterative process is not assured. It also suffers from convergence problem and large LS computational burden as the method is iterative. Within the last few years, some papers were published on improving LS and closed-form methods [16, 23, 61–65]. Based on closed-form hyperbolic-intersection method, we will explain our proposed method, which using a simple arrangement of two microphones for 2D cases and three microphones for 3D cases, can simplify non-linear equations of this method to have a linear equation. Although there have been attempts to linearize closed-form non-linear equations through algebraic means, such as [7, 16, 56, 63], our proposed method with simple pure geometrical linearization needs less microphones and features accurate localization and less processing time.

The positive root gives the square of distance from source to origin. Substituting Rs into (51), the final source coordinate will be obtained [40].

We remember again that by using a linear array, two mirror points will be produced simultaneously. This means that we can localize 2D sound source only in half-plane.

Then, we calculated d₁ and d₂ using (24) and (25), and using (37), we calculated time delay between the received signals of the two microphones. For time delay positive values, i.e., sound source nearer to the first microphone (mic1 in Figure 1), we delayed second microphone signal with respect to the first microphone signal, and for negative values, i.e., sound source nearer to the second microphone (mic2 in Figure 1), did the opposite. Then using (6) and (7), we divided the first microphone signal by d₁ and the second microphone signal by d₂ to have correct attenuation in signals according to the source distances from microphones. Finally, we tried to calculate source location using the proposed TDE-ILD method (Section 5) in a variety of SNRs for some environmental noises.

For a number of signal-to-noise ratios (SNRs) for white Gaussian, pink and babble noises from ‘NATO RSG-10 Noise Data,’ 16-bit Quantization, and 96,000-Hz sampling frequency (hence, we upsampled NATO RSG-10 from its original 190,00 to 96,000 Hz), simulation results are shown in Figure 2 for source location of (x = 10, y = 10). Assuming a high audio sampling rate, fractional delays are negligible and delays are rounded to nearest sampling point. Simulation results show larger localization error for SNRs lower than 10 dB. This issue occurs due to the use of ILD. Hence, we have to use this method for the case of only a dominant source to have accurate localization. Using spectral subtraction and source counting are useful for reducing localization error in SNRs lower than zero.

Using TDE-ILD-based method, dual microphone 2D sound source localization is applicable. However, it is known that, by using a linear array in TDE-ILD-based method, two mirror points will be produced simultaneously (half-plane localization) [40]. Also, according to TDE-ILD-based simulation results (Section 6), it is noticeable that using ILD-based method needs only one dominant high SNR source to be active in localization area. Our proposed TDE-ILD-HRTF method tries to solve these problems using source counting, noise reduction using spectral subtraction, and HRTF.

7.3. Using HRTF method

Using a linear array in TDE-ILD-based dual microphone 2D sound source localization method leads to two mirror points produced simultaneously. Adding an HRTF-inspired idea, whole-plane dual microphone 2D sound source localization would be possible. This idea was published in [68], and it is reviewed here again. Researchers have used an artificial ear with a spiral shape. This is a special type of pinna with a varying distance from a microphone placed in the center of its spiral [30]. However, we consider a half-cylinder instead of artificial ear [68]. Due to the use of such a reflector, a constant notch position is created for all variations (0 to180°) of sound signal angle of arrival (Figure 3). However, clearly, and as shown in the figure, a complete half-cylinder scatters the sound waves from sources behind it. In order to overcome this problem we consider slits on the surface of the half-cylinder (Figure 3). If d is the distance between the reflector (half-cylinder) and the microphone (placed at the center), a notch will be created when it is equal to quarter of the wavelength of sound, λ, plus any multiples of λ/2. For such wavelengths, the incident waves are reduced by reflected waves [30]:

$n\cdot \left(\frac{\lambda }{2}\right)+\left(\frac{\lambda }{4}\right)=dn=0,1,2,3....$

(73)

These notches will appear at the following frequencies:

$f=\frac{c}{\lambda }=\frac{\left(2n+1\right).c}{4d}.$

(74)

Covering only microphone 2 in Figure 1 by reflector, calculating the interaural spectral difference results in

$\begin{array}{ll}\left|\mathrm{\Delta H}\left(f\right)\right|& =\left|10{log}_{10}{H}_1\left(f\right)-10{log}_{10}{H}_2\left(f\right)\right|\\ =\left|10{log}_{10}\frac{H_1\left(f\right)}{H_2\left(f\right)}\right|.\end{array}$

(75)

High |ΔH(f)| values indicate that the sound source is in front, while negligible values indicate that sound source is at the back. One important point is that in order to have the same spectrum in both microphones when the sound source is at the back, careful design of the slits is necessary.

7.4. Extension of dimensions to three

According to the use of half-cylinder reflector in the proposed localization system, this approach is only applicable in 2D cases. Using a half-sphere reflector instead of the half-cylinder makes it usable in 3D sound source localization (Figure 4). Adding the half-sphere reflector only to microphone 2 in Figure 5 allows the localization system to localize sound sources in 3D cases.

For 3D case, we can consider a plane that passes through source position and x-axis and which makes an angle of θ with the x-y plane, and we name it source-plane (Figure 5). This plane consists of the microphones 1 and 2. According to these assumptions, using (36), we can calculate ф which is the angle of arrival in source-plane and is equal to angle of arrival in x-y plane. Then, using (59) and (60), we can also calculate x and y in 2D source-plane (not in x-y plane in 3D case) and therefore calculate sound source distance $r\sqrt{x^2+y^2}$. Introducing a new microphone (mic3 in Figure 5) with y = 0 and x = z = R helps us calculate the angle θ. Therefore, using (36), we can calculate θ as:

$\mathrm{\theta }=\mathrm{co}{\mathrm{s}}^{‒1}\left(\frac{v_{\mathrm{sound}}\cdot {\tau }_{31}}{R}\right).$

(76)

Using half-sphere reflector decreases accuracy of time delay and intensity level deference estimation between microphones 1 and 2 due to the change in the spectrum of second microphone's signal. However, covering the third microphone instead of the second microphone by half-sphere reflector only decreases the accuracy of time delay estimation between the first and third microphones (Figure 5). Multiplying the inverse function of notch-filter in third microphone's spectrum leads to increase in accuracy. Now using r, ф, and θ, we can calculate source location x, y and z in 3D case:

$\left\{\begin{array}{l}x=r.cos\left(\varphi \right).sin\left(\theta \right)\\ y=r.sin\left(\varphi \right),sin\left(\theta \right)\\ z=r.cos\left(\theta \right)\end{array}\right..$

(77)

The reasons for choosing the shape of sphere for the reflector are as follows [69]. The simplest type of reflector is a plane reflector introduced to direct signal in a desired direction. Clearly, using this type of reflector, the distance between reflector and microphone, d in (73), varies with respect to source position in 3D cases leading to a change in notch position within spectrum. The change in notch position may not be suitable as it might occur out of the spectral band of interest. To better adjust the energy in the forward direction, the geometrical shape of the plane reflector must be changed so as not to allow radiation in the back and side directions.

A possible arrangement for this purpose consists of two plane reflectors joined on one side to form a corner. This type of reflector returns the signal exactly in the same direction as it is received. Because of this unique feature, reflected wave acquired by the microphone is unique, which is our aim. Whereas having more than one reflected wave causes a higher energy value with respect to a single reflected wave at the microphone, which causes a deep notch. But using this type of reflector, d is varied with respect to the source position in 3D cases which is related to notch position in spectrum. It has been shown that if a beam of parallel waves is incident upon a parabola reflector, the radiation will focus at a spot which is known as the focal point. This point in spherical reflector does not match to the center in accordance with this fact that the center has equal distance (d) to all points of the reflector surface. Because of this feature, reflected wave which is received by the microphone is unique, whatever the position of the sound source, in 3D cases and belongs to the wave passing the center. The center of the spherical reflector with radius R is located at O (Figure 6). The sound wave axis strikes the reflector at B. From the Law of Reflection and angle geometry of parallel lines, the marked angles are equal. Hence, BFO is an equilateral triangle. Dropping the median from F which is stapled to BO and using trigonometry, the focal point is calculated as:

$\frac{R}{2b}=cos\left(\mathrm{\theta }\right)\to b=\frac{R}{2cos\mathrm{\theta }}$

(78)

$\to F=R-\frac{R}{2cos\left(\theta \right)}.$

(79)

F is not equal to R for all θ values. The half-spherical reflector scatters the sound source waves hitting it from back. Therefore, we consider some slits on its surface (Figure 4). Obviously, the slits need to be designed in a fashion that leads to the same spectrum in both microphones when sound source is in back. When a plane wave hits a barrier with a single circular slit narrower than signal wavelength (λ), the wave bends and emerges from the slit as a circular wave [70]. If L is the distance between the slit and the viewing screen and d is the slit width, based on the assumption that L> > d (Fraunhofer scattering), the wave intensity distribution observed (on a screen) at an angle θ with respect to the incident direction is given by:

$\mathit{if}k=\frac{\pi .d}{\lambda }sin\left(\theta \right)\to \frac{I\left(\theta \right)}{I\left(\theta \right)}=\frac{{sin}^2\left(k\right)}{k^2},$

(80)

where I(θ) is wave intensity in direction of observation and I(0) is maximum wave intensity of diffraction pattern (central fringe). This relationship will have a value of zero each time sin²(k) = 0. This occurs when

$k=\pm \mathit{m\pi }\mathrm{or}\frac{\pi .d}{\lambda }sin\left(\theta \right)=\pm \mathit{m\pi },$

(81)

yielding the following condition for observing minimum wave intensity from a single circular slit:

$sin\left(\theta \right)=\frac{\mathit{m\lambda }}{d}m=0,\pm 2,....$

(82)

This relationship is satisfied for integer values of m. Increasing values of m gives minima at correspondingly larger angles. The first minimum will be found for m = 1, the second for m = 2, and so forth. If $\frac{d}{\lambda }sin\left(\theta \right)$ is less than one for all values of θ, i.e., when the size of the aperture is smaller than a wavelength (d < λ), there will be no minima. As we need more than one circular slit on half-sphere reflector's surface, we consider a parallel wave incident on a barrier that consists of two closely spaced narrow circular slits S1 and S2. The narrow circular slits split the incident wave into two coherent waves. After passing through the slits, the two waves spread out due to diffraction interfere with one another. If this transmitted wave is made to fall on a screen some distance away, an interference pattern of bright and dark fringes are observed on the screen, with the bright ones corresponding to regions of maximum wave intensity while the dark ones corresponding to those of minimum wave intensity. As discussed before, for all points on the screen where the path difference is some integer multiple of the wavelength, the two waves from the slits S1 and S2 arrive in phase and bright fringes are observed. Thus, the condition for producing bright fringes is as in (82). Similarly, dark fringes are produced on the screen if the two waves arriving on the screen from slits S1 and S2 are exactly out of phase. This happens if the path difference between the two waves is an odd integer multiple of half-wavelengths:

$sin\left(\theta \right)=\frac{\left(m+\frac{1}{2}\right).\lambda }{d}m=0,\pm 1,\pm 2,....$

(83)

Of course, this condition is changed using a half-sphere instead of a plane (Figure 7), where according to the same distance R to the center, the two waves from the slits S1 and S2 arrive in phase at the center and bright fringes are observed there. This signal intensity magnification is not suitable for the case of estimating intensity level difference between two microphones where one of them is covered by this reflector. However, covering the third microphone by half-sphere reflector is suitable where there is no need to estimate the intensity level difference between two microphones 1 and 3.

In this paper, we reported on the simulation of TDE-ILD-based 2D half-plane sound source localization using only two microphones. Reduction of the microphone count was our goal. Therefore, we also proposed and implemented TDE-ILD-HRTF-based 3D entire-space sound source localization using only three microphones. Also, we used spectral subtraction and source counting methods in low-degree reverberation outdoor cases to increase localization accuracy. According to Table 3, implementation results show that the proposed method has led to less than 10% error for 3D location finding. This is a higher accuracy in source location measurement in comparison with similar researches which did not use spectral subtraction and source counting. Also, we indicated that partly covering one of the microphones by a half-sphere reflector leads to entire-space N-dimensional sound source localization using only N-microphones.

AP was born in Azerbaijan. He has a Ph.D. in Electrical Engineering (Signal Processing) from the Electrical Engineering Department, Amirkabir University of Technology, Tehran, Iran. He is now an invited lecturer at the Electrical Engineering Department, Amirkabir University of Technology and has been teaching several courses (C++ programming, multimedia systems, digital signal processing, digital audio processing, and digital image processing). His research interests include statistical signal processing and applications, digital signal processing and applications, digital audio processing and applications (acoustic modeling, speech coding, text to speech, audio watermarking and steganography, sound source localization, determined and underdetermined blind source separation and scene analyzing), digital image processing and applications (image and video coding, image watermarking and steganography, video watermarking and steganography, object tracking and scene matching) as well as BCI.

SMA received his B.S. and M.S. degrees in Electronics from the Electrical Engineering Department, Amirkabir University of Technology, Tehran, Iran, in 1984 and 1987, respectively, and his Ph.D. in Engineering from the University of Cambridge, Cambridge, U.K., in 1996, in the field of speech processing. Since 1988, he has been a Faculty Member at the Electrical Engineering Department, Amirkabir University of Technology, where he is currently an associate professor and teaches several courses and conducts research in electronics and communications. His research interests include speech processing, acoustic modeling, robust speech recognition, speaker adaptation, speech enhancement as well as audio and speech watermarking.