Expanded three-channel mid/side coding for three-dimensional multichannel audio systems

Abstract

Three-dimensional (3D) audio technologies are booming with the success of 3D video technology. The surge in audio channels makes its huge data unacceptable for transmitting bandwidth and storage media, and the signal compression algorithm for 3D audio systems becomes an important task. This paper investigates the conventional mid/side (M/S) coding method and discusses the signal correlation property of three-dimensional multichannel systems. Then based on the channel triple, a three-channel dependent M/S coding (3D-M/S) method is proposed to reduce interchannel redundancy and corresponding transform matrices are presented. Furthermore, a framework is proposed to enable 3D-M/S compress any number of audio channels. Finally, the masking threshold of the perceptual audio core codec is modified, which guarantees the final coding noise to meet the perceptual threshold constraint of the original channel signals. Objective and subjective tests with panning signals indicate an increase in coding efficiency compared to Independent channel coding and a moderate complexity increase compared to a PCA method.

Introduction

Recently, 3D audio has attracted more attention and developed fast following the booming market of 3D movie. Many 3D audio technologies are now introduced into audio-involved applications to replace the surround sound system to provide superior sound localization and an immersive feeling. Wave field synthesis (WFS), Ambisonics and vector-based amplitude panning (VBAP) are the three most well-developed technologies. WFS generally follows Huygens principle to reconstruct the original sound field [1]. Research institutions such as IDMT of Fraunhofer and IRCAM in France have an intensive study in WFS, and attempt to bring WFS into theater and live transmission of concert. Ambisonics utilizes spherical harmonic functions to recording sound field and driving loudspeakers, its loudspeakers have rigorous configuration and give a good sound field reconstruction in the center [2]. VBAP follows the tangent law in a three-dimensional space using three adjacent loudspeakers to form a sound vector. For its simplicity, VBAP is the most common algorithm in 3D signal panning [3]. A 3D system like 22.2 multichannel system proposed by NHK in Japan utilizes VBAP to generate 3D sound image [4]. The 22.2 multichannel system is also included in the developing MPEG-H standard for rendering 3D audio scene.

There is a clear trend that 3D audio technology will become mature gradually and replace stereo and surround sound [5]. However, a main and common feature of 3D audio technologies is the great number of sound channels. For instance, WFS system always contains dozens and even hundreds of audio channels. The 22.2 system has three layers and 24 audio channels. Although the Ambisonics system can have flexible order and channel number, it usually uses dozens of channels because fewer channels will cause quality deterioration. Comparing with a two-channel stereo and a 5.1 surround sound, the increasing of audio channel causes a dramatical 3D audio data increase. A report from Fraunhofer shows 37 Mbps is needed for live transmission of WFS [6]. For the 22.2 multichannel system, uncompressed data also reaches 28 Mbps [7]. Currently, storage media and transmission bandwidth can hardly afford those huge data size. So the compression of 3D multichannel audio signals becomes an important subject.

The well-known Spatial Audio Coding (SAC) models the signals as virtual sound sources in the frequency domain, extracts the interchannel level difference (ICLD) and interchannel time difference (ICTD) and interchannel coherence (IC) to represent the direction and width of virtual sound source and downmixes the multichannels to reduce redundancy [8–11]. The idea of using downmixed sources with spatial parameters was later developed into Spatial Audio Object Coding (SAOC) for efficiently coding the multiple input spatial audio objects with interactive and personalized rendering ability [12]. Recently, some other investigations have been published to increase the compression efficiency for multichannel 3D audio signals. In 2007, Goodwin and Jot proposed a PCA-based multichannel compression framework for parametric coding [13], which can enhance specific audio scenarios and provide robust spatial audio coding. In 2008, Cheng et al. proposed the Spatially Squeezed Surround Audio Coding (S³AC) for parametrically compressing the Ambisonics signal [14]. In 2009, Hellerud used an inter-channel prediction-based coding method to remove the redundancy between Ambisonics channels [15], which has low algorithm delay but high computational complexity. Tzagkarakis used a sinusoidal model and linear prediction to parameterize the separate spot microphone channels, then downmixed the residual signals. This coding scheme is more suitable for multichannel signals with weak correlation, and such scenarios require Independent channel decoding [16]. In 2010, Pinto et al. utilized a space/time-frequency transform to decompose the WFS signals into plane waves and evanescent waves. By discarding the evanescent waves and perceptually coding the plane wave signals, coding gain is obtained. Coding efficiency increases along with the number of audio channels, because the transform decomposition accuracy depends on the spatial resolution which is the number of WFS channels [17, 18]. In 2013, Cheng further proposed a Spatial Localization Quantization Point (SLQP) codec using localization cues to compress the 3D audio signals [19, 20]. Since SLQP extracts the spatial cues and downmixes the channels, it achieved high compression ratio for SLQP signals and other 3D audio systems.

In order to increase the coding efficiency at high bitrates, some non-parametric coding schemes were developed. Yang proposed a scalable multichannel codec, using the Karhunen-Loeve Transform (KLT) to remove the interchannel redundancy to realize scalable multichannel audio coding [21]. Mid/side (M/S) coding was introduced by J.D. Johnston [22] and adopted by many audio codec such as MPEG2-Layer III and MPEG4-AAC. In 2003, Liu et al. proposed a bit allocation method for M/S coding based on allocation entropy, which increases the objective quality by allocating more bits to high energy channel in M/S coding [23]. In 2008, Derrien et al. proposed an error model for M/S coding. The error model enables tuning of the quantizer used for channels M and S at the encoder with respect to the distortion of L and R at the decoder side, which increased the coding efficiency of M/S without much complexity [24]. Since M/S coding works as the simplest interchannel prediction, Krueger generalized it using linear prediction instead of M/S transformation and residual signal instead of difference signal [25]. In 2012, Schafer further developed Krueger’s method, the multichannel case, which has low algorithmic delay [26]. Recently, M/S coding was combined with parametric stereo coding at low bitrates in the MPEG-USAC standard [27] by predicting the residual channel using spatial cue-based parameters, which aimed to bridge the stereo quality gap between low bitrates and high bitrates [28]. M/S coding also works alone at high bitrates utilizing a novel complex prediction to achieve better performance [29].

The above model-based codec and parametric codec can offer a considerable compression ratio. However, those methods need to know the direction of the real audio source to do objective-oriented coding, or estimated a virtual source direction to do downmixing and parametric coding. In practice, such as live recording, it is very difficult to obtain the real audio source direction. Downmixing and parametric coding will cause interchannel interference such as ‘tone leakage’ artifacts when channel signals differ greatly [30]. Furthermore, the computational complexity of an audio codec should be acceptable while maintaining enough coding efficiency, and parametric coding can only achieve a performance gain at low bitrates. This paper focuses on the situation that only the multichannel signals of audio sources are recorded, instead of their directions. And we consider high-quality/high-bitrate application and focus on the non-parametric coding method. Section ‘M/S coding in 3D space’ describes the conventional M/S coding process and presents a three-channel Dependent M/S coding (3D-M/S) method. The main idea is to expand M/S coding to three-dimensional audio by designing a new transform matrix, which remove the redundancy of three channels in 3D space rather than just two channels in the horizontal plane. Section ‘3D-M/S psychoacoustic model’ discusses the psychoacoustic model for transformed 3D-M/S signals. Section ‘Framework for general channel configuration’ specifies a new framework enables 3D-M/S to be applied to a more general channel configuration. Section ‘Experiment’ gives a comparison of 3D-M/S coding with PCA coding and Independent channel coding to justify the performance of compression ratio and computational complexity. Section ‘Conclusion’ summarizes and concludes this paper.

M/S coding in 3D space

Conventional M/S coding

M/S coding is based on the fact that most stereo channels are strongly correlated. By simply transforming the stereo channel pair into the M/S domain, core codec encodes a summation channel and a difference channel instead of the original channels. The difference channel has much lower energy than the original channel, so more frequency bins can be quantized to similar and smaller quantized values, which leads to the entropy of the resulting quantised time-frequency samples is lower and hence lossless coding using Huffman coding achieves a higher compression rate. To illustrate how M/S coding works, a generalized sine stereo model is used. Here, a stereo pair is denoted as a vector V ₀ = (C _L ,C _R ) where

$$\begin{array}{c}{C}_L=S\text{sin}\theta \\ C_R=S\text{cos}\theta \end{array}$$

(1)

S is the virtual audio source, θ is the stereo panning angle and $\theta \in \left[0,\frac{\pi }{2}\right]$. The M/S coding can be denoted as two transform matrices M ₀ and M ₁ , the summation vector of M ₁ is denoted as ${\mathbf{V}}_{\mathbf{1}}=\left(\frac{\sqrt{2}}{2},\frac{\sqrt{2}}{2}\right)$

$${\mathbf{M}}_{\mathbf{0}}=\left[\begin{array}{c}10\\ 01\end{array}\right]{\mathbf{M}}_{\mathbf{1}}=\left[\begin{array}{c}\frac{\sqrt{2}}{2}\frac{\sqrt{2}}{2}\\ \frac{\sqrt{2}}{2}-\frac{\sqrt{2}}{2}\end{array}\right]$$

(2)

And the mid/side channels are obtained by multiplying input signals with M ₁ as (C _M ,C _S )= M ₁ V ₀ , where

$$\begin{array}{c}{C}_M=\frac{\sqrt{2}}{2}{C}_L+\frac{\sqrt{2}}{2}{C}_R=S\text{sin}\left(\theta +\frac{\pi }{2}\right)\\ C_S=\frac{\sqrt{2}}{2}{C}_L-\frac{\sqrt{2}}{2}{C}_R=S\text{sin}\left(\theta -\frac{\pi }{2}\right)\end{array}$$

(3)

In practice, subband energy or masking threshold will be used instead of C _L ,C _R in V ₀ . Only when two channels is sufficiently correlated, for example energy difference is less than a threshold Thr=2 dB as shown in (4), will the M/S mode be used to avoid being frequently transformed and recalculating the masking threshold [22].

$$\left|10{\text{log}}_{10}{\left(\frac{C_L}{C_R}\right)}^2\right|\le \text{Thr}$$

(4)

To discuss the switching condition more conveniently, the M/S switching condition will be expressed using the distance between vectors V ₀ and V ₁ . Given $\frac{C_L}{C_R}=\text{tan}\theta$, (4) can be denoted as

$$\frac{1}{\sqrt{1+{10}^{\frac{\text{Thr}}{10}}}}\le \text{cos}\theta \le \frac{1}{\sqrt{1+{10}^{-\frac{\text{Thr}}{10}}}}$$

(5)

The left side of Figure 1 shows the M/S switching condition (5) in vector space, and Figure 2 illustrates a stereo signal and corresponding transformed sum and difference signals. Equation (3) and the two figures denote that when input signal vector V ₀ is close to V ₁ , where $\text{cos}\theta \approx \frac{1}{\sqrt{2}}$ and $\theta \approx \frac{\pi }{4}$, the switching condition (4) will be satisfied. The difference signal has less amplitude than the original signals, and M/S coding will be used. Since θ is the angle of V ₀ , switching condition (4) can be represented by the inner product between signal vector V ₀ and summation vector V ₁

Figure 1

Switching condition of M/S and 3D-M/S method. Conventional M/S switching condition in vector space (left). 3D-M/S swiching condition in vector space (right).

Figure 2

Two-dimensional stereo and three-dimensional VBAP signals and their transformed signals. The left and right channels of a stereo signal and its summation (M) and difference (S) signals (left). The sound source moves from the left loudspeaker to the right by the panning angle θ. When sound source is close to one loudspeaker and θ tends to be 0 and $\frac{\pi }{2}$, the original signal has less dynamic range. When sound source is close to the center and θ tends to be $\frac{\pi }{4}$, the transformed signal has less dynamic range. The three channels of a VBAP signal and its 3D-M/S signals (right). The sound source moves from A to B, B to C and C to A shown in Figure 3. When sound source is close to the center of two channels, transformed signal C _T has less dynamic range with corresponding matrix M ₁ , M ₂ , and M ₃ .

$$\frac{〈{\mathbf{V}}_{\mathbf{0}},{\mathbf{V}}_{\mathbf{1}}〉}{\left|{\mathbf{V}}_{\mathbf{0}}\right|\left|{\mathbf{V}}_{\mathbf{1}}\right|}\ge {\text{Thr}}_v$$

(6)

This is an equivalent expression to the energy condition. It indicates that only when the input signal vector is close enough to the summation vector of a M/S transform matrix, this matrix will be used. This idea will be helpful when later discussing the 3D-M/S coding where more than one transform matrix exists. Here, Thr _v is the corresponding switching threshold of Thr in vectorial distance and

$${\text{Thr}}_v=\frac{\sqrt{2}}{2}\left(\frac{1}{\sqrt{1+{10}^{\frac{\text{Thr}}{10}}}}+\frac{1}{\sqrt{1+{10}^{-\frac{\text{Thr}}{10}}}}\right)$$

(7)

M/S coding in three-dimensional space

For diffuse audio sources like ambient sound, the multichannel signals are weakly correlated. For directional audio sources, the signals are highly correlated which make it possible to reduce the interchannel redundancy. To maintain the stability of the sound image, in a stereo and surround audio system, the virtual source is always being panned or recorded by the two most adjacent channels. So two adjacent channels always have the maximum similarity, and M/S coding and parametric coding are performed based on a two-channel unit. But for Ambisonics, VBAP and 22.2 multichannel systems, sound channels are spherically configured in 3D space as shown in Figure 3. In a VBAP system, three adjacent channels form a directional sound image and have the maximum correlation. In other 3D systems such as Ambisonics, three channels cover a basic area of 3D space and adjacent three channels also have the most similar signals [31]. If the multichannel signals are still grouped and compressed on the channel pair basis, like Ando et al. recent work for coding 22.2 multichannel signals with the AAC codec [32], there will be more than one adjacent channel for each channel. It will require extra computation to select the channel pair but may still lead to a possible to mismatch. For example, in Figure 3, channel C₁, C₂ and C₃ formed a virtual source. If C₁ and C₂ were grouped for parametric or M/S coding, C₃ would be grouped with another less correlated channels, which decreases the coding performance. And if we dynamically group the signals, we have to use complex correlation analysis algorithm to analyze its six adjacent channels. Because the channel pair grouping is based on frequency subbands, it will not only increase the codec complexity dramatically but will also be unable to reduce the overall redundancy that exist in more than two channels. In brief, the conventional channel pair unit should be redesigned for 3D audio systems.

Figure 3

The 22.2 multichannel 3D audio system. A loudspeaker configuration of the 22.2 multichannel system on the sphere surface in 3D space and its plane triangle structure.

Considering the basic unit for any 3D surface is a triangle, a spherical multichannel configuration can be easily unfolded to a plane triangle structure as shown in Figure 3. The above 3D systems also utilize three or more channels to produce 3D audio effects, which leads to interchannel signal redundancy existing in or more than three channels. Hence, channel triples should be the basic unit to remove interchannel redundancy rather than channel pair in conventional coding schemes. More specifically, in a VBAP system, the input signal V ₀ = (C₁,C₂,C₃) is calculated following the tangent model in 3D space as

$$\begin{array}{l}{C}_1=S\text{sin}\theta \text{cos}\phi \\ C_2=S\text{sin}\theta \text{sin}\phi \\ C_3=S\text{cos}\theta \end{array}$$

(8)

where $\theta ,\phi \in \left[0,\frac{\pi }{2}\right]$, which determine the gain factor of the three channels.

There are infinite possible situations, where a virtual source can be located. But for the VBAP model, those possibilities can be reduced to three basic situations. First, the virtual source is located near the position of one channel. This situation corresponds to the source panned mainly using one channel, or one channel forms a virtual source with another two channels which are out of current three loudspeakers. This situation is similar to stereo audio with only one active channel, so no transform is performed and M ₀ will be used. Second, the virtual source is located between two channels. This situation corresponds to the source panned mainly using two channels, or two channels form a virtual source with one other channel out of the current three channels. This situation is similar to conventional stereo audio, and M/S coding can be applied. However, the M/S transform matrix must be modified to adapt to the three channels condition which is expressed in Equation 9. Third, the source is panned using all the three channels. This is a new situation that stereo audio never contains. To remove the interchannel redundancy, a new transform matrix M ₄ is designed following the rule of conventional M/S coding. The first vector is the summation of three channels, and the rest vectors are orthogonal with the first vector. To guarantee the conservation of energy after transformation, unit vectors are used. This matrix realizes the sum-difference processing for 3D channel, and guarantees that when three channel signals are nearly the same, two channels primarily contain the difference signal.

$$\begin{array}{l}{\mathbf{M}}_{\mathbf{0}}=\left[\begin{array}{lll}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]{\mathbf{M}}_{\mathbf{1}}=\left[\begin{array}{lll}1& 0& 0\\ 0& \frac{\sqrt{2}}{2}& \frac{\sqrt{2}}{2}\\ 0& \frac{\sqrt{2}}{2}& -\frac{\sqrt{2}}{2}\end{array}\right]\\ {\mathbf{M}}_{\mathbf{2}}=\left[\begin{array}{lll}\frac{\sqrt{2}}{2}& 0& \frac{\sqrt{2}}{2}\\ 0& 1& 0\\ \frac{\sqrt{2}}{2}& 0& -\frac{\sqrt{2}}{2}\end{array}\right]{\mathbf{M}}_{\mathbf{3}}=\left[\begin{array}{lll}\frac{\sqrt{2}}{2}& \frac{\sqrt{2}}{2}& 0\\ \frac{\sqrt{2}}{2}& -\frac{\sqrt{2}}{2}& 0\\ 0& 0& 1\end{array}\right]\\ {\mathbf{M}}_{\mathbf{4}}=\left[\begin{array}{lll}\frac{\sqrt{3}}{3}& \frac{\sqrt{3}}{3}& \frac{\sqrt{3}}{3}\\ \frac{\sqrt{2}}{2}& -\frac{\sqrt{2}}{2}& 0\\ \frac{\sqrt{6}}{6}& \frac{\sqrt{6}}{6}& -\frac{\sqrt{6}}{3}\end{array}\right]\end{array}$$

(9)

An example is shown in Figure 2. It can be observed that when the source is close to the center of two or three channels, a corresponding matrix can produce difference signals with lower dynamic range compared to the original channel signals. Under a certain masking threshold, far less bits are required for quantizing the difference signals which brings the coding gain.

Then we discuss the switching condition for 3D-M/S as shown on right side of Figure 1, and the transformed mid/second/third channels are shown in Table 1. Firstly, if all three channel signals are almost the same which means the input vector is close to spherical triangle center V ₄ ($\text{cot}\theta \approx \frac{\sqrt{2}}{2}$ and $\phi \approx \frac{\pi }{4}$), matrix M ₄ will be chosen to give two difference channels. Secondly, if there are only two channels satisfying the conventional M/S switching condition and the projection of input vector is close to V ₁ (cot θ ≈ sin φ) or V ₂ (cot θ ≈ cos φ) or V ₃ ($\phi \approx \frac{\pi }{4}$), 3D-M/S will select the matrix having the nearest distance from the input vector. The distance is measured by vector distance following expression (6). Compared with conventional switching condition, it can be seen that conventional M/S coding works on the two-dimensional space and has two switching areas. 3D-M/S switching condition is an expansion of M/S coding, where its input vector works in three-dimensional space and has five switching areas. Following the vector distance switching condition, the switching rule of 3D-M/S can be denoted as

Table 1 Transformed channel signals with five matrices

$$\text{mode}=\left\{\begin{array}{l}{M}_4,\text{if}\frac{〈{\mathbf{V}}_{\mathbf{0}},{\mathbf{V}}_{\mathbf{4}}〉}{\left|{\mathbf{V}}_{\mathbf{0}}\right|\left|{\mathbf{V}}_{\mathbf{4}}\right|}\ge {\text{Thr}}_v\\ M_i,\text{else}\text{if}\frac{〈{\mathbf{V}}_{\mathbf{\text{0i}}},{\mathbf{V}}_{\mathbf{i}}〉}{\left|{\mathbf{V}}_{\mathbf{\text{0i}}}\right|\left|{\mathbf{V}}_{\mathbf{i}}\right|}\ge {\text{Thr}}_v\\ \text{and}\frac{〈{\mathbf{V}}_{\mathbf{\text{0i}}},{\mathbf{V}}_{\mathbf{i}}〉}{\left|{\mathbf{V}}_{\mathbf{\text{0i}}}\right|\left|{\mathbf{V}}_{\mathbf{i}}\right|}\ge \frac{〈{\mathbf{V}}_{\mathbf{\text{0j}}},{\mathbf{V}}_{\mathbf{j}}〉}{\left|{\mathbf{V}}_{\mathbf{\text{0j}}}\right|\left|{\mathbf{V}}_{\mathbf{j}}\right|},\forall j\ne i\\ M_0,\text{else}\end{array}\right.$$

(10)

where i,j∈{1,2,3}, V ₀₁ = (0,C₂,Y C₃), V ₀₂ = (C₁,0,C₃), V ₀₃ = (C₁,C₂,0) are the two channel projections of input vector V ₀ . ${\mathbf{V}}_{\mathbf{1}}=\left(0,\frac{\sqrt{2}}{2},\frac{\sqrt{2}}{2}\right)$, ${\mathbf{V}}_{\mathbf{2}}=\left(\frac{\sqrt{2}}{2},0,\frac{\sqrt{2}}{2}\right)$, ${\mathbf{V}}_{\mathbf{3}}=\left(\frac{\sqrt{2}}{2},\frac{\sqrt{2}}{2},0\right)$, ${\mathbf{V}}_{\mathbf{4}}=\left(\frac{\sqrt{3}}{3},\frac{\sqrt{3}}{3},\frac{\sqrt{3}}{3}\right)$ are the summation vectors of each transform matrix.

3D-M/S psychoacoustic model

The perceptual threshold is the maximum amount of noise for an audio signal that will not degrade its subjective quality. 3D-M/S coding transforms the original signals into summation and difference signals and then sends these to the core codec. The perceptual thresholds calculated by the core codec are for the transformed signals, which can not guarantee the noise in original signals is imperceptible. So after transformation, the perceptual threshold calculation in the core encoder needs a redesign. An example is the conventional M/S stereo coding, where the masking threshold model for the main channel and side channel is revised to avoid perceptible noise in reconstructed signals [23]. To derive the masking threshold for 3D-M/S signals, we reference the deducing method in [23] and expand it to channel triple case. Consider the transformed signals $(C_M,C_S,C_T)={\mathbf{M}}_{\mathbf{4}}{\mathbf{V}}_{\mathbf{0}}^T$ as

$$\begin{array}{l}{C}_M=\frac{C_1+C_2+C_3}{\sqrt{3}}\\ C_S=\frac{C_1-C_2}{\sqrt{2}}\\ C_T=\frac{C_1+C_2-2C_3}{\sqrt{6}}\end{array}$$

(11)

where C _M is the first signal after transformation and C _S and C _T are the second and third signals, respectively. After core codec quantization, independent noise is introduced into the three signals which is denoted as N _M , N _S and N _T . So at the decoder side:

$$\begin{array}{l}{C}_M+N_M=\frac{{Ĉ}_1+{Ĉ}_2+{Ĉ}_3}{\sqrt{3}}\\ C_S+N_S=\frac{{Ĉ}_1-{Ĉ}_2}{\sqrt{2}}\\ C_T+N_T=\frac{{Ĉ}_1+{Ĉ}_2-2{Ĉ}_3}{\sqrt{6}}\end{array}$$

(12)

where ${Ĉ}_1$, ${Ĉ}_2$ and ${Ĉ}_3$ are the reconstructed signals. Compared with the original signals shown in (11), the noise energy for the original signals can be obtained

$$\begin{array}{l}{\sigma }_{C_1-{Ĉ}_1}^2=\frac{1}{3}{N}_M^2+\frac{1}{2}{N}_S^2+\frac{1}{6}{N}_T^2\le T_{C_1}\\ {\sigma }_{C_2-{Ĉ}_2}^2=\frac{1}{3}{N}_M^2+\frac{1}{2}{N}_S^2+\frac{1}{6}{N}_T^2\le T_{C_2}\\ {\sigma }_{C_3-{Ĉ}_3}^2=\frac{1}{3}{N}_M^2+\frac{2}{3}{N}_T^2\le T_{C_3}\end{array}$$

(13)

where ${\sigma }_{C_1-{Ĉ}_1}^2$, ${\sigma }_{C_2-{Ĉ}_2}^2$ and ${\sigma }_{C_3-{Ĉ}_3}^2$ are the expected noise energy for original signals. $T_{C_1}$, $T_{C_2}$ and $T_{C_3}$ are the masking thresholds calculated for the original signals. The sufficient conditions that guarantee the expected noise energy will not exceed masking threshold are $N_M^2,N_S^2,N_T^2\le \text{min}(T_{C_1},T_{C_2},T_{C_3})$. This means the noise energy for any transformed signal must be less than the minimum threshold of their original signals. So the masking thresholds for the transformed signals can be derived from the masking threshold of the original signals as

$$T_{C_M}=T_{C_S}=T_{C_T}\le \text{min}(T_{C_1},T_{C_2},T_{C_3})$$

(14)

The same results can be deduced for other matrices.

Framework for general channel configuration

3D-M/S only works for three channels, actually all 3D audio systems contain dozens of channels. A general channel configuration is shown in Figure 4, where channel C corresponds to a loudspeaker. Here, a framework is proposed based on 3D-M/S coding as shown in Figure 5. Because all spatially placed loudspeakers can be decomposed into basic triangle units, this structure will enable 3D-M/S coding to work for arbitrary channel configurations. The framework processes the audio channels triangle by triangle until all channels are coded. C _M is the summation channel and C _S and C _T are the second and third channels, respectively. Every 3D-M/S unit shares two channels with the previous unit and only one new channel is added in. So, it only needs to compress the channel which contains the signal of the new channel. For all matrices M₀, M₁, M₂, M₃ and M₄, C _T is the third channel after 3D-M/S transform. Because every unit outputs only one that contains a new input channel, the whole coding framework keeps the number of channels exactly the same as original input signals. And because the output channel contains either the difference signal or original signal, coding gain can be obtained. The original signals can be obtained by multiplying 3D-M/S inverse transform matrix subband by subband at the decoder side. This framework is also suitable for other methods. For example, replacing the 3D-M/S with PCA, the codec can achieve better interchannel redundancy removing performance.

Figure 4

The general plane structure of 3D multichannel systems. A general plane structure for 3D multichannel audio systems and experimental sound source setting.

Figure 5

Three-channel unit-based coding framework. The framework using 3D-M/S coding for N channels.

Experiment

The experiment used five channels (C₁, C₂, C₃, C₄, C₅) in spherical 22.2 multichannel configuration as shown in Figure 4. Considering that PCA is the best decorrelation transform theoretically and Independent channel coding is widely used for 22.2 multichannel compression, the experiment compared the proposed 3D-M/S method with PCA and Independent channel coding in bitrate, complexity and objective quality. Three MPEG test sequences (es01 voice signal, sc03 symphony music signal, si02 castanets transient signal, mono 48-kHz sampling) were used as the moving virtual sources following the VBAP rule, four sequences (si03, si01, sc01, es02) were used as the discrete fixed-position virtual sources. The virtual sources and respective azimuth and altitude panning angle are generated on a per-frame basis. Here, only point virtual sources were used to test the best performance of three methods, as subband signals can be regarded as point sources in subband coding when bandwidths are small enough. Signals with decorrelated elements are beyond the scope of VBAP model and will decrease the coding performance, for its difference signals retains high energy which depends on the correlation and the energy of the decorrelated elements. Uncorrelated signals with independent audio content is tested in the end.

Three basic virtual sound movements were used to cover some basic possible virtual source locations. The three movements in a triangle are movement 1 from point to point (virtual source es01), movement 2 from point to edge (virtual source sc03) and movement 3 from edge to edge (virtual source si02). Three movements are as shown in Figure 4. Different sources were used for different panning patterns, because the experiments were designed to compare three methods on the same condition and change the condition simultaneously to see how three methods work on different kinds of sources and locations, instead of comparing the performance of one method on different virtual source locations with the same source. Considering the symmetry in the triangle, 10 discrete virtual source positions in $\frac{1}{6}$ triangle as shown in Table 2 and Figure 4 are used which equally divided the $\frac{1}{6}$ triangle: si03 was panned to $\left(\theta =\frac{11\pi }{36},\phi =\frac{7\pi }{32}\right)$; si01 was divided into two sequences and panned to $\left(\theta =\frac{13\pi }{36},\phi =\frac{7\pi }{32}\right)$, $\left(\theta =\frac{13\pi }{36},\phi =\frac{5\pi }{32}\right)$; sc01 was divided into three sequences and panned to $\left(\theta =\frac{15\pi }{36},\phi =\frac{7\pi }{32}\right)$, $\left(\theta =\frac{15\pi }{36},\phi =\frac{5\pi }{32}\right)$ and $\left(\theta =\frac{15\pi }{36},\phi =\frac{3\pi }{32}\right)$; es02 was divided into four sequences and panned to $\left(\theta =\frac{17\pi }{36},\phi =\frac{7\pi }{32}\right)$, $\left(\theta =\frac{17\pi }{36},\phi =\frac{5\pi }{32}\right)$, $\left(\theta =\frac{17\pi }{36},\phi =\frac{3\pi }{32}\right)$ and $\left(\theta =\frac{17\pi }{36},\phi =\frac{\pi }{32}\right)$.

Table 2 Discrete virtual source setting and objective results

The 3D-M/S and PCA was used in each subband in the frequency domain. The three encoders were realized based on FAAC-1.28, and decoders were based on FAAD2-2.7. AAC-LC was used as the core codec and only the long window was enabled for simplification. To avoid the influence of dynamic bandwidth setting of the FAAC, the experiment fixed the bandwidth at 12 kHz with 35 subbands.

Independent channel coding: Audio signals were sent into the core codec and compressed directly.

3D-M/S: The vector was calculated using the subband energy of three channels from AAC psychoacoustic module with no extra energy computation. Then 3D-M/S matrix switching was performed and 3 bits were used per mode parameter. The transformed signals were sent into the core codec, and the masking threshold was modified accordingly.

PCA: The eigenvectors were calculated for each subband. Subband signals were transformed using eigenvector matrix and then sent into core codec. The covariance matrix was quantized and transmitted to the decoder following a previous KLT-based multichannel audio coding scheme [21], with 4 bits per non-redundant element.

Objective evaluation

Complexity was measured by the running time of each method on PC (CPU: Intel Core2 Duo P8600 2.53GHz, RAM: 8GB). The main application scenario of the proposed method is medium and high bitrate, and the sound pressure at the center listening point is the most important in multichannel audio and the VBAP panning method. Here we compared the sound pressure SNR (original and reconstructed signal sound pressure in the center listening point) to measure the objective quality following the sound pressure equation in [33]

$$p\left(\omega \right)=G\sum _{j=1}^5\frac{e^{-\mathit{\text{ik}}{r}_j}}{r_j}{C}_j\left(\omega \right),$$

(15)

where p (ω) is the sound pressure, G is a proportionality coefficient, $k=\frac{\omega }{c}$ is the wave number and c is the wave speed. r is the distance from the loudspeaker to the listening point, and in the spherical 22.2 multichannel, all loudspeakers have the same distance. And the SNR is calculated by

$$\text{SNR}=\frac{\underset{0}{\overset{2\pi }{\int }}{\left|p\left(\omega \right)\right|}^{2}d\omega }{\underset{0}{\overset{2\pi }{\int }}{\left|p\left(\omega \right)-\widehat{p}\left(\omega \right)\right|}^2\mathrm{d\omega }}$$

(16)

Firstly, the bitrates of all three methods have been set to be nearly identical to compare the performance shown in Table 3. The bitrates were adopted by modifying the masking threshold in core codec and iterating two nested loops until rate constrain satisfied. Figure 6 shows the SNR curves for each method, where each SNR curve was smoothed to omit the details. The smoothing method is a typical average filtering over 20 frames length. An overall SNR was presented in Table 3 to give an overview by averaging the SNR for all frames. The three SNR curves have a downtrend because transient signal and symphony signal have a wider bandwidth than the voice signal, which require more bits to achieve the similar SNR. When the virtual source came close to the middle of the two channels (between C₁, C₂ around the 200th frame in Mov. 1, between C₃, C₄ for all frames in Mov. 2, between C₃, C₄ around the start frames, between C₃, C₅ around the 100th frame and between C₄, C₅ around the end frames in Mov. 3), 3D-M/S gets a higher SNR than the Independent channel coding and close to PCA. Moveover, around the 200th frame in Mov. 1 and Mov. 2, where all two and three channels are nearly the same, M ₃ and M ₄ can remove redundancy to the largest extent and outperform the PCA method. This is because some transformed subband signals came below the masking threshold and more bits were reserved for summation channel. The same results can be seen in the discrete virtual sources in Table 2, where $\phi =\frac{7\pi }{32}$ ($\phi \approx \frac{\pi }{4}$) and ($\theta =\frac{11\pi }{36},\phi =\frac{7\pi }{32}$) ($\theta =\frac{13\pi }{36},\phi =\frac{5\pi }{32}$) ($\theta =\frac{15\pi }{36},\phi =\frac{3\pi }{32}$) ($\theta =\frac{17\pi }{36},\phi =\frac{\pi }{32}$) (cot θ ≈ sin φ). But when the virtual source located beyond the middle of two channels, such as ($\theta =\frac{15\pi }{36},\phi =\frac{5\pi }{32}$), M/S coding cannot bring coding gain. In conclusion, if the input signals are located in one of the five switching areas, the coding gain can be obtained by transforming them into summation and difference signals.

Table 3 Bitrate setup and overall SNR

Figure 6

Objective evaluation results. Average SNR (12 kHz bandwidth) per frame for Independent channel coding, 3D-M/S and PCA methods in movements 1, 2 and 3, uncorrelated content and discrete positions 1, 2, 3 and 4.

Secondly, the PCA parameter bitrate of 39.3 kbps/channel is considerably higher than 3D-M/S. If the three channels have little correlation (e.g. channels with different contents or ambient sound), the transformed signals will not save any bits and cause the decrease of coding efficiency. To test the three methods under such condition, the virtual sources of three different signals were fixed at three channels and coded all at 64 kbps. The experimental result is shown in Figure 6. We can see Independent channel coding achieves the best performance in this case; meanwhile, 3D-M/S degrades about 1 dB and PCA degrades nearly 7 dB. It is because, for PCA requirement, too many bits are used for parameters which now cannot bring any coding gain. But for 3D-M/S, parameter bits for modes are only 4.9 kbps/channel. It will not reduce the coding efficiency much for medium and high bitrate conditions, which is the main application scenario of M/S coding. Although the high bitrate for PCA can be alleviated by reducing the refresh rate of PCA parameters, but it will decrease the coding performance on VBAP signals at the same time.

Finally, Table 4 demonstrates the complexity of each method. Due to the computation for covariance matrix and eigenvector matrix, PCA increases about 30% complexity compared with the original AAC codec. For 3D-M/S, the matrix switching and signal transformation increased by about 11% complexity.

Table 4 Time complexity

Subjective evaluation

Eight subjects who are actively working in the domain of audio compression participated in the subjective test based on ITU MUSHRA [34] method using a 3.5-kHz low-pass filtered original channels as anchor, and the test was carried out in a quite room with five-spherical channel configuration as shown in Figure 7. Subjects were required to evaluate the sound quality and sound orientation separately, and can draw the perceived sound position and movement to help their rating.

Figure 7

Subjective test setup. Subjective test environment and configuration.

The MUSHRA test results are shown in Figure 8. For the sound quality, the subjective result generally matches the SNR result where PCA and 3D-M/S got a subjective quality improvement compared with the Independent channel coding. For the sound orientation, subjects reported that the voice and castanet signals were easier to locate. Although the 3.5-kHz anchor signal was poor in sound quality test, some subjects indicated its virtual source position was not so bad in the movement than the discrete virtual source. This may be because it is easier to distinguish the direction change for fixed sound source than the moving sound source. For 3D-M/S, subjects felt the virtual source had a sharper position in the center of two loudspeakers compared with Independent channel coding. On the whole, the subjective improvement in sound orientation exists but is not as obvious as the sound quality.

Figure 8

Subjective evaluation results. MUSHRA test results on sound quality and orientation for Independent channel coding, 3D-M/S and PCA methods in movements 1, 2 and 3 and discrete positions 1, 2, 3 and 4 with 95% confidence interval and the mean.

From the above results on three point sources and uncorrelated signals, it can be observed that both PCA and 3D-M/S method get about 13% SNR improvement for each channel. But the complexity of 3D-M/S is much lower than PCA to achieve similar performance. It can be explained that the fixed matrix transform can be regarded as some special vectors in PCA. The special vectors are chosen based on the assumption that channel signals are either quite similar or quite different. This assumption may not be always true for the diversity of subband signals, but it makes a good compromise between coding efficiency and complexity.

Conclusion

This paper proposed a 3D-M/S coding method, which inherits the low complexity of conventional M/S coding. Moreover, 3D-M/S performs the sum and difference coding triple by triple, rather than couple by couple of the conventional method. This structure is more suitable for a 3D multichannel audio configuration, because adjacent three channels form a triangle and will have the maximum redundancy in spatial configured 3D audio channels. Besides, it is also convenient to unfold 3D audio multichannel structure into plane triangles. Combining the proposed framework, 3D-M/S and PCA methods can be applied to more than three channels. An experiment on VBAP signals indicates the performance of proposed method with relatively low complexity, comparing to the PCA and independent channel coding. Considering the development of 3D audio technology and its requirement for compression efficiency, a low complexity 3D audio codec will be promising and preferable for practical application.