A multimodal tempo and beat-tracking system based on audiovisual information from live guitar performances

2.1 Definition of the musical ensemble with guitar

Our targeted musical ensemble consists of a melody player and a guitarist and assumes quadruple rhythm for simplicity of the system. Our beat-tracking method can accept other rhythms by adjusting the hand's trajectory model explained in Section 3.2.3.

At the beginning of a musical ensemble, the guitarist gives some counts to synchronize with a co-player as he would in real ensembles. These counts are usually given by voice, gestures or hit sounds from the guitar. We determine the number of counts as four and consider that the tempo of the musical ensemble can be only altered moderately from the tempo implied by counts.

Our method estimates the beat timings without prior knowledge of the co-player's score. This is because (1) many guitar scores do not specify beat patterns but only melody and chord names, and (2) our main goal focuses on improvisational sessions.

Guitar playing is mainly categorized into two styles: stroke and arpeggio. Stroke style consists of hand waving motions. In arpeggio style, however, a guitarist pulls strings with their fingers mostly without moving their arms. Unlike most beat-trackers in the literature, our current system is designed for a much more limited case where the guitar is strummed, not in a finger picked situation. This limitation allows our system to perform well in a noisy environment, to follow sudden tempo changes more reliably and to address single instrument music pieces.

Stroke motion has two implicit rules, (1) beginning with a down stroke and (2) air strokes, that is, strokes with a soundless tactus, to keep the tempo stable. These can be found in the scores, especially pattern 4 for air strokes, in Figure 1. The arrows in the figure denote the stroke direction, common enough to appear on instruction books for guitarists. The scores say that strokes at the beginning of each bar go downward, and the cycle of a stroke usually lasts the length of a quarter note (eight beats) or of an eighth note (sixteen beats). We assume music with eight-beat measures and model the hand's trajectory and beat locations.

No prior knowledge on the color of hands is assured in our visual-tracking. This is because humans have various hand colors and such colors vary according to the lighting conditions. The motion of the guitarist's arm, on the other hand, is modeled with prior knowledge: the stroking hand makes the largest movement in the body of a playing guitarist. The conditions and assumptions for guitar ensemble are summarized below:

Conditions and assumptions for beat-tracking

2.2 Beat-tracking conditions

Our beat-tracking method estimates the tempo and bar-position, the location in the bar at which the performer is playing at a given time from audio and visual beat features. We use a microphone and a camera embedded in the robot's head for the audio and visual input signal, respectively. We summarize the input and output specifications in the following box:

Input-output

2.3 Challenges for guitar beat-tracking

A human guitar beat-tracking must overcome three problems to cope with tempo fluctuation, beat pattern complexity, and environmental noise. The first problem is that, since we do not assume a professional guitarist, a player is allowed to play fluid tempos. Therefore, the beat-tracking method should be robust to such changes of tempo.

The second problem is caused by (1) beat patterns complicated by upbeats (syncopation) and (2) the sparseness of onsets. We give eight typical beat patterns in Figure 1. Patterns 1 and 2 often appear in popular music. Pattern 3 contains triplet notes. All of the accented notes in these three patterns are down beats. However, the other patterns contain accented upbeats. Moreover, all of the accented notes of patterns 7 and 8 are upbeats. Based on these observations, we have to take into account how to estimate the tempos and bar-positions of the beat patterns with accented upbeats.

The sparseness is defined as the number of onsets per time unit. We illustrate the sparseness of onsets in Figure 3. In this paper, guitar sounds consist of a simple strum, meaning low onset density, while popular music has many onsets as is shown in the Figures. The figure shows a 62-dimension mel-scaled spectrogram of music after the Sobel filter [5]. The Sobel filter is used for the enhancement of onsets. Here, the negative values are set to zero. The concentration of darkness corresponds to strength of onset. The left one, from popular music, has equal interval onsets including some notes between the onsets. On the other hand, the right one shows an absent note compared with the tactus. Such absences mislead a listener of the piece as per the blue marks in the figure. What is worse, it is difficult to detect the tactus in a musical ensemble with few instruments because there are few supporting notes to complement the syncopation; for example, the drum part may complement the notes in larger ensembles.

As for the third problem, the audio signal in beat-tracking of live performances includes two types of noise: stationary and non-stationary noise. In our robot application, the non-stationary noise is mainly caused by the robot joints' movement. This noise, however, does not affect beat-tracking, because it is small--6.68 dB in signal-to-noise ratio (SNR)--based on our experience so far. If the robot makes loud noise when moving, we may apply Ince's method [6] to suppress such ego noise. The stationary noise is mainly caused by fans on the computer in the robot and environmental sounds including air-conditioning. Such noise degrades the signal-to-noise ratio of the input signal, for example, 5.68dB in SNR, in our experiments with robots. Therefore, our method should include a stationary noise suppression method.

We have two challenges for visual hand tracking: false recognition of the moving hand and low time resolution compared with the audio signal. A naive application of color histogram-based hand trackers is vulnerable to false detections caused by the varying luminance of the skin color and thus captures other nearly skin-colored objects. While optical-flow-based methods are considered suitable for hand tracking, we have difficulty in employing this method because flow vectors include some noise from the movements of other parts of the body. Usually, audio and visual signals have different sampling rates from one another. According to our setting, the temporal resolution of a visual signal is about one-quarter compared to an audio signal. Therefore, we have to synchronize these two signals to integrate them.

problems

2.4 Related research and solution of the problems

3.1 Audio beat feature extraction with STPM

where peak is a variable for normalization and is updated under the local peak of onsets. The N _F denotes the number of dimensions of onset vectors used in NCC and N _P denotes the frame size of pattern matching. We set these parameters to 62 dimensions and 87 frames (equivalent to 1 sec.) according to Murata et al. [3].

3.2 Visual beat feature extraction with hand tracking

We extract the visual beat features, that is, the temporal sequences of hand positions with these three methods: (1) hand candidate area estimation by optical flow, (2) hand position estimation by mean shift, and (3) hand position tracking.

3.2.1 Hand candidate area estimation by optical flow

We use Lucas-Kanade (LK) method [21] for fast optical-flow calculation. Figure 4 shows an example of the result of optical-flow calculation. We define the center of hand candidate area as a coordinate of the flow vector, which has the length and angle nearest from the middle values of flow vectors. This is because the hand motion should have the largest flow vector according to the assumption (3) in Section 2.1, and this allows us to remove noise vectors with calculating the middle values.

3.2.3 Hand position tracking

Let (h _x,t , h _y,t ) be the hand coordination calculated by the mean shift. Since a guitarist usually moves their hand near the neck of their guitar, we define r _t , a hand position at t time frame, as the relative distance between the hand and the neck as follows:

$r_t={\rho }_t-\left(h_{x,t}\text{cos}{\theta }_t+h_{y,t}\text{sin}{\theta }_t\right),$

(8)

where ρ _t and θ _t are the parameters of the line of the neck computed with Hough transform [23] (see Figure 5a for an example). In Hough transform, we compute 100 candidate lines, remove outliers with RANSAC [24], and get the average of Hough parameters. Positive values indicate that a hand is above the guitar; negative values indicate below. Figure 5b shows an example of the sequential hand positions.

Now, let ω _t and θ _t be a beat interval and bar-position at the t th time frame, where a bar is modeled as a circle, 0 ≤ θ _t < 2π and ω _t is inversely proportional to the angle rate, that is, tempo. With assumption 3 in Section 2.1, we presume that down strokes are at θ _t = nπ/2 and up strokes are at θ _t = nπ/2 + π/4(n = 0,1,2,3). In other words, zero crossover points of hand position are at these θ. In addition, since a hand stroking is in a smooth motion to keep the tempo stable, we assume that the sequential hand position can be represented with a continuous function. Thus, hand position r _t is defined by

$r_t=-a\text{sin}\left(4{\theta }_t\right),$

(9)

where a is a constant value of hand amplitude and is set to 20 in this paper.

4.1 Overview of the particle-filter model

The graphical representation of the particle-filter model is outlined in Figure 6. The state variables, ω _t and θ _t , denote the beat interval and bar-position, respectively. The observation variables, R _t (k), F _t , and r _t denote inter-frame correlation with k frames back, normalized onset summation, and hand position, respectively. The $w_t^{\left(i\right)}$ and ${\theta }_t^{\left(i\right)}$ are parameters of the i th particle. Now, we will explain the estimation process with the particle filter.

4.2 State transition with sampling

where b denotes a constant for transforming from beat interval into an angle rate of the bar-position.

We will now discuss Eqs. (10) and (11). In Eq. (10), the first term R _t (k) is multiplied with two window functions of different means. The first is calculated from the previous frame and the second is from the counts. In Eq. (11), penalty(θ|r, F) is the result of five multiplied multipeaked window functions. Each function has a condition. If it is satisfied, the function is defined by the von Mises distribution; otherwise, it shows 1 in any θ. This penalty function pulls the peak of the θ distribution into its own peak and modifies the distribution to match it with the assumptions and the models. Figure 7 shows the change in the θ distribution by multiplying the penalty function.

All β parameters are set experimentally through a trial and error process. thresh. is a threshold that determines whether F _t is constant noise or not. Eqs. (14) and (15) are determined with the assumption of zero crossover points of stroking. Eqs. (16) and (17) are determined with the stroking directions. These four equations are based on the model of the hand's trajectory presented in Eq. (9). Equation (18) is based on eight beats; that is, notes should be on the tops of the modified von Mises function which has eight peaks.

4.3 Weight calculation

The terms of the numerator in Eq. (19) are called a state transition model function and a observation model function. The more the values of a particle match each model, the larger value its weight has with the high probabilities of these functions. The denominator is called a proposal distribution. When a particle of low probability is sampled, its weight increases with the low value of the denominator.

where N _th is a threshold for resampling and is set to 1.

5.1 Experimental setup

We asked four guitarists to perform one of each eight kinds of the beat patterns given in Figure 1, at three different tempos (70, 90, and 110), for total of 96 samples. The beat patterns are enumerated in order of beat pattern complexity; a smaller index number indicates that the pattern includes more accented down beats which is easily tracked, while a larger index number indicates that the pattern includes more accented upbeats that confuse the beat-tracker. A performance consists of four counts, seven repetitions of the beat pattern, one whole note and one short note, shown in Figure 8. The average length of each sample was 30.8[sec] for 70 bpm, 24.5[sec] for 90 bpm and 20.7[sec] for 110. The camera recorded frames at about 19 [fps]. The distance between the robot and a guitarist was about 3 [m] so that the entirety of the guitar could be placed inside the camera frame. We use a one-channel microphone and the sampling parameters shown in Section 3.1 Our method uses 200 particles unless otherwise stated. It was implemented in C++ on a Linux system with an Intel Core2 processor. Table 1 shows the parameters of this experiment. The unit of the parameter relevant to θ is [deg] that ranges from 0 to 360. They all are defined experimentally through a trial and error process.

In order to evaluate the accuracy of beat-tracking methods, we use the following thresholds to define successful beat detection and tempo estimations from ground truth: 150 msec for detected beats and 10 bpm for estimated tempos, respectively.

where N _e , N _d , and N _c correspond to the number of correct estimates, whole estimates and correct beats, respectively. AMLc is the ratio of the longest continuous correctly tracked section to the length of the music, with beats at allowed metrical levels. For example, one inaccuracy in the middle of a piece leads to 50% performance. This represents that the continuity is in correct beat detections and is critical factor in the evaluation of musical ensembles.

The beat detection errors are divided into three classes: substitution, insertion and deletion errors. Substitution error means that a beat is poorly estimated in terms of the tempo or bar-position. Insertion errors and deletion errors are false-positive and false-negative estimations. We assume that a player does not know the other's score, thus one estimates score position by number of beats from the beginning of the performance. Beat insertions or deletions undermine the musical ensemble because the cumulative number of beats should be correct or the performers will lose synchronization. Algorithm 1 shows how to detect inserted and deleted beats. Suppose that a beat-tracker correctly detects two beats with a certain false estimation between them. When the method just incorrectly estimates a beat there, we regard it as a substitution error. In the case of no beat or two beats there, they are counted as a deleted or inserted beats, respectively.

5.2 Comparison of audiovisual particle filter, audio only particle filter, and Murata's method

Table 2 and Figure 9 summarize the precision, recall and F-measure of each pattern with our audiovisual integrated beat-tracking (Integrated), audio only particle filter (Audio only) and Murata's method (Murata). Murata does not show any variance in its result, that is, no error bars in result figures because its estimation is a deterministic algorithm, while the first two plots show variance due to the stochastic nature of particle filters. Our method Integrated stably produces moderate results and outperforms Murata for patterns 4-8. These patterns are rather complex with syncopations and downbeat absences. This demonstrates that Integrated is more robust against beat patterns than Murata. The comparison between Integrated and Audio only confirms that the visual beat features improve the beat-tracking performance; Integrated improves precision, recall, and F-measure by 24.9, 26.7, and 25.8 points in average from Audio only, respectively.

(a) Precision (%)
Beat Pattern	1	2	3	4	5	6	7	8	Ave.
Integrated	69.9	75.7	71.1	65.1	48.3	46.8	74.0	40.1	61.4
Audio only	43.6	46.6	45.6	28.7	24.7	18.1	43.6	41.5	36.5
Murata	86.3	82.4	83.2	44.1	39.9	22.4	25.5	22.3	50.8
(b) Recall (%)
Beat Pattern	1	2	3	4	5	6	7	8	Ave.
Integrated	70.3	75.8	71.9	66.0	47.7	45.6	74.5	39.7	61.4
Audio only	40.8	43.9	42.5	28.7	23.4	17.9	41.6	38.8	34.7
Murata	89.6	87.1	87.0	48.8	43.7	26.7	27.2	24.4	54.3
(c) F-measure (%)
Beat Pattern	1	2	3	4	5	6	7	8	Ave.
Integrated	70.1	75.7	71.5	65.5	48.0	46.1	74.2	39.9	61.4
Audio only	42.2	45.2	44.0	28.7	24.0	18.0	42.6	40.1	35.6
Murata	87.9	84.7	85.1	46.3	41.7	24.3	26.3	23.3	52.5

Bold numbers represent the largest results for each beat pattern.

The F-measure scores of the patterns 5, 6, and 8 decrease for Integrated. The following mismatch causes this degradation; though these patterns contain sixteenth beats that make the hand move at double speed, our method assumes that the hand always moves downward only at quarter note positions as Eq. (9) indicates. To cope with this problem, we should allow for downward arm motions at eighth notes, that is, sixteen beats. However, a naive extension of the method would result in degraded performances with other patterns.

The average of F-measure for Integrated shows about 61%. The score is deteriorated due to these two reasons: (1) the hand's trajectory model does not match the sixteen-beat patterns, and (2) the low resolution and the error in estimating visual beat feature extraction do not make the penalty function effective in modifying the θ distribution.

Table 3 and Figure 10 present the AMLc comparison among the three method. As well as the F-measure result, Integrated is superior to Murata for patterns 4-8. The AMLc results of patterns 1 and 3 are not so high despite the high F-measure score. Here, we define result rate as the ratio of the AMLc score to the F-measure one. In patterns 1 and 3, the result rates are not so high, 72.7 and 70.8. Likewise the F-measure results, the result rates of patterns 4 and 5 remark lower scores, 48.9 and 55.8. On the other hand, the result rates of patterns 2 and 7 show still high percentage as 85.0 and 74.7. The hand's trajectory of patterns 2 and 7 is approximately the same with our model, a sign curve. In pattern 3, however, the triplet notes affect the trajectory to be late in the upward movement. In pattern 1, no upbeats, that is, no constraints in the upward movement allow the hand to move loosely upward in comparison with the trajectories in other patterns. To conclude, the result rate has a relationship with the similarity of a hand's trajectory of each pattern with our model. The model should be refined to raise scores in our future work.

Beat Pattern	1	2	3	4	5	6	7	8	Ave.
Integrated	49.9	64.2	50.0	43.6	22.8	25.3	54.8	26.5	42.1
Audio only	18.6	18.0	17.6	16.8	14.7	18.5	18.3	16.6	17.4
Murata	84.2	68.9	78.6	24.1	11.0	8.4	19.4	16.9	38.9

Bold numbers represent the largest results for each beat pattern.

In Figure 11, Integrated demonstrates less errors than Murata with regard to the total errors of insertions and deletions. A detailed analysis shows that Integrated has less deletion errors than Murata in some patterns. On the other hand, Integrated has more insertion errors than Murata, especially in sixteen beats. However, the adaption to sixteen beats would produce fewer insertions in Integrated.

5.3 The influence of the number of particles

Table 4 also shows that the F-measures differ by only about 1.3% between 400 particles showing the maximum result and 200 particles where the system works in real-time. This suggests that our system is capable of real-time processing with almost saturated performance.

5.4 Results with various subjects

Figure 12 indicates that we can observe only little difference among the subjects except Subject 3. In the case of Subject 3, the similarity of the skin color to the guitar caused frequent loss of the hand's trajectory. To improve the estimation accuracy, we should tune the algorithm or parameters to be more robust against such confusion.

5.5 Evaluation using a robot

Our system was implemented on a humanoid robot HRP-2 that plays an electronic instrument called the theremin as in Figure 13. The video is available on Youtube [26]. The humanoid robot HRP-2 plays the theremin with a feed-forward motion control developed by Mizumoto et al. [27]. HRP-2 captures a mixture of sound consisting of its own theremin performance and human partner's guitar performance with its microphones. HRP-2 first suppresses its own theremin sounds by using the semi-blind ICA [28] to obtain the audio signal played by the human guitarist. Then, our beat-tracker estimates the tempo of the human performance and predicts the tactus. According to the predicted tactus, HRP-2 plays the theremin. Needless to say, this prediction is coordinated to absorb the delay of the actual movement of the arm.