Combined perception and control for timing in robotic music performances
 Umut Şimşekli^{1}Email author,
 Orhan Sönmez^{1},
 Barış Kurt Kurt^{1} and
 Ali Taylan Cemgil^{1}
https://doi.org/10.1186/1687472220128
© Şimşekli et al; licensee Springer. 2012
Received: 16 April 2011
Accepted: 3 February 2012
Published: 3 February 2012
Abstract
Interaction with human musicians is a challenging task for robots as it involves online perception and precise synchronization. In this paper, we present a consistent and theoretically sound framework for combining perception and control for accurate musical timing. For the perception, we develop a hierarchical hidden Markov model that combines event detection and tempo tracking. The robot performance is formulated as a linear quadratic control problem that is able to generate a surprisingly complex timing behavior in adapting the tempo. We provide results with both simulated and real data. In our experiments, a simple Lego robot percussionist accompanied the music by detecting the tempo and position of clave patterns in the polyphonic music. The robot successfully synchronized itself with the music by quickly adapting to the changes in the tempo.
Keywords
hidden Markov models Markov decision processes Kalman filters robotic performance1 Introduction
With the advances in computing power and accurate sensor technologies, increasingly more challenging tasks in humanmachine interaction can be addressed, often with impressive results. In this context, programming robots that engage in music performance via realtime interaction remained as one of the challenging problems in the field. Yet, robotic performance is criticized for being to mechanical and robotic [1]. In this paper, we therefore focus on a methodology that would enable robots to participate in natural musical performances by mimicking what humans do.
Humanlike musical interaction has roughly two main components: a perception module that senses what other musicians do and a control module that generates the necessary commands to steer the actuators. Yet, in contrast to many robotic tasks in the real world, musical performance has a very tight realtime requirement. The robot needs to be able to adapt and synchronize well with the tempo, dynamics and rhythmic feel of the performer and this needs to be achieved within hard realtime constraints. Unlike repetitive and dull tasks, such expressive aspects of musical performance are hard to formalize and realize on real robots. The existence of humans in the loop makes the task more challenging as a human performer can be often surprisingly unpredictable, even on seemingly simple musical material. In such scenarios, highly adaptive solutions, that combine perception and control in an effective manner, are needed.
Our goal in this paper is to illustrate the coupling of perception and control modules in music accompaniment systems and to reveal that even with the most basic hardware, it is possible to carry out this complex task in real time.
In the past, several impressive demonstrations of robotic performers have been displayed, see Kapur [2] as a recent survey. The improvements in the field of humancomputer interaction and interactive computer music systems influenced the robotic performers to listen and respond to human musicians in a realistic manner. The main requirement for such an interaction is a tempo/beat tracker, which should run in realtime and enable the robot to synchronize well with the music.
As a pioneering work, Goto and Muraoka [3] presented a realtime beat tracking for audio signals without drums. Influenced by the idea of an untrained listener can track the musical beats without knowing the names of the chords or the notes being played, they based their method on detecting the chord changes. The method performed well on popular music; however, it is hard to improve or adapt the algorithm for a specific domain since it was built on top of many heuristics. Another interesting work on beat tracking was presented in Kim et al. [4], where the proposed method estimates the tempo of rhythmic motions (like dancing or marching) through a visual input. They first capture the 'motion beats' from sample motions in order to capture the transition structure of the movements. Then, a new rhythmic motion synchronized with the background music is synthesized using this movement transition information.
An example of an interactive robot musician was presented by Kim et al. [5], where the humanoid robot accompanied the playing music. In the proposed method, they used both audio and visual information to track the tempo of the music. In the audio processing part, an autocorrelation method is employed to determine the periodicity in the audio signal, and then, a corresponding tempo value is estimated. Simultaneously, the robot tracks the movements of a conductor visually and makes another estimation for the tempo [6]. Finally, the results of these two modules are merged according to their confidences and supplied to the robot musician. However, this approach lacks an explicit feedback mechanism which is supposed to handle the synchronization between the robot and the music.
In this paper, rather than focusing on a particular piece of custom build hardware, we will focus on a deliberately simple design, namely a Lego robot percussionist. The goal of our percussionist will be to follow the tempo of a human performer and generate a pattern to play in sync with the performer. A generic solution to this task, while obviously simpler than that for an acoustic instrument, captures some of the central aspects or robotic performance, namely:

Uncertainties in human expressive performance

Superpositionsounds generated by the human performer and robot are mixed

Imperfect perception

Delays due to the communication and processing of sensory data

Unreliable actuators and hardwarenoise in robot controls causes often the actual output to be different than the desired one.
Our ultimate aim is to achieve an acceptable level of synchronization between the robot and a human performer, as can be measured via objective criteria that correlate well with human perception. Our novel contribution here is the combination of perception and control in a consistent and theoretically sound framework.
For the perception module, we develop a hierarchical hidden Markov model (a changepoint model) that combines event detection and tempo tracking. This module combines the template matching model proposed by Şimşekli and Cemgil [7] and the tempo tracking model by Whiteley et al. [8] for event detection in sound mixtures. This approach is attractive as it enables to separate sounds generated by the robot or a specific instrument of the human performer (clave, hihat) in a supervised and online manner.
The control model assumes that the perception module provides information about the human performer in terms of an observation vector (a bar position/tempo pair) and an associated uncertainty, as specified possibly by a covariance matrix. The controller combines the observation with the robots state vector (here, specified as an angularposition/angularvelocity pair) and generates an optimal control signal in terms of minimizing a cost function that penalizes a mismatch between the "positions" of the robot and the human performer. Here, the term position refers to the score position to be defined later. While arguably more realistic and musically more meaningful cost functions could be contemplated, in this paper, we constrain the cost to be quadratic to keep the controller linear.
A conceptually similar approach to ours was presented by Yoshii et al. [9], where the robot synchronizes its steps with the music by a realtime beat tracking and a simple control algorithm. The authors use a multiagent strategy for realtime beat tracking where several agents monitor chord changes and drum patterns and propose their hypotheses; the most reliable hypothesis is selected. While the robot keeps stepping, the step intervals are sent as control signals from a motion controller. The controller calculates the step intervals in order to adjust and synchronize the robots stepping tempo together with beat timing. Similar to this work, Murata et al. [10] use the same robotic platform and controller with an improved beattracking algorithm that uses a spectrotemporal pattern matching technique and echo cancelation. Their tracking algorithm deals better with environmental noise and responds faster to tempo changes. However, the proposed controller only synchronizes the beat times without considering which beat it is. This is the major limitation of these systems since it may allow phase shifts in beats if somebody wants to synchronize a whole musical piece with the robot.
Our approach to tempo tracking is also similar to the musical accompaniment systems developed by Dannenberg [11], Orio [12], Cemgil and Kappen [13], Raphael [14], yet it has two notable novelties. The first one is a novel hierarchical model for accurate online tempo estimation that can be tuned to specific events, while not assuming the presence of a particular score. This enables us to use the system in a natural setting where the sounds generated by the robot and the other performers are mixed. This is in contrast to existing approaches where the accompaniment only tracks a target performer while not listening to what it plays. The second novelty is the controller component, where we formulate the robot performance as a linear quadratic control problem. This approach requires only a handful of parameters and seems to be particularly effective for generating realistic and humanlike expressive musical performances, while being fairly straightforward to implement.
The paper is organized as follows. In the sequel, we elaborate on the perception module for robustly inferring the tempo and the beat from polyphonic audio. Here, we describe a hierarchical hidden Markov model. Section 3 introduces briefly the theory of optimal linear quadratic control and describes the robot performance in this framework. Sections 4 describes simulation results. Section 5 describes experiments with our simple Lego robot system. Finally Section 6 describes the conclusions, along with some future directions for further research.
2 The perception model
In this study, the aim of the perception model is to jointly infer the tempo and the beat position (score position) of a human performer from streaming polyphonic audio data in an online fashion. Here, we assume that the observed audio includes a certain instrument that carries the tempo information such as a hihat or a bass drum. We assume that this particular instrument is known beforehand. The audio can include other instrument sounds, including the sound of the percussion instrument that the robot plays.
As the scenario in this paper, we assume that the performer is playing a clave pattern. The claves is the name for both a wooden percussive instrument and a rhythmic pattern that organizes the temporal structure and forms the rhythmic backbone in AfroCuban music. Note that, this is just an example, and our framework can be easily used to track other instruments and/or rhythmic patterns in a polyphonic mixture.
In the sequel, we will construct a probabilistic generative model which relates latent quantities, such as acoustic event labels, tempi, and beat positions, to the actual audio recording. This model is an extension that combines ideas from existing probabilistic models: the bar pointer model proposed by Whiteley et al. [8] for tempo and beat position tracking and an acoustic event detection and tracking model proposed by Şimşekli and Cemgil [7].
In the following subsections, we explain the probabilistic generative model and the associated training algorithm. The main novelty of the current model is that it integrates tempo tracking with minimum delay online event detection in polyphonic textures.
2.1 Tempo and acoustic event model
where p_{ n } is the probability of a change in velocity. When the velocity is at the boundaries, in other words if n_{ τ } = 1 or n_{ τ } = N, the velocity does not change with probability, p_{ n } , or transitions respectively to n_{τ+1}= 2 or n_{τ+1}= N  1 with probability 1  p_{ n } . The modulo operator reflects the periodic nature of the model and ensures that the position variables stay in the set {0, . . . , M  1}.
Essentially, this transition model assumes that the claves hits can only occur on the beat positions, which are defined by the clave pattern. A similar idea for clave modeling was also proposed in Wright et al. [16].
Note that, in the original bar pointer model definition, there are also other variables such as the meter indicator and the rhythmic pattern indicator variables, which we do not use in our generative model.
2.2 Signal model
Şimşekli and Cemgil presented two probabilistic models for acoustic event tracking in Şimşekli and Cemgil [7] and demonstrated that these models are sufficiently powerful to track different kinds of acoustic events such as pitch labels [7, 17, 18] and percussive sound events [19]. In our signal model, we use the same idea that was presented in the acoustic event tracking model [7]. Here, the audio signal is subdivided into frames and represented by their magnitude spectrum, which is calculated with discrete Fourier transform. We define x_{ν,τ}as the magnitude spectrum of the audio data with frequency index ν and time frame index τ, where ν ∈ {1, 2, . . . , F} and τ ∈ {1, 2, . . . , T}.
The main idea of the signal model is that each acoustic event (indicated by r_{ τ } ) has a certain characteristic spectral shape which is rendered by a specific hidden volume variable, v_{ τ } . The spectral shapes, socalled spectral templates, are denoted by t_{ν,i}. The ν index is again the frequency index, and the index i indicates the event labels. Here, i takes values between 1 and R, where R has been defined as the number of different acoustic events. The volume variables v_{ τ } define the overall amplitude factor, by which the whole template is multiplied.
where pa(χ) denotes the parent nodes of χ.
The Poisson model is chosen to mimic the behavior of popular NMF models that use the KL divergence as the error metric when fitting a model to a spectrogram [20, 21]. We also choose Gamma prior on v_{ τ } to preserve conjugacy and make use of the scaling property of the Gamma distribution.
Since we have a standard HMM from now on, we can run the forwardbackward algorithm in order to compute the filtering or smoothing densities. Also, we can estimate the most probable state sequence by running the Viterbi algorithm. A benefit of having a standard HMM is that the inference algorithm can be made to run very fast. This lets the inference scheme to be implemented in realtime without any approximation [22]. Detailed information about the forward backward algorithm can be found in "Appendix A".
One point here deserves attention. The Poisson observation model described in this section is not scale invariant; i.e., turning up the volume can affect the performance. The Poisson model can be replaced by an alternative that would achieve scale invariance. For example, instead of modeling the intensity of a Poisson, we could assume conditionally Gaussian observations and model the variance. This approach corresponds to using a ItakuraSaito divergence rather than the KullbackLeibler divergence [23]. However, in practice, scaling the input volume to a specific level is sufficiently good enough for acceptable tempo tracking performance.
2.3 Training
In order to learn the spectral templates, in this study, we utilize the expectationmaximization (EM) algorithm. This algorithm iteratively maximizes the loglikelihood via two steps:
where 〈 f (x)〉_{p(x)}= ∫ p (x) f (x) dx is the expectation of the function f(x) with respect to p(x).
Intuitively, we can interpret this result as the weighted average of the normalized audio spectra with respect to v_{ τ } .
3 The control model
The goal of the control module is to generate the necessary control signals to accelerate and decelerate the robot such that the performed rhythm matches the performance by its tempo and relative position. As observations, the control model makes use of the bar position and velocity (tempo) estimates m_{ τ } and n_{ τ } inferred by the perception module and possibly their associated uncertainties. In addition, the robot uses additional sensor readings to determine its own state, such as the angular velocity and angular position of its rotating motors axis that is connected directly to the drum sticks.
3.1 Dynamic linear system formulation
Formally, at each discrete time step τ, we represent the robot state by the motors angular position ${\widehat{m}}_{\tau}\in \left[0,2\pi \right)$ and angular velocity ${\widehat{n}}_{\tau}>0$. In our case, we assume these quantities are observed exactly without noise. Then, the robot has to determine the control action u_{ τ } , which corresponds to an angular acceleration/deceleration value of its motor.
Intuitively, the control state represents the drift of the robot relative to the performer; the goal of the controller will be to force the control state toward zero.
where u_{ τ } ∈ ℝ is the control signal to accelerate the motor and ε_{ τ } is the zeromean transition noise with Σ _{ A } covariance. Here, the first coordinate of s_{ τ } give the amount of difference in the score position of the performer and the robot.
For example, consider a case where the robot is lagging behind, so Δm_{ τ } < 0. If the velocity difference Δn_{ τ } is also negative, i.e., the robot is "slower", then in subsequent time steps, the difference will grow in magnitude and the robot would lag further behind.
To complete our control model, we need to specify an appropriate cost function. While one can contemplate various attractive choices, due to computational issues, we constrain ourselves to the quadratic case. The cost function should capture two aspects. The first one is the amount of difference in the score position. Explicitly, we do not care too much if the tempo is off as long as the robot can reproduce the correct timing of the beats. Hence, in the cost function, we only take the position difference into account. The second aspect is the smoothness of velocity changes. If abrupt changes in velocity are allowed, the resulting performance would not sound realistic. Therefore, we also introduce a penalty on large control changes.
where κ ∈ ℝ^{+} is a penalty parameter to penalize large magnitude control signals.
Hence, after defining the corresponding linear dynamic system, the aim of the controller is to determine the optimal control signal, namely the acceleration of the robot motor u_{ τ } given the transition and the control matrices and the cost function.
3.2 Linearquadratic optimal control
In contrast to the general stochastic optimal control problems defined for general Markov decision processes (MDPs), linear systems with quadratic costs have an analytical solution.
for stationary transition matrix A, control maxtrix B and state cost matrix Q.
Thus, in order to calculate the gain matrix L*, a fixedpoint iteration method with an initial point of K_{0} = Q is used to find the converged K value of K* = lim_{t → ∞}K_{ t } .
3.3 Imperfect knowledge case
In the previous section, both perceived and sensor values are assumed to be true and noise free. However, possible errors of the perception module and noise of the sensors can be modeled as an uncertainty over the states. Actually, the perception module already infers a probability density over possible tempi and score positions. So, instead of a single point value, we can have a probability distribution as our belief state. However, this would bring us out of the framework of the linearquadratic control into the more complicated general case of partially observed Markov decision processes (POMDPs) [24].
This expectation is with respect to the filtering density of s_{ τ } . Since the system still behaves as a linear dynamical system due to the linearquadratic Gaussian case assumption, this filtering density can be calculated in closed form using the Kalman filter [24].
where Σ_{perception} is the estimated covariance of the tempo and position values inferred by the perception module by moment matching and Σ_{robot} is the covariance of the sensor noises specific to the actuators.
4 Simulation results
Before implementing the whole system, we have evaluated our perception and the control models via several simulation scenarios. We have first evaluated the perception model on different parameter and problem settings, and then simulated the robot itself in order to evaluate the performance of both models and the synchronization level between them. At the end, we combine the Lego robot with the perception module and evaluate their joint performance.
4.1 Simulation of the perception model
In order to understand the effectiveness and the limitations of the perception model, we have conducted several experiments by simulating realistic scenarios. In our experiments, we generated the training and the testing data by using a MIDI synthesizer. We first trained the templates offline, and then, we tested our model by utilizing the previously learned templates.
At the training step, we run the EM algorithm which we described in Section 2.3, in order to estimate the spectral templates. For each acoustic event, we use a short isolated recording where the acoustic events consist of the claves hit, the conga hit (that is supposed to be produced by the robot itself), and silence. We also use templates in order to handle the polyphony in the music.
4.2 Simulation of the robot
4.2.1 Practical issues
Additionally, even though the optimal control u_{ τ } could be in ℝ^{+}, due to the physical properties of the robot, it is actually in a bounded set such as [0, u_{max}] during the experiments with robot. Hence, its value is truncated when working with the robot in order to keep it in the constrained set. However, while this violates our theoretical assumptions, the simulations are not affected from this nonlinearity.
4.2.2 Results
in order to make the robot explicitly track velocity in addition to bar position. However, as in Figure 10c and 10d it was easily affected by the perception module errors and fluctuate a lot before converging. This behavior mainly occurs because the initial velocity of the robot is zero and the robot tends to accelerate quickly in order to track the tempo of the music. However, with this rapid increase in the velocity, its bar position gets ahead of the bar position of the music. As a response the controller would decelerate, and this would cause the fluctuating behavior until the robot reaches a stable tracking position.
As a general conclusion about the control module, it could not track the performer in the first bar of the songs, because the estimations of the perception module are not yet accurate, and the initial position of the robot is arbitrary. However, as soon as the second bar starts, control state, expected normalized difference between the robot state and the music state, starts to converge to the origin.
Also note that, when κ is chosen close to 0, velocity values of the robot tend to oscillate a lot. Even sometimes they became 0 as in Figure 10a and 10c. This means that the robot has to stop in order to wait the performer because of its previous actions with high magnitudes.
In the experiments, we observe that the simulated system is able to converge quickly in a variety of parameter settings, as can be seen from control state diagrams. We omit quantitative results for the synthetic model at this stage and provide those for the Lego robot. In this final experiment, we combine the Lego robot with the perception module and run an experiment with a monophonic claves example with steady tempo. Here, we estimate the tempo and score position and try to synchronize the robot via optimal control signals. We also compare the effects of different cost functions provided that the clave is played in steady tempo, and the other parameters are selected to be similar to the ones that are described in synthetic data experiments. While perceptually more relevant measures can be found, for simplicity, we just monitor and report the mean square error.
Here, Q penalizes both the position and velocity error, where Q^{pos} penalizes only the position. The results seem to confirm our intuition: the control cost parameter κ needs to be chosen carefully to tradeoff elasticity versus rigidity. The figures visualize the corresponding control behaviors for the three different parameter regimes: converging with early fluctuations, closetooptimal converging and converging slowly, respectively.
We also observe that the cost function taking into account only the score position difference is competitive generally. Considering the tempo estimate Δn_{ τ } does not significantly improve the tracking performance other than the extremely small chosen κ < 1 which actually is not an appropriate choice for κ.
5 Experiments with a Lego robot
Since the microcontroller used on the robot is not powerful enough to run the perception module, the perception module runs on the central computer. The perception module sends the tempo and bar position information to the robot through a Bluetooth connection. On the other hand, the control module runs on the robot by taking into account its internal motor speed and position sensors and the tempo and bar position information. The central computer also controls a MIDI synthesizer that plays the other instrumental parts upon the rhythm.
5.1 The robot
The conga player robot is designed with Lego Mindstorm NXT programmable robotics kit. The kit includes a 48MHz, 32bits microcontroller with 64 KB memory. The controller is capable of driving 3 servo motors and 4 sensors of different kinds. The controller provides a USB and a Bluetooth communication interface.
5.2 Evaluation of the system
We evaluated the realtime performance of our robot controller by feeding the tempo and score position estimates directly from the listening module. In the first experiment, we generated synthetic data that simulate a rhythm starting at a tempo of 60 bpm; initially accelerating followed by a ritardando. These data, without any observation noise, are sent to the robot in real time; e.g., the bar position and velocity values are sent in every 23 ms. The controller algorithm is run on the robot. While the robot rotates, we monitor its tachometer as an accurate estimate of its position and compare it with target bar position.
Meansquared errors for realtime robot performance
Δm  Δn  

Synthetic data  0.04  1.98 × 10^{6} 
Tempotracker data  0.08  2.83 × 10^{5} 
6 Conclusions
In this paper, we have described a system for robotic interaction, especially useful for percussion performance that consists of a perception and a control module. The perception model is a hierarchical HMM that does online event detection and separation while the control module is based on linearquadratic control. The combined system is able to track the tempo quite robustly and respond in real time in a flexible manner.
One important aspect of the approach is that it can be trained to distinguish between the performance sounds and the sounds generated by the robot itself. In synthetic and real experiments, the validity of the approach is illustrated. Besides, the model incorporates domainspecific knowledge and contributes to the area of Computational Ethnomusicology[25].
We also realized that and we will investigate another platform for such demonstrations and evaluations as a future work.
While our approach to tempo tracking is conceptually similar to the musical accompaniment systems reviewed earlier, our approach here has a notable novelty, where we formulate the robot performance as a linear quadratic control problem. This approach requires only a handful of parameters and seems to be particularly effective for generating realistic and humanlike expressive musical performances, while being straightforward to implement. In some sense, we circumvent a precise statistical characterization of expressive timing deviations and still are able to generate a variety of rhythmic "feels" such as rushing or lagging quite easily. Such aspects of musical performance are hard to quantify objectively, but the reader is invited to visit our web page for audio examples and a video demonstration at http://www.cmpe.boun.edu.tr/~umut/orumbata/. As such, the approach has also potential to be useful in generating MIDI accompaniments that mimics a real human musicians behavior, control of complicated physical sound synthesis models or control of animated visual avatars.
Clearly, a Lego system is not solid enough to create convincing performances (including articulation and dynamics); however, our robot is more a proof of concept rather than a complete robotic performance system, and one could anticipate several improvements in the hardware design. One possible improvement for the perception model is to introduce different kinds of rhythmic patterns, i.e., clave patterns, to the perception model. This can be done by utilizing the rhythm indicator variable, which is presented in Whiteley et al. [8]. One other possible improvement is to introduce continuous state space for bar position and the velocity variables in order to have more accurate estimates and eliminate the computational needs of the large state space of the perception model. However, in that case exact inference will not be tractable, therefore, one should resort to approximate inference schemata, as discussed, for example in Whiteley et al. [26]. As for the control system, it is also possible to investigate POMDP techniques to deal with more diverse cost functions or extend the set of actions for controlling, besides timing, other aspects of expressive performance such as articulation, intensity, or volume.
Appendix
A Inference in the perception model
Inference is a fundamental issue in probabilistic modeling where we ask the question "what can be the hidden variables as we have some observations?" [27]. For online processing, we are interested in the computation of the socalled filtering density: p(n_{ τ } , m_{ τ } , r_{ τ }  x_{1:F},_{1:τ}), that reflects the information about the current state {n_{ τ } , m_{ τ } , r_{ τ } } given all the observations so far x_{1:F,1:τ}. The filtering density can be computed online, however the estimates that can be obtained from it are not necessarily very accurate as future observations are not accounted for.
Here, L is a specified lag and it determines the trade off between the accuracy and the latency.
This quantity requires that we accumulate all data before estimation and should give a high accuracy at the cost of very long latency.
Briefly, the goal of inference in the HMM is computing the filtering and the (fixedlag) smoothing distributions and the (fixedlag) Viterbi path. These quantities can be computed by the wellknown forwardbackward and the Viterbi algorithms.
For big values of N, M, and R this matrix becomes extremely large, but sufficiently sparse so that making exact inference is viable.
where ∝ denotes the proportionality up to a multiplicative constant. Besides, the Viterbi path is obtained by replacing the summations over r_{ τ } by maximization in the forward recursion.
Declarations
Acknowledgements
We are grateful to Prof. Levent Akin and the members of the AI lab for letting us to use their resources (lab space and Lego^{©} robots) during this study. We also thank Antti Jylhä and Cumhur Erkut of the acoustics labs Aalto University, Finland for the fruitful discussions. We also want to thank Sabanc³ University Music Club (Müzikus) for providing the percussion instruments. We would like to also thank Ömer Temel and Alper Güngörmüşler for their contributions in program development. We thank the reviewers for their constructive feedback. This work is partially funded by The Scientific and Technical Research Council of Turkey (TÜBİTAK) grant number 110E292, project "Bayesian matrix and tensor factorisations (BAYTEN)" and Boğaziçi University research fund BAP 5723. The work of Umut Şimşekli and Orhan Sönmez is supported by the Ph.D. scholarship (2211) from TÜBİTAK.
Authors’ Affiliations
References
 Otsuka T, Nakadai K, Takahashi T, Ogata T, Okuno HG: Realtime audiotoscore alignment using particle filter for coplayer music robots. EURASIP J Adv Signal Process 2011, 2011: 2:12:13.View ArticleGoogle Scholar
 Kapur A: A history of robotic musical instruments. International Computer Music Conference (ICMC) 2005.Google Scholar
 Goto M, Muraoka Y: Realtime beat tracking for drumless audio signals: chord change detection for musical decisions. Speech Commun 1999, 27(34):311335. 10.1016/S01676393(98)000764View ArticleGoogle Scholar
 Kim Th, Park SI, Shin SY: Rhythmicmotion synthesis based on motionbeat analysis. ACM SIGGRAPH 2003 Papers 2003, 392401.View ArticleGoogle Scholar
 Kim YE, Batula AM, Grunberg D, Lofaro DM, Oh J, Oh P: Developing humanoids for musical interaction. International Conference on Intelligent Robots and Systems 2010.Google Scholar
 Lofaro DM, Oh P, Oh J, Kim Y: Interactive musical participation with humanoid robots through the use of novel musical tempo and beat tracking techniques in the absence of auditory cues. 2010 10th IEEERAS International Conference on Humanoid Robots (Humanoids) 2010, 436441.View ArticleGoogle Scholar
 Şimşekli U, Cemgil AT: Probabilistic models for realtime acoustic event detection with application to pitch tracking. J New Music Res 2011, 40: 175185. 10.1080/09298215.2011.573561View ArticleGoogle Scholar
 Whiteley N, Cemgil AT, Godsill SJ: Bayesian modelling of temporal structure in musical audio. Proceedings of International Conference on Music Information Retrieval 2006.Google Scholar
 Yoshii K, Nakadai K, Torii T, Hasegawa Y, Tsujino H, Komatani K, Ogata T, Okuno HG: IROS. 2007, 17431750.Google Scholar
 Murata K, Nakadai K, Yoshii K, Takeda R, Torii T, Okuno HG, Hasegawa Y, Tsujino H: A robot uses its own microphone to synchronize its steps to musical beats while scatting and singing. IROS 2008, 24592464.Google Scholar
 Dannenberg R: An online algorithm for realtime accompaniment. International Computer Music Conference 1984, 193198.Google Scholar
 Orio N: An automatic accompanist based on hidden markov models. Proceedings of the 7th Congress of the Italian Association for Artificial Intelligence on Advances in Artificial Intelligence ser. AI*IA 01. London: Springer; 2001, 6469. [http://portal.acm.org/citation.cfm?id=648152.751104]Google Scholar
 Cemgil AT, Kappen HJ: Monte carlo methods for tempo tracking and rhythm quantization. J Artif Intell Res 2003, 18: 4581.MATHGoogle Scholar
 Raphael C: Music plus one and machine learning. International Conference on Machine Learning 2010, 2128.Google Scholar
 Völkel T, Abeßer J, Dittmar C, Großmann H: Automatic genre classification of latin american music using characteristic rhythmic patterns. Proceedings of the 5th Audio Mostly Conference: A Conference on Interaction with Sound ser. AM '10. New York, NY, USA: ACM; 2010, 16: 116:7. [http://doi.acm.org/10.1145/1859799.1859815]View ArticleGoogle Scholar
 Wright M, Schloss WA, Tzanetakis G: Analyzing afrocuban rhythms using rotationaware clave template matching with dynamic programming. ISMIR 2008, 647652.Google Scholar
 Şimşekli U: Bayesian methods for realtime pitch tracking. Master's thesis, Boğaziçi University; 2010.Google Scholar
 Şimşekli U, Cemgil AT: A comparison of probabilistic models for online pitch tracking. Proceedings of the 7th Sound and Music Computing Conference (SMC) 2010.Google Scholar
 Şimşekli U, Jylhä A, Erkut C, Cemgil AT: Realtime recognition of percussive sounds by a modelbased method. EURASIP J Adv Signal Process 2011, in press.Google Scholar
 Févotte C, Bertin N, Durrieu JL: Nonnegative matrix factorization with the ItakuraSaito divergence. with application to music analysis. Neural Comput 2009, 21(3):793830. 10.1162/neco.2008.0408771MATHView ArticleGoogle Scholar
 Vincent E, Bertin N, Badeau R: Harmonic and inharmonic nonnegative matrix factorization for polyphonic pitch transcription. ICASSP 2008.Google Scholar
 Alpaydin E: Introduction to Machine Learning (Adaptive Computation and Machine Learning). Cambridge: The MIT Press; 2004.Google Scholar
 Fevotte C, Cemgil AT: Nonnegative matrix factorisations as probabilistic inference in composite models. Proceedings of the 17th European Signal Processing Conference (EUSIPCO'09) 2009.Google Scholar
 Bertsekas D: Dynamic Programming and Optimal Control. Belmont: Athena Scientific; 1995.MATHGoogle Scholar
 Tzanetakis G, Kapur A, Schloss WA, Wright M: Computational ethnomusicology. J Interdiscip Music Stud 2007, 1(2):124.Google Scholar
 Whiteley N, Cemgil AT, Godsill S: Sequential inference of rhythmic structure in musical audio. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2007 2007, 4: IV1321IV1324.Google Scholar
 Cappé O, Moulines E, Ryden T: Inference in Hidden Markov Models (Springer Series in Statistics). Secaucus: Springer; 2005.Google Scholar
 Barber D, Cemgil AT: Graphical models for time series. IEEE Signal Process Mag Special Issue Graph. Models 2010, 27(27):1828.Google Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.