Force-feedback interaction with a neural oscillator model: for shared human-robot control of a virtual percussion instrument

Abstract

A study on force-feedback interaction with a model of a neural oscillator provides insight into enhanced human-robot interactions for controlling musical sound. We provide differential equations and discrete-time computable equations for the core oscillator model developed by Edward Large for simulating rhythm perception. Using a mechanical analog parameterization, we derive a force-feedback model structure that enables a human to share control of a virtual percussion instrument with a "robotic" neural oscillator. A formal human subject test indicated that strong coupling (STRNG) between the force-feedback device and the neural oscillator provided subjects with the best control. Overall, the human subjects predominantly found the interaction to be "enjoyable" and "fun" or "entertaining." However, there were indications that some subjects preferred a medium-strength coupling (MED), presumably because they were unaccustomed to such strong force-feedback interaction with an external agent. With related models, test subjects performed better when they could synchronize their input in phase with a dominant sensory feedback modality. In contrast, subjects tended to perform worse when an optimal strategy was to move the force-feedback device with a 90° phase lag. Our results suggest an extension of dynamic pattern theory to force-feedback tasks. In closing, we provide an overview of how a similar force-feedback scenario could be used in a more complex musical robotics setting.

1 Introduction

1.1 Interactive music

Although any perceivable sound can be synthesized by a digital computer [1], most sounds are generally considered not to be musically interesting, and many are even unpleasant to hear [2]. Hence, it can be argued that new music composers and performers are faced with a complex control problem--out of the unimaginably large wealth of possible sounds, they need to somehow specify or select the sounds they desire. Historically the selection process has been carried out using acoustic musical instruments, audio recording, direct programming, input controllers, musical synthesizers, and combinations of these.

One particularly engaging school of thought is that music can be created interactively in real time. In other words, a human can manipulate input controllers to a "virtual" computer program that synthesizes sound according to an (often quite complicated) algorithm. The feedback from the program influences the inputs that the human provides back to the program. Consequently, the human is part of the feedback control loop. Figure 1 depicts one example, in which a human plays a virtual percussion instrument using a virtual drumstick via an unspecified input coupling. The human receives auditory, visual, and haptic feedback from a virtual environment (see Figure 1). In an ideal setting, the feedback inspires the human to experiment with new inputs, which cause new output feedback to be created, for example for the purpose of creating new kinds of art [3].

Figure 1

Interactive scenario enabling a performer to play a virtual percussion instrument.

The concept of interactive music has also been explored in the field of musical robotics. Human musicians perform with musical instruments and interact with robotic musicians, who also play musical instruments (not shown). For example, Ajay Kapur has designed a robotic drummer that automatically plays along with real human performers, such as sitar players [4]. Similarly, researchers at the Georgia Institute of Technology have been studying how robots can be programmed to improvise live with human musicians [5]. As the community learns how to design robots that behave more like humans, more knowledge is created about human-computer interaction, human-robot interaction, new media art, and the human motor control system.

Our study focuses specifically on force-feedback robotic interactions. For our purposes, it is sufficient for a human to interact with a virtual robot as depicted in Figure 2, which simplifies the experimental setup. The key research question motivating this particular article is, "How can we implement shared human-robot control of a virtual percussion instrument via a force-feedback device?" More specifically, "How can these agents be effectively linked together (see the ?-box in Figure 2) in the context of a simple rhythmic interaction?" The study is part of a larger research project on studying new, extended interaction paradigms that have become possible due to advances in force-feedback interaction technology and virtual reality simulation [6].

Figure 2

Scenario for shared human-robot control of a virtual percussion instrument via an unspecified coupling.

We believe that the interaction can be more effective if the human is able to coordinate with the virtual robot. In the human-robot interaction literature, Ludovic et al. suggest that if robots are designed to make motions in the same ways that humans make motions, humans will be able to coordinate more easily with the motion of the robots [7]. For this reason, we seek to endow our virtual robot with some kind of humanlike yet very elementary rhythm perception ability, which can be effectively employed in a force-feedback context. There is evidence that neural oscillators are involved in human rhythm perception [8], so we will use one in our model. Future study will involve extending the virtual robot to incorporate multiple coupled neural oscillators to enhance its abilities, but the challenge in the present study lies in implementing high-quality force-feedback interaction with a single neural oscillator.

It is desirable to prevent force-feedback instability in this context. One approach is to employ mechanical analog models when designing robotic force feedback so that the interactions preserve energy [9]. This is one reason why our laboratory has been employing mechanical analog models since as early as 1981 in our designs [10, 11]. In the present study, we employ a computable mechanical analog model of a neural oscillator for implementing force-feedback interaction.

A linear-only version of the mechanical analog model was proposed earlier by Claude Cadoz and Daniela Favaretto. They presented an installation documenting the study at the Fourth International Conference on Enactive Interfaces in Grenoble, France in 2007 [12]. In the present study, we relate interaction scenarios within the framework of human-robot shared control in Section 1, we review prior research on neural oscillators to form a basis for the model in Section 2, we develop a mechanical analog for the "Large" neural oscillator in Section 3, we calibrate six versions of the model and we perform two human subject tests to evaluate them in Section 4. Finally, following the conclusions in Section 5, the appendices provide some additional details as well as a motivating introduction into how the model can be applied to robotic musicianship and force-feedback conducting.

2 Related evidence of neural oscillation and coordination

2.1 Perception of rhythm

The reaction time of the human motor system lies approximately in the range 120-180 ms [13]; however, by predicting the times of future events, humans are able to synchronize their motor control systems to external periodic stimuli with much greater temporal accuracy, for example as is necessary during musical performance or team rowing. Humans can even track rhythms despite changes in tempo, perturbations, and complex syncopation, and humans can maintain a pulse even after the external stimulus ceases [14]. Brain imaging studies reveal neural correlates of rhythm perception in the brain. In particular, musical rhythms trigger bursts of high-frequency neural activity [8].

2.2 Central pattern generators (CPGs) for locomotion

Animals operate their muscles in rhythmic patterns for fundamental tasks such as breathing and chewing and also for more strongly environment-dependent tasks such as locomotion. Neural circuits responsible for generating these patterns are referred to as central pattern generators (CPGs) and can operate without rhythmic input. The CPGs located in the spines of vertebrates produce basic rhythmic patterns, while parameters for adjusting these patterns are received from higher-level centers such as the motor cortex, cerebellum, and basal ganglia [15]. This explains why, with some training, a cat's hind legs can walk on a treadmill with an almost normal gait pattern after the spine has been cut [16]. In fact, the gait pattern (for instance, run vs. walk) of the hind legs can be caused to change depending on the speed of the treadmill for decerebrated cats [17].

Similar experiments have been carried out with other animals. However, it should be noted that in reality, higher cognitive levels do play a role in carrying out periodic tasks [18]. For example, humans do not exhibit locomotion after the spine has been cut--it is argued that the cerebrum may be more dominant compared to the spine in humans compared to cats [17]. Nonetheless, in some animals, the CPG appears to be so fundamental that gait transitions can be induced via electrical stimulation [15].

CPGs can be modeled for simulating locomotion of vertebrates and controlling robots. Figure 3 depicts a model of a Salamander robot with a CPG consisting of ten neural oscillators, each controlling one joint during locomotion. The figure presents one intriguing scenario that could someday be realized in multiple degree-of-freedom extensions of this study. Imagine if a human could interact using force-feedback with the state variables of a Salamander robot CPG. For example, in an artistic setting, the motion of the joints could be sonified, while a live human could interact with the model to change the speed of its motion, change the direction, and or gait form.

Figure 3

Intriguing scenario: force-feedback interaction with a neural oscillator in a Salamander CPG.

2.3 Motor coordination in animals

CPGs could also provide insight into motor coordination in animals. For example, humans tend to coordinate the movement of both of the hands, even if unintended. Bimanual tasks which do not involve basic coordination of the limbs tend to be more difficult to carry out, such as

• patting the head with one hand while rubbing the stomach in a circle with the other hand, or

• performing musical polyrhythms [13], such as playing five evenly spaced beats with one hand while playing three evenly spaced beats with the other hand.

Unintended coordinations can also be asymmetric. For example, humans tend to write their name more smoothly in a mirror image with the non-dominant hand if the dominant hand is synchronously writing the name forwards [13].

The theory of dynamic patterns suggests that during continuous motion, the motor control system state evolves over time in search of stable patterns. Even without knowledge of the state evolution of microscopic quantities, more readily observable macroscopic quantities can clearly affect the stability of certain patterns. When a macroscopic parameter change causes an employed pattern to become unstable, the motor control system can be thought to evolve according to a self-organized process to find a new stable pattern [13].

For example, consider the large number of microscopic variables necessary to describe the state evolution of a quadruped in locomotion. Gait patterns such as trot, canter, and gallop differ significantly; however, the macroscopic speed parameter clearly affects the stability of these patterns. For example, at low speeds, trotting is the most stable, and at high speeds galloping is the most stable [13].

Dynamic patterns in human index finger motion can be similarly analyzed. For example, Haken, Kelso, and Bunz describe dynamic patterns made by test subjects when asked to oscillate the two index fingers back and forth simultaneously. At low frequencies, both the symmetric (0°) and anti-symmetric (180°) patterns appear to be stable. However, at higher frequencies, the symmetric (0°) pattern becomes significantly more stable. As a consequence, when subjects begin making the anti-symmetric (180°) pattern at low frequencies, they eventually spontaneously switch to the symmetric (0°) pattern after being asked to gradually increase the frequency of the oscillation. Thus, the frequency of oscillation is a macroscopic parameter [19]. The theory of dynamic patterns can also be employed to describe human coordination with external agents, which we describe next.

2.4 Coordination with external agents

2.4.1 Unintended coordination

Humans tend to coordinate motion automatically with external agents, even when not intended. For example, pairs of test subjects completing rhythmic tasks were found to coordinate with one another when provided with visual information about each others' movements despite being given no instructions to coordinate. Subjects showed some tendency toward moving in either a 0° or 180° phase relationship [20]. In fact, even when explicitly instructed not to coordinate, test subject pairs still showed a statistical tendency toward 0° phase-alignment of arm motions [21].

Unintended interpersonal coordination is related to the theory of motor resonance. This theory argues that similar parts of the brain are activated when a human makes a movement as when an external agent makes the same movement [7, 22]. Motor resonance could also be involved with social behaviors such as the chameleon effect, which describes the

"nonconscious mimicry of the postures, mannerisms, facial expressions, and other behaviors of one's interaction partners, such that one's behavior passively and unintentionally changes to match that of others in one's current social environment [23]."

There are some indications that the strength of motor resonance may depend on whether the external agent is perceived to be more or less human [24]. Consequently, Marin et al. argue that the motor response of humanoid robots should mimic that of humans to promote bidirectional unintentional motor coordination between robots and humans [7]. We assume a similar approach in Sections 3 and 4, where we design a force feedback system for coordinating with a human.

2.4.2 Intended coordination

Of course interpersonal coordinations can also be intended. Many researchers seek to fit dynamical models to human coordination of simple motor tasks. In the case of bidirectional interpersonal coordination between two humans swinging pendulums, a neuro-mechanical dynamical model can be fit to the performance of participants, which shows that participants meet both in phase and at a frequency which lies in between their own natural frequencies [25].

We briefly point out how that model could be adopted to this article's context. Figure 4 depicts two humans playing percussion instruments with drumsticks. Because they coordinate their motions using auditory, visual, and haptic feedback (not shown), the humans behave as if a weak coupling spring were effectively connected between their drumsticks to exert a synchronizing influence (see Figure 4).

Figure 4

Two human percussionists coordinating in playing a steady beat at approximately the same frequency--when asked to coordinate, two humans will synchronize the motion of the drumsticks as if a coupling spring were effectively linking them together.

3 Neural oscillator model

3.1 The Large oscillator

In the present study, we employ the "Large" neural oscillator introduced to the literature by Edward Large [26]. With no inputs, the Large oscillator in its most basic nonlinear form can be written as the following [26]:

$$\dot{\mathbf{z}}=\mathbf{z}\left(\alpha +i\omega +b{\mid \mathbf{z}\mid }^2\right)$$

(1)

The variable z (t) ∈ ℂ rotates about the origin of the complex plane at radial frequency ω ∈ ℝ. The damping parameter α ∈ ℝ is chosen positive to cause the equilibrium point at the origin of the complex plane to be unstable, so that when subjected to some perturbations, the Large oscillator will self-oscillate.

The parameter b ∈ ℝ causes the system to tend to a limit cycle with magnitude $r_{lim}=\sqrt{-\alpha /b}$ for b < 0 as can be shown by transforming into polar coordinates using the identity z (t) = r(t)e^{i ϕ (t)}. The system can then be decoupled into the following two independent differential equations [26]:

$$ṙ=\mathit{r}\left(\alpha +b{r}^2\right)$$

(2)

and

$$\dot{\varphi }=\omega .$$

(3)

More complex terms can also be incorporated into (1), for which the nonlinear differential equation can also be separated into two real parts, but this more complicated work is not necessary for the present study [27].

Because the phase, as described by (3), evolves independently of the amplitude (see (2)), the output position of the Large oscillator tends to be approximately sinusoidal, even if the amplitude is changing relatively quickly. This characteristic is especially useful for our musical application as explained in Appendix C. In contrast, many other commonly employed neural oscillator models have a complex interaction between the magnitude and phase [19, 25, 28, 29]. Furthermore, we employ the Large oscillator in this study also because it is a key part of a model for human perception of rhythm [26], implying that a robot incorporating Large oscillators could theoretically perceive rhythm similarly to a human.

3.2 Mechanical analog of Large oscillator

In order to facilitate robust force-feedback interaction with the Large oscillator, we obtain mechanical analog parameters for it. The easiest way to do so is to temporarily linearize the Large oscillator by setting b = 0 and relating its differential equation to the following differential equation for a damped harmonic oscillator:

$$m_D\mathit{ẅ}+R\mathit{ẇ}+kw=F_{ext},$$

(4)

with mass m _D in kg, stiffness k in N/m, and damping R in N/(m/s), with an external force F _ext in Newtons acting on the mass.

Then for the Large oscillator, we incorporate a general input term x ∈ ℂ:

$$\dot{\mathbf{z}}=\mathbf{z}\left(\alpha +i\omega \right)+\mathbf{x}.$$

(5)

By separating the equation into its real w ∈ ℝ and imaginary u∈ ℝ parts such that z = w + iu and x = x₁ + ix₂, we can write

$$\mathit{ẇ}=\alpha w-\omega u+x_1$$

(6)

$$\dot{u}=\alpha u+\omega w+x_2,$$

(7)

which results in the following after taking the derivative of both sides of (6) and substituting using (7):

$$m_D\mathit{ẅ}-2\alpha mẇ+m\left({\alpha }^2+{\omega }^2\right)w=m_D\left({\mathit{ẋ}}_1-\alpha x_1-\omega x_2\right),$$

(8)

where we have also multiplied both sides by the virtual mass m _D .

Comparing with (4), we have that the equivalent mass is m _D , the equivalent damping R = -2αm _D , and the equivalent stiffness k = (α² + ω²). F _ext can be implemented by choosing inputs x₁ and x₂ such that $m_D\left({\mathit{ẋ}}_1-\alpha x_1-\omega x_2\right)=F_{ext}.$

3.3 Force-feedback interaction

We focus now on designing the lowest-order virtual model that can provide a human with high quality force, auditory, and visual feedback. The simplest design involves making the virtual robot incorporate only one neural oscillator--in this case, the robot is the neural oscillator.

Then for simplicity, the drumstick can either be connected directly to the human or to the neural oscillator. For stability reasons, it is easier to connect the drumstick directly to the neural oscillator. In this case, a virtual spring k _C can be employed to limit the impedance presented to the human [30]. Simultaneously, the spring k _C couples the human to the neural oscillator in the same spirit as shown in Figure 4, which we believe should promote the ability to coordinate and share control. The derived model structure is depicted in Figure 5, drawn to emphasize the fact that the elements are assumed to move only vertically for the purpose of conducting simple experiments.

Figure 5

Model structure for a human sharing control of a virtual percussion instrument with a neural oscillator.

4 Evaluation of the interaction using subject tests

We conducted two formal subject tests in order to evaluate how effectively human subjects could share control of the virtual percussion instrument.

4.1 Setup

Each subject gripped a single degree-of-freedom force-feedback device that moved vertically as represented in Figure 5. The subject heard the vibration of the virtual percussion instrument and saw the position of the force- feedback device, the neural oscillator, and the virtual percussion instrument on a screen. The virtual musical instrument consisted of a simple damped resonator. The instrument sounded once per oscillation period as the drumstick passed through the center position moving in the negative direction. The CORDIS-ANIMA formalism and the ERGOS platform and force-feedback device were employed [11, 31–33]. For any reader who may wish to implement the model, we provide in Appendix A explicit discrete-time equations for simulation of the Large oscillator within the CORDIS-ANIMA paradigm.

4.2 Quantitative subject test with the linearized Large oscillator

4.2.1 Design

The model structure incorporated many parameters, so we performed a quantitative human subject test to help determine how effective models should be adjusted. During this stage, we focused on the following research questions:

• Does force feedback provide the subject with better control over the oscillator?

• Is it necessary for the spring k _C to be so strong that the oscillator and the force-feedback device remain in phase?

• When rendering visual feedback, is it necessarily optimal to plot the positions of the force-feedback device and the oscillator, as would be the case with real-world "physical" force-feedback interaction with a haptic-rate resonator? Or could some other visual representation be more helpful for the subjects?

These questions did not target specifically the neural oscillator but more generally the whole setup at hand. Hence, for the sake of simplicity in the first subject test, we employed a linearized version of the neural oscillator, that is a simple oscillator obtained using the same model structure and applying b = α = 0.

We found informally that it was generally easy to increase the amplitude of the oscillation, and it was often relatively easy to speed up the oscillator or slow it down, but it tended to be more difficult to decrease the amplitude or stop the oscillator. For this reason, we decided to study how well a subject could coordinate with the neural oscillator's motion in such a manner as to stop it, showing evidence of truly sharing control with it in all interaction modes. In particular, we focused on the situation in which the oscillator was started from the home position with an initial negative velocity, and the subject was asked to try to stop the output sound in as few oscillation "bounces" as possible. To promote high-fidelity force-feedback interaction, the unloaded natural frequency of the neural oscillator was set to a haptic rate of ω = 5.0 rad/sec, corresponding to about 0.8 Hz.

First Four Models

We calibrated five different models, for which we planned to later estimate and compare their "intrinsic difficulties" relating to stopping the oscillator. The first four models differed only in the implementation of k _C , allowing to adjust how strong the force-feedback link between the force-feedback device and oscillator was. k _C ranged from a small but non-negligible value for WEAK, to a medium-sized value for MED, to large enough to force the device and oscillator position to remain phase-locked for the STRNG "strong" model. Figures 6, 7, and 8 provide some intuition into how the positions of the force-feedback device and of the neural oscillator influence each other, ranging from the WEAK model, to the MED "medium" model, to the STRNG model. The plots are shown only for subject two, but the coupling affected all of the subjects in the same manner. In the NF "no force-feedback" model, k _C had the same value as MED except that the force-feedback was disabled.

Figure 6

Part of subject two's performance for WEAK.

Figure 7

Part of subject two's performance for MED.

Figure 8

Subject two's performance for STRNG.

Fifth Model NF-HINT

The fifth model was somewhat different. We included it to study how a visual cue providing a strategy could help the subject perform the task better given weak or non-existent force feedback, where the positions of the force-feedback device and the oscillator might not be well correlated.

In the following analysis, we assumed that the force-feedback device would move according to a decaying sinusoid at ω rad/sec. Even though no test subject produced this trajectory perfectly, many were similar, and the assumption allowed for a simple analysis that provided important insight into the optimal phase relationship. When force feedback is sufficiently weak (e.g., for the NF and WEAK models), then because the "spring" force on the neural oscillator is proportional to the difference in between its position and the position of the force-feedback device, the most energy-efficient strategy for stopping the oscillations the fastest is for the test subject to force the device along a position trajectory that lags that of the neural oscillator's position by 90°. However, according to the theory of dynamic patterns, a 90° visual phase relationship should be difficult for test subjects to maintain because it is considered "unstable" (see Section 2.3) [13, 19].

Hence, we designed NF-HINT to be the same as the NF model, except that, instead of displaying the position of the Large oscillator on screen in yellow, we displayed, in green, a ball that moved in proportion to the negative velocity of the oscillator. Then an energy-optimal solution for the subject would be to perfectly follow the green ball. This 0° visual phase relationship should be more stable for the human motor control system, at least for visually dominated coordination tasks. In other words, the motion of the green ball represented the most effective strategy. Although subjects would not be able to perfectly follow the green ball, we reasoned that in attempting to do so, they would be successful in stopping the oscillator and could gain further insight into the dynamics of the task, reducing the training time for the experiment.

Procedure

Eleven test subjects were recruited from the laboratory. Some had no experience in manipulating a force-feedback device, while others had used and even programmed them before. Only subject eight was left handed, and two subjects were women. One subject was eliminated who was gave up in stopping the sound after 317 bounces for the NF model. All of the other test subjects were successful.

For a copy of the instructions given to the participants, please see Appendix B. We were aware that the task of stopping the bouncing could be challenging, so we presented the models to the test subjects always in the following order during the training phase: NF-HINT to immediately provide insight into an optimal strategy, followed by NF, MED, WEAK, and STRNG. During the testing phase, each of the ten successful subjects received the same five models ordered according to a balanced Latin square to minimize first-order residual learning effects during testing. If a subject made a mistake, the subject could repeat the test trial until satisfied with his or her test trial.

4.2.2 Number of bounces

Table 1 shows B (n, c), the number of bounces that the n th subject required to stop the oscillator from making sound for the model c. The STRNG model clearly linked the force-feedback device to the oscillator so well that the subject was able to stop the oscillator much faster than for the other models.

Table 1 Number of bounces observed for the five models

In general, the outliers were mostly relatively large numbers of bounces (see Table 1). These trials tended to correspond to instances in which the test subject made one or more suboptimal movements, which added so much energy to the oscillator, that significantly more bounces were required to remove enough energy from the oscillator to stop the sound. We noted that taking the logarithm of the number of bounces would reduce the numerical impact of the outliers (see (10)).

From visual inspection of the data in Table 1, the reader will recognize that certain subjects tended to require more bounces to stop the oscillator. Other subjects may have been more skilled at interacting with dynamical systems. For instance, subject number three was a dexterous percussionist who attained the lowest (i.e., best) number of bounces for each model.

4.2.3 Analysis

Prior to testing, some subjects may have learned more than others, implying that some subjects may have exhibited more skill than others at stopping the oscillator during testing. The differing skill levels of the subjects made it harder to infer the intrinsic difficulty of each of the test models directly from the data shown in Table 1. Consequently, we developed a model for estimating how much each subject's skill level and how much each model's intrinsic difficulty contributed to the number of bounces observed:

$$B\left(n,c\right)=\frac{D\left(c\right)}{S\left(n\right)}\cdot N_s,$$

(9)

where S (n) was the skill level of the n th subject, D (c) was the intrinsic difficulty of the model c, and N _s was a random noise variable. By taking the natural logarithm of both sides of (9), we arrived at a linear equation in the log-variables:

$$\mathsf{\text{log}}B\left(n,c\right)=\mathsf{\text{log}}D\left(c\right)-\mathsf{\text{log}}S\left(n\right)+\mathsf{\text{log}}{N}_s.$$

(10)

We noted that taking the log of the noise N _s made its histogram more symmetrical. We applied least squares linear regression to the log-variables in (10) to estimate log D (c) and log S (n). We labeled the estimates log $\widehat{D}\left(c\right)$ and log $Ŝ\left(n\right)$, respectively. This step enabled to plot $B\left(n,c\right)Ŝ\left(n\right)$, the observed number of bounces normalized by the estimated skill level of each subject, as shown with the blue x's in Figure 9. The same figure also shows the estimated intrinsic difficulty $\widehat{D}\left(c\right)$ of each model with a black o.

Figure 9

Observed number of bounces normalized by estimated skill $B\left(n,c\right)Ŝ\left(n\right)$ (blue x 's) and estimated intrinsic difficulty $\widehat{D}\left(c\right)$ (black o 's).

Lilliefors' composite goodness-of-fit test indicated that taking the log of the normalized bounces tended to make the values seem more normally distributed. Then, using the repeated measures analysis of variance test, we concluded that the data for the different models was not all drawn from the same distribution. Finally, we applied the two-sample Kolmogorov-Smirnov goodness-of-fit hypothesis test to the data in order to evaluate the statistical significance of differences between pairs of models. Using a 5% significance level, we concluded that only the pairs (NF, WEAK) and (NF-HINT, MED) were not significantly different.

4.2.4 Stronger link provided better control

The intrinsic difficulties $\widehat{D}\left(WEAK\right)$, $\widehat{D}\left(MED\right)$, and $\widehat{D}\left(STRNG\right)$ were all pairwise significantly different. In fact, each subject performed better with STRNG compared to MED and with MED compared to WEAK, implying that a stronger coupling spring k _C , which helped keep the subject and the neural oscillator approximately in phase (recall Figures 6, 7, and 8), promoted more effective coordination with the neural oscillator. Indeed, this was in agreement with motor resonance, and more specifically the theory of dynamic patterns, which suggested that the subject would coordinate with an external haptic-rate oscillator best when the dynamic pattern is stable, and prior experiments had showed that a 0° phase relationship tends to be the most stable (see Section 2.3) [13, 19].

4.2.5 Non-physical visual feedback can be better

When humans watch passive objects vibrating mechanically in nature, they typically observe displacements and not velocities. In this sense, the NF-HINT model could be thought of as non-physical because the movement of the ball represented the oscillator's negative velocity and not its position. Hence, at first consideration, one might assume that test subjects would have had relatively little success at interacting with the non-physical model. However, the situation required further consideration because the task was especially difficult. As discussed in Section 4.2.1, the test subject could damp the oscillator the fastest by moving the force-feedback device 90° behind the position of the oscillator, which is an unstable pattern according to the theory of dynamic patterns (see Section 2.3).

On a statistically significant level, subjects performed the task of stopping the oscillator more successfully when the negative velocity of the ball was plotted on the screen (compare NF-HINT and NF in Figure 9). We believe subjects performed more successfully because the ball provided them with a strategy--they were taught in the training phase to "follow the green ball." Furthermore, they could then follow the green ball with a 0° phase lag, which is much more stable from the dynamic patterns perspective.

This result also showed that a theory from visual-only human coordination experiments could be extended to situations involving also auditory feedback: non-physical visual feedback could enable a subject to complete an otherwise impossible or very difficult task, if the visualization revealed an inner state or otherwise unseen strategy that provided a human test subject with assistance [18]. Indeed, some subjects commented that they could not really understand what they were doing, but they nonetheless performed successfully with NF-HINT.

4.2.6 Benefit of appropriate force feedback

As suggested by Figure 9, subjects may have exhibited a tendency to perform worse with weak force-feedback (WEAK) in comparison with no force-feedback at all (NO-FF). Although this effect was not determined to be statistically significant, this possibility could be investigated further in future study with larger numbers of participants. We note that weak force-feedback could possibly distract the subject from successfully employing a certain strategy, in particular due to the 90° phase relationship. Force-feedback may not be beneficial in all situations.

However, the medium strength (MED) and strong (STRNG) force-feedback models produced statistically significant improvements over the basic no force-feedback model (NF), and (STRNG) even over (NF-HINT), in which a strategy was explicitly provided to the test subject. This result strongly underscores the utility of incorporating force-feedback into systems that implement human interaction with virtual dynamical systems.

4.2.7 Perspective

Subjects were asked to fill out a questionnaire to describe their experience. Since the subjects had been instructed to attempt to follow the green ball for NF-HINT during the training phase, they initially gained some intuition into the difficulty and dynamics of the task. The subjects all reported that they attempted to follow the green ball for the NF-HINT model during testing (see the relatively low numbers of bounces in the NF-HINT column of Table 1). However, the green ball was not present for the other four models. Many of the subjects adapted this strategy more or less successfully for the NF, WEAK, and even MED models. For example, subject # 5 even reported attempting to imagine where the green ball would have been in order to produce mental guidance for stopping the oscillator for NF.

Other subjects reported "incorrect" strategies, particularly for NF, such as keeping the force-feedback device 180° out-of-phase with the position of the Large oscillator. This strategy, if implemented perfectly, would not have damped the Large oscillator's motion. In fact, participants would commonly move the force-feedback device slightly fewer than 180° (instead of an optimal precisely 90°) behind the Large oscillator's motion, resulting in only modest damping.

Finally, even though STRNG resulted in the best performance for all of the test subjects, one subject reported in his comments that he preferred the MED spring coupling level k _C . For MED, the coupling was weak enough that he felt it was easier for him to command the motion of his hand; however, the coupling was nevertheless strong enough that he could clearly feel the motion of the virtual oscillator.

This was one of the motivating factors in designing the next subject test, with which we wanted to investigate more fully the subjects' perceptions of the force-feedback interaction with the strong coupling level k _C present in the STRNG model.

4.3 Qualitative subject test with the non-linear Large oscillator

We created the STRNG-NL model by starting from the STRNG model and adjusting the parameters to make the model nonlinear. We believed that then the oscillator would behave more like a real, biological neural oscillator. First we made the model nonlinear by increasing b from zero. We increased b until the model could not oscillate with an amplitude large enough to attempt to push the force-feedback device beyond its workspace. Then we increased α such that the system would readily self-oscillate. The system had one equilibrium point at the home position, but this equilibrium point was unstable [26]. As before, the unloaded natural frequency of the Large oscillator remained set to ω = 5.0 rad/sec, or about 0.8 Hz.

4.3.1 Our own perception of the model

Anecdotally, we found the model to be curiously intriguing. We considered interacting with it to be akin to being set into the shoes of a child drummer who likes to play a drum periodically by him or herself, but who is also very capable of cooperating with external agents to synchronize frequency of oscillation and amplitude. We can report that in our opinion, the system was satisfying in the sense that we were able to share control with a neural oscillator via an exciting coupling to play a simple rhythm.

We found that the nonlinear part of the model provided a strange feeling that one typically does not encounter in nature: when one attempted to move the force-feedback device sufficiently far away from the home (center) position, the damping increased rapidly. The consequence was that the device did not immediately tend back toward the home position, but rather any further motion away from the center position was strongly damped, and then further perturbations could easily, but not necessarily, contribute to the force-feedback device being pushed back toward the center position. The reader can gain some more intuition into the STRNG-NL model behavior by watching the video at the bottom of the project website: https://ccrma.stanford.edu/~eberdahl/Projects/NO/

Since negative damping was strong near the home position, it was difficult to stop the force-feedback device from moving when held in this region. But after moving the force-feedback device further from the home position, the nonlinear damping in combination with the damping from the subject's hand could be employed to stop the motion of the device.

An analysis of the dynamics showed that it was relatively easy for the subject to increase the frequency of oscillation simply by increasing the stiffness that his or her hand presented to the force-feedback device. However, in our opinion, it was more difficult to slow down the frequency of oscillation, simply because no human could passively cause the hand to have a negative stiffness, rather, any human would need to actively exert forces on the force-feedback device to counter its motion such that its frequency of oscillation decreased.

4.3.2 Subject test

We designed a subject test in order to study human subjects' perception of interacting with STRNG-NL. Indeed, in nature, one does not have the opportunity to reach into the brain or spinal column and adjust the internal states of the neural oscillators directly by applying mechanical forces (see Figure 3 for one depiction), so we suspected that subjects would find the force-feedback interaction to be strange; however, we thought that they might consider it to be intuitive. After all, we do have many neural oscillators inside our bodies, and we use them constantly throughout our day-to-day life.

For the subject test, we recruited ten members of the laboratory, two of them female, and one of them left-handed. Eight of the participants had prior experience manipulating a force-feedback device, the other two participants were new master's degree students at the laboratory. Each of the subjects was given a questionnaire and encouraged to interact with STRNG-NL via the device while answering the questions.

On the questionnaire, some questions pertained to the subjects' perceptions of the interaction, while other questions pertained more directly to perceptions of the force-feedback device. One of the reviewers suggested that the subjects' perceptions of the device itself could be dependent on its visual appearance and description. Because we had made no effort to dress up the force-feedback device, make it look cute, or describe it as a puppet-like entity or house pet, we decided to analyze only the answers to the questions regarding the interaction.

4.3.3 No unstable explosions

The choice of the model parameters (particularly b) as well as the implementation were successful in the sense that no subject was able to destabilize the force-feedback device to cause any discomfort or large-amplitude oscillations that would have required disabling the device, stopping the experiment, or similar.

4.3.4 Force-feedback interaction was "fun"

Eight out of ten subjects found the experience to be "enjoyable," and "fun" or "entertaining" to investigate. Eight out of ten subjects also considered the interaction to be intuitive, even though it does not exist in nature.

4.3.5 Sharing control with external agent

While most test subjects were immediately comfortable with giving up some control with the robotic neural oscillator, one subject reported not feeling comfortable giving up some control to share with the neural oscillator. However, this subject also reported that he would prefer to reduce the strength of the coupling spring k _C . Humans typically limit their force-feedback interactions with strangers to only occasional occurrences, typically with relatively low force levels. Hence, the situation created in the experiment could be considered somewhat strange. One subject likened it to "interacting with a dog['s tail]" although there are clearly also many differences.

4.3.6 Most subjects discovered how to change oscillation frequency

Nine out of ten subjects reported that they were able to play faster than the free resonance frequency, and eight out of ten could play slower. However, we had never instructed the subjects on how they might go about trying to change the frequency. This result implies that test subjects acquired enactive knowledge simply by interacting with the models, so they needed less explicit instruction. In fact, as we observed in the preceding subject test, subjects were sometimes able to successfully adopt a strategy without being able to accurately describe what the strategy was.

4.3.7 Sharing control with neural oscillator

Although only eight out of ten of the subjects considered the interaction to be "intuitive," all of the subjects reported they were able to "cooperate" in some understandable manner with the oscillator.

4.4 Summary of results

All of the test subjects found that they were able to share control with the neural oscillator. Only one subject reported feeling uncomfortable giving up some control, but this subject also suggested to reduce the strength of the coupling.

In general, the results seemed quite promising. Our own perception of the models, and also the subject test we carried out, allowed us to state that force-feedback interaction with a neural oscillator enabled innovative coupling for exploring a new middle ground between intuitiveness and strangeness.

5 Conclusion

The study underscores that it is necessary to understand how the human motor control system works in order to design effective active force-feedback interactions for humans. Building upon ideas inherited from human studies of neural oscillators, we presented an explicitly computable expression for the Large neural oscillator, and we introduced its mechanical analog. Following consideration of some different scenarios for humans and robots interacting in real and virtual environments, we described a simple model structure for enabling a human and a virtual robot, which consisted of a single Large neural oscillator, to share control of a virtual percussion instrument. We implemented the model structure with a system providing concurrent force, auditory, and visual feedback. We calibrated six different models, and we performed formal subject tests with the models in order to gain insight into how to tune them and how human subjects would perceive the interactions.

We found that force feedback can be useful in helping a human share control of a virtual percussion instrument with a virtual neural oscillator (see Figure 5). The force feedback must be carefully designed and calibrated. For instance, the force feedback should not induce a contradictory percept in conjunction with visual and auditory feedback, such as some subjects might have experienced with the WEAK model. Furthermore, we found that visual feedback must not necessarily be computed completely faithfully according to a coherent physical model. Instead, if the visual feedback is non-physical but provides the test subject with a good strategy for completing a difficult task, then superior performance can be observed (compare NF-HINT and NF). In particular, if (at least a visual) 0° phase relationship can result in successfully completing a task, then a human subject will more likely be able to coordinate successfully than if the optimal dominant phase relationship is 90°[13].

In the context of coupling to the virtual oscillator, we found that reducing the amplitude of an oscillation is more difficult than increasing the amplitude or changing the frequency of oscillation. For this reason, we focused primarily on studying human subjects' ability to stop the oscillation. We found that increasing the coupling spring k _C enabled subjects to more effectively share control with the neural oscillator (compare STRNG, MED, and WEAK). Since a stronger coupling spring k _C reduces the phase lag in between the position of the force-feedback device and the oscillator position, a nearly-0° phase relationship (e.g., with STRNG) seems to result in a more stable dynamic pattern than a 90° pattern (e.g., with WEAK). Hence, this result suggests an extension to force-feedback tasks of dynamic pattern theory, which has previously been applied to humans performing visual tasks [13].

However, some subjects preferred to have a somewhat weaker coupling spring k _C . We could attribute this phenomenon to the fact that humans typically limit their force-feedback interactions with strangers to only occasional occurrences, typically with relatively low force levels. For instance, we hypothesize that a human could tend to feel more comfortable if the interaction seems more akin to interacting with a dog or an active puppet, rather than resembling holding hands tightly with a stranger. Nevertheless, eight out of ten participants found the force-feedback interaction with the neural oscillator in STRNG-NL to be "enjoyable" and "fun" or "entertaining," which we hope is often the case for musical interactions in general.

6 Final words

The case with the strong coupling spring k _C seems particularly intriguing as it provides human subjects with the best control over a virtual neural oscillator. Very low-latency feedback control systems are required for digital implementation of such strong couplings to the force-feedback device [30]. For our experiment, we were able to achieve a delay on the order of a single sample, thanks to a feedback control sampling rate of 44.1 kHz. Such technology is not available in many laboratories, but it will be necessary for further investigation along the lines of the present study.

We would like to graciously thank researchers in the field of robotic musical instruments for inspiring us to complete this study, and we hope that with this article, we can inspire them to learn more about the application of virtual modeling in this field. We hypothesize that robotic musical performances can become more expressive as the simulated models measure up more closely to human capabilities. Such systems could be capable not only of exhibiting behavior that somehow resembles human behavior, but which somehow also exceeds human abilities, resulting in some situations in new "superhuman" music [34].

Appendices

A Large oscillator discrete-time equations

The Large oscillator can be described in the form of the following two coupled real-valued nonlinear differential equations:

$$\mathit{ẇ}=\left(\alpha +b\left(w^2+u^2\right)\right)w-\omega u+x_1$$

(11)

and

$$\dot{u}=\left(\alpha +b\left(w^2+u^2\right)\right)u+\omega w+x_2$$

(12)

where the output position of the oscillator is w.

In the spirit of Cordis Anima, the coupled nonlinear equations can be discretized in time using Forward Euler Integration to arrive at [11, 33]:

$$\begin{array}{c}w\left[n+1\right]=w\left[n\right]+\frac{1}{f_S}\left[\left(\alpha +b\left(w^2\left[n\right]+u^2\left[n\right]\right)\right)w\left[n\right]\right.\\ \left.-\omega \cdot u\left[n\right]+\frac{F_{ext}\left[n\right]}{m_D{f}_S}\right]\end{array}$$

(13)

and

$$\begin{array}{c}u\left[n+1\right]=u\left[n\right]+\frac{1}{f_S}\left[\left(\alpha +b\left(w^2\left[n\right]+u^2\left[n\right]\right)\right)u\left[n\right]\right.\\ \left.+\omega \cdot w\left[n\right]+\frac{F_{ext}\left[n\right]}{m_D\omega }\left(1+\frac{\alpha }{f_S}\right)\right],\end{array}$$

(14)

where the particular form of the input via F _ext [n] is employed to allow for only a single-sample delay between the input force and its effect on the output position, while nevertheless preventing any unintended filtering of F _ext [n] (see the ${\mathit{ẋ}}_1$ term in (8)).

B Instructions for the quantitative subject test

In this section, we provide the same instructions that were given to test subjects, along with a picture similar to Figure 5 to provide orientation prior to the experiment itself.

Holding a force-feedback device in your dominant hand, you will interact with a virtual percussionist. It will be initialized so that it is already playing regular beats. Your task is to stop the virtual percussionist in as few beats as possible. You may employ any strategy that you prefer. There are the following five experimental conditions:

- NF-HINT

You will receive visual and auditory feedback in a "standard" configuration. If you carefully follow the green ball (which, by the way, represents the negative velocity of the virtual percussionist), it will help you stop the sound.

- NF

The same as NF-HINT, except that the green ball is replaced by a yellow ball showing the position of the virtual percussionist.

- MED

Same as NF but with force feedback.

- WEAK

Same as MED but with a weaker force-feedback link to the virtual percussionist.

- STRNG

Same as MED but with a stronger force-feedback link to the virtual percussionist.

Note: Do not worry--if a condition is too difficult, you can give up!

The procedure is as follows:

• 1. Prepare: Please read these instructions as well as the questionnaire that you will be asked to fill out the following experiment.

• 2. Practice: First you will practice each condition in the order shown above. You can practice each as many times as you like.

• 3. Test: When you are ready to test your performance, the conditions will be presented to you in a random but known order. For each condition, you should attempt to stop the virtual percussionist in as few beats as possible. You can attempt each test until you are satisfied with your performance.

• 4. Questionnaire: Following the experiment, you will be asked to please fill out a questionnaire to describe the strategies you employed for each condition.

C Application: robotic music performance

C.1 Robotic performance using humanlike generalized motor programs (GMPs)

Thus far in the article we have presented and studied a model structure enabling a human to share control with a neural oscillator to play a virtual percussion instrument. In this appendix, we provide a brief summary of how this technology can endow a robotic system with the ability to play a real musical instrument.

To this end, we consider a robotic system implemented using a force-feedback device, which can be programmed with a model of GMPs [35] to play musical instruments (see the force-feedback device playing a musical shaker in Figure 10, left). It is believed that GMPs are essential for controlling rapid human motor movements, for which the motor reaction time via the communication loop to the brain is too long to provide a robust feedback control mechanism. Schmidt and Lee define a GMP as "a program with invariant (a) sequencing among muscles, (b) relative timing, and (c) relative forces among the contractions [13]." For a robot, aspects of a GMP can be simulated using a stored waveform with some adjustable gain and offset parameters.

Figure 10

Force-feedback device plays a musical shaker on the left as directed by a human conductor via the force-feedback device on the right.

C.2 Force-feedback interaction for shared control

A human gripping a second force-feedback device (see Figure 10, right) can speed up or slow down the tempo through force-feedback interaction if the devices are linked together using a neural oscillator. For instance, see the paradigm suggested in Figure 11. The system mimics a human playing a musical shaker using periodic patterns according to a tempo. The tempo is controlled not only by the parameter values of the Large neural oscillator, but also by the actual motion of the force-feedback devices.

Figure 11

Scenario for a neural oscillator employed to link together a human-operated force-feedback device (right) and a force-feedback device playing a brown shaker (left).

C.3 Force-feedback conducting

In order to enable the human performer to "conduct" the music using a traditional conducting gesture, it is merely necessary to implement a method for linking the human-operated force-feedback device's position to the conducting gesture. For example, the force-feedback device operated by the human could be linked to a conductor's gesture using a tunnel paradigm [36–38] for counting in four beats to the bar, while the other robot responds by playing a GMP waveform to the shaker repeatedly. Some example videos created using this scenario are available at the top of the project website: https://ccrma.stanford.edu/~eberdahl/Projects/NO/