Yamaha VRM technology

I would love to write a “PJ explains all about VRM” article, but the more I learn, the more I recognize the breadth and depth of the technology. If you would like to know more about Yamaha’s Virtual Resonance Modeling from a player’s perspective, please see my earlier article about VRM

Sampling and modeling

We know that a relatively number of (short) piano samples by themselves sound flat. Virtual piano instruments such as those in the Vienna Symphonic Library go to great lengths to create rich, deep, evolving and playable piano tones. The VSL CFX grand boasts “up to 4,200 samples per key for maximal authenticity and liveliness.” The end result is a ginormous 115GB installed CFX library. Oh, hell, why not go for the full library at 240GB? 🙂

Massive libraries such as these would steer hardware instrument designers into solutions that really aren’t intended for low-power, light-weight electronic instruments. PC technology, that is. You could characterize the VSL approach as space-intensive. Sample playback also depends upon stable, available (disk-to-)memory bandwidth and sample caching in order to reliably stream samples as commanded by the player.

Physical modeling, such as MODARTT Pianoteq, takes a compute-intensive approach. Pianoteq’s installed size is about 50MB. That’s 3 orders of magnitude smaller than VSL! It is overly simplistic to say, but Pianoteq computes what VSL plays back. Pianoteq is its own witness to the quality and faithfulness of its internal mathematical models.

Like massive sample libraries, physical modeling requires compute resources that are at odds with a low-power embedded system like a digital musical instrument. High throughput means high power, bigger power supplies, fans, cooling, and so forth. In a manner analogous to sample playback, physical modeling requires stable, available compute capacity to maintain a consistent polyphony spec.

Why VRM?

Yamaha’s digital piano synthesis is a hybrid approach. They build on (literally) decades of experience in sampling and sound design. This experience was forged in the era of small read-only memories when pianos where packed into memories that we now regard as ridiculously small (e.g., 8MB). The CLP-685 fits the CFX, Bosendorfer, EPs and whatnot into 512MB of NAND flash memory — 2 orders of magnitude less than VSL (roughly speaking).

Instead of sampling all possible playing/tone situations, VRM adds certain dynamic qualities back into the sample playback stream. Full VRM has five components:

Damper resonance
String resonance
Body resonance
Duplex scale resonance
Damper noise

VRM reacts to player gestures in real-time — the keys that are held down, pedaling, and so forth. Thus, VRM produces a more lively experience where sample playback along might sound monotonous.

VRM is not alone in its fight against monotonous tone. Higher-end Yamaha digital pianos add techniques like transducers and soundboard emulation to enhance the playing experience. Digital pianos also rely on effects to add the spatial ambience (e.g., mic’ing and reverb) captured in ginormous sample libraries.

Compute resources

VRM involves a lot of real time computations; it, too, is compute-intensive.

VRM was first introduced in the Clavinova CLP-575 and CLP-585 digital pianos. The standard workhorse compute engine at that time was the Yamaha SWX08 processor. The SWX08 combines an SH-2A CPU core (host computer) and an unspecified number of AWM2 architecture tone generation channels. Yamaha added a heavy-weight SSP2 to the CLP-575 and CLP-585 designs for VRM processing alone. The SWX08 processor alone was not enough for VRM; a second SH-2A core (SSP2) was required.

I will say more about the internal designs in a future article. Suffice it to say, Yamaha eventually issued an SWX09 processor which is capable of both sample playback and VRM.

Yamaha VRM patents

I found two patents assigned to Yamaha by Eiji (Hidetsugu) Tominaga:

U.S. Patent 8,115,092 B2, February 14, 2012, Method for synthesizing tone signal and tone signal generating system
U.S. Patent 8,895,831 B2, November 25, 2014, Method for synthesizing tone signal and tone signal generating system

Tominaga-san is Yamaha’s resident expert in piano modeling. The two patents are very similar and both lay out the essentials of VRM.

Be forewarned — the mathematics are complex. The computations must be performed in real time and there are a lot of them. Yamaha does not add an extra processor to a product design just for kicks!

You might also enjoy these other Yamaha patents on related piano sound technology:

U.S. Patent Application 2014/0150624 A1, Yuji Fujiwara, et al, Recording and reproduction of waveform based on sound board vibrations, June 5, 2014
U.S. Patent 8,106,287 B2, Masahiko Hasebe, Tone control apparatus and method using virtual damper position, January 31, 2012
U.S. Patent 8,878,045 B2, Shinya Kosecki, Acoustic effect impartment apparatus and piano, November 4, 2014
U.S. Patent 8,729,376, Masahiro Kakishita, Musical sound synthesizing apparatus, May 10, 2014

The last patent is rather interesting. It describes a “sound deadener” to eliminate the effect of the sound board or other vibrating strings. You might want to add VRM to a stream of “pure” samples to avoid doubling up sympathetic resonance?

Virtual Resonance Modeling

So, what of the VRM models themselves?

The fully elaborated system consists of five, coupled, physical models:

Damper model
Hammer model
String model
Instrument body model
Air model

Results from each model are sent to one or more other models.

U.S. Patent 8,895,831 B2 Piano models [Yamaha]

The models take the actual piano structure into account, e.g., hammers hit one or more strings depending upon soft pedal behavior, etc. The piano cabinet, sound board, frame, bridges, bearings, and other vibratory components form the instrument body.

The models are driven by four input (controller) signals:

Key stroke data
Hammer velocity
Damper pedal stroke
Soft pedal stroke

The four input signals — controller data — vary over time. Results are computed for each time delta (a small time interval).

As to the mathematics, I recommend reading the patents. The math is dense and each model is fairly complicated.

The patents describe alternative embodiments (implementations). The full embodiment has the five models mentioned above. Other embodiments drop one or more models — no damper modeling, for example.

Modeling research

Yamaha aren’t finished yet. 🙂 If you’re curious about Yamaha’s on-going research, check out their Technologies page. One of the featured investigations is Physical Modeling and Simulation of a Piano. Yes, Tominaga-san and his compatriots at work. The page has some nifty visualizations of sound board behavior, air models and so forth.

“Physical model and simulation of piano touch”, Hidetsugu Tominaga and Juichi Sato, Tribologist Vol.62 No.10 pp. 623-628
“Physical model and simulation of a piano, Hidetsugu Tominaga, Juichi Sato, Makoto Minoda, Music Acoustic Research Group Materials 36(4), pp. 133-138

“Since the essence of piano performance lies in the interaction between the piano and the player, our ultimate goal is to build a physical model simulation technology in which the piano, air, and the player (fingers, ears, intellect, etc.) are one system.”

Sand, software and sound

Electronics and computing for the fun of it