Yamaha multi-touch patents

I was on a quest for information about Virtual Resonance Modeling (VRM) when I encountered two interesting Yamaha patents. Both patents have the same obtuse title: “Musical sound information outputting apparatus, musical sound producing apparatus, method for generating musical sound information.” The U.S. Patent numbers are:

  • U.S. 11,398,210 B2, July 26, 2022
  • U.S. 11,657,791 B2, May 23, 2023

Both patents, essentially, cover the same technology — a kind of multi-touch keyboard and control scheme for synthesis. Here’s my synopsis.

Black and white keys are divided into two (or three) detection regions each as shown in the diagram below. Each region detects touch and release. Pretty simple, eh?

Yamaha U.S. Patents 11,398,210 and 11,657,791

The sound generator reacts in repsonse to touch and release differently according to key region. Here are a few examples mentioned in the patent:

  • A touch in white (black) key region Wa (Ba) initiates a tone.
  • A slide in white (black) key region Wb (Bb) applies an effect to the tone.
  • A release in Wa (Ba) stops the tone.
  • A release in Wb (Bb) stops application of the effect.

The word “effect” is meant rather broadly, including vibrato, modulation and so forth. Touch and release generate typical MIDI-like information such as note number and velocity.

Please note that a player can touch more than one key region simultaneously. The player chooses the gesture — with one or more fingers of a single hand or both hands. The sound generator could respond differently as to which region is touched first or held. Similarly, the sound generator could react differently depending upon the order (or temporal relationship) of release.

The inventors describe possibilities afforded by the invention when the sound generator is producing guitar tones. Depending upon touch gestures, the sound generator may produce a fingered tone, a plucked (picked) tone, hammer-on, fret noise or mute. Possibilities abound: one region initiates a single tone, the second region initiates a chord.

Obviously, if the idea works for two key regions, why not three or more?

The actual mechanism for key region detection depends upon the chosen key technology itself. Much of the patent describes implementation with a tablet touch screen, e.g., electrostatic sensing. They also mention a “pantograph-type elevating structure” where the player can depress independently the front and rear parts of a key.

What really caught my eye is the list of inventors: Masahiko Hasabe, Shinichi Ito, Kenichi Nishida, Masahire Kakishita, and Shinichi Ohta. These folks are heavy-hitters and have made many technical contributions to Yamaha products and its patent portfolio. Shinichi Ohta, for example, was the manager in charge of MONTAGE development!

So, make of it what you will. Yamaha has a broad and deep patent portfolio and not all of its patented tech gets into product. However, that is a rather large amount of talent interested in multi-touch, multi-zone keyboard control.

Copyright © 2023 Paul J. Drongowski

Yamaha YC series: Tonewheels

overall Speculation about future Yamaha product reminded me of some unfinished business — analyzing the design of the Yamaha YC stage organ series.

Design of the YC series put some of Yamaha’s best minds to work including Dr. Toshifumi Kunimoto. “Dr. K” and his team are well-known for Virtual Circuit Modeling (VCM) and physical modeling (VL). Before reading ahead, it’s worth reviewing my post summarizing YC61 Developers’ comments. The article has link to a (subtitled) interview with Dr. K, Takashi Mori and Akinobu Shibuya. One big take-away is how the developers took a system-wide approach to emulation the Hammond sound.

The YC61 Owner’s Manual cites six specific innovations:

  • Natural, organic harmonies when playing chords — thanks to a matrix circuit that connects the keyboard, tone wheels, and drawbars.
  • Percussion sound with presence — based on vacuum tube circuit analysis.
  • Key clicks and leakage sounds — based on electrical circuit analysis.
  • Natural sound distortion — simulating vintage vacuum tube pre-amplifiers.
  • Vibrato/Chorus effect — from scanner-based vibrato circuitry.
  • Changes in frequency characteristics and drive amount that responds dynamically to operation of the expression pedal.

These innovations are all in the realm of VCM and are needed to re-create the overall Hammond sound.

I assumed that Yamaha modeled the tonewheels, too. Now, I’m not so sure. I think the tonewheel waveforms are sampled and a modified form of AWM2 synthesis generates the basic, uneffected tonewheel signal (in digital form, of course). Here is my justification.

The interview and YC-series documentation

Yamaha are always honest about what they say even if they don’t say everything. Neither the developers’ interview or Yamaha documentation mention modeled tonewheels.

The YC specifications provide an important clue. Yamaha specify YC polyphony as:

VCM Organ + AWM2: 128 (Total of VCM Organ and AWM2), FM: 128

YC series keyboards have a single SWP70 tone generator (TG) integrated circuit (IC). Like the MODX design, the YC splits AWM2 and FM-X tone generation duties. It’s clear from the polyphony spec that the “VCM Organ” and AWM2 voices split resources, i.e., the AWM2 tone generation channels.

In AWM2 synthesis, each active voice element is assigned to an SWP70 tone generation channel. Genos and the upper-end PSR — also AWM2- and SWP70-based — assign a single drawbar waveform to an element (so-called “Organ Flutes” mode). Organ emulation on MODX (Montage) is similar.

Clearly, the AWM2 pipeline is involved in “VCM Organ” synthesis in some way.

Oh, the complexity!

Everyone is familiar with the 100,000 foot view of the Hammond tonewheel generator. A synchronous motor drives an assembly which spins the tonewheels. Each tonewheel has a pick-up that produces a fluctuating sine-like waveform. The waveforms pass through a key switching matrix and drawbars producing a mixed-down, composite organ tone. The tone is sent to the vibrato scanner, reverb, Leslie speaker, etc.

When it comes to modeling, the devil is in the details. I highly recommend reading one of the excellent Hammond tonewheel deep-dives on the Web:

When reading, please think about what is would take to write a mathematical model of this wonderful electro-mechanical contraption! It ain’t as trivial as summing up a bunch of sine waves. 🙂

The tonewheel assembly itself is closer to Charles Babbage’s mechanical Analytical Engine, than it is to an electronic home organ. The twelve (24, really) fundamental pitches are determined by integer gear ratios which approximate equal temperment. The tone wheels themselves have 2, 4, 8, 16, 32, 64, 128, 192 notches, producing subpitches at (near) octave intervals, derived from the fundamental scale pitches.

We know from our own experience that other aspects of the Hammond and Leslie organ system affect the final sound more than the basic tonewheel tones. If I were a developer, I would say, “Memory is cheap,” sample the tonewheels, move on and concentrate on the scanner, vacuum tube distortion, rotary speaker, etc.

Patents

Except, there is the issue of phase relationships when samples are played back. The Hammond tonewheel generator is a mechanical system with fixed relationships between tonewheel positions. This must be taken into account. Naive sample playback moves phase all over the place in an un-Hammond-like manner. Sample playback should be positionally aligned to preserve the fixed relationships present in a real, physical Hammond tonewheel generator.

Dr. K refers to “phase interference:”

“While collecting a range of different pitch waveforms, combining them, and including some non-linear additions, we also had to deal with phase interference between them. It turns out that this interference is not constant, and while balanced over the entire pitch of the instrument, the pitches do shift in subtle and inconsistent ways. … [T]his disordered yet harmonious behavior” is essential and necessary.

I believe that Yamaha have solved this problem by fetching and combining sampled tonewheel waveforms in a different way than everyday AWM2. Here are some patents to consider:

  • US Patent 10,388,290 B2 Multifunctional audio signal generation apparatus, August 20, 2019, Inventor: Taro Shirahama, Yamaha.
  • Japanese Patent 6360692 B2, Audio signal generation apparatus, July 4, 2018.

Yamaha could be aligning tonewheel waveforms when samples are fetched, thereby eliminating phase errors with respect to Hammond behavior. The sampled waveforms, of course, must also preserve the near-equal temperment of integer Hammond gear ratios. The end result is “Natural, organic harmonies when playing chords.”

I also want to draw attention to:

  • European patent application 20214572.8, Rotary speaker emulation — Device, musical instrument, method and program, December 16, 2020, Inventors: Yuji YAMADA and Takashi MORI, Yamaha.

This patent may summarizes Yamaha’s most recent work on rotary speaker emulation although the patent seems to be written as to obfuscate its intent. Yamaha has covered this territory before including:

Please note the inventors!

Copyright © 2022 Paul J. Drongowski

Noted in passing

A recent post about the Venova™ wind instrument reminded me about a few patent applications from last summer.

In addition to technology patents, Yamaha have been actively filing for and obtaining design patents on the physical industrial design of their products. Here are a half-dozen recently awarded design patents:

When I first saw the Venova, I thought, “Man, that looks like the physical manifestation of a virtual VL wind instrument.” Yamaha filed two patent applications for both the Venova and a close cousin: US Patent Application 2018/0254026 and 2018/0190245. If you’re interested in the gory details about the Venova design, dig in.

Ah, physics.

Copyright © 2019 Paul J. Drongowski

Time stretching applied to rotary speaker sound

“Sí, sí, I am very intrigued.”

With Summer NAMM 2018 one week away, I cast the net to see what I can catch. I did a quick sweep of recent patents and came up with a good ‘un.

When folks mention Yamaha, “tonewheel clone” does not immediately come to mind. Other players like Nord, Hammond Suzuki, etc. seem to be ahead in the clone market. So, I was a little surprised to find US Patent 9,899,016 B2, “Musical sound signal generation apparatus that generates sound emulating sound emitting from a rotary speaker.” This patent was issued and assigned to Yamaha on February 20, 2018. It is based on the Japanese patent 2015-171065 issued August 31, 2015.

Yamaha currently use two sample-based methods to generate the basic organ sound:

  • Playback and mix of waveforms for each individual tone wheel. On Montage and Genos, for example, the musician can adjust the level of each footage using the sliders to mimic drawbars. The generated sound is passed through a rotary speak DSP effect.
  • Playback of waveforms for “full up” organ registrations with and without the rotary speaker effect “sampled in.” The resulting sound may also be passed through a rotary speaker DSP effect.

In the first case, especially, the overall impression of a genuine B-3 depends upon the quality of the DSP rotary speaker effect. The up-side of the DSP effect is the ability to ramp up and ramp down the rotary speaker speed. So far, reaction to Yamaha’s rotary speaker effects has been mixed.

In the second case, one is not likely to put the sound through a rotary DSP effect — the swirling mass would just not be realistic. The “sampled in” approach can sound more realistic than the rotary DSP effect, but it has two major drawbacks:

  1. The rotary speaker speed cannot ramp up and down between slow and fast rotation.
  2. Sample playback does not align (synchronize) the rotary speaker position, so some noted are “rotating” faster than others and the true spatial characteristics of the horn and rotor are lost.

The second drawback is perhaps the worst of the two since it introduces audible artifacts which are not part of the true rotary speaker sound.

The method in the patent is a different take on sample-based synthesis of tone wheel sound which seeks to eliminate these problems. The notes are sampled for each tone wheel footage after a real world rotary speaker rotating at a particular rate. In each case, wavfeorms are sampled and saved for various rotational angles of the rotary speaker. Thus, the rotary speaker effect is “sampled in.”

Let’s quote from the patent:

Also, the electronic musical instrument has a time stretching function. The time stretching function is a function of changing the length of a sound while maintaining the pitch and formant of the sound. In other words, with the time stretching function, it is possible to extend and shorten a sound in a time axis direction, or in other words, it is possible to change only the reproduction speed (speed with which time advances) of the musical sound signal. The electronic musical instrument uses the time stretching function to extend and shorten each piece of waveform data in the time axis direction by the same extension and shortening rates.

Time stretching is applied to each of the tone wheel samples during playback. Thanks to time stretching, the instrument can reproduce the SLOW and FAST sound, and everything in between when the rotation speed ramps up or down. “A known pitch synchronous overlap and add method is used to achieve the time stretching function.”

The rest of the method — and it is both exhaustive and exhausting! — deals with the synchronization of the waveforms during playback, that is, the alignment of each waveform in accordance with the current virtual position (rotational angle) of the rotary speaker. Throw in separate treatment of the horn and rotor, stereo channels, etc.

The end result is a unique sample-based method that eliminates the problems of “sampled in” rotary speaker effects. I wish that patents came with audio demo files as it would be a treat to hear the method in action and to judge with one’s own ears. Maybe someday in a product?

Copyright © 2018 Paul J. Drongowski