CTP431- Music and Audio Computing Sound Synthesis Graduate School - PowerPoint PPT Presentation

CTP431- Music and Audio Computing Sound Synthesis Graduate School of Culture Technology KAIST Juhan Nam 1

Musical Sound Synthesis § Modeling the patterns of musical tones and generating them § (Typical) musical tones – Time-wise: amplitude envelope (ADSR) à Waveform – Frequency-wise : harmonic distribution à Spectrogram § How are musical tones different from other sounds (e.g. speech)? 2

Types of Tones § Musical tones have more diverse types – Harmonic: guitar, flute, violin, organ, singing voice (vowel) – Inharmonic: piano, vibraphone – Non-harmonic: drum, percussion, singing voice (consonant) [From Klapuri’s slides] Vibraphone *Inharmonicity in Piano 3

Information in Tones § Musical tones have the main information in “pitch” – Speech Tones have it mainly in “formant (i.e. spectral envelop)” § Examples – Original music : – Reconstruction from timbre features (MFCC) and using white-noise as a source): – Reconstruction from tonal features (Chroma): 4

Pitch Scale and Range § In music, pitch is arranged on a tuning system and the range is much wider 4000 3500 3000 2500 frequency − Hz 2000 1500 1000 500 0 10 20 30 40 50 time [second] 5

Control of Tones § Musical Instrument Note Number , Music Velocity, Output Synthesizer Duration (+ Expressions) § Speech Speech Phonemes Output Synthesizer (+ Expressions) 6

Overview of Sound Synthesis Techniques § Signal model (analog / digital) – Additive Synthesis – Subtractive Synthesis – Modulation Synthesis: ring modulation, frequency modulation – Distortion Synthesis: non-linear § Sample model (digital) – Sampling Synthesis – Granular Synthesis – Concatenative Synthesis § Physical model (digital) – Digital Waveguide Model 7

Theremin § A sinusoidal tone generator § Two antennas are remotely controlled to adjust pitch and volume Theremin ( by Léon Theremin, 1928)

Theremin (Clara Rockmore) https://www.youtube.com/watch?v=pSzTPGlNa5U 9

Additive Synthesis § Synthesize sounds by adding multiple sine oscillators – Also called Fourier synthesis OSC Amp (Env) OSC Amp (Env) + . . . . . . OSC Amp (Env) 10

Hammond Organ § Drawbars – Control the levels of individual tonewheels 11

Hammond Organ https://www.youtube.com/watch?v=2rqn4bYFUZU 12

Sound Examples § Web Audio Demo – http://femurdesign.com/theremin/ – http://www.venlabsla.com/x/additive/additive.html – http://codepen.io/anon/pen/jPGJMK § Examples (instruments) – Kurzweil K150 • https://soundcloud.com/rosst/sets/kurzweil-k150-fs-additive – Kawai K5, K5000 13

Subtractive Synthesis § Synthesize sounds by filtering wide-band oscillators – Source-Filter model – Examples • Analog Synthesizers: oscillators + resonant lowpass filters • Voice Synthesizers: glottal pulse train + formant filters 20 20 20 10 10 10 0 0 0 Magnitude (dB) Magnitude (dB) Magnitude (dB) − 10 − 10 − 10 − 20 − 20 − 20 − 30 − 30 − 30 − 40 − 40 − 40 − 50 − 50 − 50 − 60 − 60 − 60 5 10 15 20 5 10 15 20 0 0.5 1 1.5 2 2.5 Frequency (kHz) Frequency (kHz) Frequency (kHz) 4 x 10 Filtered Source Source Filter 14

Moog Synthesizers Soft Envelope LFO Control Keyboard Physical Envelope Control Wheels Slides Pedal Parameter Parameter Parameter Audio Path Amp Oscillators Filter (e.g. filter Parameter = offset + depth*control cut-off frequency) (static value) (dynamic value) 15

Oscillators § Classic waveforms 2 2 1 0 0 0 − 2 − 1 − 2 0 50 100 150 200 0 50 100 150 200 0 50 100 150 200 20 20 20 Magnitude (dB) Magnitude (dB) Magnitude (dB) − 12dB/oct − 6dB/oct 0 − 6dB/oct 0 0 − 20 − 20 − 20 − 40 − 40 − 40 − 60 − 60 − 60 5 10 15 20 5 10 15 20 5 10 15 20 Frequency (kHz) Frequency (kHz) Frequency (kHz) Triangular Sawtooth Square § Modulation – Pulse width modulation – Hard-sync – More rich harmonics 16

Amp Envelop Generator § Amplitude envelope generation – ADSR curve: attack, decay, sustain and release – Each state has a pair of time and target level Amplitude Attack Decay Sustain (dB) Release Note On Note Off 17

Examples § Web Audio Demos – http://www.google.com/doodles/robert-moogs-78th-birthday – http://webaudiodemos.appspot.com/midi-synth/index.html – http://aikelab.net/websynth/ – http://nicroto.github.io/viktor/ § Example Sounds – SuperSaw – Leads – Pad – MoogBass – 8-Bit sounds: https://www.youtube.com/watch?v=tf0-Rrm9dI0 – TR-808: https://www.youtube.com/watch?v=YeZZk2czG1c 18

Modulation Synthesis § Modulation is originally from communication theory – Carrier: channel signal, e.g., radio or TV channel – Modulator: information signal, e.g., voice, video § Decreasing the frequency of carrier to hearing range can be used to synthesize sound § Types of modulation synthesis – Amplitude modulation (or ring modulation) – Frequency modulation § Modulation is non-linear processing – Generate new sinusoidal components 19

Ring Modulation / Amplitude Modulation § Change the amplitude of one source with another source – Slow change: tremolo – Fast change: generate a new tone OSC OSC Modulator Modulator OSC OSC x + x Carrier Carrier (1 + a m ( t )) A c cos(2 π f c t ) a m ( t ) A c cos(2 π f c t ) Ring Modulation Amplitude Modulation 20

Ring Modulation / Amplitude Modulation § Frequency domain – Expressed in terms of its sideband frequencies – The sum and difference of the two frequencies are obtained according to trigonometric identity – If the modulator is a non-sinusoidal tone, a mirrored-spectrum with regard to the carrier frequency is obtained carrier sideband sideband a m ( t ) = A m sin(2 π f m t )) f c -f m f c f c +f m 21

Examples § Tone generation – SawtoothOsc x SineOsc – https://www.youtube.com/watch?v=yw7_WQmrzuk § Ring modulation is often used as an audio effect – http://webaudio.prototyping.bbc.co.uk/ring-modulator/ 22

Frequency Modulation § Change the frequency of one source with another source – Slow change: vibrato – Fast change: generate a new (and rich) tone – Invented by John Chowning in 1973 à Yamaha DX7 OSC Modulator A c cos(2 π f c t + β sin(2 π f m t )) frequency OSC β = A m Carrier Index of modulation f m 23

Frequency Modulation § Frequency Domain – Expressed in terms of its sideband frequencies – Their amplitudes are determined by the Bessel function – The sidebands below 0 Hz or above the Nyquist frequency are folded k = −∞ ∑ y ( t ) = A c J k ( β )cos(2 π ( f c + kf m ) t ) k = −∞ carrier sideband1 sideband1 sideband2 sideband2 sideband3 sideband3 f c -3f m f c -f m f c f c +f m f c -2f m f c +2f m f c +3f m 24

Bessel Function ( − 1) n ( β 2 ) k + 2 n ∞ ∑ J k ( β ) = n !( n + k )! n = 0 1 Carrier Sideband 1 Sideband 2 Sideband 3 Sideband 4 0.5 J_(k) 0 − 0.5 0 50 100 150 200 250 300 350 beta 25

Bessel Function 26

The Effect of Modulation Index 1 1 Amplitude Amplitude 0 0 − 1 − 1 0 500 1000 1500 2000 0 500 1000 1500 2000 Time (Sample) Time (Sample) 20 20 Magnitude (dB) Magnitude (dB) Beta = 0 Beta = 1 0 0 − 20 − 20 − 40 − 40 − 60 − 60 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 Frequency (kHz) Frequency (kHz) 1 1 Amplitude Amplitude 0 0 − 1 − 1 0 500 1000 1500 2000 0 500 1000 1500 2000 Time (Sample) Time (Sample) 20 20 Magnitude (dB) Magnitude (dB) Beta = 10 Beta = 20 0 0 − 20 − 20 − 40 − 40 − 60 − 60 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 Frequency (kHz) Frequency (kHz) f c = 500, f m = 50 27

Yamaha DX7 (1983) 28

“Algorithms” in DX7 http://www.audiocentralmagazine.com/yamaha-dx-7-riparliamo-di-fm-e-non-solo-seconda-parte/yamaha-dx7-algorithms/ 29

Examples § Web Audio Demo – http://www.taktech.org/takm/WebFMSynth/ § Sound Examples – Bell – Wood – Brass – Electric Piano – Vibraphone 30

Non-linear Synthesis (wave-shaping) § Generate a rich sound spectrum from a sinusoid using non-linear transfer functions (also called “distortion synthesis”) § Examples of transfer function: y = f(x) – y = 1.5x’ – 0.5x’ 3 x’=gx: g correspond to the “gain knob” of the distortion – y = x’/(1+|x’|) – y = sin(x’) T 0 (x)=1, T 1 (x)=x, – Chebyshev polynomial: T k+1 (x) = 2xT k (x)-T k-1 (x) T 2 (x)=2x 2 -1, T 2 (x)=4x 3 -3x 1 1 Amplitude Amplitude 1 0 0 0.5 − 1 0 50 100 150 200 − 1 Amplitude 0 50 100 150 200 Time (Sample) Time (Sample) 0 20 Magnitude (dB) 20 Magnitude (dB) 0 0 − 0.5 − 20 − 20 − 40 − 1 − 40 − 60 − 1 − 0.5 0 0.5 1 5 10 15 20 Time (Sample) − 60 Frequency (kHz) 5 10 15 20 Frequency (kHz) 31

Physical Modeling § Modeling Newton’s laws of motion (i.e. 𝐺 = 𝑛𝑏 ) on musical instruments – Every instrument have a different model § The ideal string ' ( ) ' ( ) – Wave equation: 𝐺 = 𝑛𝑏 à 𝐿 ( 𝐿 : tension, 𝜁 : linear mass density) '* ( = 𝜁 '* ( * * – General solution: 𝑧 𝑢, 𝑦 = 𝑧 0 (𝑢 − 3 ) + 𝑧 5 (𝑢 + 3 ) à Left-going traveling wave and right-going traveling wave 32