CTP431- Music and Audio Computing Sound Synthesis Graduate School - PowerPoint PPT Presentation

CTP431- Music and Audio Computing Sound Synthesis Graduate School of Culture Technology KAIST Juhan Nam 1

Outlines § Additive Synthesis § Subtractive Synthesis – Analog synthesizers – Singing voice synthesis § Nonlinear Synthesis – Ring modulation / Frequency modulation – Wave-shaping § Physical Modeling § Sample-based Synthesis 2

Additive Synthesis § Synthesize sounds by adding multiple sine oscillators – Also called Fourier synthesis OSC Amp (Env) OSC Amp (Env) + . . . . . . OSC Amp (Env) 3

Telharmonium § The first additive synthesizer using electro-magnetic “tone wheels” (Cahill, 1897) § Transmitted through telephone lines – Subscription only – The business failed

Theremin § A sinusoidal tone generator § Two antennas are remotely controlled to adjust pitch and volume Theremin ( by Léon Theremin, 1928)

Theremin (Clara Rockmore) https://www.youtube.com/watch?v=pSzTPGlNa5U 6

Hammond Organ § Drawbars – Control the levels of individual tonewheels 7

Sound Examples § Web Audio Demo – http://femurdesign.com/theremin/ – http://www.venlabsla.com/x/additive/additive.html § Examples (instruments) – Kurzweil K150 • https://soundcloud.com/rosst/sets/kurzweil-k150-fs-additive – Kawai K5, K5000 8

Subtractive Synthesis § Synthesize sounds by filtering wide-band oscillators – Source-Filter model – Examples • Analog Synthesizers: oscillators + resonant lowpass filters • Voice Synthesizers: glottal pulse train + formant filters 20 20 20 10 10 10 0 0 0 Magnitude (dB) Magnitude (dB) Magnitude (dB) − 10 − 10 − 10 − 20 − 20 − 20 − 30 − 30 − 30 − 40 − 40 − 40 − 50 − 50 − 50 − 60 − 60 − 60 5 10 15 20 5 10 15 20 0 0.5 1 1.5 2 2.5 Frequency (kHz) Frequency (kHz) Frequency (kHz) 4 x 10 Filtered Source Source Filter 9

Moog Synthesizers 10

https://www.youtube.com/watch?v=usl_TvIFtG0

Moog Synthesizers Soft Envelope LFO Control Keyboard Physical Envelope Control Wheels Slides Pedal Parameter Parameter Parameter Audio Path Amp Oscillators Filter (e.g. filter Parameter = offset + depth*control cut-off frequency) (static value) (dynamic value) 12

Oscillators § Classic waveforms 2 2 1 0 0 0 − 2 − 1 − 2 0 50 100 150 200 0 50 100 150 200 0 50 100 150 200 20 20 20 Magnitude (dB) Magnitude (dB) Magnitude (dB) − 12dB/oct − 6dB/oct 0 − 6dB/oct 0 0 − 20 − 20 − 20 − 40 − 40 − 40 − 60 − 60 − 60 5 10 15 20 5 10 15 20 5 10 15 20 Frequency (kHz) Frequency (kHz) Frequency (kHz) Triangular Sawtooth Square § Modulation – Pulse width modulation – Hard-sync – More rich harmonics 13

Amp Envelop Generator § Amplitude envelope generation – ADSR curve: attack, decay, sustain and release – Each state has a pair of time and target level Amplitude Attack Decay Sustain (dB) Release Note On Note Off 14

Examples § Web Audio Demos – http://www.google.com/doodles/robert-moogs-78th-birthday – http://webaudiodemos.appspot.com/midi-synth/index.html – http://aikelab.net/websynth/ – http://nicroto.github.io/viktor/ § Example Sounds – SuperSaw – Leads – Pad – MoogBass – 8-Bit sounds: https://www.youtube.com/watch?v=tf0-Rrm9dI0 – TR-808: https://www.youtube.com/watch?v=YeZZk2czG1c 15

Modulation Synthesis § Modulation is originally from communication theory – Carrier: channel signal, e.g., radio or TV channel – Modulator: information signal, e.g., voice, video § Decreasing the frequency of carrier to hearing range can be used to synthesize sound § Types of modulation synthesis – Amplitude modulation (or ring modulation) – Frequency modulation § Modulation is non-linear processing – Generate new sinusoidal components 16

Ring Modulation / Amplitude Modulation § Change the amplitude of one source with another source – Slow change: tremolo – Fast change: generate a new tone OSC OSC Modulator Modulator OSC OSC x + x Carrier Carrier (1 + a m ( t )) A c cos(2 π f c t ) a m ( t ) A c cos(2 π f c t ) Ring Modulation Amplitude Modulation 17

Ring Modulation / Amplitude Modulation § Frequency domain – Expressed in terms of its sideband frequencies – The sum and difference of the two frequencies are obtained according to trigonometric identity – If the modulator is a non-sinusoidal tone, a mirrored-spectrum with regard to the carrier frequency is obtained carrier sideband sideband a m ( t ) = A m sin(2 π f m t )) f c -f m f c f c +f m 18

Examples § Tone generation – SawtoothOsc x SineOsc – https://www.youtube.com/watch?v=yw7_WQmrzuk § Ring modulation is often used as an audio effect – http://webaudio.prototyping.bbc.co.uk/ring-modulator/ 19

Frequency Modulation § Change the frequency of one source with another source – Slow change: vibrato – Fast change: generate a new (and rich) tone – Invented by John Chowning in 1973 à Yamaha DX7 OSC Modulator A c cos(2 π f c t + β sin(2 π f m t )) frequency OSC β = A m Carrier Index of modulation f m 20

Frequency Modulation § Frequency Domain – Expressed in terms of its sideband frequencies – Their amplitudes are determined by the Bessel function – The sidebands below 0 Hz or above the Nyquist frequency are folded k = −∞ ∑ y ( t ) = A c J k ( β )cos(2 π ( f c + kf m ) t ) k = −∞ carrier sideband1 sideband1 sideband2 sideband2 sideband3 sideband3 f c -3f m f c -f m f c f c +f m f c -2f m f c +2f m f c +3f m 21

Bessel Function ( − 1) n ( β 2 ) k + 2 n ∞ ∑ J k ( β ) = n !( n + k )! n = 0 1 Carrier Sideband 1 Sideband 2 Sideband 3 Sideband 4 0.5 J_(k) 0 − 0.5 0 50 100 150 200 250 300 350 beta 22

Bessel Function 23

The Effect of Modulation Index 1 1 Amplitude Amplitude 0 0 − 1 − 1 0 500 1000 1500 2000 0 500 1000 1500 2000 Time (Sample) Time (Sample) 20 20 Magnitude (dB) Magnitude (dB) Beta = 0 Beta = 1 0 0 − 20 − 20 − 40 − 40 − 60 − 60 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 Frequency (kHz) Frequency (kHz) 1 1 Amplitude Amplitude 0 0 − 1 − 1 0 500 1000 1500 2000 0 500 1000 1500 2000 Time (Sample) Time (Sample) 20 20 Magnitude (dB) Magnitude (dB) Beta = 10 Beta = 20 0 0 − 20 − 20 − 40 − 40 − 60 − 60 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 Frequency (kHz) Frequency (kHz) f c = 500, f m = 50 24

Yamaha DX7 (1983) 25

“Algorithms” in DX7 http://www.audiocentralmagazine.com/yamaha-dx-7-riparliamo-di-fm-e-non-solo-seconda-parte/yamaha-dx7-algorithms/ 26

Examples § Web Audio Demo – http://www.taktech.org/takm/WebFMSynth/ § Sound Examples – Bell – Wood – Brass – Electric Piano – Vibraphone 27

Non-linear Synthesis (wave-shaping) § Generate a rich sound spectrum from a sinusoid using non-linear transfer functions (also called “distortion synthesis”) § Examples of transfer function: y = f(x) – y = 1.5x’ – 0.5x’ 3 x’=gx: g correspond to the “gain knob” of the distortion – y = x’/(1+|x’|) – y = sin(x’) T 0 (x)=1, T 1 (x)=x, – Chebyshev polynomial: T k+1 (x) = 2xT k (x)-T k-1 (x) T 2 (x)=2x 2 -1, T 2 (x)=4x 3 -3x 1 1 Amplitude Amplitude 1 0 0 0.5 − 1 0 50 100 150 200 − 1 Amplitude 0 50 100 150 200 Time (Sample) Time (Sample) 0 20 Magnitude (dB) 20 Magnitude (dB) 0 0 − 0.5 − 20 − 20 − 40 − 1 − 40 − 60 − 1 − 0.5 0 0.5 1 5 10 15 20 Time (Sample) − 60 Frequency (kHz) 5 10 15 20 Frequency (kHz) 28

Physical Modeling § Modeling Newton’s laws of motion (i.e. 𝐺 = 𝑛𝑏 ) on musical instruments – Every instrument have a different model § The ideal string ' ( ) ' ( ) – Wave equation: 𝐺 = 𝑛𝑏 à 𝐿 ( 𝐿 : tension, 𝜁 : linear mass density) '* ( = 𝜁 '* ( * * – General solution: 𝑧 𝑢, 𝑦 = 𝑧 0 (𝑢 − 3 ) + 𝑧 5 (𝑢 + 3 ) à Left-going traveling wave and right-going traveling wave 29

Physical Modeling § Waveguide Model – With boundary condition (fixed ends) -0.99 -1.0 § The Karplus-Strong model Delay Line + x(n) Z -M y(n) Noise Burst Lowpass Filter 30

Physical Modeling § The Extended Karplus -Strong model https://ccrma.stanford.edu/~jos/pasp/Extended_Karplus_Strong_Algorithm.html 31

Sample-based Synthesis § The majority of digital sound and music synthesis today is accomplished via the playback of stored waveforms – Media production: sound effects, narration, prompts – Digital devices: ringtone, sound alert – Musical Instruments • Native Instrument Kontakt5: 43+ GB (1000+ instruments) • Synthogy Ivory II Piano: 77GB+ (Steinway D Grand, ….) Synthogy Ivory II Piano Foley (filmmaking) Ringtones 32

Why Don’t We Just Use Samples? § Advantages – Reproduce realistic sounds (needless to say) – Less use of CPU § Limitations – Not flexible: repeat the same sound again, not expressive – Can require a great deal of storage – Need high-quality recording – Limited to real-world sounds § Better ways – Modify samples based on existing sound processing techniques • Much richer spectrum of sounds – Trade-off: CPU, memory and programmability 33