13.01.2020 Shimon An Intelligent Music-Playing Robot Capable of Improvising with Humans Vincent Rolfs Shimon: An intelligent music-playing robot capable of 1 improvising with humans
Once upon a time... Vincent Rolfs Shimon: An intelligent music-playing robot capable of 2 improvising with humans
A “player piano” [1] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 3 improvising with humans
Topics for today Why musical robots? ● Introduction to Shimon ● Physical architecture ● Algorithms for Musicianship ● Related work and outlook ● Vincent Rolfs Shimon: An intelligent music-playing robot capable of 4 improvising with humans
Why musical robots? Vincent Rolfs Shimon: An intelligent music-playing robot capable of 5 improvising with humans
Motivation: Why musical robots? Overcome human Play in band settings but Interdisciplinary shortcomings in music with full musical control education Vincent Rolfs Shimon: An intelligent music-playing robot capable of 6 improvising with humans
Introduction to Shimon Vincent Rolfs Shimon: An intelligent music-playing robot capable of 7 improvising with humans
What is Shimon? [2, 3, 4, 5] An intelligent music-playing Developed by Guy Hoffmann Plays full range of marimba robot and Gil Weinberg using four arms Vincent Rolfs Shimon: An intelligent music-playing robot capable of 8 improvising with humans
What can Shimon do? Play set melodies alone Vincent Rolfs Shimon: An intelligent music-playing robot capable of 9 improvising with humans
Shimon playing a simple melody [6] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 10 improvising with humans
What can Shimon do? Play set melodies with others Play set melodies alone in a call-and-response fashion Vincent Rolfs Shimon: An intelligent music-playing robot capable of 11 improvising with humans
Shimon playing call-and response (slow) [7] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 12 improvising with humans
Shimon playing call-and response (fast) [7] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 13 improvising with humans
What can Shimon do? Play set melodies with others Play improvisation together Play set melodies alone in a call-and-response fashion with other actors Vincent Rolfs Shimon: An intelligent music-playing robot capable of 14 improvising with humans
Shimon improvising [8] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 15 improvising with humans
Physical Architecture Vincent Rolfs Shimon: An intelligent music-playing robot capable of 16 improvising with humans
Physical architecture: Goals Large movements for visibility Fast movements for virtuosity Wide range of note combinations Vincent Rolfs Shimon: An intelligent music-playing robot capable of 17 improvising with humans
Physical architecture: High-level view [5] Four arms , running ● along a shared rail using linear actuators Each arm can ● reach an acceleration of up to 3g (105 km/h per second) Each arm can ● cover a full octave in 0.25 seconds Vincent Rolfs Shimon: An intelligent music-playing robot capable of 18 improvising with humans
Physical architecture: Arms in detail [5] Each arm contains two ● mallets , one for “black”, one for “white” keys Both mallets are ● controlled by an ON/OFF rotary solenoid These actuators are ● positioned at the crosshairs in the figure Vincent Rolfs Shimon: An intelligent music-playing robot capable of 19 improvising with humans
Problem: Collision avoidance [5] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 20 improvising with humans
Controlling of the arm motors Collisions of arms Movement of arms must be avoided should look natural Solution: Slider Manager Compute position update Avoid collisions using Animation System gives of each arm simulated springs natural movements Vincent Rolfs Shimon: An intelligent music-playing robot capable of 21 improvising with humans
Algorithms for Musicianship Vincent Rolfs Shimon: An intelligent music-playing robot capable of 22 improvising with humans
The three types of interaction modules Call-and-response Opportunistic overlay Rhythmic phrase-matching improvisation Vincent Rolfs Shimon: An intelligent music-playing robot capable of 23 improvising with humans
The three types of interaction modules Plays a set melody in response to ● a musical sequence played at an Call-and-response arbitrary tempo Beat-matched and synchronized , ● starting on time, without delay Opportunistic overlay Rhythmic phrase-matching improvisation Vincent Rolfs Shimon: An intelligent music-playing robot capable of 24 improvising with humans
The three types of interaction modules Call-and-response Focuses on choreographic ● movement Opportunistic overlay Plays a sparse improvisation that ● is beat-matched, synchronized and chord-adaptive Rhythmic phrase-matching improvisation Vincent Rolfs Shimon: An intelligent music-playing robot capable of 25 improvising with humans
Shimon using the opportunistic overlay module [7] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 26 improvising with humans
The three types of interaction modules Call-and-response Opportunistic overlay Beat-matched and ● chord-synchronized Rhythmic phrase-matching improvisation improvisation Tries to match style and density of ● human player Vincent Rolfs Shimon: An intelligent music-playing robot capable of 27 improvising with humans
Shimon using the improvisation module [8] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 28 improvising with humans
The improvisation algorithm: Finding probabilities Quantize the bar into 16 beats and declare probabilities p i, k for arm i to strike at beat k . 1 2 Listen to one bar of human play and quantize the playing into 16 beats. For each of the 16 beats, cluster the notes played in that beat into 4 bins (one of for each 3 arm) by pitch. Vincent Rolfs Shimon: An intelligent music-playing robot capable of 29 improvising with humans
The improvisation algorithm: Finding probabilities Vincent Rolfs Shimon: An intelligent music-playing robot capable of 30 improvising with humans
The improvisation algorithm: Finding probabilities Quantize the bar into 16 beats and declare probabilities p i, k for arm i to strike at beat k . 1 2 Listen to one bar of human play and quantize the playing into 16 beats. For each of the 16 beats, cluster the notes played in that beat into 4 bins (one of for each 3 arm) by pitch. Declare h i, k as 1 if the human played a note in bin i at beat k , and 0 otherwise. Then apply 4 p i, k ← λ h i, k + (1 - λ ) p i, k . Vincent Rolfs Shimon: An intelligent music-playing robot capable of 31 improvising with humans
The improvisation algorithm: Playing 1 Detect the current chord played by comparing human play to all chords in the piece. 2 Position the arms according to the detected chord. Each arm now targets one key. Strike arm i at beat k with probability p i, k . 3 …Profit! 4 Vincent Rolfs Shimon: An intelligent music-playing robot capable of 32 improvising with humans
Related work and outlook Vincent Rolfs Shimon: An intelligent music-playing robot capable of 33 improvising with humans
Compressorhead, a band consisting only of robots Vincent Rolfs Shimon: An intelligent music-playing robot capable of 34 improvising with humans
Compressorhead, a band consisting only of robots [9] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 35 improvising with humans
Haile, a robotic percussionist by Weinberg et al. Vincent Rolfs Shimon: An intelligent music-playing robot capable of 36 improvising with humans
Haile, a robotic percussionist by Weinberg et al. [10] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 37 improvising with humans
A “hyperinstrument” by Kapur et al. Vincent Rolfs Shimon: An intelligent music-playing robot capable of 38 improvising with humans
A “hyperinstrument” by Kapur et al. [11] Vincent Rolfs Shimon: An intelligent music-playing robot capable of 39 improvising with humans
Strengths and weaknesses of Shimon Can improvise in a variety of Algorithms depend strongly genres on chord classification Is able to play in sync with Tempo detection is very humans in a natural way simplistic Can improvise on the fly and Improvisation does not invent new melodies improve over time Shimon is continually Many new advances are not improved published Vincent Rolfs Shimon: An intelligent music-playing robot capable of 40 improvising with humans
The future of Shimon [12] Shimon’s head: DeepShimon: Shimon as a movie score Further research Using neural networks composer Vincent Rolfs Shimon: An intelligent music-playing robot capable of 41 improvising with humans
Thank you for your attention! Vincent Rolfs Shimon: An intelligent music-playing robot capable of 42 improvising with humans
Recommend
More recommend
