Quantifying Algorithmic Improvements over Time Lars Kotthofg - PowerPoint PPT Presentation

Quantifying Algorithmic Improvements over Time Lars Kotthofg University of Wyoming larsko@uwyo.edu 1 IJCAI, 17 July 2018 1 joint work with Alexandre Fréchette, Tomasz Michalak, Talal Rahwan, Holger H. Hoos, Kevin Leyton-Brown

Contributions – Standalone Performance 1 dual pivot (2009) 798602199 dual pivot (2009) median 9 (1993) 798501630 median 9 (1993) median 9 random (1993) 798470169 median 9 random (1993) 798466233 mid (1978) mid (1978) 798461169 median 3 random (1978) median 3 random (1978) random (1961) 798360514 random (1961) median 3 (1978) 794178118 median 3 (1978) first (1961) 784476788 first (1961) insertion (1946) 671833 insertion (1946) Standalone Performance

Contributions – Marginal Performance How much does an algorithm contribute to the state of the art (defjned by a coalition of all other algorithms)? Xu, Hutter, Hoos, Leyton-Brown. “Evaluating Component Solver Contributions to Portfolio-Based Algorithm Selectors.” SAT 2012 2 φ i = v ( C i ∪ { i } ) − v ( C i )

Contributions – Marginal Performance 3 dual pivot (2009) 798602199 median 9 (1993) 798501630 798470169 median 9 random (1993) mid (1978) 798466233 median 3 random (1978) 798461169 798360514 random (1961) median 3 (1978) 794178118 first (1961) 784476788 98900 dual pivot (2009) 18 median 9 (1993) 5 median 3 random (1978) 5 median 9 random (1993) 3 mid (1978) 1 median 3 (1978) 0 first (1961) 0 insertion (1946) insertion (1946) 671833 0 random (1961) Standalone Performance Marginal Performance

Contributions – Temporal Marginal Performance How much does an algorithm contribute to the state of the art (defjned by a coalition of all other algorithms available at the time)? 4 φ ≻ i = v ≻ ( C i ∪ { i } ) − v ≻ ( C i ) where ≻ is a relation that encodes temporal precedence.

Contributions – Temporal Marginal Performance 5 dual pivot (2009) 798602199 median 9 (1993) 798501630 median 9 random (1993) 798470169 798466233 mid (1978) 798461169 median 3 random (1978) random (1961) 798360514 median 3 (1978) 794178118 first (1961) 784476788 13212030 random (1961) 671833 insertion (1946) 98900 dual pivot (2009) 20497 median 9 (1993) 15703 mid (1978) 6907 median 3 random (1978) 552 median 3 (1978) 541 median 9 random (1993) insertion (1946) 671833 137 first (1961) Standalone Performance Temporal Marginal Performance

Shapley Value How much does an algorithm contribute to all possible coalitions Algorithm Portfolios.” In 30th AAAI Conference on Artifjcial Intelligence, 2016. Leyton-Brown, and Tomasz P. Michalak. “Using the Shapley Value to Analyze Fréchette, Alexandre, Lars Kotthofg, Talal Rahwan, Holger H. Hoos, Kevin Games, 1953. We can compute this in polynomial time. 6 of other algorithms? φ i = 1 ∑ v ( C π i ∪ { i } ) − v ( C π i ) | Π | π ∈ Π N Shapley. “A Value for n -person Games.” In Contributions to the Theory of

Shapley Value How much does an algorithm contribute to all possible coalitions Algorithm Portfolios.” In 30th AAAI Conference on Artifjcial Intelligence, 2016. Leyton-Brown, and Tomasz P. Michalak. “Using the Shapley Value to Analyze Fréchette, Alexandre, Lars Kotthofg, Talal Rahwan, Holger H. Hoos, Kevin Games, 1953. We can compute this in polynomial time. 6 of other algorithms? φ i = 1 ∑ v ( C π i ∪ { i } ) − v ( C π i ) | Π | π ∈ Π N Shapley. “A Value for n -person Games.” In Contributions to the Theory of

Contributions – Shapley Value 7 dual pivot (2009) 798602199 median 9 (1993) 798501630 median 9 random (1993) 798470169 798466233 mid (1978) 798461169 median 3 random (1978) random (1961) 798360514 median 3 (1978) 794178118 first (1961) 784476788 100267058 100167412 98900 dual pivot (2009) 100153715 18 median 9 (1993) 100153384 5 median 3 random (1978) 100151097 5 median 9 random (1993) 100131186 3 mid (1978) 99434662 1 median 3 (1978) 98059604 0 first (1961) 0 insertion (1946) insertion (1946) 671833 84173 0 random (1961) Standalone Performance Shapley Value Marginal Performance

Temporal Shapley Value How much does an algorithm contribute to all possible coalitions 2018. Intelligence (IJCAI) Special Track on the Evolution of the Contours of AI, Improvements over Time.” In 27th International Joint Conference on Artifjcial Holger H. Hoos, and Kevin Leyton-Brown. “Quantifying Algorithmic Kotthofg, Lars, Alexandre Fréchette, Tomasz P. Michalak, Talal Rahwan, We can compute this in polynomial time as well. 8 of other algorithms, taking temporal precedence into account? 1 φ ≻ ∑ v ≻ ( C π i ∪ { i } ) − v ≻ ( C π i = i ) | Π ≻ | π ∈ Π ≻

Contributions – Temporal Shapley Value 9 798602199 dual pivot (2009) median 9 (1993) 798501630 median 9 random (1993) 798470169 mid (1978) 798466233 median 3 random (1978) 798461169 random (1961) 798360514 median 3 (1978) 794178118 first (1961) 784476788 405450356 392238462 100267058 100167412 13212030 random (1961) 100153715 671833 insertion (1946) 100153384 671833 98900 dual pivot (2009) 100151097 98900 20497 median 9 (1993) 100131186 57198 15703 mid (1978) 99434662 50411 6907 median 3 random (1978) 98059604 22506 552 median 3 (1978) 10074 541 median 9 random (1993) insertion (1946) 671833 84173 2550 137 first (1961) Shapley Value Temporal Marginal Performance Standalone Performance Temporal Shapley Value

Quicksort Over Time 10 Sum of temporal Shapley values 1e+08 1e+05 1e+02 1946 1961 1978 1993 2009 year

SAT Competition 11 dimetheus_2.100_2014 gnovelty+_2007 144.25268 ● 78.42398 ● ranov_2007 BalancedZ_2014 144.08601 ● 63.90765 ● adaptnovelty_2007 139.91915 ● ● 55.75744 CCgscore_2014 TNM_2009 CSCCSat2014_SC2014_2014 57.41773 ● 55.15192 ● sparrow2011_2011 56.33454 ● 52.43065 ● probSAT_sc14_2014 sapsrt_2007 Ncca+_v1.05_2014 51.75097 ● ● 50.72959 March − KS_2007 CCA2014_2.0_2014 44.43418 ● ● 45.33413 KCNFS_2007 sattime_2014 43.43408 ● ● 44.61692 dimetheus_2.100_2014 YalSAT_03l_2014 35.08748 ● ● 44.50427 hybridGM3_2009 sparrow2011_2011 28.8338 ● 36.26344 ● iPAWS_2009 TNM_2009 27.08395 ● 31.60638 ● sattime2011_2011 sattime2011_2011 19.16712 ● ● 30.53198 BalancedZ_2014 adaptg2wsat2011_2011 19.13952 ● ● 30.41135 adaptg2wsat2011_2011 MPhaseSAT_M − 2011 − 02 − 16_2011 16.3337 ● ● 28.4814 DEWSATZ − 1A_2007 CSHCrandMC_2013 12.43352 ● 25.76449 ● CSCCSat2014_SC2014_2014 ranov_2007 10.19402 ● 21.82523 ● CCgscore_2014 iPAWS_2009 9.69765 ● ● 20.7125 probSAT_sc14_2014 gnovelty+_2007 9.0321 ● ● 20.49854 Ncca+_v1.05_2014 adaptnovelty_2007 8.30224 ● 20.15654 ● CSHCrandMC_2013 hybridGM3_2009 8.0001 ● 19.71357 ● gnovelty+2_2009 gnovelty+2_2009 7.75017 ● 18.71084 ● YalSAT_03l_2014 gNovelty+ − T_2009 7.03675 ● ● 17.82205 CCA2014_2.0_2014 march_br_sat+unsat_2013 6.64388 ● 16.86642 ● sattime_2014 gNovelty+GCwa_1.0_2013 4.87001 ● 16.31361 ● MPhaseSAT_M − 2011 − 02 − 16_2011 march_rw − 2011 − 03 − 02_2011 4.50018 ● 15.36641 ● minisat − SAT_2007 march_hi_2011 4.26672 ● 14.86641 ● MXC_2007 March − KS_2007 4.26672 ● ● 14.18306 gNovelty+ − T_2009 KCNFS_2007 3.91678 ● ● 13.18303 csls − pnorm − 8cores_2011 csls − pnorm − 8cores_2011 2.00002 ● 9.83857 ● march_br_sat+unsat_2013 sapsrt_2007 2.00001 ● ● 8.74145 march_rw − 2011 − 03 − 02_2011 DEWSATZ − 1A_2007 1.83337 ● ● 4.81501 MiraXT − v3_2007 minipure_1.0.1_2013 1.16668 ● ● 3.56499 march_hi_2011 MXC_2007 0.83336 ● 1.90815 ● gNovelty+GCwa_1.0_2013 minisat − SAT_2007 4e − 05 ● ● 1.88997 minipure_1.0.1_2013 MiraXT − v3_2007 0 ● ● 0.53389 Solver43a_a_2013 Solver43a_a_2013 0 ● 0.45055 ● Solver43b_b_2013 Solver43b_b_2013 0 ● 0.14286 ● strangenight_satcomp11 − st_2013 strangenight_satcomp11 − st_2013 0 ● ● 0 Temporal Shapley Value Shapley Value

12 SAT Competition Over Time Sum of temporal Shapley Values 600 400 200 0 2007 2009 2011 2013 2014 year