On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg and Patrick Johnson Artifjcially Intelligent Manufacturing Center University of Wyoming larsko,pjohns27@uwyo.edu UCC, 02 April 2019 1
Big Picture techniques intelligently – automatically 2 ▷ advance the state of the art through meta-algorithmic ▷ rather than inventing new things, use existing things more ▷ invent new things through combinations of existing things
Motivation – Performance Difgerences Hurley, Barry, Lars Kotthofg, Yuri Malitsky, and Barry O’Sullivan. “Proteus: A Hierarchical Portfolio of Solvers and Transformations.” In CPAIOR, 2014. 3 1000 100 Virtual Best SAT 10 1 0.1 0.1 1 10 100 1000 Virtual Best CSP
Motivation – Performance Improvements Hutter, Frank, Domagoj Babic, Holger H. Hoos, and Alan J. Hu. “Boosting Verifjcation by Automatic 27–34. Washington, DC, USA: IEEE Computer Society, 2007. Tuning of Decision Procedures.” In FMCAD ’07: Proceedings of the Formal Methods in Computer Aided Design, 4 4 10 SPEAR, optimized for SWV (s) 3 10 2 10 1 10 0 10 −1 10 −2 10 −2 10 −1 10 0 10 1 10 2 10 3 10 4 10 SPEAR, original default (s)
What to Tune – Parameters constraint decomposition 5 ▷ anything you can change that makes sense to change ▷ e.g. search heuristic, variable ordering, type of global ▷ not random seed, whether to enable debugging, etc. ▷ some will afgect performance, others will have no efgect at all
Automated Parameter Tuning Frank Hutter and Marius Lindauer, “Algorithm Confjguration: A Hands on Tutorial”, AAAI 2016 6
General Approach workings intensifjcation/exploitation 7 ▷ evaluate algorithm as black-box function ▷ observe efgect of parameters without knowing the inner ▷ decide where to evaluate next ▷ balance diversifjcation/exploration and ▷ repeat until satisfjed
Grid and Random Search Bergstra, James, and Yoshua Bengio. “Random Search for Hyper-Parameter Optimization.” J. Mach. Learn. Res. 13, no. 1 (February 2012): 281–305. 8 ▷ evaluate certain points in parameter space
Local Search quality achieved 9 ▷ start with random confjguration ▷ change a single parameter (local search step) ▷ if better, keep the change, else revert ▷ repeat, stop when resources exhausted or desired solution ▷ restart occasionally with new random confjgurations
Local Search Example graphics by Holger Hoos 10 (Initialisation)
Local Search Example graphics by Holger Hoos 11 (Initialisation)
Local Search Example graphics by Holger Hoos 12 (Local Search)
Local Search Example graphics by Holger Hoos 13 (Local Search)
Local Search Example graphics by Holger Hoos 14 (Perturbation)
Local Search Example graphics by Holger Hoos 15 (Local Search)
Local Search Example graphics by Holger Hoos 16 (Local Search)
Local Search Example graphics by Holger Hoos 17 (Local Search)
Local Search Example graphics by Holger Hoos 18 ? Selection (using Acceptance Criterion)
Surrogate-Model-Based Search on this quality achieved 19 ▷ evaluate small number of initial (random) confjgurations ▷ build surrogate model of parameter-performance surface based ▷ use model to predict where to evaluate next ▷ repeat, stop when resources exhausted or desired solution ▷ allows targeted exploration of promising confjgurations
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 20 Iter = 1, Gap = 1.9909e−01 0.8 ● ● y 0.4 ● type ● init ● prop 0.0 ● type 0.025 y 0.020 yhat ei 0.015 ei 0.010 0.005 0.000 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 21 Iter = 2, Gap = 1.9909e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 ● seq type 0.03 y yhat 0.02 ei ei 0.01 0.00 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 22 Iter = 3, Gap = 1.9909e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 seq ● type 0.006 y yhat 0.004 ei ei 0.002 0.000 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 23 Iter = 4, Gap = 1.9992e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 seq ● 8e−04 type y 6e−04 yhat ei ei 4e−04 2e−04 0e+00 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 24 Iter = 5, Gap = 1.9992e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 seq ● type y 2e−04 yhat ei ei 1e−04 0e+00 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 25 Iter = 6, Gap = 1.9996e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 seq ● 0.00012 type y 0.00009 yhat ei 0.00006 ei 0.00003 0.00000 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 26 Iter = 7, Gap = 2.0000e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 seq ● 5e−05 type y 4e−05 yhat 3e−05 ei ei 2e−05 1e−05 0e+00 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 27 Iter = 8, Gap = 2.0000e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 seq ● type 2.0e−05 y 1.5e−05 yhat ei ei 1.0e−05 5.0e−06 0.0e+00 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 28 Iter = 9, Gap = 2.0000e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 seq ● 1.0e−05 type y 7.5e−06 yhat ei 5.0e−06 ei 2.5e−06 0.0e+00 −1.0 −0.5 0.0 0.5 1.0 x
Surrogate-Model-Based Search Example Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. “MlrMBO: A http://arxiv.org/abs/1703.03373. Modular Framework for Model-Based Optimization of Expensive Black-Box Functions,” March 9, 2017. 29 Iter = 10, Gap = 2.0000e−01 0.8 ● ● y type 0.4 ● ● init ● prop 0.0 seq ● type 4e−07 y yhat 3e−07 ei ei 2e−07 1e−07 0e+00 −1.0 −0.5 0.0 0.5 1.0 x
Two-Slide MBO # http://www.cs.uwyo.edu/~larsko/mbo.py [0. 1, 1e-06,0.9333333333333333]] # [1000000000.0, 1e-07, 1.0], # # [[1.0, 0.1, 1.0], for pars in random.sample(param_grid, initial_samples): return np.median(cross_val_score(clf, iris.data, iris.target, cv = 10)) 30 def est_acc(pars): random.seed(1) evals = 10 initial_samples = 3 # [{'C': 0.01, 'gamma': 1e-09}, {'C': 0.01, 'gamma': 1e-08}...] for y in params['gamma'] ] 'gamma': np.logspace(-9, 3, 13) } params = { 'C': np.logspace(-2, 10, 13), param_grid = [ { 'C': x, 'gamma': y } for x in params['C'] clf = svm.SVC(**pars) data = [] acc = est_acc(pars) data += [ list(pars.values()) + [ acc ] ]
Two-Slide MBO regr = RandomForestRegressor(random_state = 0) for evals in range(0, evals): regr.fit(df[:,0:2], df[:,2]) print ("{}: best predicted {} for {}, actual {}" .format(evals, round(preds[i], 2), param_grid[i], round(acc, 2))) i = np.array(data)[:,2].argmax() print ("Best accuracy ({}) for parameters {}".format(data[i][2], data[i][0:2])) 31 df = np.array(data) preds = regr.predict([ list(pars.values()) for pars in param_grid ]) i = preds.argmax() acc = est_acc(param_grid[i]) data += [ list(param_grid[i].values()) + [ acc ] ]
Recommend
More recommend
