9/3/2018 Visualization of machine learning algorithms Why is this hard? Why is this hard? Machine learning Machine learning Visualization Visualization Fast algorithms Multi-dimensional spaces Sufficient data Comparing complex data Automatic learning Showing uncertainty http://localhost:8080/slides.html#/ 46/127
9/3/2018 Visualization of machine learning algorithms Vis helping ML Vis helping ML http://localhost:8080/slides.html#/ 47/127
9/3/2018 Visualization of machine learning algorithms Vis helping ML Vis helping ML How do they work together? Building models Validating models Understanding models http://localhost:8080/slides.html#/ 48/127
9/3/2018 Visualization of machine learning algorithms Building models Building models http://localhost:8080/slides.html#/ 49/127
9/3/2018 Visualization of machine learning algorithms Building models Building models Meta parameters Model selection http://localhost:8080/slides.html#/ 50/127
9/3/2018 Visualization of machine learning algorithms What are meta parameters? What are meta parameters? Meta parameters control how learning takes place Learning rate Number and size of network layers Slack variables Stopping conditions http://localhost:8080/slides.html#/ 51/127
9/3/2018 Visualization of machine learning algorithms Why study meta-parameters? Why study meta-parameters? http://localhost:8080/slides.html#/ 52/127
9/3/2018 Visualization of machine learning algorithms Why study meta-parameters? Why study meta-parameters? http://localhost:8080/slides.html#/ 53/127
9/3/2018 Visualization of machine learning algorithms Manual method Manual method 1.0 0.5 0.0 0.5 1.0 http://localhost:8080/slides.html#/ 54/127
9/3/2018 Visualization of machine learning algorithms Manual method Manual method 1.0 0.5 0.0 0.5 1.0 http://localhost:8080/slides.html#/ 55/127
9/3/2018 Visualization of machine learning algorithms How to study them? How to study them? run a bunch of models and examine outputs paramorama design galleries http://localhost:8080/slides.html#/ 56/127
9/3/2018 Visualization of machine learning algorithms Paramorama Paramorama Automated sampling Manual sampling Pretorius, A. Johannes, Mark-Anthony P. Bray, Anne E. Carpenter, and Roy A. Ruddle. “Visualization of parameter space for image analysis,” 2011. http://localhost:8080/slides.html#/ 57/127
9/3/2018 Visualization of machine learning algorithms Paramorama Paramorama Pretorius, A. Johannes, Mark-Anthony P. Bray, Anne E. Carpenter, and Roy A. Ruddle. “Visualization of parameter space for image analysis,” 2011. http://localhost:8080/slides.html#/ 58/127
9/3/2018 Visualization of machine learning algorithms Paramorama Paramorama Pretorius, A. Johannes, Mark-Anthony P. Bray, Anne E. Carpenter, and Roy A. Ruddle. “Visualization of parameter space for image analysis,” 2011. http://localhost:8080/slides.html#/ 59/127
9/3/2018 Visualization of machine learning algorithms Design galleries Design galleries Marks, Joe, Brad Andalman, Paul A. Beardsley, William Freeman, Sarah Gibson, Jessica Hodgins, Thomas Kang, et al. “Design Galleries: A general approach to setting parameters for computer graphics and animation,” 1997. http://localhost:8080/slides.html#/ 60/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 61/127
9/3/2018 Visualization of machine learning algorithms How to study them? How to study them? use a more principled approach http://localhost:8080/slides.html#/ 62/127
9/3/2018 Visualization of machine learning algorithms Objective measures Objective measures Ground truth - - Dice: 0.85 ... Image Error: 0.25 http://localhost:8080/slides.html#/ 63/127
9/3/2018 Visualization of machine learning algorithms Visual parameter space exploration Visual parameter space exploration conceptual pipeline Michael Sedlmair, Christoph Heinzl, Stefan Bruckner, Harald Piringer, and Torsten Möller "Visual parameter space analysis: A conceptual framework" IEEE Transactions on Visualization and Computer Graphics. 20(12) 2014. http://localhost:8080/slides.html#/ 64/127
9/3/2018 Visualization of machine learning algorithms Tuner Tuner - - Torsney-Weir, Thomas, Ahmed Saad, Torsten Möller, Britta Weber, Hans-Christian Hege, Jean-Marc Verbavatz, and Steven Bergner. “Tuner: Principled parameter finding for image segmentation algorithms using visual response surface exploration,” 2011. http://localhost:8080/slides.html#/ 65/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 66/127
9/3/2018 Visualization of machine learning algorithms Tuner Tuner Torsney-Weir, Thomas, Ahmed Saad, Torsten Möller, Britta Weber, Hans-Christian Hege, Jean-Marc Verbavatz, and Steven Bergner. “Tuner: Principled parameter finding for image segmentation algorithms using visual response surface exploration,” 2011. http://localhost:8080/slides.html#/ 67/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 68/127
9/3/2018 Visualization of machine learning algorithms Tuner Tuner Torsney-Weir, Thomas, Ahmed Saad, Torsten Möller, Britta Weber, Hans-Christian Hege, Jean-Marc Verbavatz, and Steven Bergner. “Tuner: Principled parameter finding for image segmentation algorithms using visual response surface exploration,” 2011. http://localhost:8080/slides.html#/ 69/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 70/127
9/3/2018 Visualization of machine learning algorithms Tuner Tuner Torsney-Weir, Thomas, Ahmed Saad, Torsten Möller, Britta Weber, Hans-Christian Hege, Jean-Marc Verbavatz, and Steven Bergner. “Tuner: Principled parameter finding for image segmentation algorithms using visual response surface exploration,” 2011. http://localhost:8080/slides.html#/ 71/127
9/3/2018 Visualization of machine learning algorithms Building models Building models Meta parameters can have a large influence on performance Almost all ML algorithms require tuning Manual tuning is time consuming and error prone http://localhost:8080/slides.html#/ 72/127
9/3/2018 Visualization of machine learning algorithms Validating and verifying models Validating and verifying models http://localhost:8080/slides.html#/ 73/127
9/3/2018 Visualization of machine learning algorithms What do we mean? What do we mean? How do we know our models are working? model selection Committee on Mathematical Foundations of Verification, Validation, and Uncertainty Quantification; Board on Mathematical Sciences and Their Applications, Division on Engineering and Physical Sciences, National Research Council. Assessing the reliability of complex models: Mathematical and statistical foundations of verification, validation, and uncertainty quantification , 2012. http://www.nap.edu/openbook.php?record_id=13395 . http://localhost:8080/slides.html#/ 74/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 75/127
9/3/2018 Visualization of machine learning algorithms Validating and verifying models Validating and verifying models Summary statistics are not always enough Balancing multiple objectives is difficult Certain training points might be very important http://localhost:8080/slides.html#/ 76/127
9/3/2018 Visualization of machine learning algorithms Examples Examples HyperMoVal - local inspection Sliceplorer - global inspection Tuner - error inspection http://localhost:8080/slides.html#/ 77/127
9/3/2018 Visualization of machine learning algorithms HyperMoVal HyperMoVal Piringer, Harald, Wolfgang Berger, and Jurgen Krasser. “HyperMoVal: Interactive visual validation of regression models for real-time simulation,” 2010. http://localhost:8080/slides.html#/ 78/127
9/3/2018 Visualization of machine learning algorithms Sliceplorer views Sliceplorer views Single layer NN (26 nodes) Dual layer NN (5 and 3 nodes) SVM (polynomial kernel) SVM (RBF kernel) Torsney-Weir, Thomas, Michael Sedlmair, and Torsten Möller. “Sliceplorer,” 2017. http://localhost:8080/slides.html#/ 79/127
9/3/2018 Visualization of machine learning algorithms Tuner error views Tuner error views Examining multi-dimensional functions error view shows where model is unsure can visually verify the model Prediction Prediction Optimization Optimization Error view Error view http://localhost:8080/slides.html#/ 80/127
9/3/2018 Visualization of machine learning algorithms Validating and verifying models Validating and verifying models Understand fit for individual samples Visual inspection to understand extrapolation Uncertainty can help to understand quality of prediction http://localhost:8080/slides.html#/ 81/127
9/3/2018 Visualization of machine learning algorithms Understanding models Understanding models http://localhost:8080/slides.html#/ 82/127
9/3/2018 Visualization of machine learning algorithms Who needs this? Who needs this? models are complex the business world likes spreadsheets because they can walk through the calculations http://localhost:8080/slides.html#/ 83/127
9/3/2018 Visualization of machine learning algorithms Simple vs complex models Simple vs complex models Simple Simple Complex Complex few factors multi-layer neural network small integer factors Gaussian process model low-depth trees non-linear many decisions http://localhost:8080/slides.html#/ 84/127
9/3/2018 Visualization of machine learning algorithms What does complexity buy us? What does complexity buy us? Global vs local models Deep-learning networks can deal with feature selection Can deal with edge cases http://localhost:8080/slides.html#/ 85/127
9/3/2018 Visualization of machine learning algorithms Understanding models Understanding models Just an answer is not enough (show your work) Humans have trouble understanding complex models Interactivity can bring people into the model http://localhost:8080/slides.html#/ 86/127
9/3/2018 Visualization of machine learning algorithms Methods Methods interaction walkthroughs simpler models ala LIME (Ribeiro et al. 2016) direct inspection http://localhost:8080/slides.html#/ 87/127
9/3/2018 Visualization of machine learning algorithms Examples Examples regression: Muhlbacher and Piringer clustering: Dis-function text processing: TagRefinery smaller models: Explanation explorer http://localhost:8080/slides.html#/ 88/127
9/3/2018 Visualization of machine learning algorithms Mühlbacher and Piringer Mühlbacher and Piringer Directly interact with the model building process Mühlbacher, Thomas, and Harald Piringer. “A partition-based framework for building and validating regression models,” 2013. Best Paper Award. http://localhost:8080/slides.html#/ 89/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 90/127
9/3/2018 Visualization of machine learning algorithms Dis-function Dis-function Build a distance function interactively Brown, Eli T, Jingjing Liu, Carla E Brodley, and Remco Chang. “Dis-Function: Learning Distance Functions Interactively,” 2012. http://localhost:8080/slides.html#/ 91/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 92/127
9/3/2018 Visualization of machine learning algorithms TagRefinery TagRefinery Tutorial/walkthrough system Text processing pipeline Kralj, Christoph, Mohsen Kamalzadeh, and Torsten Möller. “TagRefinery: A visual tool for tag wrangling,” 2017. http://localhost:8080/slides.html#/ 93/127
9/3/2018 Visualization of machine learning algorithms TagRefinery TagRefinery Kralj, Christoph, Mohsen Kamalzadeh, and Torsten Möller. “TagRefinery: A visual tool for tag wrangling,” 2017. http://localhost:8080/slides.html#/ 94/127
9/3/2018 Visualization of machine learning algorithms TagRefinery TagRefinery Kralj, Christoph, Mohsen Kamalzadeh, and Torsten Möller. “TagRefinery: A visual tool for tag wrangling,” 2017. http://localhost:8080/slides.html#/ 95/127
9/3/2018 Visualization of machine learning algorithms LIME method LIME method Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. “‘Why should I trust you?’: Explaining the predictions of any classifier,” 2016. http://localhost:8080/slides.html#/ 96/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 97/127
9/3/2018 Visualization of machine learning algorithms Explanation explorer Explanation explorer Krause, Josua, Aritra Dasgupta, Jordan Swartz, Yindalon Aphinyanaphongs, and Enrico Bertini. “A workflow for visual diagnostics of binary classifiers using instance-level explanations,” 2017. http://localhost:8080/slides.html#/ 98/127
9/3/2018 Visualization of machine learning algorithms http://localhost:8080/slides.html#/ 99/127
9/3/2018 Visualization of machine learning algorithms Direct inspection Direct inspection e.g. hidden states in a neural network http://localhost:8080/slides.html#/ 100/127
Recommend
More recommend
