Prediction Cubes Bee-Chung Chen, Lei Chen, Yi Lin and Raghu - PowerPoint PPT Presentation

Prediction Cubes Bee-Chung Chen, Lei Chen, Yi Lin and Raghu Ramakrishnan University of Wisconsin - Madison

Big Picture • We are not trying to build a single accuracy “model” • We want to find interesting subsets of the dataset – Interestingness: Defined by the “model” built on a subset – Cube space: A combination of dimension attribute values defines a candidate subset (just like regular OLAP) • We are not using regular aggregate functions as the measures to summarize subsets • We want the measures to represent decision/prediction behavior – Summarize a subset using the “model” built on it – Big difference from regular OLAP!! 2

One Sentence Summary • Take OLAP data cubes, and keep everything the same except that we change the meaning of the cell values to represent the decision/prediction behavior – The idea is simple, but it leads to interesting and promising data mining tools 3

Example (1/5): Regular OLAP Z : Dimensions Y : Measure Goal: Look for patterns of unusually Location Time # of App. high numbers of applications … … ... AL, USA Dec, 04 2 04 03 … … … … Coarser CA 100 90 … WY, USA Dec, 04 3 USA 80 90 … regions … … … … 2004 … Jan … Dec … Roll up AB 20 15 15 … Drill CA … 5 2 20 … 5 3 15 … 2004 2003 … down YT 55 … … … Jan … Dec Jan … Dec … AL CA 30 20 50 25 30 … … USA … 5 … … USA 70 2 8 10 … … … WY 10 … … … … … … … … … … … … … … … … … Cell value: Number of loan applications Finer regions 4

Example (2/5): Decision Analysis Goal: Analyze a bank’s loan decision process w.r.t. two dimensions: Location and Time Fact table D Z : Dimensions X : Predictors Y : Class cube subset Location Time Race Sex … Approval Model h ( X , σ Z ( D )) AL, USA Dec, 04 White M … Yes … … … … … … E.g., decision tree WY, USA Dec, 04 Black F … No Location Time All All Country Japan USA Norway State AL W Y 5

Example (3/5): Questions of Interest • Goal: Analyze a bank’s loan decision process with respect to two dimensions: Location and Time • Target: Find discriminatory loan decision • Questions: – Are there locations and times when the decision making was similar to a set of discriminatory decision examples (or similar to a given discriminatory decision model )? – Are there locations and times during which Race or Sex is an important factor of the decision process? 6

Example (4/5): Prediction Cube 2004 2003 … 1. Build a model using data Jan … Dec Jan … Dec … from USA in Dec., 1985 0.4 0.8 0.9 0.6 0.8 … … CA 2. Evaluate that model USA 0.2 0.3 0.5 … … … Measure in a cell: … … … … … … … … • Accuracy of the model • Predictiveness of Race Data σ [ USA , Dec 04 ] ( D ) measured based on that model Location Time Race Sex … Approval • Similarity between that AL , USA Dec, 04 White M … Y model and a given model … … … … … … WY, USA Dec, 04 Black F … N Model h ( X , σ [ USA , Dec 04 ] ( D )) E.g., decision tree 7

Example (5/5): Prediction Cube 04 03 … Roll up 2004 2003 … CA 0.3 0.2 … Jan … Dec Jan … Dec … 0.2 0.3 … USA 0.4 0.1 0.3 0.6 0.8 … … CA … … … … USA 0.7 0.4 0.3 0.3 … … … … … … … … … … … 2004 2003 … Cell value: Predictiveness of Race Jan … Dec Jan … Dec … AB 0.4 0.2 0.1 0.1 0.2 … … 0.1 0.1 0.3 0.3 … … … CA … YT 0.3 0.2 0.1 0.2 … … … 0.2 0.1 0.2 … … … … AL Drill down 0.3 0.1 0.1 … … … USA … WY 0.9 0.7 0.8 … … … … … … … … … … … … … 8

Outline • Motivating example • Definition of prediction cubes • Efficient prediction cube materialization • Experimental results • Conclusion 9

Prediction Cubes • User interface: OLAP data cubes – Dimensions, hierarchies, roll up and drill down • Values in the cells: → Test-set accuracy cube – Accuracy → Model-similarity cube – Similarity → Predictiveness cube – Predictiveness 10

Test-Set Accuracy Cube Given: Data table D - Data table D Location Time Race Sex … Approval - Test set ∆ AL, USA Dec, 04 White M … Yes … … … … … … WY, USA Dec, 04 Black F … No 2004 2003 … Jan … Dec Jan … Dec … CA 0.4 0.2 0.3 0.6 0.5 … … Build a model 0.2 0.3 0.9 … … … USA Accuracy … … … … … … … … Prediction Level: [ Country , Month ] Race Sex … Approval Yes Yes White F … … … … … … The decision model of USA during Dec 04 Yes Black M … No had high accuracy when applied to ∆ Test set ∆ 11

Model-Similarity Cube Given: Data table D - Data table D Location Time Race Sex … Approval - Target model h 0 ( X ) AL, USA Dec, 04 White M … Yes - Test set ∆ w/o labels … … … … … … WY, USA Dec, 04 Black F … No 2004 2003 … Jan … Dec Jan … Dec … CA 0.4 0.2 0.3 0.6 0.5 … … Build a model 0.2 0.3 0.9 … … … USA … … … … … … … … Similarity Race Sex … Level: [ Country , Month ] White F … Yes Yes … … … … … The loan decision process in USA during Dec 04 Black M … No Yes Test set ∆ was similar to a discriminatory decision model h 0 ( X ) 12

Predictiveness Cube Given: Data table D - Data table D Location Time Race Sex … Approval - Attributes V AL, USA Dec, 04 White M … Yes - Test set ∆ w/o labels … … … … … … WY, USA Dec, 04 Black F … No 2004 2003 … Jan … Dec Jan … Dec … Yes Yes CA 0.4 0.2 0.3 0.6 0.5 … … No No Build models . . 0.2 0.3 0.9 … … … USA . . Yes No … … … … … … … … h ( X − V ) h ( X ) Race Sex … Level: [ Country , Month ] Predictiveness of V White F … … … … Race was an important factor of loan approval Black M … Test set ∆ decision in USA during Dec 04 13

Outline • Motivating example • Definition of prediction cubes • Efficient prediction cube materialization • Experimental results • Conclusion 14

One Sentence Summary • Reduce prediction cube computation to data cube computation – Somehow represent a data-mining model as a distributive or algebraic (bottom-up computable) aggregate function, so that data-cube techniques can be directly applied 15

Full Materialization Full Materialization Table [ All , All ] Level Location Time Cell Value [ All , All ] 0.7 ALL ALL [ Country , All ] [ All , Year ] 0.4 CA ALL [ Country , Year ] [ Country , All ] … ALL … 0.9 USA ALL 0.8 ALL 1985 [ All , Year ] [ All , All ] [ All , Year ] ALL … … 1985 1986 … 2004 All 0.3 ALL 2004 All All 0.9 CA 1985 1985 1986 … 2004 All CA 1986 0.2 CA CA [ Country , Year ] … … … … … USA USA 0.8 USA 2004 [ Country , Year ] [ Country , All ] 16

Bottom-Up Data Cube Computation 1985 1986 1987 1988 All 47 107 76 67 297 All All 1985 1986 1987 1988 All Norway 10 30 20 24 Norway 84 23 45 14 32 114 … … 14 32 42 11 99 USA USA Cell Values: Numbers of loan applications 17

Functions on Sets • Bottom-up computable functions: Functions that can be computed using only summary information • Distributive function: α ( X ) = F ({ α ( X 1 ), …, α ( X n )}) – X = X 1 ∪ … ∪ X n and X i ∩ X j = ∅ – E.g., Count ( X ) = Sum ({ Count ( X 1 ), …, Count ( X n )}) • Algebraic function: α ( X ) = F ({ G ( X 1 ), …, G ( X n )}) – G ( X i ) returns a length-fixed vector of values – E.g., Avg ( X ) = F ({ G ( X 1 ), …, G ( X n )}) • G ( X i ) = [ Sum ( X i ), Count ( X i )] • F ({[ s 1 , c 1 ], …, [ s n , c n ]}) = Sum ({ s i }) / Sum ({ c i }) 18

Scoring Function • Represent a model as a function of sets. • Conceptually, a machine-learning model h ( X ; σ Z ( D )) is a scoring function Score ( y , x ; σ Z ( D )) that gives each class y a score on test example x – h ( x ; σ Z ( D )) = argmax y Score ( y , x ; σ Z ( D )) – Score ( y , x ; σ Z ( D )) ≈ p ( y | x , σ Z ( D )) – σ Z ( D ): The set of training examples (a cube subset of D ) 19

Bottom-up Score Computation • Key observations: – Observation 1: Score ( y , x ; σ Z ( D )) is a function of cube subset σ Z ( D ); if it is distributive or algebraic , the data cube bottom-up technique can be directly applied – Observation 2: Having the scores for all the test examples and all the cells is sufficient to compute a prediction cube • Scores ⇒ predictions ⇒ cell values • Details depend on what each cell means (i.e., type of prediction cubes); but straightforward 20