Towards Sustainable Insights Tim Kraska <tim_kraska@brown.edu>
A New Study shows: A Glass Of Red Wine Is The Equivalent To An Hour At The Gym [Fox News 02/15 and others] http://www.huffingtonpost.co.uk/2016/01/08/a-glass-of-red- wine-is-the-equivalent-to-an-hour-at-the-gym-says-new- study_n_7317240.html
A new study shows: Secret to winning a nobel prize? Eat More Chocolate [Time 10/12 ]
A new study shows: Secret to winning a nobel prize? Eat More Chocolate [Time 10/12 ]
Scientists find the secret of longer life for men (The bad news: castration is the key) [Daily Mail UK, 09/12] http://www.dailymail.co.uk/sciencetech/article-2207981/Scientists-secret- living-life-men-bad-news-Castration-key.html
on of There has been an ex explos osion (data-driven) discoveries, many of which being qu questionabl ble .
Reasons are manifold, but… the database community … and many others works hard on to be not left out (again)
Let me introduce (vi (virtual) ) Re Revi viewe wer 2 : The paper's shortcomings are in its motivation, solution, and presentation. The part of the paper that I did like was the examples given in Sec 2.2.2. A note for Reviewer 2 : We actually liked your comments and it helped us to sharpen our points. If you feel in any way offended by this talk, this was not my intention and I am more than happy to make it up to you with a lot of whisky. Just come to me after the talk and say we need to drink. Knowing this crowd, enough people will do it and I will even never find out your identity if you do not wish so.
Outline Par art I: I: The problem with: A. A. In Interac active Dat ata a Explorat ation B. B. Vi Visualization Recommendation Sy System ems C. C. Hy Hypothes hesis Gener enerator A. A. Par art II: II: Solutions
A) Interactive Data Exploration Tools (Vizdom as an Example)
Why Visualizations contribute to the problem If If a a visual alizati ation provides an any insight, t, it t is an an hy hypothes hesis t tes est (just one where you not necessarily know if it is statistical significant) Otherwise, visualizations have just to be taken as pretty pictures about (potentially) random facts A B C count count Male Female Other count count count gender Male Female Other True False Male Female Other True False gender gender salary over 50k salary over 50k count Male Female Other gender E F count count count count count HS Bachelor Master PhD HS Bachelor Master PhD Married Never Not Widowed Married Never Not Widowed True False Married Married Married Married education education salary over 50k marital status marital status
If visualizations are used to find something interesting, the user is doing multiple hypothesis testing A B C D count Male Female Other count count count count gender count count Male Female Other True False Male Female Other True False gender HS Bachelor Master PhD Married Never Not Widowed gender salary over 50k salary over 50k Married Married education count marital status Male Female Other gender count E F 10 20 30 40 50 60 70 80 90 age count p count 0.011 count count count count t-test HS Bachelor Master PhD HS Bachelor Master PhD Married Never Not Widowed Married Never Not Widowed True False True False Married Married Married Married education education marital status salary over 50k marital status salary over 50k count 10 20 30 40 50 60 70 80 90 age
Running Example: Survey on Amazon Mechanical Turk
Our goal: To find good indicators (correlations) that somebody knows who Mike Stonebraker is.
And after searching for a bit, one of my favorites Pearson correlation significance-level p < 0.05
But Why Does the DB community make the situation worse?
So What Did Reviewer 2 say? Blaming the multiple-comparison problem on fast visualization- generation is like blaming fast cars for child driver casualties due to car accidents… But…
2) Visual Recommendation Systems (SeeDB as an Example) Target Uninteresting 1 Aggr(Collumn A) 0.8 Normalized 0.6 0.4 Reference 0.2 1 0 Aggr(Collumn A) 0.8 V1 V2 Normalized 0.6 Collumn B (filtered Column C = V?) 0.4 0.2 0 Target V1 V2 1 Interesting Collumn B (filtered Column C = V?) Aggr(Collumn A) 0.8 Normalized 0.6 0.4 0.2 0 V1 V2 Collumn B (filtered Column D = V?)
What is different The The system em aut utomatically gener enerates es tho hous usand nds of of visualization ons and d ranks them som omehow ow (e (e.g .g., b ., base sed e effe fect si t size ze)
SeeDB on Our Survey Data % Cheddar & Sour Cream % Cheddar & Sour Cream Potato Chips vs Workspace Preference Potato Chips vs Workspace Preference 0.8 0.4 0.6 0.3 0.4 0.2 0.2 0.1 0 0 Startup Corporation Startup Corporation Filter: All Filter: Prefer Blow Hair Drying % Cheddar & Sour Cream Potato Chips vs Workspace Preference % Cheddar & Sour Cream Potato Chips vs Workspace Preference 0.8 1 0.6 0.4 0.5 0.2 0 0 Startup Corporation Startup Corporation …I did like […] the Filter: Disbelief in Alien Existence Filter: Belief in Alien Existence example …
What is the Problem? The user is in the dark what the system did. The system might have “tested” thousands of potential visualization, just to find something interesting.
What did Reviewer 2 say? These systems are not designed for an average person to run and get insights that they can publish medical articles on! The end users are still analysts. The only difference is that they automate hypotheses generation and NOT hypotheses testing ,…
My suggestions, papers should include in the future a a warning like WARNING Afterusingthetool, throwawaythedata . Itisnotsafe ! 1 1 To be more precise: you do not have to throw it all away, but you can not use the same data anymore for significance testing
3) Real Hypothesis Generators (Data Polygamy as an Example)
(Data) Polygamy is bad, especially if you do not know what is going on.
Outline Par art I: I: The problem with: A. A. In Interac active Dat ata a Explorat ation B. B. Vi Visualization Recommendation Sy System ems C. C. Hy Hypothes hesis Gener enerator A. A. Par art II: II: Solutions
Should we stop working on IDE, Recommenders, etc? NO • Actively inform the user about the risk factors • Try your techniques over random data with different data sizes • If possible , split data into a exp exploration a n and nd a a va validation s n set et. . • Be aware, si significantly lowers s the power • Everything on the validation data set has to be carefully handled (i.e., use multi-hypothesis control) • If possible , use ad addit itional ional exp exper erim iment ents s (e.g., A/B testing) • Requires a small number of hypothesis and careful design • Might not always be possible or is very expensive Be Bette tter: r: con ontr trol ol th the multi ti-hy hypot othes hesis prob oblem em from om the he star art
Our Interactive Data Exploration Stack (BIDES) Quantifying the Uncertainty in QUDE Data Exploration With hypothesis Mlbase2 IDEA Legacy control Systems Interactive Data Exploration Accelerator Python BigDAWG
Many Interesting Open Problems We are just at the beginning • Trans nsparent ent hyp hypothes hesis t tes esting ng how to automatically derive what the hypothesis is the user is testing • How t to co convey t nvey the m he mea eani ning ng t to t the us he user er (e.g., FDR vs family-wise error) • Sa Safe r e reco ecommend ender er t techni echniques ues (we are currently exploring new techniques based VC-dimensions to control the error) • Incr ncrem ement ental m mul ultiple-hyp hypothes hesis co cont ntrol t techni echniques ues (for example, see ”Controlling False Discoveries During Interactive Data Exploration” CoRR abs/1612.01040 how we use new alpha-investing policies to do that) • Dep epend endenci encies es b bet etween hyp een hypothes hesis (this can safe ”hypothesis budget”) • …
A Final Note from Reviewer 2 on Is Is the Si he Situat uation on real eally s so B o Bad ad? .., the systems that are criticized by this paper are essentially three tools [4,6,28] … So the problem is not really as serious as it might seem as none of these systems are used by anyone in practice
Tim Kraska <tim_kraska@brown.edu> Special thanks to: A last note to Reviewer 2: 1 st I sincerely hope you are not one of my letter writers for my tenure case :) 2 nd Your comments actually helped us to improve the paper and helped with the talk. So thank you! 3 rd I am happy to pay for your drinks tonight to make it up to you.
Recommend
More recommend
