High-Dimensional Distribution Testing Constantinos Costis - PowerPoint PPT Presentation

Aug 04, 2023 •374 likes •679 views

High-Dimensional Distribution Testing Constantinos Costis Daskalakis CSAIL and EECS, MIT What properties do your BIG distributions have? e.g. 1 Testing Uniformity e.g.2: Linkage Disequilibrium Genome locus 1 locus 2 Single

High-Dimensional Distribution Testing Constantinos “Costis” Daskalakis CSAIL and EECS, MIT
What properties do your BIG distributions have?
e.g. 1 Testing Uniformity • …
e.g.2: Linkage Disequilibrium Genome locus 1 locus 2 Single Nucleotide Polymorphisms (SNPs), are they independent? 1000 samples (you patients)
e.g.3: Behavior in a Social Network Q: Are nodes behaving independently or far from independently? Q’: Do adopted technologies exhibit weak or strong network effects? 1 sample
Problem formulation • TV ( c.f. G’s talk)
What do we really know about our BIG distributions of interest?
Inspecting the LB Instance • u.a.r.
Today’s Menu • Motivation • Testing Bayesian Networks • Testing Ising Models • Closing Thoughts
Today’s Menu • Motivation • Testing Bayesian Networks • Testing Ising Models • Closing Thoughts
Bayesian Networks •
Testing Bayesian Networks
Testing Bayesian Networks (cont’d)
Testing Bayesian Networks (cont’d)
Today’s Menu • Motivation • Testing Bayesian Networks • Testing Ising Models • Closing Thoughts
Ising Model •
Ising Model: Strong vs weak ties “low temperature regime” “high temperature regime” Forces
Testing Ising Models • product measures
Testing Ising Models • product measures
Testing Ising Models • Low temperature. How about high temperature?
High Temperature Ising •
Ising Model: Strong vs weak ties “low temperature regime” “high temperature regime” Exponential mixing of the Glauber dynamics
Testing Ising Models •
Concentration of Measure •
Using Concentration to Test •
Testing Weak vs Strong Network Ties e.g. Who listens to the Beatles? Q: Given one sample (from last.fm dataset) of who does/doesn’t listen to a particular band, can we reject the hypothesis that this decision comes from high-temperature Ising model (lack of long range correlation)? A: we can for Taylor Swift, Britney Spears, Katy Perry, Rihanna, Lady Gaga; we cannot for Beatles and Muse
Conclusions • Testing properties of high-dimensional distributions requires exponentially many samples • Making assumptions about the distribution being sampled gives leverage • [w/ Pan COLT’17]: Testing Bayes nets with linearly many samples • [w/ Dikkala, Kamath SODA’18]: Testing Ising models with polynomially many samples • [w/ Dikkala, Kamath NIPS’17]: Testing weak vs strong ties from one sample
Testing from a Single Sample • Given one social network, one brain, etc., how can we test the validity of a certain generative model? • Ongoing with Aliakbarpour-Rubinfeld-Zampetakis, testing preferential attachment models
Testing Markov Chains • How to quantify distance between Markov chains? Thanks!

Recommend

Testing High-Dimensional Distributions: Subcube Conditioning, Random Restrictions, and Mean

Testing High-Dimensional Distributions: Subcube Conditioning, Random Restrictions, and Mean Testing Cl ement Canonne (IBM Research) February 25, 2020 Joint work with Xi Chen, Gautam Kamath, Amit Levi, and Erik Waingarten 1 Outline

935 views • 60 slides

proofs of proximity for distribution testing (Distribution testing now with proofs!) Tom Gur

proofs of proximity for distribution testing (Distribution testing now with proofs!) Tom Gur (UC Berkeley) October 14, 2017 FOCS 2017 Workshop: Frontiers in Distribution Testing Joint work with Alessandro Chiesa (UC Berkeley) proofs of

2.25k views • 158 slides

Constructive universal high-dimensional distribution generation through deep ReLU networks Dmytro

Constructive universal high-dimensional distribution generation through deep ReLU networks Dmytro Perekrestenko July 2020 joint work with Stephan M uller and Helmut B olcskei Motivation Deep neural networks are widely used as generative

380 views • 17 slides

What can we say about high- dimensional objects from a low-dimensional representation? 2

Dimension Reduction with Certainty Rasmus Pagh IT University of Copenhagen S CALABLE S IMILARITY ECML-PKDD S EARCH September 21, 2016 Slides: goo.gl/hZoRWo 1 What can we say about high- dimensional objects from a low-dimensional

940 views • 75 slides

Hypothesis Testing for High-Dimensional Regression: Nearly Optimal Sample Size Adel Javanmard

Hypothesis Testing for High-Dimensional Regression: Nearly Optimal Sample Size Adel Javanmard Stanford University- UC Berkeley Based on joint work with Andrea Montanari January 2015 Adel Javanmard (Stanford- UC Berkeley) Hypothesis Testing

820 views • 62 slides

On corrections of classical multivariate tests for high-dimensional data Jian-feng Yao with

Introduction Sample covariance matrices Random Fisher matrices Testing covariance matrices I Testing covariance matrices II Multivariate regressions Conclusions On corrections of classical multivariate tests for high-dimensional data

612 views • 50 slides

Testing Monotone Continuous Distributions on High-dimensional Real Cubes Michal Adamaszek DIMAP,

Introduction Monotone distributions Conclusions Testing Monotone Continuous Distributions on High-dimensional Real Cubes Michal Adamaszek DIMAP, University of Warwick Joint work with Artur Czumaj and Christian Sohler Michal Adamaszek

492 views • 38 slides

High Dimensional Data, Covariance Matrices High Dimensional Data Examples and Application to

Outline Motivation High Dimensional Data, Covariance Matrices High Dimensional Data Examples and Application to Genetics Theoretical Underpinnings Random Matrices Shrinkage Estimation Decision Theory Samprit Banerjee, Ph.D Bayesian

1.34k views • 80 slides

Statistics for High-Dimensional Data: Selected Topics Peter B uhlmann Seminar f ur

Statistics for High-Dimensional Data: Selected Topics Peter B uhlmann Seminar f ur Statistik, ETH Z urich June 2014 High-dimensional linear model: basic concepts in methodology, theory and computation High-dimensional data

998 views • 48 slides

High Dimensional Data Alark Joshi High dimensional data Data with multiple dimensions,

High Dimensional Data Alark Joshi High dimensional data Data with multiple dimensions, multiple variables or multiple attributes Cars dataset Economy Cylinders Displacement Power Weight Mph Year Scatterplots

1.12k views • 65 slides

High-Dimensional Nearest Neighbor Search High-Dimensional Nearest Neighbor Search Who?

High-Dimensional Nearest Neighbor Search High-Dimensional Nearest Neighbor Search Who? About Cliqz and me What? Problem statement Why? Applications How? Exact solutions in low dimensions Approximate

690 views • 19 slides

High-dimensional and infinite-dimensional hyperbolic crosses and their applications in

High-dimensional and infinite-dimensional hyperbolic crosses and their applications in approximation and uncertainty quantification Dinh D ung Vietnam National University, Hanoi, Vietnam Workshop on Information-Based Complexity and

996 views • 42 slides

Three-dimensional Radial Visualization of High-dimensional Continuous or Discrete Datasets Fan

Three-dimensional Radial Visualization of High-dimensional Continuous or Discrete Datasets Fan Dai, Yifan Zhu and Ranjan Maitra Department of Statistics Iowa State University {fd43,yifanzhu,maitra} @ iastate.edu Motivation Multivariate

640 views • 34 slides

Learning the Density Structure of High-Dimensional Data Yoshua Bengio Work done with Martin

Learning the Density Structure of High-Dimensional Data Yoshua Bengio Work done with Martin Monperrus Real Goals of Statistical Learning Given a set D of l examples x t coming from an unknown distribution or process. Discover

208 views • 18 slides

Multi-level Thresholding Tests for High Dimensional Means and Covariance Matrices Song Xi Chen

Multi-level Thresholding Tests for High Dimensional Means and Covariance Matrices Song Xi Chen Guanghua School of Management Center for Statistical Science Peking University Joint work with Bin Guo and Yumou Qiu Two Sample Testing and Signal

875 views • 70 slides

Deep Neural Network Mathematical Mysteries for High Dimensional Learning Stphane Mallat

Deep Neural Network Mathematical Mysteries for High Dimensional Learning Stphane Mallat cole Normale Suprieure www.di.ens.fr/data High Dimensional Learning High-dimensional x = ( x (1) , ..., x ( d )) R d : Classification:

1.34k views • 87 slides

A Spline Dimensional Decomposition for High-Dimensional Uncertainty Quantification Sharif Rahman

INTRODUCTION SDD EXAMPLES CLOSURE A Spline Dimensional Decomposition for High-Dimensional Uncertainty Quantification Sharif Rahman and Ramin Jahanbin The University of Iowa, Iowa City, IA 52242 HDA 2019: 8th Workshop on HDA Zurich,

502 views • 18 slides

Motivation High Dimensional Issues Subspace Clustering Full Dimensional Clustering Issues

Motivation High Dimensional Issues Subspace Clustering Full Dimensional Clustering Issues Accuracy Issues Andrew Foss PhD Candidate Database Lab, Dept. of Computing Science University of Alberta For CMPUT 695 March 2007 Curse

457 views • 12 slides

Random projections, reweighting and half-sampling for high-dimensional statistical inference Art

MC for high-dimensional statistics 1 Random projections, reweighting and half-sampling for high-dimensional statistical inference Art B. Owen Stanford University based on joint works with: Dean Eckles Facebook Inc. Sarah Emerson Oregon

603 views • 56 slides

Distributional Results for Thresholding Estimators in High-Dimensional Gaussian Regression Ulrike

Distributional Results for Thresholding Estimators in High-Dimensional Gaussian Regression Ulrike Schneider University of G ottingen Workshop on High-Dimensional Problems in Statistics ETH Z urich September 23, 2011 Joint work with

1.03k views • 58 slides

High-dimensional consistency in score-based and hybrid structure learning Marloes Maathuis joint

High-dimensional consistency in score-based and hybrid structure learning Marloes Maathuis joint work with Preetam Nandy and Alain Hauser Structure learning We consider random variables ( X 1 , . . . , X p ) with distribution F 0 , where F 0

1.28k views • 82 slides

Gleaner Distribution Auger Testing System Agenda 1 Background 2 Design 3 Testing 4

Scott Harris Kyle Mueggenborg Logan Nightengale Jeremiah Pine Gleaner Distribution Auger Testing System Agenda 1 Background 2 Design 3 Testing 4 Results and Conclusions 1 Gleaner S77 2 Gleaner Transverse Combine 3 Project

531 views • 30 slides

Statistics for high-dimensional data: p-values and confidence intervals Peter B uhlmann

Statistics for high-dimensional data: p-values and confidence intervals Peter B uhlmann Seminar f ur Statistik, ETH Z urich June 2014 High-dimensional data Behavioral economics and genetics (with Ernst Fehr, U. Zurich) n = 1

653 views • 41 slides

High-dimensional covariance estimation based on Gaussian graphical models Shuheng Zhou

High-dimensional covariance estimation based on Gaussian graphical models Shuheng Zhou Department of Statistics, The University of Michigan, Ann Arbor IMA workshop on High Dimensional Phenomena Sept. 26, 2011 Joint work with Philipp R

615 views • 46 slides