Three right directions and three wrong directions for tensor research Michael W. Mahoney Stanford University ( For more info, see: http:// cs.stanford.edu/people/mmahoney/ or Google on “Michael Mahoney”)
Lots and lots of large data! • High energy physics experimental data • Hyperspectral medical and astronomical image data • DNA microarray data and DNA SNP data • Medical literature analysis data • Collaboration and citation networks • Internet networks and web graph data • Advertiser-bidded phrase data • Static and dynamic social network data
“Scientific” and “Internet” data SNPs … AG AG AG AG AA CC GG AG CG AC CC AA CC AA GG TT AG CT CG CG CG AT CT CT AG CT … individuals … AA AG AG AG AA CC AG GG CC AC CC AA CG AA GG TT AG CT CG CG CG AT CT CT AG CT … … AA GG GG GG AA CT GG AA CC AC CG AA CC AA GG TT GG CC CG CG CG AT CT CT AG CT … … AG AG AG AG AA CT GG AG CC CC CG AA CC AA GT TT AG CT CG CG CG AT CT CT AG CT … … AA AG AG AG AA CC AG AG CG AA CC AA CG AA GG TT AA TT GG GG GG TT TT CC GG TT …
Algorithmic vs. Statistical Perspectives Lambert (2000) Computer Scientists • Data: are a record of everything that happened. • Goal: process the data to find interesting patterns and associations. • Methodology: Develop approximation algorithms under different models of data access since the goal is typically computationally hard. Statisticians • Data: are a particular random instantiation of an underlying process describing unobserved patterns in the world. • Goal: is to extract information about the world from noisy data. • Methodology: Make inferences (perhaps about unseen events) by positing a model that describes the random variability of the data around the deterministic model.
Matrices and Data SNPs … AG AG AG AG AA CC GG AG CG AC CC AA CC AA GG TT AG CT CG CG CG AT CT CT AG CT … individuals … AA AG AG AG AA CC AG GG CC AC CC AA CG AA GG TT AG CT CG CG CG AT CT CT AG CT … … AA GG GG GG AA CT GG AA CC AC CG AA CC AA GG TT GG CC CG CG CG AT CT CT AG CT … … AG AG AG AG AA CT GG AG CC CC CG AA CC AA GT TT AG CT CG CG CG AT CT CT AG CT … … AA AG AG AG AA CC AG AG CG AA CC AA CG AA GG TT AA TT GG GG GG TT TT CC GG TT … Matrices provide simple representations of data: • A ij = 0 or 1 (perhaps then weighted), depending on whether word i appears in document j • A ij = -1,0,+1, if homozygous for the major allele, heterozygous, or homozygous for the minor allele Can take advantage of “nice” properties of vector spaces: • structural properties: SVD, Euclidean geometry • algorithmic properties: “everything” is O(n 3 ) • statistical properties: PCA, regularization, etc.
Graphs and Data Interaction graph model of Common variations include: networks: • Directed graphs • Nodes represent entities • Weighted graphs • Edges represent interaction • Bipartite graphs between pairs of entities
Why model data as graphs and matrices? Graphs and matrices - • provide natural mathematical structures that provide algorithmic, statistical, and geometric benefits • provide nice tradeoff between rich descriptive framework and sufficient algorithmic structure • provide regularization due to geometry, either explicitly due to R n or implicitly due to approximation algorithms
What if graphs/matrices don’t work? Employ more general mathematical structures: • Hypergraphs • Attributes associated with nodes • “Kernelize” the data using, e.g., a similarity notion • Generalized linear or hierarchical models • Tensors!! These structures provide greater descriptive flexibility, that typically comes at a (moderate or severe) computational cost.
What is a tensor? (1 of 3) See L.H.Lim’s tutorial on tensors at MMDS 2006.
What is a tensor? (2 of 3)
What is a tensor? (3 of 3) IMPORTANT: This is similar to NLA --- but, there is no reason to expect the “subscript manipulation” methods, so useful in NLA, to yield anything meaningful for more general algebraic structures.
Tensor ranks and data analysis (1 of 3)
Tensor ranks and data analysis (2 of 3) IMPORTANT: These ill-posedness results are NOT pathological--- they are ubiquitous and essential properties of tensors.
Tensor ranks and data analysis (3 of 3) THAT IS: To get a “simple” or “low-rank” tensor approximation, we focus on exceptions to fundamental ill-posedness properties of tensors (i.e., rank-1 tensors and 2-mode tensors).
Historical Perspective on NLA • NLA grew out of statistics (among other areas) (40s and 50s) • NLA focuses on numerical issues (60s, 70s, and 80s) • Large-scale data generation increasingly common (90s and 00s) • NLA has suffered due to the success of PageRank and HITS. • Large-scale scientific and Internet data problems invite us to take a broader perspective on traditional NLA: revisit algorithmic basis of common NLA matrix algorithms revisit statistical underpinnings of NLA expand traditional NLA view of tensors
The gap between NLA and TCS Matrix factorizations: • in NLA and scientific computing - used to express a problem s.t. it can be solved more easily. • in TCS and statistical data analysis - used to represent structure that may be present in a matrix obtained from object-feature observations. TCS: NLA: • motivated by large data applications • emphasis on optimal conditioning, • space-constrained or pass-efficient • backward error analysis issues, models • is running time a large or small • over-sampling and randomness as constant multiplied by n 2 or n 3 . computational resources. MMDS06, MMDS08, … were designed to “bridge the gap” between NLA, TCS, and data applications.
How to “bridge the gap” (Lessons from MMDS) • In a vector space, “everything is easy,” multi-linear captures the inherent intractability of NP-hardness. • Convexity is an appropriate generalization of linear nice algorithmic framework, as with kernels in Machine Learning • Randomness, over-sampling, approximation ... are powerful algorithmic resources but you need to have a clear objective you are solving • Geometry of combinatorial objects (e.g., graphs, tensors, etc.) has positive algorithmic, statistical, and conceptual benefits • Approximate computation induces implicit statistical regularization
Examples of “tensor data” (Acar and Yener 2008) • Chemistry: model fluorescence excitation-emission data in food science: A ijk is samples x emission x excitation . • Neuroscience: EEG data as patients, doses, conditions, etc. varied: A ijk is time samples x frequency x electrodes . • Social network and Web analysis: to discover hidden structures: A ijk is webpages x webpages x anchor text . A ijk is users x queries x webpages . A ijk is advertisers x bidded-phrases x time . • Computer Vision: image compression and face recognition: A ijk is pixel x illumination x expression x viewpoint x person . • Quantum mechanics, large-scale computation, hyperspectral data, climate data, ICA, nonnegative data, blind source separation, NP-hard problems, … “Tensor-based data are particularly challenging due to their size and since many data analysis tools based on graph theory and linear algebra do not easily generalize.” -- MMD06
Three Right Directions
Three Right Directions 1. Understand statistical and algorithmic assumptions s.t. tensor methods work. (NOT just independence.)
Three Right Directions 1. Understand statistical and algorithmic assumptions s.t. tensor methods work. (NOT just independence.) 2. Understand the geometry of tensors. (NOT of vector spaces you unfold to.)
Three Right Directions 1. Understand statistical and algorithmic assumptions s.t. tensor methods work. (NOT just independence.) 2. Understand the geometry of tensors. (NOT of vector spaces you unfold to.) 3. Understand WHY tensors work in physical applications and what this says about less structured data applications (and vice-versa, which has been very fruitful for matrices*.) *(E.g., low-rank off-diagonal blocks are common in matrices -- since the world is 3D, which is not true in less structured applications -- this has significant algorithmic implications.)
Four! Right Directions 1. Understand statistical and algorithmic assumptions s.t. tensor methods work. (NOT just independence.) 2. Understand the geometry of tensors. (NOT of vector spaces you unfold to.) 3. Understand WHY tensors work in physical applications and what this says about less structured data applications (and vice-versa, which has been very fruitful for matrices*.) 4. Understand “unfolding” as a process of defining features. (Since this puts you in a nice algorithmic place.) *(E.g., low-rank off-diagonal blocks are common in matrices -- since the world is 3D, which is not true in less structured applications -- this has significant algorithmic implications.)
Three Wrong directions
Three Wrong directions 1. Viewing tensors as matrices with additional subscripts. (That may be true, but it hampers you, since R n is so nice.)
Three Wrong directions 1. Viewing tensors as matrices with additional subscripts. (That may be true, but it hampers you, since R n is so nice.) 2. Using methods that damage geometry and enhance sparsity. (BTW, you will do this if you don’t understand the underlying geometric and sparsity structure.)
Recommend
More recommend
