SEEKING STABLE CLUSTERS IN THE BLOGOSPHERE VLDB 2007, VIENNA - PowerPoint PPT Presentation

SEEKING STABLE CLUSTERS IN THE BLOGOSPHERE VLDB 2007, VIENNA Nilesh Bansal, Fei Chiang, Nick Koudas Nilesh Bansal, Fei Chiang, Nick Koudas University of Toronto Frank Wm. Tompa p University of Waterloo

The Blogosphere The Blogosphere 2 � The new way to communicate � Millions of text articles posted daily � From all over the globe � A wide variety of topics, from sports to politics y p , p p � Forms a huge repository of human generated content � A high volume temporally ordered stream of text � A high volume temporally ordered stream of text documents � Challenge: discover persistent chatter Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

BlogScope BlogScope 3 � Live blog search and analysis engine � Tracking over 13 million blogs, 100 million posts � Serves thousands of daily visitors � Visit: www.blogscope.net � Visit: www.blogscope.net Demo Today: 4:30 - 6:00 pm Nilesh Bansal Nick Koudas BlogScope A Nilesh Bansal, Nick Koudas, BlogScope: A System for Online Analysis of High Volume Text Streams, VLDB 2007, Demonstration Proposal Nilesh Bansal, Nick Koudas, Searching the Blogosphere, WebDB 2007 Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Persistent Chatter Persistent Chatter 4 � Apple iPhone – January 2007 � Jan first week: Anticipation of iPhone release � Jan 9 th : iPhone release at Macworld � Jan 10 th : Lawsuit by Cisco � Jan 10 : Lawsuit by Cisco � Jan third week: Decrease in chatter about iPhone in chatter about iPhone Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Keyword Clusters Keyword Clusters 5 � When there is a lot of discussion on a topic, a set of keywords will become correlated � Elements in this keyword set will frequently appear together � These keywords form a cluster � Keyword clusters are transient � Keyword clusters are transient � Associated with time interval � As topics recede these clusters will dissolve � As topics recede, these clusters will dissolve Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Stable Clusters - Apple iPhone Stable Clusters Apple iPhone 6 � Persistent for 4 days � Topic drifts � Starts with � Starts with discussion about Apple in general pp g � Moves towards the Cisco lawsuit Note: All keywords are stemmed Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Gap in Clusters Gap in Clusters 7 � Three clusters are shown for Jan 6, 9 and 10 2007; no clusters were discovered for Jan 7 and 8 (related to this topic) � English FA cup soccer game between Liverpool and Arsenal E li h FA b Li l d A l with double goal by Rosicky at Anfield on Jan 6. The same two teams played again on Jan 9 with goals by Bapista and teams played again on Jan 9,with goals by Bapista and Fowler Note: keywords are stemmed Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Why Stable Clusters Why Stable Clusters 8 � Information Discovery � Monitor the buzz in the Blogosphere � “What were bloggers talking about in April last year?” � Query refinement and expansion � Query refinement and expansion � If the query keyword belongs to one of the cluster � Visualization? Vi li ti ? � Show keyword clusters directly to the user � Or show matching blogs Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Overview Overview 9 � Efficient algorithm to identify keyword clusters � BlogScope data contains over 13M unique keywords � Applicable to other streaming text sources � Flickr tags, News articles � Formalize the notion of stable clusters � Efficient algorithms to identify stable clusters � Efficient algorithms to identify stable clusters � BFS, DFS and TA � Amenable to online computation over streaming data � Amenable to online computation over streaming data � Experimental evaluation Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Pipeline Pipeline 10 day 1 day 1 Cluster graph graph day 2 day 3 Keyword Keyword Keyword Keyword d documents graph clusters Stable clusters Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Keyword Graph Keyword Graph 11 Crawler day 1 day 2 day 3 george bush oil � One undirected graph for each day 9 4 � Each keyword forms a node � Each keyword forms a node 8 8 2 2 1 1 5 usa � Edge weight = number of 3 6 iraq war documents in which both the documents in which both the 2 1 keywords occur saddam i th d Graph for i th day G h f Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Pruning the Graph Pruning the Graph 12 � Keep only strong keyword associations � Assess two way association between keyword pairs y y p [Manning & Schutze, 1999] � Pearson Chi-square test � Pearson Chi square test � Correlation coefficient Date File Size # keywords # edges Jan 6 2007 3027MB 2.8 million 138 million Jan 7 2007 Jan 7 2007 2968MB 2968MB 2 8 million 2.8 million 135 million 135 million Keyword graph – after stemming, and removing stop words Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Chi square and Correlation Chi-square and Correlation 13 � Perform a single pass on the graph � For each edge (keyword pair), compute g ( y p ), p day i d i � Chi-square � If confidence is low, delete the edge If confidence is low, delete the edge � Correlation Coefficient � If less than threshold, delete the edge � If less than threshold, delete the edge � Only strong associations remain after pruning pruning Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Segmenting the Keyword Graph Segmenting the Keyword Graph 14 � Graph clustering algorithms [KK’98, FRT’05] � We don’t know the number of clusters � High computational complexity � Graph may not fit in main memory G p y y � Correlation clustering [BBC’04] - expensive � Bi-connected components Bi t d t � An articulation point in a graph is a vertex such that its removal makes the graph disconnected. A graph with l k h h d d A h h at least two edges is bi-connected if it contains no articulation points. ti l ti i t Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Bi-connected Components Bi connected Components 15 � Segment the graph S h h � Find maximal bi-connected components keyword keyword graph k keyword clusters d l Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Finding Bi-connected Components Finding Bi connected Components 16 � Efficient algorithm exists – single pass � Realizable in secondary storage [CGGTV’05] � Perform a DFS on the graph � Maintain two numbers, un and low, with each node un=1 a low=1 Bi-connected Bi t d a b un=2 un=4 Components: low=1 low=4 b d 1. (f,d) (e,f) (d,e) c d e 2. (c,a) (b,c) (a,b) un=5 low=4 c e un=3 f low=1 un=6 un=6 low=4 f Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Cluster Graph Cluster Graph 17 � We have a set of clusters for each time step (day) � Each cluster is a set of keywords � Similarity between two clusters can be assessed � Intersection i e number of common keywords � Intersection, i.e., number of common keywords � Jaccard coefficient � Aim is to find clusters that persist over time Ai i t fi d l t th t i t ti � A graph of clusters over time can be constructed � Undirected graph with edge weight equal to similarity between the keyword clusters Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Example Cluster Graph Example Cluster Graph 18 � Graph over clusters from three time steps G h l f h i � Max temporal gap size, g=1 � Three keyword clusters on each time step � Each node is a keyword cluster � Add a dummy source and sink, and make edges directed � Edge weights represent similarity between clusters Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Formal Problem Definitions Formal Problem Definitions 19 � Weight of path = sum of participating edge weights � Definition: kl-Stable clusters � Find top-k paths of length l with highest weight � Definition: normalized stable clusters � Definition: normalized stable clusters � Find top-k paths of minimum length l min of i i l th l f highest weight normalized by their lengths day 1 day 2 day 3 Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Algorithms for kl-Stable Clusters Algorithms for kl Stable Clusters 20 � Breadth First Search � Fastest, but requires significant amounts of memory � Depth First Search � Slower but has low memory requirements � Slower, but has low memory requirements � Adaptation of the Threshold Algorithm [FLN’01] � Exponential number of I/Os, very slow E i l b f I/O l Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net

Pipeline Pipeline 21 Cluster graph Cluster graph day 1 day 1 BFS, DFS, TA Aggregate or day 2 Normalized Normalized day 3 Keyword Keyword Keyword Keyword d documents clusters graph Stable clusters Seeking Stable Clusters in the Blogosphere, VLDB 2007 www.blogscope.net