Motivation Computation and Aggregation of Quantiles Application at - PowerPoint PPT Presentation

Motivation Computation and Aggregation of Quantiles • Application at Lucent Technologies: from Data Streams software to monitor distributed IP-based services. John Chambers, David James, • Goal: characterize various metrics (e.g. e- Diane Lambert, Scott Vander Wiel mail transaction times), locally and aggregated, updated over time. Vienna, June 17, 2006 • Constraint: computing at the node, amount of data transmitted to server. (related article to appear with discussion in “Statistical Science”) The Idea Quantile Estimation (Approximate, Update, Aggregate) Metrics are often unusually distributed (long tails, bimodal, ...) • Approximate the empirical distribution for each metric & node (agent) • Update each approximation periodically for new data at the node. • Aggregate the ecdfs for relevant groupings of nodes (e.g., regions) Need to estimate quantiles (often in tail).

Update for each agent Aggregate agent records X 1 , X 2 , X 3 , ... Agent Summary fill fill D : Data Buffer D : Data Buffer update Q : Quantile Buffer .10 .25 .50 .75 .90 .95 .98 .99 .10 .25 .50 .75 .90 .95 update report Q : Quantile Buffer Agent Summary .05 .10 .25 .50 .75 .90 .95 .98 .99 .995 .10 .25 .50 .75 .90 .95 report Server Summary .10 .25 .50 .75 .90 .95 Software Software • Objects represent each evolving quantile • R simplifies large-scale simulation studies, estimate: a <- seqQuants(....) • OOP-style functions to simulate updating, with varying statistical assumptions. • R also helps in the algorithm development in aggregating: a$merge(data) (modifies a ) • Using R closures (object contains functions C, by calling an R tracer from C. with a shared environment for updates).

Summary • An example of the productive interaction between applications and research, typical of Bell Labs research (in the old days). • An interesting algorithmic study to estimate distributions with distributed, ongoing data. • The productive computing environment centered on R essential for productivity.