on the complex network clustering using dryadlinq
play

On the complex network clustering using DryadLINQ Stojan Trajanovski - PowerPoint PPT Presentation

Feb 28, 2023 •310 likes •390 views

Data Centric Networking (R202) Open source project study On the complex network clustering using DryadLINQ Stojan Trajanovski ( st508 ) MPhil in Advanced Computer Science Motivation Why going parallel in complex networks analysis? Online

  1. Data Centric Networking (R202) Open source project study On the complex network clustering using DryadLINQ Stojan Trajanovski ( st508 ) MPhil in Advanced Computer Science

  2. Motivation Why going parallel in complex networks analysis? • Online social networks, Internet graph o millions of users (Facebook, Twitter …) o increased computational complexity • Why is prospective? o some actions are fully independent o increased hardware performance - multi-core - network clusters, global cloud clusters Data Centric Networking (R202) presenter: Stojan Trajanovski (st508) 2

  3. Motivation Why using PLINQ/DryadLINQ? • Inherited LINQ behaviour o declarative and imperative programming o T-SQL syntax in your code - no more SQL server store-procedures - optimized performance - inherited SELECT, GROUP/ORDER BY • + Dryad/Parallel processing o optimized job management Data Centric Networking (R202) presenter: Stojan Trajanovski (st508) 3

  4. Why not (mainly pure technical reasons)? • problems even with Microsoft concepts o requires .NET environment anyway o evaluated only on newest Microsoft OSs o head node: - > Windows Server ’08 OS (problems with ‘03) - more than 500G HD, 8 MB memory o computational nodes (at least Windows 7) o no Windows Azure support � o Someone mentioned Linux/MacOS? ☺ Data Centric Networking (R202) presenter: Stojan Trajanovski (st508) 4

  5. My application/solution? Using PLINQ/DryadLINQ for network clustering? • K-means clustering o parallel performs better o the approach: - parallelize the method o the results - significantly better time performance o TO DO - more clustering approaches, comparison … Data Centric Networking (R202) presenter: Stojan Trajanovski (st508) 5

  6. Some plots Parallel vs. non parallel LINQ ( dataset: ) different values of N= { 100,200,500,1000} Data Centric Networking (R202) presenter: Stojan Trajanovski (st508) 6

  7. o Questions?? o Short Discussion - still work in progress ... Data Centric Networking (R202) presenter: Stojan Trajanovski (st508) 7

Recommend

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find a finite set of representatives that provides good coverage of a complex, possibly infinite, high-dimensional space. Finding meaningful

396 views • 36 slides
DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language Overview Motivation for DryadLINQ Design Implementation Performance Q & A Motivation More machines + more code =

680 views • 18 slides
Bi-clustering and co-clustering Going further in cluster analysis and classifjcation: publics ou

Bi-clustering and co-clustering Going further in cluster analysis and classifjcation: publics ou

HAL Id: hal-01810380 manant des tablissements denseignement et de Summer School on Clustering, Data Analysis and Visualization of Complex Data, May 2018, Catania, Going further in cluster analysis and classifjcation: Bi-clustering and

1.23k views • 67 slides
The game of go as a complex network The game of go as a complex network Bertrand Georgeot,

The game of go as a complex network The game of go as a complex network Bertrand Georgeot,

The game of go as a complex network The game of go as a complex network Bertrand Georgeot, Olivier Giraud, Vivek Kandiah supported by EC FET Open project NADINE B.G. and O. Giraud, Europhysics Letters 97 68002 (2012) Quantware group Laboratoire

236 views • 19 slides
Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering? Aykut Erdem May 2016 Hacettepe University Last time K-Means An iterative clustering algorithm - Initialize: Pick K random points as cluster

932 views • 64 slides
Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and Computational Genomics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity

772 views • 40 slides
Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns in data, or grouping similar groups of data-points together into clusters . Clustering algorithms for numeric data: Lloyds K-means, EM

688 views • 19 slides
DryadLINQ by Yuan Yu et al., OSDI08 Ilias Giechaskiel Cambridge University, R212

DryadLINQ by Yuan Yu et al., OSDI08 Ilias Giechaskiel Cambridge University, R212

DryadLINQ by Yuan Yu et al., OSDI08 Ilias Giechaskiel Cambridge University, R212 ig305@cam.ac.uk January 28, 2014 Conclusions Takeaway Messages SQL cannot express iteration Unsuitable for machine learning, graph processing, etc.

236 views • 20 slides
Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics

Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics

Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity and high separation Many

768 views • 43 slides
Discovering Packet Structure through Lightweight Hierarchical Clustering Abdulrahman Hijazi

Discovering Packet Structure through Lightweight Hierarchical Clustering Abdulrahman Hijazi

Understanding Network Traffic Approach Network Traffic Clustering: ADHIC Evaluation Discovering Packet Structure through Lightweight Hierarchical Clustering Abdulrahman Hijazi Hajime Inoue Ashraf Matrawy P .C. van Oorschot Anil Somayaji

540 views • 29 slides
Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine Learning 10-601B Seyoung Kim Many of these slides are derived from William Cohen,

660 views • 46 slides
Distributed Aggregation for Data- Parallel Computing Interfaces and Implementations Yuan Yu

Distributed Aggregation for Data- Parallel Computing Interfaces and Implementations Yuan Yu

Distributed Aggregation for Data- Parallel Computing Interfaces and Implementations Yuan Yu Pradeep Kumar Gunda Michael Isard Microsoft Research Silicon Valley Dryad and DryadLINQ Automatic query plan generation by DryadLINQ Automatic

721 views • 30 slides
Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering Mixture models Hierarchical clustering Dimensionality reduction Principal component analysis Multidimensional scaling Isomap

1.61k views • 17 slides
Transaction clustering using network traffic blockchains analysis for Bitcoin and derived

Transaction clustering using network traffic blockchains analysis for Bitcoin and derived

Transaction clustering using network traffic analysis for Bitcoin and derived Transaction clustering using network traffic blockchains analysis for Bitcoin and derived blockchains Biryukov, Tikhomirov Introduction Network privacy Alex

491 views • 30 slides
Clustering A Categorization of Major Clustering Methods Partitioning Methods

Clustering A Categorization of Major Clustering Methods Partitioning Methods

Clustering What is Clustering? Types of Data in Cluster Analysis Clustering A Categorization of Major Clustering Methods Partitioning Methods Hierarchical Methods 1 2 What is Clustering? What is Clustering? Typical

647 views • 18 slides
Practical Advances in Complex Root Clustering Collaborative and ongoing works R. Imbach 1 , V. Pan

Practical Advances in Complex Root Clustering Collaborative and ongoing works R. Imbach 1 , V. Pan

Practical Advances in Complex Root Clustering Collaborative and ongoing works R. Imbach 1 , V. Pan 2 , M. Pouget 3 , C. Yap 1 1 Courant Institute of Mathematical Sciences, New York University, USA 2 Lehman College, City University of New York, USA

1.27k views • 100 slides
Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative

Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative

LUDWIG- MAXIMILIANS- INSTITUTE DATABASE UNIVERSITT FOR SYSTEMS MNCHEN MNCHEN INFORMATICS INFORMATICS GROUP GROUP Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative Clustering, Multiview

513 views • 9 slides
Context-Aware Clustering for SDN Enabled Network Ran Duo Celimuge Wu Tsutomu

Context-Aware Clustering for SDN Enabled Network Ran Duo Celimuge Wu Tsutomu

Context-Aware Clustering for SDN Enabled Network Ran Duo Celimuge Wu Tsutomu Yoshinaga Yusheng Ji The University of Electro-Communications, National Institute of Informatics CONTENT Introduction 1 Related woks

666 views • 22 slides
network Complex Networks Complex Networks experience for professional or social purposes : a

network Complex Networks Complex Networks experience for professional or social purposes : a

Overview of Overview of Outline Complex Networks Complex Networks Complex Networks Principles of Complex Systems Basic definitions Basic definitions Basic definitions Examples of Examples of Course CSYS/MATH 300, Fall, 2009 Complex

754 views • 30 slides
DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level

DryadLINQ A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language Arman Idani 14 Feb 2012 R202 Data Centric Networking Background Major Distributed Computing Frameworks MapReduce Dryad

494 views • 28 slides
network Complex Networks Complex Networks Prof. Peter Dodds Nutshell Nutshell noun

network Complex Networks Complex Networks Prof. Peter Dodds Nutshell Nutshell noun

Overview of Overview of Complex Networks Complex Networks Overview of Complex Networks Principles of Complex Systems Basic definitions Basic definitions Examples of Examples of CSYS/MATH 300, Spring, 2013 | #SpringPoCS2013 Complex

486 views • 9 slides
network Complex Networks Complex Networks Prof. Peter Dodds Nutshell Nutshell noun Basic

network Complex Networks Complex Networks Prof. Peter Dodds Nutshell Nutshell noun Basic

Overview of Thesaurus deliciousness: Overview of Complex Networks Complex Networks Complex Networks Principles of Complex Systems Basic definitions Basic definitions Examples of Examples of CSYS/MATH 300, Fall, 2011 Complex Networks

563 views • 21 slides
Unsupervised clustering with growing self-organizing neural network A comparison with non-neural

Unsupervised clustering with growing self-organizing neural network A comparison with non-neural

Unsupervised clustering with growing self-organizing neural network A comparison with non-neural approach Martin Hynar, Michal Burda, Jana Sarmanov a Department of Computer Science, V SB Technical University of Ostrava, 17.

473 views • 24 slides
Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering

Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering

Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering Clustering by Partitioning, e.g. k-Means Density Based Clustering, e.g. DBScan Grid Based Clustering Compendium slides for Guide to Intelligent

1.32k views • 82 slides

More recommend