dspace clustering dspace clustering
play

DSPACE CLUSTERING DSPACE CLUSTERING VIA PUPPET, HAPROXY AND CEPHFS - PowerPoint PPT Presentation

Mar 20, 2023 •219 likes •480 views

DSPACE CLUSTERING DSPACE CLUSTERING VIA PUPPET, HAPROXY AND CEPHFS VIA PUPPET, HAPROXY AND CEPHFS DSpace-Clustering Bernd Nicklas, Paul Mnch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0 TABLE OF CONTENTS TABLE OF CONTENTS 1. What we

  1. DSPACE CLUSTERING DSPACE CLUSTERING VIA PUPPET, HAPROXY AND CEPHFS VIA PUPPET, HAPROXY AND CEPHFS DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  2. TABLE OF CONTENTS TABLE OF CONTENTS 1. What we want 2. How we get what we want 3. Code example DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  3. WHAT WE WANT WHAT WE WANT DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  4. Automatic, customizable and secure all-in-one con�guration DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  5. Automatic, customizable cluster con�guration which allows for Horizontal scaling, improved availability and improved maintainability DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  6. WHAT DSPACE NEEDS WHAT DSPACE NEEDS DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  7. SOFTWARE STACK SOFTWARE STACK Apache web server Tomcat application server PostgreSQL database server Solr index server DSpace web application File system DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  8. HOW WE GET WHAT WE WANT HOW WE GET WHAT WE WANT DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  9. SOFTWARE STACK SOFTWARE STACK GitLab: source code management & CI & CD Puppet: con�guration management & orchestration Custom Puppet module for DSpace DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  10. CUSTOM PUPPET MODULE FOR DSPACE CUSTOM PUPPET MODULE FOR DSPACE Handles installation & con�guration of all DSpace components Handles �rewall, orchestration and communication in cluster setting DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  11. GITLAB + PUPPET GITLAB + PUPPET DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  12. PUPPET AGENT ON A DSPACE APPLICATION NODE PUPPET AGENT ON A DSPACE APPLICATION NODE Gets con�guration from Puppet Master Clones/pulls the source code Fills and places template �les Runs Maven and Ant Restarts Tomcat DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  13. DSPACE COMPONENTS DSPACE COMPONENTS DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  14. SHARED RESOURCES IN A CLUSTER SETTING SHARED RESOURCES IN A CLUSTER SETTING Database Index AssetStore directory (we use CephFS ) Other shared directories (e.g. exports or oai ; we use CephFS ) Session data? No, actually we just pin the client on a worker DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  15. CHARACTERISTICS OF OUR CLUSTERING CHARACTERISTICS OF OUR CLUSTERING The components portal, application, database and index are installed and con�gured on designated nodes All DSpace-speci�c server components listen to local host Web server listens on the public interface In cluster mode all components are transparently connected via HAProxy Secure �rewall and monitoring con�guration DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  16. CONFIGURATION AND CUSTOMIZATION VIA PUPPET MODULE CONFIGURATION AND CUSTOMIZATION VIA PUPPET MODULE General Con�guration of System/Java/Tomcat environment variables Con�guration of general DSpace-related settings Git repository with slightly modi�ed basic DSpace source code, e.g. hot �xes Git repository with different branches of customized code (maven overlay mechanics) DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  17. CODE EXAMPLE CODE EXAMPLE DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  18. CONFIGURATION CLASS CONFIGURATION CLASS DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  19. ALL-IN-ONE INSTALLATION ALL-IN-ONE INSTALLATION DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  20. CLUSTER INSTALLATION CLUSTER INSTALLATION DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  21. CLUSTER INSTALLATION CLUSTER INSTALLATION DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  22. CLUSTER INSTALLATION CLUSTER INSTALLATION DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  23. CLUSTER INSTALLATION CLUSTER INSTALLATION DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  24. CLUSTER INSTALLATION CLUSTER INSTALLATION DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

  25. THANK YOU! THANK YOU! Visit us on GitHub . Soon on Puppet Forge . DSpace-Clustering Bernd Nicklas, Paul Münch OR2019 Kontakt: dspace@uni-marburg.de CC BY 4.0

Recommend

Getting Started with DSpace 7: Advanced Training Andrea Bollini, 4Science Art Lowel, Atmire

Getting Started with DSpace 7: Advanced Training Andrea Bollini, 4Science Art Lowel, Atmire

Getting Started with DSpace 7: Advanced Training Andrea Bollini, 4Science Art Lowel, Atmire Tim Donohue, DuraSpace Workshop Schedule DSpace 7 UI deep dive (Angular) Customizing UI (beyond branding) DSpace 7 REST API deep dive

1.67k views • 156 slides
Into DSpace using SWORD & GLOBUS Lee Taylor, University of Exeter, UK 11 July 2013 Exeter

Into DSpace using SWORD & GLOBUS Lee Taylor, University of Exeter, UK 11 July 2013 Exeter

Moving BIG DATA Into DSpace using SWORD & GLOBUS Lee Taylor, University of Exeter, UK 11 July 2013 Exeter Landscape 2011 DSpace in use for three independent repositories A brand new ~1M Petabyte data store established to hold all

408 views • 12 slides
Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering?

Lecture 23: Spectral clustering Hierarchical clustering What is a good clustering? Aykut Erdem May 2016 Hacettepe University Last time K-Means An iterative clustering algorithm - Initialize: Pick K random points as cluster

932 views • 64 slides
Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and

Clustering Hierarchical clustering, k-mean clustering Genome 559: Introduction to Statistical and Computational Genomics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity

772 views • 40 slides
Digital Libraries@UC with DSpace/GLAM Universidade de Coimbra (since 1290, Portugal) Ana Lusa

Digital Libraries@UC with DSpace/GLAM Universidade de Coimbra (since 1290, Portugal) Ana Lusa

Digital Libraries@UC with DSpace/GLAM Universidade de Coimbra (since 1290, Portugal) Ana Lusa Silva, Ana Maria Miguis, Bruno Neves, Mrio Bernardes Joint Library Service UC (SIBUC) 13.Jun.2019 Digital Libraries@UC with DSpace/GLAM UC

487 views • 14 slides
Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns

Graph Clustering Graph Clustering What is clustering? What is clustering? Finding patterns in data, or grouping similar groups of data-points together into clusters . Clustering algorithms for numeric data: Lloyds K-means, EM

688 views • 19 slides
Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics

Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics

Clustering Hierarchical clustering and k-mean clustering Genome 373 Genomic Informatics Elhanan Borenstein A quick review The clustering problem: partition genes into distinct sets with high homogeneity and high separation Many

768 views • 43 slides
Supervisor: Jonathan Emery PMSM driving and control 15 V Power dSPACE DS103 SLAVE I/O

Supervisor: Jonathan Emery PMSM driving and control 15 V Power dSPACE DS103 SLAVE I/O

Permanent Magnet Synchronous Motors (PMSM) DRIVING and CONTROL Supervisor: Jonathan Emery PMSM driving and control 15 V Power dSPACE DS103 SLAVE I/O Interface board (CP31) -15V Supply dSPACE to driver +15V P1 P2 P3 230 V 230 V

422 views • 6 slides
Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find

Clustering DSE 210 Clustering in R d Two common uses of clustering: Vector quantization Find a finite set of representatives that provides good coverage of a complex, possibly infinite, high-dimensional space. Finding meaningful

396 views • 36 slides
Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine

Clustering: Hierarchical Clustering and K- Means Clustering Machine Learning 10-601B Seyoung Kim Many of these slides are derived from William Cohen,

660 views • 46 slides
Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering

Clustering and Dimensionality Reduction Preview Clustering K -means clustering Mixture models Hierarchical clustering Dimensionality reduction Principal component analysis Multidimensional scaling Isomap

1.61k views • 17 slides
Sydney eScholarship Repository and DSpace Sten Christensen & Gary Browne Sydney eScholarship

Sydney eScholarship Repository and DSpace Sten Christensen & Gary Browne Sydney eScholarship

Sydney eScholarship Repository and DSpace Sten Christensen & Gary Browne Sydney eScholarship Repository Software: DSpace Version: 1.4.1 Standard customisations ses.library.usyd.edu.au Sydney eScholarship Repository An open access

363 views • 9 slides
Clustering A Categorization of Major Clustering Methods Partitioning Methods

Clustering A Categorization of Major Clustering Methods Partitioning Methods

Clustering What is Clustering? Types of Data in Cluster Analysis Clustering A Categorization of Major Clustering Methods Partitioning Methods Hierarchical Methods 1 2 What is Clustering? What is Clustering? Typical

647 views • 18 slides
Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative

Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative

LUDWIG- MAXIMILIANS- INSTITUTE DATABASE UNIVERSITT FOR SYSTEMS MNCHEN MNCHEN INFORMATICS INFORMATICS GROUP GROUP Subspace Clustering Ensemble Clustering Subspace Clustering, Ensemble Clustering, Alternative Clustering, Multiview

513 views • 9 slides
Reinventing the Wheel? - The Case for the Development of an alternative DSpace Submission

Reinventing the Wheel? - The Case for the Development of an alternative DSpace Submission

Reinventing the Wheel? - The Case for the Development of an alternative DSpace Submission Assistant for Psychological Science Open Repositories 2019 in Hamburg Peter Weiland, Christiane Baier, Roland Ramthun, Johannes Hhmann

593 views • 9 slides
Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering

Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering

Finding Clusters Types of Clustering Approaches: Linkage Based, e.g. Hierarchical Clustering Clustering by Partitioning, e.g. k-Means Density Based Clustering, e.g. DBScan Grid Based Clustering Compendium slides for Guide to Intelligent

1.32k views • 82 slides
Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering

Introduction to Machine Learning, Clustering and EM Barnab s P czos Contents Clustering K-means Mixture of Gaussians Expectation Maximization Variational Methods 2 Clustering 3 K- means clustering What is clustering?

1.32k views • 66 slides
Batch Metadata Editing in DSpace 1.6+ Maureen P. Walsh, The Ohio State University Libraries

Batch Metadata Editing in DSpace 1.6+ Maureen P. Walsh, The Ohio State University Libraries

Batch Metadata Editing Batch Metadata Editing in DSpace 1.6+ Maureen P. Walsh, The Ohio State University Libraries DSpace User Group Meeting at SPARC DRM November 10, 2010 J Batch Metadata Editing Batch Metadata Editing

681 views • 47 slides
API Ruby on Rails UI ES API Hedtek Wijiti API API Elasticsearch Depositing user Build

API Ruby on Rails UI ES API Hedtek Wijiti API API Elasticsearch Depositing user Build

Consuming user API Ruby on Rails UI ES API Hedtek Wijiti API API Elasticsearch Depositing user Build index Updates DSpace Solr Discovery Stats Depositing Consuming user user Ruby on Rails UI DSpace REST API Updates DSpace

172 views • 5 slides
The Angular UI from a user perspective Ignace Deroost & Art Lowel www.atmire.com DSpace

The Angular UI from a user perspective Ignace Deroost & Art Lowel www.atmire.com DSpace

The Angular UI from a user perspective Ignace Deroost & Art Lowel www.atmire.com DSpace Users Visitors Authors / submitters Administrators Machines Public UI Retrievability Speed Architecture Web server Web Browser

588 views • 38 slides
PAC-Bayesian Analysis of Co-clustering, Graph Clustering and Pairwise Clustering Yevgeny Seldin

PAC-Bayesian Analysis of Co-clustering, Graph Clustering and Pairwise Clustering Yevgeny Seldin

PAC-Bayesian Analysis of Co-clustering, Graph Clustering and Pairwise Clustering Yevgeny Seldin Motivation Example Clustering cannot be analyzed without specifying what it will be used for! be used for! Example Cluster then pack

293 views • 28 slides
Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means

Clustering kMeans, Expectation Maximization, Self-Organizing Maps Outline K-means clustering Hierarchical clustering Incremental clustering Probability-based clustering Self-Organising Maps Classification vs. Clustering

1k views • 67 slides
Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat

Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat

Chapter 7: Clustering (Unsupervised Data Organization) 7.1 Hierarchical Clustering 7.2 Flat Clustering 7.3 Embedding into Vector Space for Visualization 7.4 Applications Clustering: unsupervised grouping (partitioning) of objects into

763 views • 49 slides
Graph Clustering Why graph clustering is useful? Distance matrices are graphs as useful as

Graph Clustering Why graph clustering is useful? Distance matrices are graphs as useful as

Graph Clustering Why graph clustering is useful? Distance matrices are graphs as useful as any other clustering Identification of communities in social networks Webpage clustering for better data management of web data Outline

607 views • 41 slides

More recommend