lecture 21 parallel filesystems
play

Lecture 21: Parallel Filesystems Abhinav Bhatele, Department of - PowerPoint PPT Presentation

Apr 06, 2023 •335 likes •455 views

Introduction to Parallel Computing (CMSC498X / CMSC818X) Lecture 21: Parallel Filesystems Abhinav Bhatele, Department of Computer Science Announcements Interim report was due yesterday Assignment 4 is due next week Abhinav Bhatele

  1. Introduction to Parallel Computing (CMSC498X / CMSC818X) Lecture 21: Parallel Filesystems Abhinav Bhatele, Department of Computer Science

  2. Announcements • Interim report was due yesterday • Assignment 4 is due next week Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 2

  3. Parallel filesystem • Home directories and scratch space typically on a parallel file system • Mounted on all login and compute nodes • Also referred to as I/O sub-system http://wiki.lustre.org/Introduction_to_Lustre Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 3

  4. Links between cluster and filesystem 36 total 9 9 9 9 9 9 Leaf Switch Each SU (1 management node, 1 login node, 2 LNET router nodes, 2 gateway nodes) Compute node LNET router node Object storage server (OSS) 9 9 Leaf Switch 9 total Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 4

  5. Different parallel filesystems • Lustre: open-source (lustre.org) • GPFS: General Parallel File System from IBM, now called Spectrum Scale • PVFS: Parallel Virtual File System Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 5

  6. Tape drive • Store data on magnetic tapes • Used for archiving data • Use robotic arms to access the right tape: https://www.youtube.com/watch?v=d- eWDuEo-3Q Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 6

  7. Burst buffer • Fast, intermediate storage between compute nodes and the parallel filesystem • Two designs: • Node-local burst buffer • Remote (shared) burst buffer Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 7

  8. I/O libraries • High-level libraries: HDF5, NetCDF • Middleware: MPI-IO • Low-level: POSIX IO Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 8

  9. Different I/O patterns • One process reading/writing all the data • Multiple processes reading/writing data from/to shared file • Multiple processes reading/writing data from/to different files • Different performance depending upon number of readers/writers, file sizes, filesystem etc. Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 9

  10. I/O profiling tools • Darshan • Lightweight profiling tool from Argonne National Lab • Recorder • Research prototype from UIUC Abhinav Bhatele (CMSC498X/CMSC818X) LIVE RECORDING 10

  11. Abhinav Bhatele 5218 Brendan Iribe Center (IRB) / College Park, MD 20742 phone: 301.405.4507 / e-mail: bhatele@cs.umd.edu

Recommend

Data Management Parallel Filesystems Dr David Henty HPC Training and Support

Data Management Parallel Filesystems Dr David Henty HPC Training and Support

Data Management Parallel Filesystems Dr David Henty HPC Training and Support d.henty@epcc.ed.ac.uk +44 131 650 5960 Overview Lecture will cover Why is IO difficult Why is parallel IO even worse Lustre GPFS Performance

496 views • 25 slides
Taming Metadata Storms in Parallel Filesystems with MetaFS Tim Shaffer Motivation A

Taming Metadata Storms in Parallel Filesystems with MetaFS Tim Shaffer Motivation A

Taming Metadata Storms in Parallel Filesystems with MetaFS Tim Shaffer Motivation A (well-meaning) user tried to run a bioinformatics pipeline to analyze a batch of genomic data. MAKER 2 Motivation Shared filesystem performance became

507 views • 35 slides
Introduction to serial HDF5 Matthieu Haefele Saclay, April 2018, Parallel filesystems and

Introduction to serial HDF5 Matthieu Haefele Saclay, April 2018, Parallel filesystems and

Introduction to serial HDF5 Matthieu Haefele Saclay, April 2018, Parallel filesystems and parallel IO libraries PATC@MdS Matthieu Haefele Training outline Day 1: AM: Serial HDF5 (M. Haefele) PM: Parallel IO and parallel HDF5 (M. Haefele)

1.09k views • 47 slides
Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems

Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems

Applying Machine Learning to Understand Write Performance of Large-scale Parallel Filesystems Presented by Bing Xie Bing Xie, Zilong Tan, Philip Carns, Jeff Chase, Kevin Harms, Jay Lofstead, Sarp Oral, Sudharshan S. Vazhkudai, Feiyi Wang ORNL

222 views • 18 slides
Mixing Hadoop and HPC Workloads on Parallel Filesystems Esteban Molina-Estolano * , Maya Gokhale

Mixing Hadoop and HPC Workloads on Parallel Filesystems Esteban Molina-Estolano * , Maya Gokhale

Mixing Hadoop and HPC Workloads on Parallel Filesystems Esteban Molina-Estolano * , Maya Gokhale , Carlos Maltzahn * , John May , John Bent , Scott Brandt * * UC Santa Cruz, ISSDM, PDSI Lawrence Livermore National Laboratory

516 views • 22 slides
Parallel Data Migration Between GPFS Filesystems via the iRODS Rule Engine Ilari Korhonen, PDC

Parallel Data Migration Between GPFS Filesystems via the iRODS Rule Engine Ilari Korhonen, PDC

Parallel Data Migration Between GPFS Filesystems via the iRODS Rule Engine Ilari Korhonen, PDC Center for High Performance Computing The 12th Annual iRODS Users Group Meeting June 9th 2020, The Internet Background PDC is a HPC center based

236 views • 19 slides
I/O 2 / Filesystems 1 1 Changelog Changes made in this version not seen in fjrst lecture: 13

I/O 2 / Filesystems 1 1 Changelog Changes made in this version not seen in fjrst lecture: 13

I/O 2 / Filesystems 1 1 Changelog Changes made in this version not seen in fjrst lecture: 13 November: Correct cluster number on FAT directory entry slide. 1 last time page replacement modifjcations for scanning Linux: guess fjle pages used

2.03k views • 95 slides
This time we'll talk about filesystems. We'll start out by looking at disk partitions, which are

This time we'll talk about filesystems. We'll start out by looking at disk partitions, which are

Linux System Administration SSU: Disks and Filesystems 1 This time we'll talk about filesystems. We'll start out by looking at disk partitions, which are the traditional places to put filesystems. Then we'll take a look at logical

711 views • 58 slides
Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Moreno Baricevic Gilberto Daz Axel Kohlmeyer Stefano Cozzini ULA ICTP CNR-IOM DEMOCRITOS Merida, VENEZUELA Trieste, ITALY Trieste, ITALY Introduction Introduction to storage and to storage and filesystems filesystems Introduction

1.19k views • 51 slides
Parallel IO concepts (MPI-IO and pHDF5) Matthieu Haefele Saclay, April 2018, Parallel

Parallel IO concepts (MPI-IO and pHDF5) Matthieu Haefele Saclay, April 2018, Parallel

Parallel IO concepts (MPI-IO and pHDF5) Matthieu Haefele Saclay, April 2018, Parallel filesystems and parallel IO libraries PATC@MdS Matthieu Haefele Outline Day 1 Morning: HDF5 in the context of Input/Output (IO) HDF5 Application

431 views • 32 slides
1 Analysis of sequential algorithms: The PRAM Model a Parallel RAM RAM model (Random Access

1 Analysis of sequential algorithms: The PRAM Model a Parallel RAM RAM model (Random Access

Outline Lecture 1: Multicore Architecture Concepts Lecture 2: Parallel programming with threads and tasks Design and Analysis Lecture 3: Shared memory architecture concepts of Parallel Programs Lecture 4: Design and analysis of parallel

356 views • 8 slides
Lecture 10: Parallel Patterns: The What and How of Parallel Programming G63.2011.002/G22.2945.001

Lecture 10: Parallel Patterns: The What and How of Parallel Programming G63.2011.002/G22.2945.001

Lecture 10: Parallel Patterns: The What and How of Parallel Programming G63.2011.002/G22.2945.001 November 9, 2010 Embarrassing Partition Pipelines Reduction Scan Tentative Plan for Rest of Class Today: Parallel Patterns Nov 16:

989 views • 86 slides
Lecture 2: Parallel Architectures Lecture 2: Parallel Architectures and Programming Models

Lecture 2: Parallel Architectures Lecture 2: Parallel Architectures and Programming Models

ECE 451/566 - Intro. to Parallel & Distributed Prog. ECE-451/ECE-566 - Introduction to Parallel and Distributed Programming Lecture 2: Parallel Architectures Lecture 2: Parallel Architectures and Programming Models Department of Electrical

480 views • 32 slides
Shared Memory Programming with OpenMP Lecture 3: Parallel Regions Parallel region directive

Shared Memory Programming with OpenMP Lecture 3: Parallel Regions Parallel region directive

Shared Memory Programming with OpenMP Lecture 3: Parallel Regions Parallel region directive Code within a parallel region is executed by all threads. Syntax: Fortran: !$OMP PARALLEL block !$OMP END PARALLEL C/C++: #pragma omp

321 views • 15 slides
STORAGE@TGCC & LUSTRE FILESYSTEMS WORKING & BEST PRACTICES Philippe DENIEL | |

STORAGE@TGCC & LUSTRE FILESYSTEMS WORKING & BEST PRACTICES Philippe DENIEL | |

STORAGE@TGCC & LUSTRE FILESYSTEMS WORKING & BEST PRACTICES Philippe DENIEL | | CEA/DAM/DIF PATC PARALLEL I/O APRIL 25TH, 2018 23 AVRIL 2018 CEA | 10 AVRIL 2012 | PAGE 1 AGENDA TGCC storage architecture TGCC storage workspaces

484 views • 26 slides
An Introduction to the Lustre Parallel File System Tom Edwards tedwards@cray.com C O M P U T E

An Introduction to the Lustre Parallel File System Tom Edwards tedwards@cray.com C O M P U T E

An Introduction to the Lustre Parallel File System Tom Edwards tedwards@cray.com C O M P U T E | S T O R E | A N A L Y Z E Agenda Introduction to storage hardware RAID Parallel Filesystems Lustre Mapping

565 views • 35 slides
An Introduction to the Lustre Parallel File System Tom Edwards tedwards@cray.com C O M P U T E

An Introduction to the Lustre Parallel File System Tom Edwards tedwards@cray.com C O M P U T E

An Introduction to the Lustre Parallel File System Tom Edwards tedwards@cray.com C O M P U T E | S T O R E | A N A L Y Z E Agenda Introduction to storage hardware RAID Parallel Filesystems Lustre Mapping

639 views • 33 slides
Lecture 20: Parallel Matrix Multiplication Abhinav Bhatele, Department of Computer Science

Lecture 20: Parallel Matrix Multiplication Abhinav Bhatele, Department of Computer Science

High Performance Computing Systems (CMSC714) Lecture 20: Parallel Matrix Multiplication Abhinav Bhatele, Department of Computer Science Summary of last lecture Parallel sorting is used in many HPC applications Two categories of parallel

377 views • 10 slides
Lecture Notes on Parallel Scientific Computing Tao Yang Department of Computer Science

Lecture Notes on Parallel Scientific Computing Tao Yang Department of Computer Science

Lecture Notes on Parallel Scientific Computing Tao Yang Department of Computer Science University of California at Santa Barbara Contents 1 Design and Implementation of Parallel Algorithms 2 1.1 A simple model of parallel computation . . .

716 views • 42 slides
Parallel Computation Patterns Scan (Prefix Sum) Objective To master parallel scan (prefix

Parallel Computation Patterns Scan (Prefix Sum) Objective To master parallel scan (prefix

Lecture Lecture 4.4 4.4 Parallel Computation Patterns Scan (Prefix Sum) Objective To master parallel scan (prefix sum) algorithms frequently used for parallel work assignment and resource allocation A key primitive in many

498 views • 10 slides
Hard State Revisited: Network Filesystems Hard State Revisited: Network Filesystems Jeff Chase

Hard State Revisited: Network Filesystems Hard State Revisited: Network Filesystems Jeff Chase

Hard State Revisited: Network Filesystems Hard State Revisited: Network Filesystems Jeff Chase CPS 212, Fall 2000 Network File System (NFS) Network File System (NFS) server client syscall layer user programs VFS syscall layer NFS VFS

430 views • 29 slides
Lecture 3 Log into Linux Questions about Homework 1? Reminder: Additional on-line

Lecture 3 Log into Linux Questions about Homework 1? Reminder: Additional on-line

Lecture 3 Log into Linux Questions about Homework 1? Reminder: Additional on-line references Thursday, September 2 CS 375 UNIX System Programming - Lecture 3 1 Outline Filesystems BASH - Bourne Again SHell Redirection

520 views • 27 slides
Pipelining hazards, Parallel Data, Threads Lecture 18 CDA 3103 07-21-2014 Review Software

Pipelining hazards, Parallel Data, Threads Lecture 18 CDA 3103 07-21-2014 Review Software

Pipelining hazards, Parallel Data, Threads Lecture 18 CDA 3103 07-21-2014 Review Software Hardware Parallel Requests Warehouse Smart Assigned to computer Scale Phone e.g., Search Katz Computer Parallel Threads

1.06k views • 89 slides
Block Devices, Filesystems And Block Layer Alignment Christoph Anton Mitterer

Block Devices, Filesystems And Block Layer Alignment Christoph Anton Mitterer

. DCACHE WORKSHOP DEUTSCHES ELEKTRONEN-SYNCHROTRON DESY ZEUTHEN April 19, 2012 Block Devices, Filesystems And Block Layer Alignment Christoph Anton Mitterer christoph.anton.mitterer@lmu.de BLOCK DEVICES, FILESYSTEMS AND BLOCK LAYER

843 views • 45 slides

More recommend