an input output library for cluster of smp
play

An Input/Output LIbrary for cluster of SMP Adrien Lebre , Yves - PowerPoint PPT Presentation

Jan 15, 2024 •234 likes •629 views

An Input/Output LIbrary for cluster of SMP Adrien Lebre , Yves Denneulin { Adrien.Lebre,Yves.Denneulin } @imag.fr ID-IMAG (UMR 5132) Laboratory, Grenoble, France BULL - HPC, Echirolles, France. Slide 1/17 6th May 2005 aIOLi - CCGRID05 -

  1. An Input/Output LIbrary for cluster of SMP Adrien Lebre , Yves Denneulin { Adrien.Lebre,Yves.Denneulin } @imag.fr ID-IMAG (UMR 5132) Laboratory, Grenoble, France BULL - HPC, ´ Echirolles, France. Slide 1/17 6th May 2005 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  2. Plan 1 Introduction Context Parallel Input/Output 2 aIOLi system Preamble Principles Technical aspects 3 Results POSIX vs aIOLi MPI I/O vs aIOLi 4 Conclusion Slide 2/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  3. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context Environment Cluster of SMPs Linux High Performance Computing Intensive I/O applications CPU bounded application ⇒ I/O bounded application Remote hard drive I/O Parallel I/O Handling concurrent accesses to a same resource (a file) Accesses : different in size, in offset Example : matrix product Slide 3/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  4. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context Environment Cluster of SMPs Linux High Performance Computing Intensive I/O applications CPU bounded application ⇒ I/O bounded application Remote hard drive I/O Parallel I/O Handling concurrent accesses to a same resource (a file) Accesses : different in size, in offset Example : matrix product Slide 3/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  5. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Example P0 read(fd,buf,1024); //file position=0 Matrix product Specific parts to fetch (according read(fd,buf,1024); //file position=1024 P1 to data distribution: columns, rows, BLOCK/BLOCK, read(fd,buf,1024); //file position=2048 P2 BLOCK/CYCLIC ...) Several requests at the same time : read(fd,buf,1024); //file position=3072 P3 disjoint/contiguous SMP Client “lethal” behavior for I/O subsystem 4 independent requests but contiguous from a global point of view Slide 4/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  6. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Example Matrix product P0 read(fd,buf,1024); //file position=0 read(fd,buf,1024); //file position=4096 ... Specific parts to fetch (according read(fd,buf,1024); //file position=1024 P1 to data distribution: columns, read(fd,buf,1024); //file position=5120 rows, BLOCK/BLOCK, BLOCK/CYCLIC ...) read(fd,buf,1024); //file position=2048 P2 read(fd,buf,1024); //file position=6144 Several requests at the same time : read(fd,buf,1024); //file position=3072 P3 disjoint/contiguous read(fd,buf,1024); //file position=7168 “lethal” behavior for I/O subsystem SMP Client 4 requests have been processed ? What about the new requests ? Contiguous / Disjoint ? Slide 4/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  7. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Example Matrix product P0 read(fd,buf,1024); //file position=0 read(fd,buf,1024); //file position=4096 ... Specific parts to fetch (according read(fd,buf,1024); //file position=1024 P1 to data distribution: columns, read(fd,buf,1024); //file position=5120 ... rows, BLOCK/BLOCK, BLOCK/CYCLIC ...) read(fd,buf,1024); //file position=2048 P2 read(fd,buf,1024); //file position=6144 ... Several requests at the same time : read(fd,buf,1024); //file position=3072 P3 disjoint/contiguous read(fd,buf,1024); //file position=7168 ... “lethal” behavior for I/O subsystem SMP Client 4 requests have been processed ? What about the new requests ? Contiguous / Disjoint ? Slide 4/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  8. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Requirements / constraints Methods for disjoint data (readv) ⇒ complexity of API Collective operations ⇒ Synchronization mechanisms logical view (the files) ⇒ physical placements (block devices) Available solutions - related works Many Parallel File Systems : +/- efficient but hardware dependent “cluster compliant” : PVFS, NFSparallel, GPFS, Lustre Designed for “ Parallel I/O” : PIOUS, VESTA ... Libraries : Focus on portability aspects A lot ! : MPI I/O is the reference. Sophisticated API ⇒ Development overhead / Language bindings Slide 5/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  9. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Parallel I/O Requirements / constraints Methods for disjoint data (readv) ⇒ complexity of API Collective operations ⇒ Synchronization mechanisms logical view (the files) ⇒ physical placements (block devices) Available solutions - related works Many Parallel File Systems : +/- efficient but hardware dependent “cluster compliant” : PVFS, NFSparallel, GPFS, Lustre Designed for “ Parallel I/O” : PIOUS, VESTA ... Libraries : Focus on portability aspects A lot ! : MPI I/O is the reference. Sophisticated API ⇒ Development overhead / Language bindings Slide 5/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  10. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  11. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  12. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  13. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  14. Introduction aIOLi system Context Results Parallel Input/Output Conclusion Context summary ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn SMP Client SMP Client SMP Client ... ... ... ... ... ... P1 Pn P1 Pn P1 Pn IO server 1 IO server 2 IO server n Slide 6/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  15. Introduction Preamble aIOLi system Principles Results Technical aspects Conclusion aIOLi system Objectives Supply Parallel I/O algorithms scheduling policies aggregating access ⇒ efficiency overlapping access Only through the use of the ubiquitous POSIX calls open/creat/lseek/read/write/close ⇒ Simplicity Minimal overhead avoid expensive synchronisation mechanisms (barrier, . . . ) Slide 7/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

  16. Introduction Preamble aIOLi system Principles Results Technical aspects Conclusion Evaluation of “the Linux” I/O stack 1 GB File decomposition on a SMP (kernel 2.4.27, IDPOT cluster, NFS version 3, mpich 1.2.5) 300 1 Observations 2 4 250 8 1 process ⇒ Sequential 1 randomize read (optimal) 200 Completion time + processes ⇒ - performance (sec) 150 1 process in random 100 access ⇒ more performance for large 50 access than parallel approach 0 1 4 8 16 32 64 128 512 1024 4096 Access granularity (KBytes) Slide 8/17 aIOLi - CCGRID05 - May 2005 Adrien Lebre � Bull-ID LIPS 2004 c

Recommend

Input and Output Objectives Survey IO facilities in .NET Framework Class Library file

Input and Output Objectives Survey IO facilities in .NET Framework Class Library file

Input and Output Objectives Survey IO facilities in .NET Framework Class Library file and directory management text files binary files object serialization 2 IO Library Input/output library in System.IO namespace

364 views • 32 slides
16. Recursion 2 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Output: 12

16. Recursion 2 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Output: 12

Motivation: Calculator Goal: we build a command line calculator Example Input: 3 + 5 Output: 8 Input: 3 / 5 Output: 0.6 Input: 3 + 5 * 20 16. Recursion 2 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Output: 12 Building a

387 views • 11 slides
17. Recursion 2 Input: 3 + 5 * 20 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20

17. Recursion 2 Input: 3 + 5 * 20 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20

Motivation: Calculator Goal: we build a command line calculator Example Input: 3 + 5 Output: 8 Input: 3 / 5 Output: 0.6 17. Recursion 2 Input: 3 + 5 * 20 Output: 103 Input: (3 + 5) * 20 Output: 160 Input: -(3 + 5) + 20 Building a Calculator,

245 views • 11 slides
7. Java Input/Output User Input/Console Output, File Input and Output (I/O) 133 User Input (half

7. Java Input/Output User Input/Console Output, File Input and Output (I/O) 133 User Input (half

7. Java Input/Output User Input/Console Output, File Input and Output (I/O) 133 User Input (half the truth) e.g. reading a number: int i = In.readInt(); Our class In provides various such methods. Some of those methods have to deal with wrong

1.09k views • 33 slides
BASIC INPUT/OUTPUT Fundamentals of Computer Science I Outline: Basic Input/Output Screen

BASIC INPUT/OUTPUT Fundamentals of Computer Science I Outline: Basic Input/Output Screen

BASIC INPUT/OUTPUT Fundamentals of Computer Science I Outline: Basic Input/Output Screen Output Keyboard Input Command Line Input File Input Simple Screen Output print("The count is " + str(count)) Outputs the

523 views • 27 slides
The Stream Hierarchy Inheritance of istream and ostream from ios ios istream ostream Stream

The Stream Hierarchy Inheritance of istream and ostream from ios ios istream ostream Stream

17 . Input/Output Mechanism: Standard IO, File IO Standard Input/Output stdin cin stdout cout stderr cerr Input/Output Streams: istream & ostream Input/Output of Built-in Types Input/Output of User-defined Types The Stream Hierarchy

356 views • 4 slides
Problems A problem (in Computer Science) specifies an input/output relationship, P I O . A

Problems A problem (in Computer Science) specifies an input/output relationship, P I O . A

1/23/2010 Problems A problem (in Computer Science) specifies an input/output relationship, P I O . A Constant-Space Model of Computation for First-Order Queries (input) (output) P Steven Lindell How are input/output represented as

412 views • 5 slides
some channel models Input X P(y|x) output Y transition probabilities memoryless: - output at

some channel models Input X P(y|x) output Y transition probabilities memoryless: - output at

some channel models Input X P(y|x) output Y transition probabilities memoryless: - output at time i depends only on input at time i - input and output alphabet finite Example: binary symmetric channel (BSC) 1-p Error Source 0 0 E p Y

528 views • 23 slides
BASIC INPUT/OUTPUT Fundamentals of Computer Science Outline: Basic Input/Output Screen Output

BASIC INPUT/OUTPUT Fundamentals of Computer Science Outline: Basic Input/Output Screen Output

BASIC INPUT/OUTPUT Fundamentals of Computer Science Outline: Basic Input/Output Screen Output Keyboard Input Simple Screen Output System.out.println("The count is " + count); Outputs the sting literal "the count is

651 views • 23 slides
18. Java Input/Output e.g. reading a number: int i = In.readInt(); Our class In provides various

18. Java Input/Output e.g. reading a number: int i = In.readInt(); Our class In provides various

User Input (half the truth) 18. Java Input/Output e.g. reading a number: int i = In.readInt(); Our class In provides various such methods. Some of those methods have to deal with wrong inputs: User Input/Console Output, File Input and Output

114 views • 9 slides
Java Input/Output 6 May 2019 OSU CSE 1 Overview The Java I/O (Input/Output) package java.io

Java Input/Output 6 May 2019 OSU CSE 1 Overview The Java I/O (Input/Output) package java.io

Java Input/Output 6 May 2019 OSU CSE 1 Overview The Java I/O (Input/Output) package java.io contains a group of interfaces and classes similar to the OSU CSE components SimpleReader and SimpleWriter component families Except that

855 views • 69 slides
An example grammar Todays Agenda: Input-Output Issues s --> np, vp. np --> [john]. vp

An example grammar Todays Agenda: Input-Output Issues s --> np, vp. np --> [john]. vp

Some hints regarding input/output with your projects Macros in Prolog Leftovers: The mystery of terminals on the LHS in DCGs A useful extension of print/1 : format/2 Changing the output behavior of the top-level A hook into

504 views • 5 slides
File Input and Output File Input and Output 1 / 9 File input/output input function reads values

File Input and Output File Input and Output 1 / 9 File input/output input function reads values

File Input and Output File Input and Output 1 / 9 File input/output input function reads values entered by a user. File Input and Output 2 / 9 File input/output input function reads values entered by a user. load function read data from a

450 views • 23 slides
Input/Output Cmd Line Input Formatted I/O Formatted Output Formatted Input Volker Sorge

Input/Output Cmd Line Input Formatted I/O Formatted Output Formatted Input Volker Sorge

SSC1: I/O Volker Sorge Overview I/O Streams Basic Streams Data Streams Software Systems Components 1 Standard Streams I/O Redirection Input/Output Cmd Line Input Formatted I/O Formatted Output Formatted Input Volker Sorge Buffered

1.14k views • 59 slides
lecture 19 Input / Output (I/O) - system bus - input e.g. keyboard, mouse - output

lecture 19 Input / Output (I/O) - system bus - input e.g. keyboard, mouse - output

lecture 19 Input / Output (I/O) - system bus - input e.g. keyboard, mouse - output e.g. printer, display screen Mon. March 21, 2016 System "Bus" (up to mid 1990's only) system bus It is cheaper and simpler to have

426 views • 23 slides
Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In

Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In

2/23/2019 channels slides Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In [1]: import d2l from mxnet import nd def corr2d_multi_in(X, K): # First, traverse along the 0th dimension (channel

457 views • 6 slides
Nonlinear Control Lecture # 14 Input-Output Stability Nonlinear Control Lecture # 14 Input-Output

Nonlinear Control Lecture # 14 Input-Output Stability Nonlinear Control Lecture # 14 Input-Output

Nonlinear Control Lecture # 14 Input-Output Stability Nonlinear Control Lecture # 14 Input-Output Stability L Stability Input-Output Models: y = Hu u ( t ) is a piecewise continuous function of t and belongs to a linear space of signals The

418 views • 20 slides
some channel models Input X P(y|x) output Y 1-p Error Source 0 0 E transition

some channel models Input X P(y|x) output Y 1-p Error Source 0 0 E transition

Example: binary symmetric channel (BSC) some channel models Input X P(y|x) output Y 1-p Error Source 0 0 E transition probabilities p Y X E X = ! + 1 1 memoryless: Output Input 1-p - output at time i depends only on input at

323 views • 4 slides
Python Input, output and variables CS111 Plan Three Lectures on Python 1. Input, output and

Python Input, output and variables CS111 Plan Three Lectures on Python 1. Input, output and

Python Input, output and variables CS111 Plan Three Lectures on Python 1. Input, output and variables. 2. Loops and Conditional Statements 3. Turtle graphics Paper and Pencil These lectures (especially tomorrow) will include a few longer

516 views • 33 slides
File class in Java n Programmers refer to input/output as "I/O". n Input is

File class in Java n Programmers refer to input/output as "I/O". n Input is

10/31/15 File class in Java n Programmers refer to input/output as "I/O". n Input is received from the keyboard, mouse, files. File Input and Output output is sent to the console, monitor, files, (Savitch, Chapter 10)

314 views • 8 slides
lti The Goal Input: educational text Output: quiz lti The Goal Input:

lti The Goal Input: educational text Output: quiz lti The Goal Input:

Good Question! Statistical Ranking for Question Generation Michael Heilman and Noah A. Smith lti The Goal Input: educational text Output: quiz lti The Goal Input: educational text Output: quiz Output: ranked list

672 views • 32 slides
W j,i a j a i Chapter 20, Section 5 Input Input Activation Output Output

W j,i a j a i Chapter 20, Section 5 Input Input Activation Output Output

McCullochPitts unit Output is a squashed linear function of the inputs: j W j,i a j a i g ( in i ) = g Neural networks Bias Weight a 0 = 1 a i = g ( in i ) W 0 ,i g in i W j,i a j a i Chapter 20, Section 5

209 views • 4 slides
Input/Output Programming Chapter 3: Section 3.1, 3.2 Input and output (I/O) programming

Input/Output Programming Chapter 3: Section 3.1, 3.2 Input and output (I/O) programming

Input/Output Programming Chapter 3: Section 3.1, 3.2 Input and output (I/O) programming Communicating with I/O devices Busy-wait I/O Interrupt-driven I/O I/O devices Devices may include digital and non-digital

529 views • 33 slides
Input (part 1: devices) Where we are... Two largest aspects of building interactive systems:

Input (part 1: devices) Where we are... Two largest aspects of building interactive systems:

Input (part 1: devices) Where we are... Two largest aspects of building interactive systems: output and input Have looked at basics of output Now look at input 2 Input Generally, input is somewhat harder than output Less

825 views • 47 slides

More recommend