Tutorial: Advanced (Batch) Job Scriptjng Robert Barthel, SCC, KIT - PowerPoint PPT Presentation

Tutorial: Advanced (Batch) Job Scriptjng Robert Barthel, SCC, KIT Steinbuch Centre for Computjng (SCC) www.bwhpc-c5.de Funding:

How to read the following slides Abbreviatjon/Colour code Full meaning prompt of the interactjve shell $ c o m m a n d - o p t v a l u e $ = The full prompt may look like: u s e r @ m a c h i n e : p a t h $ The command has been entered in the interactjve shell session Placeholder for integer, string etc < i n t e g e r > < > = < s t r i n g > Metasyntactjc variables f o o , b a r $ { W O R K S H O P } / p f s / d a t a 1 / s o f t w a r e _ u c 1 / b w h p c / k i t / w o r k s h o p / 2 0 1 7 - 1 0 - 1 1 11/10/2017 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 2

Goal Be descriptjve! Header Header Comment your code e.g. via headers sectjons of script and functjons. Declaratjons Declaratjons (MSUB + defaults) Decipherable names for variables and functjons (MSUB + defaults) Functjons Functjons Organise and structure! Input handling Break complex scripts into simpler blocks Input handling e.g. use functjons Use exit codes Use standardized parameter fmags for script Main sectjon Main sectjon invocatjon. Write job script that runs interactjvely Footer → Then add part for MOAB Footer 11/10/2017 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 3

Best Practjses – Common Problems (1) Do not Run your code, applicatjon, job on login nodes / in $ { H O M E } : for interactjve jobs use msub -I -V Multjnode Job: use workspaces Producing Tbyte of scratch fjles & >10000 File: Change your applicatjon code Need help? Apply for Tiger Team Support. Singlenode Job: use $ : HowTo → Case 1 { T M P D I R } Chain jobs: HowTo → Case 2 Many sequentjal tjny jobs: Bundle to one big job: HowTo → Case 3 Handling walltjme based job aborts: HowTo → Case 4 Use of MPI/OpenMP Parallelisatjon in jobs: htups://indico.scc.kit.edu/indico/event/310/material/slides/13.pdf 11/10/2017 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 4

Rev: MOAB variables htup://www.bwhpc-c5.de/wiki/index.php/Batch_Jobs#Moab_Environment_Variables MSUB variables: # ! / b i n / b a s h # M S U B - N t e s t # M S U B - l n o d e s = 1 : p p n = 1 , m e m = 5 0 m b # M S U B - l w a l l t i m e = 0 0 : 0 5 : 0 0 # M S U B - m n # M S U B - v m y _ o w n _ v a r i a b l e s = “ a r g u m e n t s “ 11/10/2017 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 5

Case 1: Jobs @ $TMPDIR (1) If temporary fjles of job > Gbyte → Run your job at $ { T M P D I R } but ONLY if single node jobs What to do: Generate subdirectory under $ { T M P D I R } = > $ { r u n _ D I R } Copy to $ { r u n _ D I R } Change to $ & program executjon { r u n _ D I R } Copy results to start DIR How? Start with templates: $ { W O R K S H O P } / e x e r c i s e s / 0 3 / 0 1 _ j o b _ r u n _ u n d e r _ l o c a l _ t m p d i r . s h $ { W O R K S H O P } / e x e r c i s e s / 0 3 / 0 1 _ j o b _ r u n _ u n d e r _ l o c a l _ t m p d i r . s h + + $ { W O R K S H O P } / e x e r c i s e s / 0 3 / { 0 1 _ g e n _ f i l e s , 0 1 _ g e n _ f i l e s . i n p } $ { W O R K S H O P } / e x e r c i s e s / 0 3 / { 0 1 _ g e n _ f i l e s , 0 1 _ g e n _ f i l e s . i n p } 11/10/2017 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 6

Case 1: Jobs @ $TMPDIR (2) Code snip: $ { W O R K S H O P } / e x e r c i s e s / 0 3 / 0 1 _ j o b _ r u n _ u n d e r _ l o c a l _ t m p d i r . s h # ! / b i n / b a s h . . . T A S K / T o D o : 1 0 m i n T A S K / T o D o : 1 0 m i n # # a ) T u t o r i a l T o D o : l o a d m o d u l e s I N T E L + M K L * G e n e r a l i s e b l u e c o d e i f n o t l o a d e d * G e n e r a l i s e b l u e c o d e a v o i d i n g r e p e t i t i o n a v o i d i n g r e p e t i t i o n # # b ) D e f i n e y o u r r u n d i r e c t o r y u n d e r t m p d i r * W r i t e c o d e f o r a - h * W r i t e c o d e f o r a - h # # i n c o r p o r a t i n g u s e r n a m e a n d J o b I D / P I D * R e d i r e c t o u t p u t o f b i n a r y * R e d i r e c t o u t p u t o f b i n a r y m k d i r - p v " $ { T M P D I R } / $ { U S E R } . $ { M O A B _ J O B I D : - $ $ } " # # c ) T u t o r i a l T o D o : C h e c k e x i s t e n c e o f r u n d i r e c t o r y # # d ) C o p y f i l e s f r o m s u b m i t d i r e c t o r y # # t o r u n d i r e c t o r y c d $ M O A B _ S U B M I T D I R c p - p v g e n _ f i l e s . x " $ { T M P D I R } / $ { U S E R } . $ { M O A B _ J O B I D : - $ $ } " # # C h e c k i f c o p y s u c c e e d e d c p - p v g e n _ f i l e s . i n p " $ { T M P D I R } / $ { U S E R } . $ { M O A B _ J O B I D : - $ $ } " # # e ) C h a n g e t o r u n d i r e c t o r y ( c h e c k i f s u c c e e d e d ) a n d s t a r t b i n a r y + i n p u t f i l e c d " $ { T M P } / $ { U S E R } . $ { M O A B _ J O B I D } " . / 0 1 _ g e n _ f i l e s . x 0 1 _ g e n _ f i l e s . i n p # # f ) T u t o r i a l T o D o : c h e c k r u n s t a t u s # # g ) t r a n s f e r f i l e s t o s u b m i t d i r e c t o r y c p - p v f i l e s _ * . o u t " $ { M O A B _ S U B M I T D I R } " # # h ) T u t o r i a l T o D o : c l e a n u p r u n _ D I R 11/10/2017 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 7

Case 1: Jobs @ $TMP (3) Decl. + a-c: $ { W O R K S H O P } / s o l u t i o n s / 0 3 / 0 1 _ g e n e r a l i s e d _ j o b _ r u n _ u n d e r _ l o c a l _ t m p d i r . s h # # 1 ) D e f i n e f u l l p a t h o f y o u r b i n a r y E X E = " $ { M O A B _ S U B M I T D I R : - $ { P W D } } / 0 1 _ g e n _ f i l e s . x " S o l u t i o n ! S o l u t i o n ! # # 2 ) D e f i n e o u t p u t f i l e # # = N a m e o f e x e c u t a b l e + J O B I D o r P I D o u t p u t = " $ ( b a s e n a m e $ { E X E } ) _ $ { M O A B _ J O B I D : - $ $ } . l o g " # # 3 ) D e f i n e f u l l p a t h i n p u t f i l e s I n p u t = " $ { M O A B _ S U B M I T D I R : - $ { P W D } } / 0 1 _ g e n _ f i l e s . i n p " # # 4 ) D e f i n e i n p u t f i l e s t o b e c o p i e d c o p y _ l i s t = " $ { E X E } $ { i n p u t } " # # 5 ) D e f i n e f i l e s t o b e c o p i e d b a c k a f t e r r u n , i . e . o u t p u t f i l e s a v e _ l i s t = " $ { o u t p u t } f i l e s _ * . o u t " # # a ) L o a d m o d u l e s I N T E L + M K L i f n o t l o a d e d f o r m o d i n c o m p i l e r / i n t e l n u m l i b / m k l ; d o m o d u l e l i s t 2 > & 1 | g r e p " $ { m o d } " > / d e v / n u l l | | m o d u l e l o a d " $ { m o d } " d o n e # # b ) D e f i n e y o u r r u n d i r e c t o r y a n d a n d g e n e r a t e v i a m k d i r r u n _ D I R = " $ { T M P D I R } / $ { U S E R } . $ { M O A B _ J O B I D : - $ $ } " m k d i r - p v " $ { r u n _ D I R } " # # c ) C h e c k e x i s t e n c e o f r u n d i r e c t o r y i f [ ! - d " $ { r u n _ D I R } " ] ; t h e n e c h o " E R R O R : R u n D I R = $ { r u n _ D I R } d o e s n o t e x i s t " ; e x i t 1 f i 11/10/2017 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 8