Automatic Benchmarking with JUBE Sebastian Lührs Member of the Helmholtz-Association s.luehrs@fz-juelich.de Jülich Supercomputing Centre Forschungszentrum Jülich GmbH Reykjavík, August 25 th , 2017
Automatic benchmarking? Why should I spend time writing Alternatives: additional configuration files? Can you run your Manual benchmarking: benchmark every day, using ten Was the last run different Easy to use for optimization parameterizations? level three? … Time-consuming Very error-prone Benchmark specific script solution: Optimized Changes can be time-consuming Portability problems Member of the Helmholtz-Association JUBE provides a generic workflow and parameter handling environment, but also supports more flexible and specialised approaches. August 25th, 2017 Nordic High Performance Computing & Applications Workshop 2
What is JUBE? Generic, configurable environment to run, monitor and analyse benchmarks in a systematic way Also usable for testing or production scenarios configuration create result execute 35.13 compile #cpus = 2048 benchmark and input data analyse output opt = O1 #cpus = 4096 opt #cpus time [s] 18.14 O1 2048 35.13 #cpus = 8192 O1 4096 18.14 O1 8192 9.32 JUBE platform configuration specific #cpus = 2048 O3 2048 29.00 configuration O3 4096 15.00 O3 8192 7.68 opt = O3 #cpus = 4096 7.68 XML XML #cpus = 8192 Member of the Helmholtz-Association automatic workflow creation and execution August 25th, 2017 Nordic High Performance Computing & Applications Workshop 3
DEMO Member of the Helmholtz-Association August 25th, 2017 Nordic High Performance Computing & Applications Workshop 4
JUBE availability Jötunn : curl http://apps.fz- juelich.de/jsc/jube/jube2/download.php?versio n=2.1.4 > JUBE.tar.gz tar -xf JUBE.tar.gz cd JUBE-2.1.4 python setup.py install --user Other systems: www.fz-juelich.de/jsc/jube Dependency : Python 2.6 / Python 2.7 / Python 3.2, … Member of the Helmholtz-Association Examples are included August 25th, 2017 Nordic High Performance Computing & Applications Workshop 5
Command line access Start a new benchmark run jube run benchmark.xml Continue an existing benchmark run jube continue benchmark_dir [--id <id>] Analyse the benchmark data jube analyse benchmark_dir [--id <id>] Create and show result representation jube result benchmark_dir [--id <id>] Member of the Helmholtz-Association August 25th, 2017 Nordic High Performance Computing & Applications Workshop 6
Help?! Online documentation and tutorial www.fz-juelich.de/jsc/jube Info mode jube info benchmark_dir [--id <id>] [--step <stepname>] Command line accessible glossary jube help <keyword> Logs jube log benchmark_dir [--id <id>] [--command <cmd>] Debug mode jube --debug run|continue|analyse|result … Verbose mode Member of the Helmholtz-Association jube -v[vv ] run … August 25th, 2017 Nordic High Performance Computing & Applications Workshop 7
HowTo: General file layout XML header line <?xml version=“1.0” encoding=“UTF - 8” ?> <jube> JUBE root tag <benchmark name=“…” outpath =“…”> <parameterset/> benchmark area <fileset/> <substituteset/> set definitions <patternset/> <step/> steps and commands <analyse/> <result/> file analysers </benchmark> </jube> result output creation Member of the Helmholtz-Association >> jube help general_structure August 25th, 2017 Nordic High Performance Computing & Applications Workshop 8
HowTo: Sets Main JUBE information storage technique Four different types of sets are available <parameterset> Parameter storage <fileset> Define all available files <substituteset> Define file substitution <patternset> Define the analyse pattern Set names must be unique Can be initialised by using an additional configuration file Available <parameterset> , <fileset> and <substituteset> are used and combined within a <step> Available <patternset> are used within <analyse> Member of the Helmholtz-Association >> jube help <setname>_tag August 25th, 2017 Nordic High Performance Computing & Applications Workshop 9
HowTo: Command execution <do>…</do> holds the executable commands All commands must use SHELL syntax (they will be executed by using /bin/sh ) JUBE parameter can be used by using $parametername Parameter will be expanded in a pre-processing step Environment parameter can also be used JUBE stops execution if the command’s return code fails Commands will only be executed once All <do> within the same <step> shares the same environment Member of the Helmholtz-Association August 25th, 2017 Nordic High Performance Computing & Applications Workshop 10
DEMO (/home/s.luehrs/JUBE_example) Member of the Helmholtz-Association August 25th, 2017 Nordic High Performance Computing & Applications Workshop 11
Key Concept: Workflow creation Dependency driven step structure Parameter based expansion of steps < parameterset name=“ preset ”> Pre-Process Compile Execute < parameter name=“ const ”>0</ parameter > </ parameterset > const = 0 < parameterset name=“ compset ”> p1 = a p1 = a < parameter name=“p1”> a,b</ parameter > p2 = 1 </ parameterset > < parameterset name=“ execset ”> const = 0 < parameter name=“p2”>1,2</ parameter > p1 = a </ parameterset > p2 = 2 const = 0 < step name=“ preprocess ”> < use >preset</ use > const = 0 </ step > p1 = b < step name=“compile”> p2 = 1 < use >compset</ use > </ step > < step name=“execute” const = 0 depend=“ preprocess,combile ”> p1 = b p1 = b < use >execset</ use > Member of the Helmholtz-Association p2 = 2 </ step > August 25th, 2017 Nordic High Performance Computing & Applications Workshop 12
DEMO (/home/s.luehrs/JUBE_example) Member of the Helmholtz-Association August 25th, 2017 Nordic High Performance Computing & Applications Workshop 13
Key Concept: Directory and data handling Each parameter/step combination runs in a separate sandbox directory Dependent steps can be accessed using sym. links compile execname = my_exe execname = my_exe directory cppflag = -O3 cppflag = -O1 Makefile.in Makefile.in my_exe my_exe Source Source substitution substitution Makefile Makefile sym. link execute cores = cores = cores = cores = cores = cores = 2048 4096 8192 2048 4096 8192 Member of the Helmholtz-Association compile compile compile compile compile compile August 25th, 2017 Nordic High Performance Computing & Applications Workshop 14
DEMO (/home/s.luehrs/JUBE_example) Member of the Helmholtz-Association August 25th, 2017 Nordic High Performance Computing & Applications Workshop 15
Key Concept: Data re-usage Separation of platform dependent and independent configuration options export JUBE_INCLUDE_PATH=... platform.xml jube run --include_path ... JUBE <include-path>...</include-path> configuration select platform job template < parameterset name=“set” init_with =“platform.xml”>... platform.xml </ parametserset > < step ...> < use from=“platform.xml”>...</ use > < do >$submit $submit_script</ do > job template </ step > Member of the Helmholtz-Association < include from=“platform.xml” /> August 25th, 2017 Nordic High Performance Computing & Applications Workshop 16
DEMO (/home/s.luehrs/JUBE_example) Member of the Helmholtz-Association August 25th, 2017 Nordic High Performance Computing & Applications Workshop 17
HowTo: Analyse Files will be analysed by using regular expressions which are defined by the given patterns Multiple occurrences of the same pattern create statistical values (average, minimum, maximum etc.) <analyser name =“...”> analyser area <use >…</ use> used patternset <analyse step =“…”> <file >…</ file> step which should be </analyse> analysed </analyser> list of files Member of the Helmholtz-Association >> jube help analyser_tag August 25th, 2017 Nordic High Performance Computing & Applications Workshop 18
HowTo: Result creation <result> result area <use >…</ use> used analyser <table name=“…”> <column >…< column> table result type definition </table> </result> column definition Member of the Helmholtz-Association >> jube help result_tag >> jube help table_tag August 25th, 2017 Nordic High Performance Computing & Applications Workshop 19
DEMO (/home/s.luehrs/JUBE_example) Member of the Helmholtz-Association August 25th, 2017 Nordic High Performance Computing & Applications Workshop 20
Recommend
More recommend
