Center for Information Services and High Performance Computing (ZIH) Debugger Usage at HRSK 2009-03-19 Zellescher Weg 12 Willers-Bau A106 Tel. +49 351 - 463 - 31945 Matthias Lieber (Matthias.Lieber@tu-dresden.de)
Why using a Debugger? Your program… – terminates abnormally – produces wrong results – shows incomprehensible behavior � You want to know what your program is (really) doing Typical example: your program crashes with a segmentation fault % gcc myprog.c –o myprog What’s going % ./myprog wrong? Segmentation fault %
What can a Debugger do? Observe a running program: – Print variables (scalars, arrays, structures / derived types, classes) – Inform about current source code line and function (function call stack) Control the program execution: – Stop the program at a specific source code line (breakpoint) – Stop the program by evaluating variable expressions (watchpoint) – Stop the program before terminating abnormally – Execute the program line-by-line (stepping)
Function call stack program main Function call stack: information of all ... calls active functions of a process which have call print_date been called but not yet finished. ... end program main Stack: subroutine print_date #2: date_and_time ... calls #1: print_date call date_and_time #0: main ... end subroutine print_date subroutine date_and_time � program is here ...
Typical Usage of a Debugger Compile the program with the –g compiler flag – gcc –g myprog.c –o myprog Run the program under control of the debugger: – ddt ./myprog – Locate the position of the problem and examine variables – Find out why the program shows unexpected behavior Edit the source code, modify parameters, etc. Repeat until problem is solved
Debugging Tools at HRSK graphical user parallel vendor / name command Mars Deimos interface support • • Threads GNU Debugger gdb GNU Data Display • • • Threads ddd Debugger Threads, Allinea Distributed • • • ddt Debugging Tool MPI Threads, • • • Totalview totalview MPI • Valgrind Threads valgrind
Example: Allinea DDT (Distributed Debugging Tool) Help Button Process and thread selection Process control: run, stop, stepping Variables pane Source view Source file browser Output, Breakpoints, Evaluation Watchpoint, window Call stack
Example: Totalview
Example: GNU Debugger (gdb)
Basic GDB commands GDB Reference Sheet: http://users.ece.utexas.edu/~adnan/gdb-refcard.pdf Help (on <command>) h, help (<command>) Run program with options r, run <options> Print program call stack bt, backtrace Print source code (at <line>) l, list (<line>) Display value of expression (variables) p, print <expression> Step to next line (n ... over func. calls) s, step / n, next Navigate in the program call stack up / down Set breakpoint b, break <file:line> List breakpoints info break
Example: GNU DDD (GUI for gdb)
Allinea DDT Commercial debugger from Allinea DDT - Distributed Debugging Tool Support for parallel applications (Threads, OpenMP, MPI, and hybrid) Supports C, C++, Fortran Internally utilizes the GNU debugger (or other) Memory debugging and other advanced features Intuitive graphical user interface More Information: http://www.allinea.com
DDT Usage at HRSK – 3 methods to start DDT On login CPUs (not recommended) – works for short runs only (10 minutes runtime limit) As batch job – start your program like normal batch job, but under control of DDT – closing DDT finishes batch job – new job necessary for next DDT run From interactive batch session (my recommendation) – start a shell on compute CPUs and start program from there – very useful if you plan to do multiple debugging sessions: consecutive program runs in one batch job – in the interactive batch session you can work like on the login CPUs – but with your own private CPUs
DDT Usage at HRSK – serial programs Compile your code with the –g compiler flag (common across all compilers) icc –g –O0 myprog.c –o myprog Set the environment for DDT module load ddt Start the program under control of DDT – on the login CPUs (short runs only): ddt ./myprog – as batch job: bsub –W 2:00 –n 1 ddt ./myprog – from interactive batch session: bsub –W 2:00 –n 1 –Is bash ddt ./myprog
Example: Debugging with DDT on Mars
Example: Debugging with DDT on Mars Compiling myproc.c Loading module for DDT Starting interactive batch session Starting DDT to debug myprog
Example: Debugging with DDT on Mars Run and Debug a Program
Run Example: Debugging with DDT on Mars
Example: Debugging with DDT on Mars Hit “Play” Button to start program
Example: Debugging with DDT on Mars Segmentation Fault! Hit “Pause” to stop the program
Example: Debugging with DDT on Mars This is the line where the program crashed
Example: Debugging with DDT on Mars i = 291016 No the expected value!
Example: Debugging with DDT on Mars Restart the debugging session to find out what happens to “i”
Example: Debugging with DDT on Mars
Example: Debugging with DDT on Mars Adding a Breakpoint just before the while loop
Example: Debugging with DDT on Mars run the program Breakpoint is shown by red dot
Example: Debugging with DDT on Mars Hit “Pause” to examine variables
Example: Debugging with DDT on Mars Open the “Locals” tab to see all local variables
Example: Debugging with DDT on Mars Use the “step” Button to step line-by- line through the program i = 291016 before the loop starts! Forgot initialization?!
Example: Debugging with DDT on Mars Use the “step” Button to step line-by- line through the program
Example: Debugging with DDT on Mars Use the “step” Button to step line-by- line through the program
Example: Debugging with DDT on Mars
DDT Usage at HRSK – MPI parallel programs Mars Deimos (MPI Implementation: altix-mpi) (MPI Implementation: OpenMPI) Login CPUs module load ddt --- ddt ./myprog Batch Job module load ddt module load ddt bsub –W 2:00 –n8 ddt ./myprog bsub –W 2:00 –n8 –Is –aopenmpi ddt ./myprog batch session Interactive module load ddt module load ddt bsub –W 2:00 –n8 –Is bash bsub –W 2:00 –n8 –Is –aopenmpi bash ddt ./myprog ddt ./myprog
Other Debugging Options (1) Attach to a running program Useful if program shows incomprehensive behavior – Hangs – Wrong results But program was not started under debugger control Core file debugging Core dumps are memory and register state of a crashed program written to disk These files can be analyzed after program run with a debugger
Other Debugging Options (2) Memory debugging with Valgrind Detects memory management bugs – Use of uninitialized or free’d memory – Access memory out off allocated bounds – etc. Deimos only
Core file debugging Core dumps are memory and register state of a crashed program written to disk These files can be analyzed after program run with a debugger Compile with –g compiler flag: (Fortran) ifort -g myprog.f90 -o myprog (C) icc -g myprog.c -o myprog Check current core dump limit (reports kB): ulimit –c If necessary, set the limit for core dumps: (set limit to 100MB) ulimit –c 100000 Intel Fortran requires this environment variable to be set before running program: export decfort_dump_flag=yes (Intel Fortran only) Start the program without debugger: ./myprog When the program crashes, core dump(s) [HOSTNAME].[PID].core will be created Analyze the core dump with DDT: – Run and select the core dump via “ Core file…” button ddt ./myprog
Core dump of myprog Core file debugging
Open a Core File Core file debugging
Choose core file Core file debugging
Core file debugging
Memory Debugging with Valgrind (Deimos only) Detects memory management bugs: – Use of uninitialized memory – Access free’d memory – Access memory out off allocated bounds – Access inappropriate areas on the stack – Memory leaks – when pointers to allocated areas get lost forever – Mismatched use of malloc and free (C, Fortran), new and delete (C++) – Wrong use of memcpy() and related functions Available on Deimos via module load valgrind Simply run program under valgrind: valgrind ./myprog More Information: http://www.valgrind.org
Memory Debugging with Valgrind (Deimos only) The buggy program from the DDT example works on Deimos with gcc Let’s see if valgrind finds the bug…
Memory Debugging with Valgrind (Deimos only) Loading valgrind module and starting program under valgrind Valgrind reports the lines where we access uninitialized i
Recommend
More recommend
