Linux Performance Profiling and Monitoring Christoph Mitasch & - PowerPoint PPT Presentation

Linux Performance Profiling and Monitoring Christoph Mitasch & Georg Schönberger @cmitasch

Thomas-Krenn.AG _ A server manufacturer in Bavaria, Germany _ Well visited knowledge base, Thomas-Krenn Wiki (parts in English) 2

Agenda _ Collect Statistics _ Tracing Sysstat Package _ _ p e r f _ e v e n t s i o s t a t _ _ f t r a c e p i d s t a t _ _ p e r f - t o o l s _ s a r , a t o p Flame graphs _ Percona Cacti Template _ _ Watch online _ t o p _ i o t o p _ i f t o p 3

# f i n d / - t y p e f - n a m e s t a t i s t i c s

From Brendan Gregg, http://www.brendangregg.com/linuxperf.html

From Brendan Gregg, http://www.brendangregg.com/linuxperf.html

20 0

m p s t a t ( p a r t o f s y s s t a t ) _ Without Interval/Count → since system startup _ CPU usage per Core Including Hyperthreading _ # l s c p u | g r e p - E ' c o r e | s o c k e t ' T h r e a d ( s ) p e r c o r e : 2 C o r e ( s ) p e r s o c k e t : 2 _ Check how well usage is balanced # m p s t a t - P A L L L i n u x 3 . 1 3 . 0 - 4 8 - g e n e r i c ( X 2 2 0 ) 2 0 1 5 - 0 4 - 1 4 _ x 8 6 _ 6 4 _ ( 4 C P U ) 1 4 : 2 8 : 2 1 C P U % u s r % n i c e % s y s % i o w a i t % i r q % s o f t % s t e a l % g u e s t % g n i c e % i d l e 1 4 : 2 8 : 2 1 a l l 1 1 , 5 9 0 , 0 9 3 , 6 2 0 , 0 3 0 , 0 0 0 , 0 4 0 , 0 0 0 , 0 0 0 , 0 0 8 4 , 6 4 1 4 : 2 8 : 2 1 0 6 , 4 5 0 , 0 5 1 , 8 7 0 , 0 4 0 , 0 0 0 , 0 7 0 , 0 0 0 , 0 0 0 , 0 0 9 1 , 5 3 1 4 : 2 8 : 2 1 1 1 6 , 4 4 0 , 1 1 5 , 5 6 0 , 0 1 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 7 7 , 8 9 1 4 : 2 8 : 2 1 2 1 7 , 1 5 0 , 1 4 5 , 5 5 0 , 0 3 0 , 0 0 0 , 0 5 0 , 0 0 0 , 0 0 0 , 0 0 7 7 , 0 8 1 4 : 2 8 : 2 1 3 1 6 , 2 7 0 , 1 1 4 , 8 9 0 , 0 1 0 , 0 0 0 , 0 2 0 , 0 0 0 , 0 0 0 , 0 0 7 8 , 7 0 8

m p s t a t # m p s t a t - P A L L 1 2 L i n u x 3 . 1 3 . 0 - 4 8 - g e n e r i c ( X 2 2 0 ) 2 0 1 5 - 0 4 - 1 4 _ x 8 6 _ 6 4 _ ( 4 C P U ) 1 5 : 2 4 : 4 4 C P U % u s r % n i c e % s y s % i o w a i t % i r q % s o f t % s t e a l % g u e s t % g n i c e % i d l e 1 5 : 2 4 : 4 5 a l l 5 , 2 1 0 , 0 0 7 , 1 2 1 7 , 8 1 0 , 0 0 0 , 2 7 0 , 0 0 0 , 0 0 0 , 0 0 6 9 , 5 9 1 5 : 2 4 : 4 5 0 1 , 4 3 0 , 0 0 1 , 4 3 0 , 0 0 0 , 0 0 2 , 8 6 0 , 0 0 0 , 0 0 0 , 0 0 9 4 , 2 9 1 5 : 2 4 : 4 5 1 1 1 , 8 8 0 , 0 0 2 3 , 7 6 6 4 , 3 6 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 Core 1 is not idle 1 5 : 2 4 : 4 5 2 4 , 1 2 0 , 0 0 1 , 0 3 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 9 4 , 8 5 and also deals 1 5 : 2 4 : 4 5 3 3 , 0 3 0 , 0 0 1 , 0 1 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 9 5 , 9 6 with % i o w a i t 1 5 : 2 4 : 4 5 C P U % u s r % n i c e % s y s % i o w a i t % i r q % s o f t % s t e a l % g u e s t % g n i c e % i d l e 1 5 : 2 4 : 4 6 a l l 5 , 7 4 0 , 0 0 7 , 1 0 1 7 , 7 6 0 , 0 0 0 , 5 5 0 , 0 0 0 , 0 0 0 , 0 0 6 8 , 8 5 1 5 : 2 4 : 4 6 0 2 , 9 9 0 , 0 0 1 , 4 9 0 , 0 0 0 , 0 0 2 , 9 9 0 , 0 0 0 , 0 0 0 , 0 0 9 2 , 5 4 1 5 : 2 4 : 4 6 1 1 1 , 8 8 0 , 0 0 2 3 , 7 6 6 4 , 3 6 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 1 5 : 2 4 : 4 6 2 6 , 0 0 0 , 0 0 1 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 9 3 , 0 0 1 5 : 2 4 : 4 6 3 1 , 0 1 0 , 0 0 1 , 0 1 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 0 , 0 0 9 7 , 9 8 9

v m s t a t _ High Level Statistics about Virtual memory _ Swap/Paging _ I/O statistics _ System interrupts and context switches _ CPU statistics _ # v m s t a t 1 p r o c s - - - - - - - - - - - m e m o r y - - - - - - - - - - - - - s w a p - - - - - - - i o - - - - - s y s t e m - - - - - - - - c p u - - - - - r b s w p d f r e e b u f f c a c h e s i s o b i b o i n c s u s s y i d w a s t 3 0 1 7 2 3 7 1 8 5 6 1 3 7 0 8 8 3 1 2 5 6 6 4 0 0 0 1 5 3 0 6 0 7 6 1 8 7 0 5 9 1 7 9 5 6 1 7 0 3 0 1 7 2 4 1 6 5 9 6 1 3 7 0 9 6 3 1 2 5 7 0 4 0 0 0 1 6 3 4 2 0 8 6 8 9 7 4 1 9 1 1 1 0 6 1 1 7 0 0 0 1 7 2 4 5 1 7 1 6 1 3 7 0 9 6 3 0 8 9 9 1 6 0 0 0 0 3 9 6 1 8 4 8 3 1 9 6 0 0 0 0 1 7 2 4 1 3 9 1 6 1 3 7 1 0 8 3 1 1 8 7 9 6 0 0 0 5 2 5 0 2 2 2 1 8 9 2 9 0 0 0 2 0 1 7 2 3 9 9 7 5 6 1 3 7 1 0 8 3 1 1 8 8 6 0 0 0 2 8 4 8 8 4 0 1 4 8 3 0 1 0 9 4 1 1 0 1 3 6 6 1 2 0 1 1 1 7 2 3 6 4 9 4 8 1 3 7 1 0 8 3 1 1 8 9 8 8 0 0 3 1 0 7 9 2 0 1 6 2 0 4 1 2 7 3 8 2 0 1 3 5 3 1 4 0 10

v m s t a t _ Memory statistics Raw disk blocks like filesystem metadata (superblocks, _ b u f f inodes) Memory used for data -> pages with actual contents _ c a c h e $ v m s t a t 1 p r o c s - - - - - - - - - - - m e m o r y - - - - - - - - - - - - - s w a p - - - - - - - i o - - - - - s y s t e m - - - - - - - - c p u - - - - - r b s w p d f r e e b u f f c a c h e s i s o b i b o i n c s u s s y i d w a s t 0 0 1 7 2 6 0 7 7 6 0 1 8 2 1 7 2 3 3 1 3 6 8 4 0 0 1 5 9 4 9 6 1 5 4 2 2 2 1 8 6 7 6 0 0 0 0 1 7 2 6 0 7 6 2 8 1 8 2 1 7 2 3 3 1 3 6 8 4 0 0 0 5 2 3 8 7 2 0 0 8 4 2 9 5 0 0 0 0 1 7 2 6 0 7 3 4 8 1 8 2 1 7 2 3 3 1 3 6 8 4 0 0 0 0 3 9 7 2 0 3 4 4 1 9 5 0 0 0 0 1 7 2 6 0 6 4 4 8 1 8 2 1 7 2 3 3 1 3 6 8 4 0 0 0 0 3 7 8 1 8 9 6 4 2 9 4 0 0 $ f r e e t o t a l u s e d f r e e s h a r e d b u f f e r s c a c h e d M e m : 8 0 5 6 6 6 4 7 4 5 0 3 1 6 6 0 6 3 4 8 4 9 1 8 2 0 1 8 2 1 7 2 3 3 1 3 6 8 4 - / + b u f f e r s / c a c h e : 3 9 5 4 4 6 0 4 1 0 2 2 0 4 S w a p : 1 0 4 8 5 7 2 1 7 2 1 0 4 8 4 0 0 11

v m s t a t _ Process related fields The number of runnable processes (running or waiting for _ r run time) If high → indicator for saturation _ The number of processes in uninterruptible sleep _ b Mostly waiting for I/O _ # v m s t a t 1 p r o c s - - - - - - - - - - - m e m o r y - - - - - - - - - - - - - s w a p - - - - - - - i o - - - - - s y s t e m - - - - - - - - c p u - - - - - r b s w p d f r e e b u f f c a c h e s i s o b i b o i n c s u s s y i d w a s t Processes doing I/O [ . . . ] can be in waiting state 0 1 1 7 2 4 0 4 7 6 8 1 3 7 0 8 8 3 1 2 5 6 6 4 0 0 0 1 6 7 5 2 4 9 0 2 9 6 9 5 5 6 6 7 0 1 8 0 0 1 1 7 2 3 9 9 9 5 6 1 3 7 0 8 8 3 1 2 5 6 6 4 0 0 0 1 3 8 3 4 0 8 1 3 3 6 1 6 5 7 7 6 8 1 9 0 $ p s - e o p p i d , p i d , u s e r , s t a t , p c p u , c o m m , w c h a n : 3 2 | g r e p e x t 4 Kernel function process [ . . . ] is sleeping on 7 1 5 9 7 1 6 1 r o o t D s 3 . 2 f i o e x t 4 _ f i l e _ w r i t e 7 1 5 9 7 1 6 2 r o o t D s 3 . 2 f i o e x t 4 _ f i l e _ w r i t e 7 1 5 9 7 1 6 4 r o o t D s 3 . 2 f i o e x t 4 _ f i l e _ w r i t e 12

plots v m s t a t Drawing interrupts and context switches https://clusterbuffer.wordpress.com/admin-tools/vmstat-plotter/ 13

But we are not satisfied with summaries and overviews...

What is PID 9059 doing?