IgProf The ignominious profiler. A generic memory and performance - PowerPoint PPT Presentation

IgProf The ignominious profiler. A generic memory and performance profiler for linux applications. http://igprof.org Giulio Eulisse Fermi National Accelerator Laboratory

Opinions expressed in this presentation are mine, only mine, and solely mine…. buahahahah

Why a profiler?

Where I work CMS experiment @ CERN LHC

Where I work Over 4000 researchers...

Where I work 43 countries, 191 institutes Source: http://cms.web.cern.ch/content/cms-collaboration

What do we do? 5c646f63756d656e74636c6173735b613470617065722c313270745d7b61727469636c657d0a0a5c7573657061636b61676 57b657073667d0a5c7573657061636b6167657b67726170686963737d0a0a0965787477696474683d362e35696e0a5c6f64 64736964656d617267696e3d302e30696e0a5c6576656e736964656d617267696e3d302e30696e0a0a0a6577636f6d6d616 e647b5c6d65747d7b0d6d5c2c2f5c215c215c215c21455f7b547d7d0a0a6577636f6d6d616e647b5c65747d7b0d6d20455f 547d0a0a6577636f6d6d616e647b5c7a7072696d657d7b5a5e5c7072696d657d0a0a086567696e7b646f63756d656e747d0 a0a6f696e64656e747b086620546865205374616e64617264204d6f64656c3a205468652046696e616c204c616772616e67 69616e7d0a0b73706163657b302e31696e7d0a0a0a6f696e64656e740a4672616e6369732048616c7a656e2c20616e64204 16c616e20442e204d617274696e2c0a7b5c697420517561726b7320616e64204c6570746f6e733a20416e20496e74726f64 7563746f727920436f7572736520696e204d6f6465726e205061727469636c6520506879736963732c7d0a4a6f686e20576 96c6579205c2620536f6e732c202831393834292e0a0b73706163657b302e31696e7d0a0a0a6f696e64656e740a546f2073 756d6d6172697a6520746865207374616e6461726420285765696e626572672d53616c616d29206d6f64656c2c207765206 7617468657220746f67657468657220616c6c207468650a696e6772656469656e7473206f6620746865204c616772616e67 69616e2e2054686520636f6d706c657465204c616772616e6769616e2069733a0a0a086567696e7b63656e7465727d0a086 567696e7b746162756c61727d7b726c6c7d0a202020245c6d61746863616c7b4c7d203d240a202020202020202020202020 2020202620242d0c7261637b317d7b347d207b086620577d5f7b5c6d750a757d205c63646f74207b086620577d5e7b5c6d7 50a757d0a2020202020202020202020202020202020202d0c7261637b317d7b347d20202020202020425f7b5c6d750a757d 20202020202020202020202020425e7b5c6d750a757d240a2020202020202020202020202020202620245c6c6566745c7b2 0086567696e7b61727261797d7b6c7d0a20202020202020202020202020202020202020202020202020205c6d626f787b57 7d5e7b5c706d7d2c205c6d626f787b5a7d2c205c67616d6d61205c6d626f787b206b696e657469637d205c0a20202020202 020202020202020202020202020202020202020205c6d626f787b656e65726769657320616e647d20202020202020202020 20202020202020202020202020202020202020205c0a20202020202020202020202020202020202020202020202020205c6 d626f787b73656c662d696e746572616374696f6e737d0a2020202020202020202020202020202020202020202020202020 5c656e647b61727261797d0a2020202020202020202020202020202020200d696768742e240a20202020202020202020202 02020205c20262026205c0a202020202020202020202020202020262024086567696e7b61727261797d7b6c7d0a20202020 20202020202020202020202020202b0861727b4c7d5c67616d6d615e7b5c6d757d5c6c6566742820695c7061727469616c5 f7b5c6d757d0a20202020202020202020202020202020202020202020202020202020202020202020202020202020202020 + = 20202d670c7261637b317d7b327d7b0866200961757d205c63646f74207b086620577d5f7b5c6d757d0a202020202020202 0202020202020202020202020202020202020202020202020202020202020202020202020202d67270c7261637b597d7b32 7d425f7b5c6d757d0a20202020202020202020202020202020202020202020202020202020202020202020202020200d696 7687429204c205c0a2020202020202020202020202020202020202b0861727b527d5c67616d6d615e7b5c6d757d5c6c6566 742820695c7061727469616c5f7b5c6d757d0a2020202020202020202020202020202020202020202020202020202020202 020202020202020202020202020202d67270c7261637b597d7b327d425f7b5c6d757d0a2020202020202020202020202020 2020202020202020202020202020202020202020202020200d69676874292052205c0a20202020202020202020202020202 02020205c656e647b61727261797d240a2020202020202020202020202020202620245c6c6566745c7b20086567696e7b61 727261797d7b6c7d0a20202020202020202020202020202020202020202020202020205c6d626f787b6c6570746f6e20616 e6420717561726b7d202020202020202020205c0a20202020202020202020202020202020202020202020202020205c6d62 6f787b6b696e6574696320656e6572676965737d202020202020202020205c0a20202020202020202020202020202020202 020202020202020205c6d626f787b616e642074686569727d20202020202020202020202020202020205c0a202020202020 20202020202020202020202020202020202020205c6d626f787b696e746572616374696f6e7320776974687d20202020202 02020205c0a20202020202020202020202020202020202020202020202020205c6d626f787b577d5e7b5c706d7d2c205c6d 626f787b5a7d2c205c67616d6d61205c0a20202020202020202020202020202020202020202020202020205c656e647b617 27261797d0a2020202020202020202020202020202020200d696768742e240a2020202020202020202020202020205c2026 2026205c0a2020202020202020202020202020202620242b5c6c6566747c5c6c6566742820695c7061727469616c5f7b5c6 d757d0a20202020202020202020202020202020202020202020202020202020202020202d670c7261637b317d7b327d7b08 66200961757d205c63646f74207b086620577d5f7b5c6d757d0a20202020202020202020202020202020202020202020202 020202020202020202d67270c7261637b597d7b327d425f7b5c6d757d0a2020202020202020202020202020202020202020 20202020200d6967687429205c7068690a202020202020202020202020202020202020200d696768747c5e32202d2056285 c70686929240a2020202020202020202020202020202620245c6c6566745c7b20086567696e7b61727261797d7b6c7d0a20 202020202020202020202020202020202020202020202020205c6d626f787b577d5e7b5c706d7d2c205c6d626f787b5a7d2 c205c67616d6d612c205c6d626f787b20616e642048696767737d205c0a2020202020202020202020202020202020202020 Mostly, we use 1.3 terawatt hours yearly* to 2020202020205c6d626f787b6d617373657320616e647d20202020202020202020202020202020202020202020202020202 020202020202020205c0a20202020202020202020202020202020202020202020202020205c6d626f787b636f75706c696e 67737d2020202020202020202020202020202020202020202020202020202020202020202020205c0a20202020202020202 020202020202020202020202020202020205c656e647b61727261797d0a2020202020202020202020202020202020200d69 smash particles and we take digital pictures 6768742e240a2020202020202020202020202020205c20262026205c0a2020202020202020202020202020202620242d284 75f310861727b4c7d5c70686920520a20202020202020202020202020202020202020202b475f320861727b4c7d5c706869 5f6320520a20202020202020202020202020202020202020202b5c6d626f787b6865726d697469616e20636f6e6a7567617 4657d292e240a2020202020202020202020202020202620245c6c6566745c7b20086567696e7b61727261797d7b6c7d0a20 of the result with a sophisticated camera. 202020202020202020202020202020202020202020202020205c6d626f787b6c6570746f6e20616e6420717561726b7d202 05c0a20202020202020202020202020202020202020202020202020205c6d626f787b6d617373657320616e647d20202020 202020205c0a20202020202020202020202020202020202020202020202020205c6d626f787b636f75706c696e6720746f2 048696767737d205c0a20202020202020202020202020202020202020202020202020205c656e647b61727261797d0a2020 Every single collision, a “RAW Event”, is 202020202020202020202020202020200d696768742e24205c0a5c656e647b746162756c61727d0a5c656e647b63656e746 5727d0a0a0a6f696e64656e740a244c242064656e6f7465732061206c6566742d68616e646564206665726d696f6e20286c 6570746f6e206f7220717561726b2920646f75626c65742c20616e6420522064656e6f74657320610a72696768742d68616 e646564206665726d696f6e2073696e676c65742e0a0a5c656e647b646f63756d656e747d0a roughly 1 MB. LHC delivers 10^9 events per second (i.e. 1 PB) when running, which become a few hundreds evt/s after a first level of (HW) “trigger”. We stored to disk roughly 10^9 events in 2011-2013 period Source: http://home.web.cern.ch/about/engineering/powering-cern

What do we do? We process detector data with in custom written software and quickly obtain PhD theses and conference papers with it.

CMSSW... Over 5M SLOCs custom code Large C++ / Python / Fortran codebase, developed over almost two decades by over 1300 researchers ranging from master students to Nobel prize candidates. Software comprises complex pattern recognition algorithms, data analysis tools and simulations. Small core of professional software engineers mostly doing application framework, release integration and QA. Large and diverse application set Many different “workflows” depending on the the kind of simulation / analysis being done. Over 3000 shared objects chained via python configuration language. Each workflow has ~300 MB of CODE sections, sparse in ~600 shared libraries. Roughly 600’000 symbols present in the process image. Over 300 level deep call-trees are the norm. 2GB RSS footprint on average, large memory churn (up to 1M allocation per second). No single offender. Large working dataset ~12 PB of RAW data acquired in 2011-13, more than double that if we count the processed via the WLGC Grid using 100k cores sparse over 5 continents. We expect a 2-3 order of magnitude increase in data volume in the next 10 - 15 years. 9

Courtesy of Ian Bird ...the WLGC Grid... Tier-0 (CERN): data recording, nearly 170 sites, 
 reconstruction 40 countries and distribution ~350’000 cores Tier-1: 
 permanent storage, re- 500 PB of storage processing, analysis > 2 million jobs/day Tier-2: 
 Simulation, end- user analysis 10-100 Gb links WLCG: An International collaboration to distribute and analyse LHC data Integrates computer centres worldwide that provide computing and storage resource into a single infrastructure accessible by all LHC physicists.