Center for Information Services and High Performance Computing (ZIH) Sampling XOR Instrumentation? Or both? Scalable Tools Workshop, Lake Tahoe, 2015-08-04 Andreas Knüpfer, Bert Wesarg, Thomas Ilsche, Ronny Tschüter, Joseph Schuchart, Hartmut Mix, Holger Brunst from ZIH, TU Dresden, Germany
Overview Introduction and existing approaches Recording and data formats Analysis of samples and events combined – Timeline visualization – Statistics Conclusions 2
Definitions Certain terms are used almost synonymously even though they aren’t ? “Profiling” “Event Tracing” Acquisition Sampling Instrumentation Recording Summarization Logging Data representation Profile Event/Call-Path Traces 3
Existing Combinations: Sample one thing, instrument another: – Sampling of user routines or call-path tracing, instrumentation of MPI [Tallent et.al. 2011, Ilsche et.al. 2014] – Sampling of hardware counters, instrumentation of user routines and MPI Sampling of energy consumption next to instrumentation-based performance monitoring [Hackenberg et. al. 2014] Instrumentation maintains shadow stack, sampling reads it as shortcut of a stack walk [Iwainsky et.al. 2014] Very coarse- grained sampling, then “folding” over many repeated instances, instrumentation is only guiding the folding mechanism, instrumented events are not recorded [Servat, Ph.D. thesis 2015] 4
Overview Introduction and existing approaches Recordi rding g and data a formats ats Analysis of samples and events combined – Timeline visualization – Statistics Conclusions 5
Example with Instrumentation and Sampling Main Fine-grained call Phase 2 Phase 3 Phase 4 timeline from Calc Calc Calc instrumentation System System System System System Call-stack _Main _Main _Main _Main _Main representation Main Main Main Main Main Phase 2 Phase 2 Phase 3 Phase 4 “Trampolines” allow Calc Calc Calc Calc tracking uninterrupted MPI calls, reduce overhead MPI Calc Calc Main Calc Flat representation 6
Samples with Calling Context Tree Main Efficient storage with Phase 2 Phase 3 Phase 4 Calling Context Tree Calc Calc Calc Main 1 System System System System System Phase 2 _Main _Main _Main _Main _Main Main Main Main Main Main Phase 2 Phase 2 Phase 3 Phase 4 Calc 5 9 11 Calc Calc Calc Calc MPI MPI 7 7 5 9 1 11 7
Representation in OTF2: CCT and Sample Points Define calling context nodes recursively: DefCallingContext { CallingContextRef self, RegionRef region, // Routine or function SourceCodeLocationRef sourceCodeLocation, CallingContextRef parent } Use them at a sample point to specify entire call stack by single ID: CallingContextSample { <process>, <time>, CallingContextRef callingContext, uint unwindDistance, InterruptGeneratorRef interruptGenerator } 8
Now add Events from Instrumentation Main Intermix Samples (S) Phase 2 as well as Enter (E) Calc and Leave (L) events, all refer to the CCT System System System System System System _Main _Main _Main _Main _Main _Main Main Main Main Main Main Main Phase 2 Phase 2 Phase 2 Phase 2 Phase 2 Phase 2 Calc Calc Calc Calc Calc Calc MPI MPI MPI S:5 E:7 S:7 L:7 S:5 S:5 9
Representation in OTF2: Special Enter/Leave Events Introduce new form of enter and leave events: CallingContextEnter { <process>, <time>, CallingContextRef callingContext, uint32_t unwindDistance } CallingContextLeave { <process>, <time>, CallingContextRef callingContext ); Refer to CCT, easily converted to old mode for legacy purposes if needed Little to no storage overhead, but more information (e.g., hidden stack entries) … no reason to keep the old enter/leave event records referring to routines 10
Overview Introduction and existing approaches Recording and data formats Analysis ysis of sa samples es and events ts combin ined – Timelin ine visu suali lizat zation on – Statistics Conclusions 11
Combined Visualization in Timeline Events are status changes, usually drawn from “now” until “following event” Samples are points in time, but usually drawn 1 δ wide (with sample distance δ ) t Main Phase 2 Calc System System Either _Main _Main draw at or at [t, t+ δ ) Main Main Main [t-½ δ , t+½ δ ) Phase 2 Phase 2 Calc Calc Calc MPI 12
Combined Visualization in Timeline: Shift by ½ δ Unified strategy for events and samples: – draw from “now” until “following event or sample” Main Phase 2 Calc Do not suppress samples in instrumented System System function calls (see below), but _Main _Main do optimize the Main Main Main extra stack walk Phase 2 Phase 2 Calc Calc Calc 13
Overview Introduction and existing approaches Recording and data formats Analysis of samples and events combined – Timeline visualization – Statis tistic tics Conclusions 14
How to Compute Run-Time Statistics? From samples alone or from samples and events combined? Main Phase 2 Time Time Sum Calc for Calc for MPI time 2.1 ms 2.2 ms 8.1 ms Events only (10.2) 2.2 (12.4) 1 δ = 3.1 ms 1 δ 1 δ 1 δ Samples 9.3 3.1 Calc MPI Calc Calc 12.4 only = ¾ = ¼ 2.1 ms 1.0 1.2 1.9 ms 3.1 ms 3.1 ms Events AND 10.2 2.2 12.4 Calc Calc Calc samples Events OR 9.3 2.2 11.5 samples 15
How to Compute Run-Time Statistics? Cannot compute from events alone with selective instrumentation Do not compute some from events and some from samples (cherry picking) Compute from samples only: produces statistically correct results – Don’t expect sampling to be more precise than 1 δ in the first place Compute from samples and events combined: produces different correct result! – It is not more accurate than the one from sampling (max. error is the same) – Different granularity for instrumented calls may become evident What is easier to comprehend by users? What is easier to explain? Which is the expected model that brings the lesser surprise? 16
Impressions 17
Impressions 18
Conclusions & Outlook Sampling and Instrumentation should be combined – Allow a completely flexible mix from samples and events – Event tracing should adopt favorable event representation via CCT – Make sure to present it in a clear way Release plans: – Sampling records already part of OTF2 – Include sampling in next Score-P release – Visualization in Vampir release version at SC’15 19
Advertisements 9th Parallel Tools Workshop in Dresden, 2-3 September https://tools.zih.tu-dresden.de/2015/ Extreme Scale Programming Tools Workshop (ESPT) at SC’15 http://www.vi-hps.org --> News Deadline extended until 14 August 20
Recommend
More recommend
