introduction
play

Introduction SystemTap: a tool for system-wide instrumentation - PowerPoint PPT Presentation

Jul 02, 2023 •43 likes •373 views

SystemTap update & overview Josh Stone <jistone@redhat.com> Software Engineer, Red Hat Introduction SystemTap: a tool for system-wide instrumentation Inspired by Sun DTrace, IBM dprobes, etc. GPL license, open project since

  1. SystemTap update & overview Josh Stone <jistone@redhat.com> Software Engineer, Red Hat

  2. Introduction ● SystemTap: a tool for system-wide instrumentation ● Inspired by Sun DTrace, IBM dprobes, etc. ● GPL license, open project since 2005 ● Current release 1.4, for kernels 2.6.9 ... 2.6.37+ ● Release 1.5 coming Real Soon Now™ http://sourceware.org/systemtap 2 SystemTap update & overview - Josh Stone - LFCS 2011

  3. Coming up: ● Overview of SystemTap ● Development update 3 SystemTap update & overview - Josh Stone - LFCS 2011

  4. System-wide instrumentation ● The most general case: ● Look into a live, unmodified system ● Examine what's going on ● Take action as appropriate ● Operate in the background ● Tracing, Debugging, Manipulation 4 SystemTap update & overview - Josh Stone - LFCS 2011

  5. Version flexibility ● Heterogeneous computer network ● different versions of the OS and/or applications ● Patching or upgrading not always practical ● Sometimes need a tool that works across the spectrum ● SystemTap has several mechanisms to adapt/abstract 5 SystemTap update & overview - Josh Stone - LFCS 2011

  6. Example usage scenarios ● Anyone : simple tracing ● Developers : to debug or comprehend code ● stepping through code, pretty-printing variables ● Analysts : to measure performance ● measure elapsed time between events ● attribute statistics to processes ● Sysadmins : to monitor, to patch ● activity logging, constraining ● security band-aids ● remote diagnostics (tech. support) 6 SystemTap update & overview - Josh Stone - LFCS 2011

  7. Developer: monitoring statements & vars ● # stap .../examples/general/varwatch.stp \ 'kernel.statement(“do_sys_open@fs/open.c:*”)' '$$vars' open.c:1045 ... $$vars ... thread 9541 from to dfd=0xff...ff9c filename=0x3b...bb1 ... open.c:1049 ... $$vars ... thread 9541 from ... to dfd=0xff...ff9c filename=? flags=0x8000 mode=0x1 tmp=? fd=? open.c:1047 ... $$vars ... thread 9541 from ... to dfd=0xff...ff9c filename=? flags=0x8000 mode=0x1 tmp=0xffff8803c8d0a000 fd=0xffffffffc8d0a000 open.c:1052 ... $$vars ... thread 9541 ... open.c:1057 ... $$vars ... thread 9541 ... open.c:1058 ... $$vars ... thread 9541 ... open.c:1061 ... $$vars ... thread 9541 ... open.c:1045 ... $$vars ... thread 9541 ... open.c:1049 ... $$vars ... thread 9541 ... 7 SystemTap update & overview - Josh Stone - LFCS 2011

  8. Sysadmin: page faults ● # stap ... examples/memory/pfaults.stp 13927:10843:0x7fffffffefec:w:minor:16 14106:10843:0x7ffff822ed29:w:minor:3 14193:10843:0x3b0e81f0d0:w:minor:4 14643:10843:0x607348:r:major:418 14655:10843:0x7ffff8359038:r:minor:2 14683:10843:0x2b6bf2ea0018:w:minor:9 15250:10843:0x7ffff822a74c:w:minor:2 24565:10843:0x7ffff822b77f:w:minor:4 24575:10843:0x7ffff822c77f:w:minor:3 60976:10843:0x2b6bf2f20270:r:major:21323 83819:10843:0x7ffff8229ed8:w:minor:4 83866:10843:0x2b6bf8d65000:w:minor:3 tid fault address μs service time elapsed time 8 SystemTap update & overview - Josh Stone - LFCS 2011

  9. Sysadmin: monitoring ttys ● # stap ... examples/io/ttyspy.stp (maj,min, pgrp, uid) (128, 1, 0, 99) \244\263\377}#\300!})\314} }5} }(\352d\\:i\353 (136, 8, 8331, 500) ls -al\necho hello world\n (128, 8, 0, 0) Nov 23 2002 \033[01;34m.netscape6\033[0m\rd\be (128, 2, 0, 0) \033[1;1H\033[J(maj,min, pgrp, uid)\r\n(128, 9 SystemTap update & overview - Josh Stone - LFCS 2011

  10. Conceptual model ● probe points: “events when to do something” ● probe handlers: “what to do then” ● script: a collection of probe points & handlers, plus utility functions ● many scripts can run concurrently, independently 10 SystemTap update & overview - Josh Stone - LFCS 2011

  11. Probe points – low level ● Provide an operational definition of the events: ● kernel.function(“vfs_*”) ● process(“a.out”).function(“*”).return ● module(“foo”).statement(“*@file.c:2323”) ● kernel.trace(“timer_*”) ● timer.s(1) ● perf.type(0).config(3).sample(2000) ● and context variables to probe handlers ● $arg4, $ptr->field[5], $$vars 11 SystemTap update & overview - Josh Stone - LFCS 2011

  12. Probe points – high level ● Defined as aliases in the standard tapset library ● syscall.open = kernel.function(“sys_open”) { ... } ● perf.hw.bus_cycles ● hotspot.thread_start ● python.function.entry ● Provide salient values to probe handlers ● Wildcards, metavariables widely available ● Also support add-on tapsets 12 SystemTap update & overview - Josh Stone - LFCS 2011

  13. Utility functions ● also defined in standard tapset library ● provide information not specific to context of probe point ● tid(), cpu(), get_cycles(), execname(): obvious ● tz_ctime: formatted timestamp in local time zone ● indent: formatting aid for per-thread nested reports ● backtrace, ubacktrace: unwound stack frames ● symdata, usymdata: symbol table lookup by address 13 SystemTap update & overview - Josh Stone - LFCS 2011

  14. Probe handlers: where magic happens ● Variables and metavariables available from context of probe point ● Developer chooses: ● trace some values ● filter, summarize, aggregate ● or traverse data structures ● or collect statistics via global variables ● or compose report ● or change state (guru mode) 14 SystemTap update & overview - Josh Stone - LFCS 2011

  15. Probe handlers ● small safe domain-specific language ● loops, conditionals, functions ● inferred strong data typing for temporary and context variables ● arrays, global variables ● structured error handling (cleanup, try/catch) ● automatic concurrency protection ● automatic resource limits (time & space) ● optional escape hatch from safety constraints 15 SystemTap update & overview - Josh Stone - LFCS 2011

  16. Sample script fragments probe begin { printf(“hello world\n”) } # say hello function gtod() { return gettimeofday_us() } # helper probe syscall.*.return { # after every syscall errno = $return # check return value if (errno < 0) { # if it's negative e = gtod()-@entry(gtod()) # measure time # print a line printf(“tid %d %s errno %d %s after %d us \n”, tid(), name, errno, errno_str(errno), elapsed) } probe syscall.ptrace { # every syscall if (target()==pid()) { # if this is the target # print a line printf(“noptrace(%s) from pid %s\n”,argstr,pid()) $request=0xbeef # clobber code } } 16 SystemTap update & overview - Josh Stone - LFCS 2011

  17. Example scripts ● ~80 packaged along with SystemTap ● Demonstrate common uses and unusual techniques ● Starting point for new users http://sourceware.org/systemtap/examples 17 SystemTap update & overview - Josh Stone - LFCS 2011

  18. Current implementation ● Compile: (local or remote) ● translate script to constrained C code ● compile into a kernel module using ordinary system compiler ● Run: (local or remote) ● loads module ● attach to kernel instrumentation callbacks (kprobes, perf, uprobes, ...) ● at conclusion, detach, unload, clean up 18 SystemTap update & overview - Josh Stone - LFCS 2011

  19. Perhaps SystemTap is not for you if ... ● If offline data analysis is good enough, and ... ● perf? ftrace? kernelshark? ● if you only run recent upstream kernels ● for relatively simple kernel-only tracing/analysis ● for easiest deployment ● lttng? ● if you can run patched kernels ● if you need high performance bulk tracing ● holy grail – a single all-purpose tool? ● convergence not impending 19 SystemTap update & overview - Josh Stone - LFCS 2011

  20. SystemTap release history ● First release with RHEL4U2 (October 2005) ● Recent release 1.4, January 2011 ● It still works with RHEL4 ● and RHEL5, RHEL6, Fedoras ● and several distributions (suse, debian and derivatives) ● and many upstream kernels 20 SystemTap update & overview - Josh Stone - LFCS 2011

  21. Recent SystemTap releases ● 1.2 March 22, 2010 ● Support for perf events (notably PMU) ● Support for hardware breakpoints ● New syntax: @defined(), try-catch ● 1.3 July 21, 2010 ● NOP optimization for uprobes ● Improved backtracing ● Integrated compile-server client ● New syntax: @entry(), C-expr, $var$ pretty-printing 21 SystemTap update & overview - Josh Stone - LFCS 2011

  22. Recent SystemTap releases (2) ● 1.4 January 17, 2011 ● SDT v3 (fewer relocations, better args, no DWARF req) ● Other userspace-focused improvements ● Prototype remote execution ● New policy for deprecation & compatibility ● 1.5 (impending) ● Better remoting – more robust; multiple hosts ● Improved compile-server ● Powerful new option: --version 22 SystemTap update & overview - Josh Stone - LFCS 2011

  23. Userspace probing ● Probe processes & shared libraries ● System-wide or focused ● Major support for C, C++, Java ● Limited support for Perl, Python, TCL ● Out-of-tree uprobes module, based on utrace ● Upstream utrace-free uprobes getting closer... ● Zeno's paradox resolved? 23 SystemTap update & overview - Josh Stone - LFCS 2011

Recommend

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

TRAINING COURSE ON SOCIAL PROTECTION &amp; FORMALIZATION TRINIDAD AND TOBAGO MARCH 15, 2017 INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION Design of the NIS Assistance from the

547 views • 23 slides
INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

Patterns of variation in the expression of case and agreement Andrs Brny Leiden University Centre for Linguistics 9 November 2019, BaSIS Workshop a.barany@hum.leidenuniv.nl INTRODUCTION Introduction 2/42 INTRODUCTION

1.37k views • 42 slides
Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures Definition and concepts Process and product properties Principles SE Approaches Motivation Software and the economy The economies of

1.03k views • 49 slides
Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1 https://www.cdc.gov/std/stats17/infographic.htm Introduction Epidemiology in Western Colorado CPHE Surveillance Data Report, May 2018 Chlamydia Epidemiology in Western

977 views • 39 slides
DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every meeting (2) Some ideas of diagrams for section 7.1 (Introduction) of TP. Giles Barr 25 Jan 2018, Pembroke College On DUNE, the reality is there is a

474 views • 10 slides
INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

ADVANCED COMPUTER SECURITY INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT IS SECURITY? A systems security policies describe What the system is supposed to do Store and provide access

479 views • 16 slides
Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout World Wide

387 views • 14 slides
14 Introduction Introduction Bad guys can put malware into Example: hosts

14 Introduction Introduction Bad guys can put malware into Example: hosts

Introduction Introduction source Encapsulation ISO/OSI reference model message M application segment H t H t M transport network datagram H n H n H t M presentation: allow applications to frame link H l H n H t M interpret

583 views • 4 slides
Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout

322 views • 8 slides
Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Introduction Zynq Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL Processing System Processor Peripherals Data Buses AXI Bus Conclusion BTE5380 - Embedded Systems Octobre 2014 Andreas Habegger

1.81k views • 30 slides
2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04 Project References S E 5 PART.1 SMILE5 Introduction M I L Introduction STORION-SMILE5 Residential Energy Storage System Inverter 5kW Output Input

1.06k views • 25 slides
Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an application server? Why use an application server? Course introduction Services provided by CICS How CICS applications are defined Scaling CICS to meet

1.84k views • 146 slides
Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic building blocks Node Links (relationship between nodes) Spatial information Network data

584 views • 9 slides
Introduction Introduction (1) Main argument of the paper: universities are not only

Introduction Introduction (1) Main argument of the paper: universities are not only

Le jugement des pairs comme outil de management des universits Peer-review as a management tool Christine Musselin (CSO, Sciences Po et CNRS) Avril 2013, Lirrsistible ascension du capitalisme acadmique Introduction Introduction (1)

590 views • 28 slides
Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Moreno Baricevic Gilberto Daz Axel Kohlmeyer Stefano Cozzini ULA ICTP CNR-IOM DEMOCRITOS Merida, VENEZUELA Trieste, ITALY Trieste, ITALY Introduction Introduction to storage and to storage and filesystems filesystems Introduction

1.19k views • 51 slides
JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of the brand new JA-100 alarm system Visions Basic architecture System components Setting Help us with final validations You are the FIRST ones

1.69k views • 111 slides
1 Introduction Introduction Communication System Baseband an adjective that describes

1 Introduction Introduction Communication System Baseband an adjective that describes

Introduction Introduction Physical Layer Mobile network Global ISP Provides the means to transmit bits Basic Concepts of from sender to receiver, Data Transmission Defines the mechanical, electrical, and Home network timing

463 views • 7 slides
Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical

INFOGR Computer Graphics Jacco Bikker - April-July 2015 - Lecture 1: Introduction Welcome! Todays Agenda: Topic Introduction Course Introduction Team Practical Details Assignments Field Study State of

869 views • 62 slides
Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction

Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction

Virtual and Augmented Reality Introduction (00) RNDr. Martin Madaras, PhD. madaras@skeletex.xyz Introduction Introduction Who am I? What do I do here? 2 About me Short research bio: - 2014 - finished PhD at FMFI UK - 2014 2016 -

1.3k views • 39 slides
PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience

PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience

PRESENTATION The Introduction What is the Introduction ? The introduction prepares the audience for your presentation. It tells them what your presentation is about, why it is important, and finally, what to listen for in your speech. Get the

505 views • 3 slides
How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer

How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer

Introduction to Beamer Hafida Benhidour How to make a presentation with L A T EX? Introduction to L A T EX Introduction to Beamer Introduction to Beamer Hafida Benhidour Department of computer science King Saud University November

732 views • 49 slides
Shenzhen Cuilu jewelry Co., Ltd was founded in 1996 and its a large private enterprise

Shenzhen Cuilu jewelry Co., Ltd was founded in 1996 and its a large private enterprise

Introduction Introduction - - Brief Brief Introduction Introduction of of Cuilu Cuilu Group Group Introduction Introduction - - Brief Brief Introduction Introduction of of Cuilu Cuilu Group Group Shenzhen Cuilu jewelry Co., Ltd

147 views • 14 slides
Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

V olunteer C ommunications N etwork (V-C-N.org) Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org Academy Course - 2012-02.24-2208 262: Introduction to ICS-214 : V-C-N.org : Slide: 1 V olunteer C

735 views • 34 slides
Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org

V olunteer C ommunications N etwork (V-C-N.org) Introduction to ICS- -214 214 Introduction to ICS Official Unit / Incident Log - A V-C-N.org Academy Course - 2012-02.24-2208 262: Introduction to ICS-214 : V-C-N.org : Slide: 1 V olunteer C

558 views • 18 slides

More recommend