Real-Time Systems: Testing Your System in Linux Jiangnan Liu CSE - PowerPoint PPT Presentation

Real-Time Systems: Testing Your System in Linux Jiangnan Liu CSE 520S Spring 02/13/2020 With adaptions from Haoran Li

What is meant by Time? Ø Absolute time since some fixed past event, e.g.: q Seconds since start of the Unix Epoch (00:00:00 UTC on 1 January 1970) q Seconds since system boot Ø Relative time, E.g.: q Seconds between two events q Ten seconds into the future (from now) q Execution time of a program segment Ø World time, E.g.: q January 30 th , 9:00 AM Ø An OS must approximate time to provide time-based functions for users. 2

Measure Elapsed Time

Why Elapsed Time First? Ø End-to-End Latency q Get absolute time on each end? Clock 1 Clock 2 Synchronized / Asynchronized Clocks? end_time (clock 2) Start_time (clock 1) Clock 1 Ø Elapsed Time: q Round Trip Time (ping) q Same Clock Source Start_time (clock 1) Measure Time on a single Host End_time (clock 1) 4

Get a coarse-grained estimation Ø Use shell built-in command q time q “real”: Wall Time Elapsed (Just an estimation, Don’t rely on it) q “user”: Execution Time in User Space q “sys”: Execution Time in Kernel Space (syscall) Get an overview of your program’s response / execution time. 5

Measure Elapsed Time: gettimeofday() Ø gettimeofday() [http://linux.die.net/man/2/sched_setscheduler]. Wall clock time q return struct timeval, includes tv_sec and tv_usec q NOT ok for measuring overhead on standard kernel configuration q Wall clock time is subject to anomalies • User/other program(NTP) changes clock Normal Evaluation (5 seconds) Intentionally change the wall clock, when program is running 6

Don’t Relied on Wall Clock Time Ø Shell built-in command “time” When Measuring Elapse Time, don’t rely on WCT. 7

Use POSIX clock_gettime() Ø Sources: https://linux.die.net/man/3/clock_gettime q CLOCK_REALTIME: • Wall Time, affected by discontinuous jump q CLOCK_MONOTONIC: • Not affected by discontinuous jump but affected by incremental adjustments (e.g. NTP). • Clock cannot jump, but may skew q CLOCK_MONOTONIC_RAW: • Not affected by NTP • More accurate for very short intervals Ø Precision: clock_getres() For short latency measurement in a single host, try CLOCK_MONOTIC_RAW 8

FYI: Other Time Sources RTC (Real-Time Clock) q Available on most computers (not on RPi 2 or 3 unless you add it) q Low precision (as low as 0.5 seconds) Hardware Timers q Might be used to generate interrupts, might be queryable q Run at a variety of frequencies q Programmable Interval Timer (PIT) q High-Performance Event Timer (HPET) q Programmable Interrupt Controller (PIC) q Advanced Programmable Interrupt Controller (APIC) Processor Cycles q Timestamp Counter (TSC) on x86, 64-bit q Cycle Counter (CCNT) on ARM, 32-bit, 64-cycle divider, not accessible in user mode q Potentially very high accuracy Source: CSE 522S http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/03_Timers_Timing.pptx 9

Pointers Ø Time related system calls in the Linux kernel q https://0xax.gitbooks.io/linux-insides/content/Timers/timers-7.html Ø Clock_gettime() q Man 2 clock_gettime q https://linux.die.net/man/2/clock_gettime Ø RDTSC q http://www.mcs.anl.gov/~kazutomo/rdtsc.html Ø RDTSCP q https://www.intel.com/content/www/us/en/embedded/training/ia-32-ia-64- benchmark-code-execution-paper.html Ø Invariant TSC q Pitfall of TSC usage q http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/ Ø (ARM) High Resolution Timing on Raspberry Pi q https://blog.regehr.org/archives/794 10

Measure End-to-End Latency

Why End-to-End Latency? Ø End-to-End Latency Clock 1 Clock 2 end_time (clock 2) start_time (clock 1) Ø Contributor: q propagation delay: static q queuing delays q node processing delays q routing changes Need synchronization between clocks 12

How to Get Clock Synchronized? Ø NTP(Network Time Protocol) q Query NTP Server, get clock synchronized NTP Server Clock 1 Clock 2 start_time (clock 1) end_time (clock 2) 13

Background: NTP Ø Hierarchical NTP servers: Clock Strata Ø Stratum0: reference clock Ø Stratum1: primary time servers The U.S. Naval Observatory Alternate Master Clock Stratum 0: high-precision timekeeping devices atomic (cesium) clocks 14 Source: https://en.wikipedia.org/wiki/Network_Time_Protocol

Problems with NTP Ø Loose granularity: q T ens of milliseconds in public internet q Around one millisecond in Local Area Network [1] Ø Leap second handling: q Detects the announcement at least a few seconds after the refclock has started to announce the leap second Ø Security: q NTP has been used in Distributed Denial of Service attacks(DDos) [2] Set up an NTP server: https://ubuntuforums.org/showthread.php?t=862620 15 https://arstechnica.com/information-technology/2014/01/new-dos-attacks-taking-down-game-sites-deliver-crippling-100-gbps-floods/

Precision Time Protocol: Better Accuracy Ø On LAN: q sub-microsecond range q making it suitable for measurement and control systems q PTP v.s. NTP • hardware support present in various network interface controllers (NIC) and network switches. PTP master PTP slave PTP slave LAN 16

PTP v.s. NTP Ø NTP q Hierarchical NTP servers q Redundancy: Client-Server, Peer-to-Peer, Multicast q Granularity:T ens of microseconds q Built-in in EC2 Ø PTP q Grand Master-Slave(Master)-Slave q Redundancy: Fail take over q Hardware slaves (e.g. PCIe card) when possible q Granularity: Sub-microseconds (when w/ hardware support) q Not built-in in EC2, normally used for LAN 17

Pointers Ø Linux PTP q http://linuxptp.sourceforge.net/ q SO_TIMESTAMPS q Supports the Linux PTP Hardware Clock (PHC) subsystem by using the clock_gettime family of calls, including the new clock_adjtimex system call. clo Ø PTP daemon man page q sudo apt-get install ptpd q man ptpd Ø NTP v.s PTP: How do you get accuracy q http://www.atis.org/tam/presentations/ntp_vs_ptp_darnold.pdf q https://blog.meinbergglobal.com/2013/11/22/ntp-vs-ptp-network-timing-smackdown/ q https://www.redhat.com/en/blog/combining-ptp-ntp-get-best-both-worlds 18

Summary Ø Elapsed time or end-to-end latency? Ø What time is measured by the clock? (real, user, system, or, hopefully not, wall-clock?) Ø What is the granularity requirement of time? (s, ms, µs, ns) Ø Is the clock monotonic, or will it change with changes in the system time (via NTP, time zone, daylight savings time, by the user, etc.)? Ø Does the program interference the time? Ø Does your observation of the time interference the program? 19

Trace your System

Trace Your System by using ftrace Ø ftrace q Traces the internal operations of the kernel q Static tracepoints within the kernel (event tracing) • Scheduling • interrupts Ø Trace-cmd q Front-End (user-level) utility for ftrace q Example: • sudo trace-cmd record -e sched_switch ./myapp • Dump trace.dat Ø Kernel Shark q GUI trace-cmd reader • kernelshark trace.dat Kernel Shark: http://rostedt.homelinux.com/kernelshark/ 21

A Typical Trace 22

Configure and Trace your Network Ø ifconfig q Configure, manage and query network interface parameters through command line Ø netstat q List network(socket) connections connections to the system Ø tcpdump q Packet analyzer to trace packets transmitted over the network to which the computer is attached q Example: • tcpdump -i eth0 dst 172.168.0.118 -w 001.pcap Ø wireshark q GUI based alternative to tcpdump 23

Real-Time Scheduler in Linux

Scheduling in Classes Multiple schedulers are implemented as different scheduling classes . Normal: q SCHED_NORMAL/SCHED_OTHER : regular, interactive CFS tasks q SCHED_BATCH : low priority, non-interactive CFS tasks q SCHED_IDLE : very low priority tasks Real-time: q SCHED_RR : round-robin q SCHED_FIFO : first-in, first-out q SCHED_DEADLINE : earliest deadline first 25 CSE 522S – Advanced Operating Systems

Real-Time Scheduling Real-time tasks execute repeatedly (usually are periodic ) under some time constraint Task Task Task Time 0ms 5ms 10ms E.g., a task is released to execute every 5 ms, and each invocation has a deadline of 5 ms Separate priority range from nice: • Priorities range from 1 (low) to 99 (high) 26 Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx

Real-Time OS Support Goal is to achieve predictable execution: Ideal: Real world: Preempt Migrate Sources of uncertainty (and solutions): q Scheduling preemptions (real-time scheduling) q Interrupts (can mask interrupts) q Migrations (can pin tasks to cores) q OS latency & jitter (RT_PREEMPT patch set) 27 Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx

SCHED_RR Round-robin scheduling Among tasks of equal priority: Ø Rotate through all tasks Ø Each task gets a fixed time slice Task 1 Task 2 Task 3 Task 1 Task 2 Task 3 Task 1 Task 2 Task 3 Time Cannot run if higher priority tasks are runnable 28 Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx

SCHED_FIFO First-in, First-out scheduling Ø The first enqued task of highest priority executes to completion Ø A task will only relinquish a processor when it completes, yields, or blocks Task 1 Task 2 Task 3 Time 29 Source: CSE 522S – Advanced Operating Systems: http://www.cse.wustl.edu/~cdgill/courses/cse522/slides/09_real_time_sched.pptx