linux as a real time os
play

Linux as a Real-Time OS 2 Tuesday, July 9, 13 2 Linux as a - PowerPoint PPT Presentation

A Comparison of Scheduling Latency in Linux, PREEMPT_RT, and LITMUS RT Felipe Cerqueira and Bjrn Brandenburg July 9th, 2013 1 Tuesday, July 9, 13 1 Linux as a Real-Time OS 2 Tuesday, July 9, 13 2 Linux as a Real-Time OS Optimizing system


  1. No Background Tasks Similar max. and avg. latency for Linux 3.0 and LITMUS RT . Scheduling Latency ( μ s ) Average (99% conf.) Maximum 20 15.3 13.9 15 11.2 10 5 3.5 2.9 2.7 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 45 Tuesday, July 9, 13 45

  2. No Background Tasks Improved max. Similar max. and latency for avg. latency for Linux PREEMPT_RT 3.0 and LITMUS RT . Scheduling Latency ( μ s ) Average (99% conf.) Maximum 20 15.3 13.9 15 11.2 10 5 3.5 2.9 2.7 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 46 Tuesday, July 9, 13 46

  3. Second Scenario CPU-bound background tasks ✤ Tasks running an infinite loop accessing memory (read/write) ✤ Working set larger than L2 cache size 47 Tuesday, July 9, 13 47

  4. Second Scenario CPU-bound background tasks ✤ Tasks running an infinite loop accessing memory (read/write) Generates memory tra ffi c and cache contention! ✤ Working set larger than L2 cache size 48 Tuesday, July 9, 13 48

  5. CPU-bound Background Tasks Scheduling Latency ( μ s ) Average (99% conf.) Maximum 80 72.7 60 47.6 40 17.4 20 5.2 4.2 3.4 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 49 Tuesday, July 9, 13 49

  6. CPU-bound Background Tasks Scheduling Latency ( μ s ) Average (99% conf.) Maximum Latency under 80 72.7 PREEMPT_RT is significantly lower. 60 47.6 40 17.4 20 5.2 4.2 3.4 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 50 Tuesday, July 9, 13 50

  7. CPU-bound Background Tasks LITMUS RT ’s latency lower than Scheduling Latency ( μ s ) Average (99% conf.) Maximum Latency under on Linux 3.0? 80 72.7 PREEMPT_RT is significantly lower. 60 47.6 40 17.4 20 5.2 4.2 3.4 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 51 Tuesday, July 9, 13 51

  8. LITMUS RT vs . Linux 3 . 0: � CPU-bound Background Tasks log scale! Linux 3.0: scheduling latency (CPU-bound bg tasks) Linux 3.0 P-FP(LITMUS RT ) min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us avg=4.22 μ s max=72.73 μ s avg=5.17 μ s max=47.59 μ s 1e+07 samples: total=5854711 1e+06 number of samples 100000 10000 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 Scheduling Latency ( μ s) (Bin size = 1 μ s) overhead in microseconds (bin size = 1.00us) 52 Tuesday, July 9, 13 52

  9. LITMUS RT vs . Linux 3 . 0: � CPU-bound Background Tasks Linux 3.0: scheduling latency (CPU-bound bg tasks) Linux 3.0 P-FP(LITMUS RT ) min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us avg=4.22 μ s max=72.73 μ s avg=5.17 μ s max=47.59 μ s 1e+07 samples: total=5854711 1e+06 number of samples 100000 10000 1000 100 3 samples out of ~6 million 10 1 0 10 20 30 40 50 60 70 80 90 Scheduling Latency ( μ s) (Bin size = 1 μ s) overhead in microseconds (bin size = 1.00us) 53 Tuesday, July 9, 13 53

  10. LITMUS RT vs . Linux 3 . 0: � CPU-bound Background Tasks Linux 3.0: scheduling latency (CPU-bound bg tasks) Linux 3.0 P-FP(LITMUS RT ) min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us avg=4.22 μ s max=72.73 μ s avg=5.17 μ s max=47.59 μ s 1e+07 samples: total=5854711 Slightly worse latencies 1e+06 number of samples 100000 on average 10000 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 Scheduling Latency ( μ s) (Bin size = 1 μ s) overhead in microseconds (bin size = 1.00us) 54 Tuesday, July 9, 13 54

  11. LITMUS RT vs . Linux 3 . 0: � CPU-bound Background Tasks Linux 3.0: scheduling latency (CPU-bound bg tasks) Linux 3.0 P-FP(LITMUS RT ) min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us avg=4.22 μ s max=72.73 μ s avg=5.17 μ s max=47.59 μ s 1e+07 samples: total=5854711 1e+06 number of samples 100000 extra spinlock 10000 lack of low-level 1000 optimizations 100 10 1 0 10 20 30 40 50 60 70 80 90 Scheduling Latency ( μ s) (Bin size = 1 μ s) overhead in microseconds (bin size = 1.00us) 55 Tuesday, July 9, 13 55

  12. CPU-bound Background Tasks LITMUS RT incurs slightly more latency Scheduling Latency ( μ s ) Average (99% conf.) Maximum Latency under than Linux 3.0 on 80 72.7 PREEMPT_RT is average. significantly lower. 60 47.6 40 17.4 20 5.2 4.2 3.4 0 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 56 Tuesday, July 9, 13 56

  13. Third Scenario I/O-bound background tasks ✤ hackbench: Linux scheduler stress tool bonnie++: Disk and file system benchmark wget: Network activity 57 Tuesday, July 9, 13 57

  14. Third Scenario Causes a lot of system I/O-bound calls and interrupts background tasks ✤ hackbench: Linux scheduler stress tool bonnie++: Disk and file system benchmark wget: Network activity 58 Tuesday, July 9, 13 58

  15. I/O-bound Background Tasks log scale! Scheduling Latency ( μ s ) Average (99% conf.) Maximum 10,000 4300.4 3956.5 1,000 100 44.2 6.6 6.4 10 4.1 1 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 59 Tuesday, July 9, 13 59

  16. Huge impact on I/O-bound Background Tasks scheduling latency under standard Linux. Scheduling Latency ( μ s ) Average (99% conf.) Maximum 10,000 4300.4 3956.5 1,000 100 44.2 6.6 6.4 10 4.1 1 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 60 Tuesday, July 9, 13 60

  17. Huge impact on I/O-bound Background Tasks scheduling latency under standard Linux. Scheduling Latency ( μ s ) Average (99% conf.) Maximum PREEMPT_RT is not 10,000 4300.4 3956.5 a ff ected by the interrupt load! 1,000 100 44.2 6.6 6.4 10 4.1 1 P-FP (LITMUS^RT) Linux 3.0 PREEMPT_RT 61 Tuesday, July 9, 13 61

  18. Summary 1.Cost of the scheduling plugin layer 2.LITMUS RT vs. PREEMPT_RT 62 Tuesday, July 9, 13 62

  19. Summary 1.Cost of the scheduling plugin layer The overhead introduced by LITMUS RT is small 2.LITMUS RT vs. PREEMPT_RT 63 Tuesday, July 9, 13 63

  20. Summary 1.Cost of the scheduling plugin layer The overhead introduced by LITMUS RT is small 2.LITMUS RT vs. PREEMPT_RT PREEMPT_RT significantly decreases scheduling latency. 64 Tuesday, July 9, 13 64

  21. Importance of Feather-Trace ✤ cyclictest was ported to LITMUS RT . ✤ Should it become the standard tool for evaluating LITMUS RT ? 65 Tuesday, July 9, 13 65

  22. Importance of Feather-Trace ✤ cyclictest was ported to LITMUS RT . ✤ Should it become the standard tool NO! for evaluating LITMUS RT ? 66 Tuesday, July 9, 13 66

  23. Interference? OOPS... a higher priority task interrupt! k a s t u p e k w a T T T s ! scheduler invoked ISR called 67 Tuesday, July 9, 13 67

  24. Interference? interrupt! k a s t u p e k w a T T T s ! ? scheduler invoked another task picked ISR called 68 Tuesday, July 9, 13 68

  25. Interference? interrupt! k a s t u p e k w a T T T s ! P ? ... task picked ISR called scheduler invoked another task picked 69 Tuesday, July 9, 13 69

  26. Interference? interrupt! k a s t u p e k w a T T T This length of this interval depends on the execution of other tasks... s ! P ? ... task picked ISR called scheduler invoked another task picked 70 Tuesday, July 9, 13 70

  27. Interference? ..., which depends on other kinds of overhead, preemptions, interrupt! k a s t u p e k w a context switches, etc. T T T This length of this interval depends on the execution of other tasks... s ! P ? ... task picked ISR called scheduler invoked another task picked 71 Tuesday, July 9, 13 71

  28. Interference? ..., which depends on other kinds of overhead, preemptions, interrupt! k a s t u p e k w a context switches, etc. Overhead-aware schedulability T analysis is required! T T This length of this interval depends on the execution of other tasks... s ! P ? ... task picked ISR called scheduler invoked another task picked 72 Tuesday, July 9, 13 72

  29. cyclictest or Feather-Trace? BOTH! cyclictest LITMUS RT /Feather-Trace For tasks other than the Practical, easy-to- highest-priority ones, understand measure schedulability analysis is necessary. Can easily compare responsiveness Only with Feather-Trace between kernels. we obtain the data required for the analysis. 73 Tuesday, July 9, 13 73

  30. Conclusion LITMUS RT : small overheads in comparison with stock Linux PREEMPT_RT is highly necessary for Linux as a RTOS LITMUS RT will be ported to PREEMPT_RT soon ✤ Scheduling latency should not be used as the sole metric for quantifying real-time guarantees 74 Tuesday, July 9, 13 74

  31. Thank You! We also have a patch that implements Feather-Trace on top of standard Linux, enabling fine-grained measurements. 75 Tuesday, July 9, 13 75

  32. Appendix 76 Tuesday, July 9, 13 76

  33. Linux 3.0 vs. Linux 3.8.13 77 Tuesday, July 9, 13 77

  34. No Background Tasks Scheduling Latency ( μ s ) Average Maximum 25 19.7 20 13.9 15 10 5 2.9 2.9 0 Linux 3.0 Linux 3.8.13 78 Tuesday, July 9, 13 78

  35. No Background Tasks Scheduling Latency ( μ s ) Average Maximum 25 19.7 20 Similar 13.9 averages 15 10 5 2.9 2.9 0 Linux 3.0 Linux 3.8.13 79 Tuesday, July 9, 13 79

  36. No Background Tasks Linux 3.0 Linux 3.0: scheduling latency (no bg tasks) min=1.87us max=13.89us avg=2.89us median=2.77us stdev=0.51us avg=2.89 μ s max=13.89 μ s 1e+07 samples: total=5854779 1e+06 number of samples 100000 10000 1000 Similar 100 10 shapes 1 0 10 20 30 Linux 3.8.13 40 50 60 70 80 90 Linux 3.8.13: scheduling latency (no bg tasks) overhead in microseconds (bin size = 1.00us) min=1.52us max=19.73us avg=2.89us median=2.58us stdev=0.69us avg=2.89 μ s max=19.73 μ s 1e+07 samples: total=5854801 1e+06 number of samples 100000 10000 1000 100 10 1 80 0 10 20 30 40 50 60 70 80 90 Tuesday, July 9, 13 80

  37. CPU-bound Background Tasks Scheduling Latency ( μ s ) Average Maximum 80 72.7 64.5 60 40 20 4.2 4.0 0 Linux 3.0 Linux 3.8.13 81 Tuesday, July 9, 13 81

  38. CPU-bound Background Tasks Scheduling Latency ( μ s ) Average Maximum 80 72.7 64.5 60 Similar averages 40 20 4.2 4.0 0 Linux 3.0 Linux 3.8.13 82 Tuesday, July 9, 13 82

  39. CPU-bound Background Tasks Linux 3.0 Linux 3.0: scheduling latency (CPU-bound bg tasks) min=2.04us max=72.73us avg=4.22us median=3.86us stdev=1.37us avg=4.22 μ s max=72.73 μ s 1e+07 samples: total=5854711 1e+06 number of samples 100000 10000 1000 Similar 100 10 shapes 1 0 10 20 30 Linux 3.8.13 40 50 60 70 80 90 Linux 3.8.13: scheduling latency (CPU-bound bg tasks) overhead in microseconds (bin size = 1.00us) min=2.14us max=64.47us avg=4.02us median=3.67us stdev=1.20us avg=4.02 μ s max=64.47 μ s 1e+07 samples: total=5854707 1e+06 number of samples 100000 10000 1000 100 10 1 83 0 10 20 30 40 50 60 70 80 90 Tuesday, July 9, 13 83

  40. I/O-bound Background Tasks log scale! Scheduling Latency ( μ s ) Average Maximum 10,000 5464.1 4300.4 1,000 100 6.4 6.2 10 1 Linux 3.0 Linux 3.8.13 84 Tuesday, July 9, 13 84

  41. I/O-bound Background Tasks Scheduling Latency ( μ s ) Average Maximum 10,000 5464.1 4300.4 Similar 1,000 averages 100 6.4 6.2 10 1 Linux 3.0 Linux 3.8.13 85 Tuesday, July 9, 13 85

  42. I/O-bound Background Tasks Linux 3.0 Linux 3.0: scheduling latency (IO-bound bg tasks) min=1.85us max=4300.43us avg=6.39us median=4.98us stdev=13.25us avg=6.39 μ s max=4300.43 μ s 1e+07 samples: total=5854674 1e+06 number of samples 100000 10000 1000 Similar 100 10 shapes 1 0 150 300 450 600 Linux 3.8.13 750 900 1050 1200 1350 Linux 3.8.13: scheduling latency (IO-bound bg tasks) overhead in microseconds (bin size = 1.00us) min=1.85us max=5464.07us avg=6.23us median=4.60us stdev=15.91us avg=6.23 μ s max=5464.07 μ s 1e+07 samples: total=5854773 1e+06 number of samples 100000 10000 1000 100 10 1 86 0 150 300 450 600 750 900 1050 1200 1350 Tuesday, July 9, 13 86

  43. LITMUS RT ’s plugins 87 Tuesday, July 9, 13 87

  44. No Background Tasks Scheduling Latency ( μ s ) Average Maximum 30 26.2 22.5 15.1 14.3 15 7.5 3.5 3.5 3.1 0 P-FP PSN-EDF GSN-EDF 88 Tuesday, July 9, 13 88

  45. No Background Tasks Similar shapes GSN-EDF PSN-EDF LITMUS^RT G-EDF: scheduling latency (no bg tasks) LITMUS^RT P-EDF: scheduling latency (no bg tasks) avg=3.06 μ s max=14.34 μ s min=1.59us max=14.34us avg=3.06us median=2.56us stdev=1.18us min=1.76us max=26.17us avg=3.45us median=2.87us stdev=1.24us avg=3.45 μ s max=26.17 μ s 1e+07 1e+07 samples: total=5854797 samples: total=5854783 1e+06 1e+06 number of samples 100000 100000 10000 10000 1000 1000 100 100 10 10 1 1 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) P-FP LITMUS^RT P-FP: scheduling latency (no bg tasks) min=1.96us max=15.13us avg=3.45us median=3.10us stdev=1.03us avg=3.45 μ s max=15.13 μ s 1e+07 samples: total=5854818 1e+06 number of samples 100000 10000 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 89 overhead in microseconds (bin size = 1.00us) Tuesday, July 9, 13 89

  46. No Background Tasks Similar shapes GSN-EDF PSN-EDF LITMUS^RT G-EDF: scheduling latency (no bg tasks) LITMUS^RT P-EDF: scheduling latency (no bg tasks) avg=3.06 μ s max=14.34 μ s min=1.59us max=14.34us avg=3.06us median=2.56us stdev=1.18us min=1.76us max=26.17us avg=3.45us median=2.87us stdev=1.24us avg=3.45 μ s max=26.17 μ s 1e+07 1e+07 samples: total=5854797 samples: total=5854783 1e+06 1e+06 number of samples 100000 100000 10000 10000 1000 1000 2 samples 100 100 10 10 1 1 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) P-FP LITMUS^RT P-FP: scheduling latency (no bg tasks) min=1.96us max=15.13us avg=3.45us median=3.10us stdev=1.03us avg=3.45 μ s max=15.13 μ s 1e+07 samples: total=5854818 1e+06 number of samples 100000 10000 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 90 overhead in microseconds (bin size = 1.00us) Tuesday, July 9, 13 90

  47. CPU-bound Background Tasks Scheduling Latency ( μ s ) Average Maximum 80 73.3 60.2 60 47.6 40 20 5.8 5.2 5.1 0 P-FP PSN-EDF GSN-EDF 91 Tuesday, July 9, 13 91

  48. CPU-bound Background Tasks GSN-EDF PSN-EDF LITMUS^RT G-EDF: scheduling latency (CPU-bound bg tasks) LITMUS^RT P-EDF: scheduling latency (CPU-bound bg tasks) min=1.91us max=60.20us avg=5.81us median=5.39us stdev=2.51us avg=5.81 μ s max=60.20 μ s min=2.40us max=73.27us avg=5.14us median=4.21us stdev=2.95us avg=5.14 μ s max=73.27 μ s 1e+07 1e+07 samples: total=5854728 samples: total=5854739 1e+06 1e+06 number of samples 100000 100000 10000 10000 1000 1000 100 100 10 10 1 1 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) P-FP LITMUS^RT P-FP: scheduling latency (CPU-bound bg tasks) min=2.10us max=47.59us avg=5.17us median=4.37us stdev=2.75us avg=5.17 μ s max=47.59 μ s 1e+07 samples: total=5854719 1e+06 number of samples Similar 100000 10000 shapes 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 92 overhead in microseconds (bin size = 1.00us) Tuesday, July 9, 13 92

  49. CPU-bound Background Tasks GSN-EDF PSN-EDF LITMUS^RT G-EDF: scheduling latency (CPU-bound bg tasks) LITMUS^RT P-EDF: scheduling latency (CPU-bound bg tasks) min=1.91us max=60.20us avg=5.81us median=5.39us stdev=2.51us avg=5.81 μ s max=60.20 μ s min=2.40us max=73.27us avg=5.14us median=4.21us stdev=2.95us avg=5.14 μ s max=73.27 μ s 1e+07 1e+07 samples: total=5854728 samples: total=5854739 1e+06 1e+06 number of samples 100000 100000 10000 10000 1000 1000 5 samples 100 100 10 10 1 1 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) P-FP LITMUS^RT P-FP: scheduling latency (CPU-bound bg tasks) min=2.10us max=47.59us avg=5.17us median=4.37us stdev=2.75us avg=5.17 μ s max=47.59 μ s 1e+07 samples: total=5854719 1e+06 number of samples Similar 100000 10000 shapes 1000 100 10 1 0 10 20 30 40 50 60 70 80 90 93 overhead in microseconds (bin size = 1.00us) Tuesday, July 9, 13 93

  50. I/O-bound Background Tasks log scale! Scheduling Latency ( μ s ) Average Maximum 10,000 3956.5 3875.0 3905.8 1,000 100 11.0 6.6 6.6 10 1 P-FP PSN-EDF GSN-EDF 94 Tuesday, July 9, 13 94

  51. I/O-bound Background Tasks Scheduling Latency ( μ s ) Average Maximum 10,000 3956.5 3875.0 3905.8 Similar results for 1,000 P-FP and PSN-EDF 100 11.0 6.6 6.6 10 1 P-FP PSN-EDF GSN-EDF 95 Tuesday, July 9, 13 95

  52. I/O-bound Background Tasks Higher Scheduling Latency ( μ s ) Average Maximum average 10,000 3956.5 3875.0 3905.8 for GSN-EDF Similar results for 1,000 P-FP and PSN-EDF 100 11.0 6.6 6.6 10 1 P-FP PSN-EDF GSN-EDF 96 Tuesday, July 9, 13 96

  53. I/O-bound Background Tasks GSN-EDF PSN-EDF LITMUS^RT G-EDF: scheduling latency (IO-bound bg tasks) LITMUS^RT P-EDF: scheduling latency (IO-bound bg tasks) avg=10.95 μ s max=3905.79 μ s min=2.26us max=3905.79us avg=10.95us median=7.38us stdev=14.11us min=1.98us max=3874.99us avg=6.56us median=5.11us stdev=12.66us avg=6.56 μ s max=3874.99 μ s 1e+07 1e+07 samples: total=5854793 samples: total=5854606 1e+06 1e+06 number of samples 100000 100000 10000 10000 1000 1000 100 100 10 10 1 1 0 150 300 450 600 750 900 1050 1200 1350 0 150 300 450 600 750 900 1050 1200 1350 overhead in microseconds (bin size = 1.00us) overhead in microseconds (bin size = 1.00us) P-FP LITMUS^RT P-FP: scheduling latency (IO-bound bg tasks) min=1.89us max=3956.48us avg=6.60us median=5.17us stdev=12.76us avg=6.60 μ s max=3956.48 μ s 1e+07 samples: total=5854660 1e+06 number of samples Similar 100000 10000 shapes 1000 100 10 1 0 150 300 450 600 750 900 1050 1200 1350 97 overhead in microseconds (bin size = 1.00us) Tuesday, July 9, 13 97

Recommend


More recommend