compressing strings of the kernel
play

Compressing Strings of the Kernel Wolfram Sang Consultant - PowerPoint PPT Presentation

Jan 14, 2023 •460 likes •909 views

. . Compressing Strings of the Kernel Wolfram Sang Consultant 21.8.2014, LinuxCon14 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 1 / 36 The origin: CEWG project 1 kernel debug messages.

  1. . . Compressing Strings of the Kernel Wolfram Sang Consultant 21.8.2014, LinuxCon14 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 1 / 36

  2. The origin: CEWG project 1 kernel debug messages. 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) 1 http://elinux.org/Compressed_printk_messages … Timothy Miller did some work on this in 2003 … savings attractive, because they still would like to see . developers do not use it, even when they find the space option to disable all printks, but many embedded messages to save kernel runtime footprint. There is an Attempts have been made in the past to compress printk . . From the proposal: 2 / 36

  3. Timothy’s approach 2 4 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) 2 http://lwn.net/Articles/28935/ Compile again 6 . There, replace strings with tokens 5 . . Create copies of the source files . . . Compress those strings using tokenization 3 . . Filter them for printk strings 2 . . Compile kernel and keep .i -files 1 . 3 / 36

  4. Further notes no code was made public, only the description, a codebook and some results not even sure depacking at printk was ever made allyesconfig was used for the tests based on 2.4.20 and 2.5.68 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 4 / 36

  5. Author asks the golden question, too . Timothy Miller: 3 . . ”So, I ask... is this a useful savings? Is there any chance anyone would bother to increase their compile time by a factor of 5 in order to shave off 4% or 100k bytes?” 3 https://lkml.org/lkml/2003/6/6/207 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 5 / 36

  6. The Graph! Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 6 / 36

  7. The Graph! Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 6 / 36 5 4 3 compiletime size 2 1 0 before after

  8. Three problems identified . 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) Replace printk format strings 3 . Compress printk format strings . 2 . . Extract printk format strings 1 . 7 / 36

  9. Extracting . Problem: Find all printk-strings . . There are lots of functions/defines embedding printk/vprintk_emit They are nested in all ways you can think of Moving target, there will be more <new_subsys>_dev_err, … Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 8 / 36

  10. Extracting: Options I . Scan the source files . . needs to know all printk-emerging functions misses merging of literals handle all ways of string concatenation Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 9 / 36

  11. Extracting: Options II . printk strings to own section . . scales a bit better (only base functions need to be converted) no knowledge where strings came from needs changes to core functions Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 10 / 36

  12. Extracting: own section + 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) +} while (0) printk(fmt, ##args); \ + } else \ printk(__f, ##args); \ +#define __printk(fmt, args...) \ + char __f[] = fmt; \ + static const __attribute__((section("__printk"))) \ + if (__builtin_constant_p(fmt)) { \ + +do { \ 11 / 36

  13. Extracting: own section II + 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) BTW don’t redefine printk . Really, don’t! ... __printk(KERN_ALERT pr_fmt(fmt), ##__VA_ARGS__) printk(KERN_ALERT pr_fmt(fmt), ##__VA_ARGS__) #define pr_emerg(fmt, ...) \ - #define pr_alert(fmt, ...) \ __printk(KERN_EMERG pr_fmt(fmt), ##__VA_ARGS__) + printk(KERN_EMERG pr_fmt(fmt), ##__VA_ARGS__) - 12 / 36

  14. Extracting: own section III 5 ++++ 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) 5 files changed, 75 insertions(+), 60 deletions(-) | 27 +++++++++++++++------- include/linux/printk.h | 48 ++++++++++++++++++++++++++------------- include/linux/device.h include/asm-generic/vmlinux.lds.h | Author: Wolfram Sang <wsa@the-dreams.de> | 19 ++++++++++++---- include/asm-generic/bug.h | 36 ++++------------------------- drivers/base/core.c Signed-off-by: Wolfram Sang <wsa@the-dreams.de> Kernel doesn't fully build yet. Only pr_*, dev_*, BUG, and WARN are supported. WIP: move printk strings to a special section 13 / 36

  15. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 14 / 36

  16. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 14 / 36

  17. Compressing . 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) no frequency based compression (stats3) no variable length encoding (Huffman and friends) no sliding window algos (LZ and friends) . Conclusion . . no significant overhead not somewhere in the middle of packed data should be instantly available lots of small strings . . Problem: Algorithm 15 / 36

  18. Compressing . 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) no frequency based compression (stats3) no variable length encoding (Huffman and friends) no sliding window algos (LZ and friends) . Conclusion . . no significant overhead not somewhere in the middle of packed data should be instantly available lots of small strings . . Problem: Algorithm 15 / 36

  19. Compressing II tokenization is actually a good option BytePairEncoding works, too . Problem: UTF8 . . Both approaches need ’empty’ symbols which might collide with UTF8 encoding. Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 16 / 36 both achieve ≈ 50% of compression still ≈ 4% of the kernel size gained

  20. Compressing II tokenization is actually a good option BytePairEncoding works, too . Problem: UTF8 . . Both approaches need ’empty’ symbols which might collide with UTF8 encoding. Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 16 / 36 both achieve ≈ 50% of compression still ≈ 4% of the kernel size gained

  21. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 17 / 36

  22. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 17 / 36

  23. Compressing III Brainstorming 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) could save second kernel compile, too predefined codebook? . . . . what about modules? smaller kernels, smaller pool for codes allyesconfig is unrealistic for tiny systems . . Problem: Codebook 18 / 36 share codebook from kernel → tied to that build own codebook → overhead eats gain

  24. Replacing . Scanning . . run source files through filter before compiling . Own section . . Work on the section directly? Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 19 / 36

  25. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 20 / 36

  26. Upstreaming forecast Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 20 / 36

  27. Further issues printk strings are only a subset devicetree uses a lot of strings! which should be easier to tackle since they are accessed via of_* functions address this problem at a higher level all strings? all .rodata? Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 21 / 36

  28. Summary Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 22 / 36

  29. Summary Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 22 / 36

  30. This quote still makes sense . From: Managing Gigabytes 4 . . ”We find ourselves in the midst of a practically important and intelectually fascinating convergence between the desire for more and better compression and the need to learn about what ’structure’ there is in data.” 4 Witten/Moffat/Bell, 1st edition, p. 385 Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 23 / 36

  31. Analyzing the data observations from 3.16-rc5: x86-64 allyesconfig arm-cortexa8 customer kernel maybe a bit biased for device drivers Wolfram Sang (wsa@the-dreams.de) Compressing Strings of the Kernel 21.8.2014, LinuxCon14 24 / 36

  32. Print from central locations . 21.8.2014, LinuxCon14 Compressing Strings of the Kernel Wolfram Sang (wsa@the-dreams.de) devm_ioremap_resource() OOM error message removal . Examples . . bonus: consistent messages locations as possible strings should be emitted from as centralized . . Proposal 25 / 36

Recommend

Chapter 9 Strings 1 C-Strings vs C++ Strings T wo string types: C-strings Array

Chapter 9 Strings 1 C-Strings vs C++ Strings T wo string types: C-strings Array

Chapter 9 Strings 1 C-Strings vs C++ Strings T wo string types: C-strings Array with base type char End of string marked with null, \0 Older method inherited from C C++ strings Objects of string class 2

469 views • 18 slides
Strings Testing for equality with strings. Lexicographic ordering of strings. Other

Strings Testing for equality with strings. Lexicographic ordering of strings. Other

Strings Testing for equality with strings. Lexicographic ordering of strings. Other string methods. 1 Strings and Testing for Equality The == operator is also not appropriate for determining if two objects , such as strings, have

380 views • 13 slides
Python:Strings Strings

Python:Strings Strings

Python:Strings Strings Source:https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-0001-introduction-to-computer-science-and- programming-in-python-fall-2016/lecture-slides-code/ Strings String is a sequence of

293 views • 13 slides
Strings l Chapter 3s problem context is cryptography, but mostly it is about strings and

Strings l Chapter 3s problem context is cryptography, but mostly it is about strings and

Starting chapter 3 Strings l Chapter 3s problem context is cryptography, but mostly it is about strings and related ideas l Strings are basically sequences of characters l A string literal is enclosed in quotes ( '' or in Python):

292 views • 12 slides
ARM Assembler Strings Strings p. 1/16 Characters or Strings A string is a sequence of

ARM Assembler Strings Strings p. 1/16 Characters or Strings A string is a sequence of

Systems Architecture ARM Assembler Strings Strings p. 1/16 Characters or Strings A string is a sequence of characters ASCII 7-bit (American) character codes ( char ) One byte per character Unicode International character

901 views • 61 slides
Py Python Strings Python strings are immuatable: s = abc s[2] = d s = abd

Py Python Strings Python strings are immuatable: s = abc s[2] = d s = abd

2/25/20 CS 224 Introduction to Python Spring 2020 Class #14: Strings Py Python Strings Python strings are immuatable: s = abc s[2] = d s = abd These dont change the string abc s = s[:-1] + d they

444 views • 19 slides
s[i] Introduction to Computer Programming Strings CSCI-UA 2 Strings and Characters Strings are

s[i] Introduction to Computer Programming Strings CSCI-UA 2 Strings and Characters Strings are

Introduction to Computer Programming Strings CSCI-UA 2 Strings and Characters s[i] Introduction to Computer Programming Strings CSCI-UA 2 Strings and Characters Strings are one of Pythons primary Strings data types Strings can be used

242 views • 10 slides
Listing Bit Strings List all bit strings of length 3. 000, 001, 010, 011, 100, 101, 110, 111.

Listing Bit Strings List all bit strings of length 3. 000, 001, 010, 011, 100, 101, 110, 111.

Listing Bit Strings List all bit strings of length 3. 000, 001, 010, 011, 100, 101, 110, 111. Now do it while only flipping one bit at a time! Today: Finish graphs and talk about numbers. Forests A forest is an acyclic graph. Each connected

534 views • 26 slides
Strings A string is an array of characters s = 'abc' MATLAB Strings is equivalent to s =

Strings A string is an array of characters s = 'abc' MATLAB Strings is equivalent to s =

Strings A string is an array of characters s = 'abc' MATLAB Strings is equivalent to s = [ 'a' 'b' 'c' ] All operations that apply to vectors and Selim Aksoy arrays can be used together with Bilkent University strings as well

39 views • 3 slides
Chapter 9: Strings (To avoid confusion, C-style strings will be referred to as C-string,

Chapter 9: Strings (To avoid confusion, C-style strings will be referred to as C-string,

Chapter 9: Strings (To avoid confusion, C-style strings will be referred to as C-string, regular C++ strings from the string class will be referred to as String). C-Strings Made up of character arrays that stores strings of

690 views • 3 slides
Lempel- -Ziv Ziv- -Welch (LZW) Welch (LZW) Lempel Data Compressing Model Data Compressing

Lempel- -Ziv Ziv- -Welch (LZW) Welch (LZW) Lempel Data Compressing Model Data Compressing

Lempel- -Ziv Ziv- -Welch (LZW) Welch (LZW) Lempel Data Compressing Model Data Compressing Model Martin Chakravorti Information Information What is information? Any interaction What is information? Any interaction between objects, when

754 views • 15 slides
Strings, Languages, and Regular expressions Lecture 2 1 Strings 2 Definitions for strings

Strings, Languages, and Regular expressions Lecture 2 1 Strings 2 Definitions for strings

Strings, Languages, and Regular expressions Lecture 2 1 Strings 2 Definitions for strings e.g., = {0,1}, = { , , , } , = set of ascii characters alphabet = finite set of symbols string = finite

371 views • 36 slides
Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel Fernndez V, David P. Woodruff, Taisuke Yasuda Kernel Method Many machine learning tasks can be expressed as a function of the inner product

389 views • 22 slides
Strings Digital Medicine I Lists, strings, loops Repetition Hans-Joachim Bckenhauer Dennis

Strings Digital Medicine I Lists, strings, loops Repetition Hans-Joachim Bckenhauer Dennis

Departement Informatik Strings Digital Medicine I Lists, strings, loops Repetition Hans-Joachim Bckenhauer Dennis Komm Autumn 2020 October 15, 2020 Strings Strings Operators Addition Strings are lists of characters (there

268 views • 13 slides
1 Kernel methods &amp; optimization One example of a kernel that is frequently used in practice

1 Kernel methods &amp; optimization One example of a kernel that is frequently used in practice

Machine Learning Class Notes 9-25-12 Prof. David Sontag 1 Kernel methods &amp; optimization One example of a kernel that is frequently used in practice and which allows for highly non-linear discriminant functions is the Gaussian kernel,

241 views • 5 slides
C-Style Strings CS2253 Owen Kaser, UNBSJ Strings In C and some other low-level languages,

C-Style Strings CS2253 Owen Kaser, UNBSJ Strings In C and some other low-level languages,

C-Style Strings CS2253 Owen Kaser, UNBSJ Strings In C and some other low-level languages, strings are just consecutive memory locations that contain characters. A special null character (ASCII code 0) terminates the string.

813 views • 17 slides
Compressing IP Forwarding Tables: Towards Entropy Bounds and Beyond Revised on Feb 10, 2014

Compressing IP Forwarding Tables: Towards Entropy Bounds and Beyond Revised on Feb 10, 2014

Compressing IP Forwarding Tables: Towards Entropy Bounds and Beyond Revised on Feb 10, 2014 Gbor Rtvri, Jnos Tapolcai, Attila K orsi, Andrs Majdn, Zaln Heszberger Budapest Univ. of Technology and Economics Dept. of

244 views • 23 slides
STRINGS AND FACTORS Jeff Goldsmith, PhD Department of Biostatistics 1 Strings vs Factors

STRINGS AND FACTORS Jeff Goldsmith, PhD Department of Biostatistics 1 Strings vs Factors

STRINGS AND FACTORS Jeff Goldsmith, PhD Department of Biostatistics 1 Strings vs Factors They both look like character vectors, but: Strings are just strings Factors have an underlying numeric structure with character labels sitting

241 views • 8 slides
Strings in Python Computers store text as strings &gt;&gt;&gt; s = &quot;GATTACA&quot; 0 1 2

Strings in Python Computers store text as strings &gt;&gt;&gt; s = &quot;GATTACA&quot; 0 1 2

Strings in Python Computers store text as strings &gt;&gt;&gt; s = &quot;GATTACA&quot; 0 1 2 3 4 5 6 G A T T A C A s Each of these are characters Why are strings important? Sequences are strings ..catgaaggaa ccacagccca

712 views • 17 slides
HANDOUT 1 Strings STRINGS Weve already introduced the string data type a few lectures ago.

HANDOUT 1 Strings STRINGS Weve already introduced the string data type a few lectures ago.

HANDOUT 1 Strings STRINGS Weve already introduced the string data type a few lectures ago. Strings are subtypes of the sequence data type. Strings are written with either single or double quotes encasing a sequence of characters. s1 =

470 views • 25 slides
Compressing IP Forwarding Tables: Towards Entropy Bounds and Beyond Gbor Rtvri, Jnos

Compressing IP Forwarding Tables: Towards Entropy Bounds and Beyond Gbor Rtvri, Jnos

Compressing IP Forwarding Tables: Towards Entropy Bounds and Beyond Gbor Rtvri, Jnos Tapolcai, Attila K orsi, Andrs Majdn, Zaln Heszberger Budapest Univ. of Technology and Economics Dept. of Telecomm. and Media Informatics

418 views • 24 slides
Mixing Between Two Compressing Cylinders Steven H. Batha Los Alamos National Laboratory K. W.

Mixing Between Two Compressing Cylinders Steven H. Batha Los Alamos National Laboratory K. W.

LA-UR-01-6658 Mixing Between Two Compressing Cylinders Steven H. Batha Los Alamos National Laboratory K. W. Parker, # C. W. Barnes,* A. M. Dunne, # N. E. Lanier,* G. R. Magelssen,* T. J. Murphy,* S. D. Rothman, # J. M. Scott,* and D. L. Youngs #

619 views • 16 slides
Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel

Tight Kernel Query Complexity of Kernel Ridge Regression and Kernel -means Clustering Manuel Fernndez V, David P. Woodruff, Taisuke Yasuda Overview Preliminaries Kernel ridge regression Kernel -means clustering

476 views • 25 slides
Debugging the Linux Kernel with GDB Kieran Bingham Debugging the Linux Kernel with GDB Many

Debugging the Linux Kernel with GDB Kieran Bingham Debugging the Linux Kernel with GDB Many

Debugging the Linux Kernel with GDB Kieran Bingham Debugging the Linux Kernel with GDB Many of us need to debug the Linux kernel Proprietary tools like Trace32 and DS-5 are $$$ Open source debuggers like GDB lack kernel

884 views • 40 slides

More recommend