clang-scan-deps Fast Dependency Scanning For Explicit Modules - PowerPoint PPT Presentation

clang-scan-deps • • Fast Dependency Scanning For Explicit Modules Alex Lorenz, Michael Spencer, Apple LLVM Developers’ Meeting, Brussels, Belgium, April 2019 � 1

• Clang Modules • Dependency Scanning • Fast Dependency Scanning • Dependency Extraction • Future Work � 2

Clang Modules • Replace the textual preprocessor inclusions with an import of an AST • Widely used in SDKs shipped with Xcode • Implicit modules: Clang builds modules as they’re included • Users don’t have to specify modular dependencies • Requires a build system in the compiler 😬 � 3

Compiler Discovered Build System Known Implicit Modules A.cpp B.cpp C.cpp D.cpp � 4

Compiler Discovered Build System Known Implicit Modules Transforms A.cpp B.cpp C.cpp D.cpp � 5

Implicit Modules Module Maps module LLVM_Transforms { requires cplusplus umbrella "Transforms" module * { export * } } � 6

Compiler Discovered Build System Known Implicit Modules Transforms A.cpp B.cpp C.cpp D.cpp � 7

Compiler Discovered Build System Known Implicit Modules Transforms A.cpp B.cpp C.cpp D.cpp � 8

Compiler Discovered Build System Known Implicit Modules Analysis Analysis Transforms A.cpp B.cpp C.cpp D.cpp � 9

Compiler Discovered Build System Known Implicit Modules Analysis Transforms A.cpp B.cpp C.cpp D.cpp � 10

Compiler Discovered Build System Known Implicit Modules IR Analysis Transforms A.cpp B.cpp C.cpp D.cpp � 11

Compiler Discovered Build System Known Implicit Modules IR Analysis Transforms A.cpp B.cpp C.cpp D.cpp � 12

Compiler Discovered Build System Known Implicit Modules IR Analysis Transforms A.cpp B.cpp C.cpp D.cpp E.cpp -DFOO � 13

Compiler Discovered Build System Known Implicit Modules IR Analysis Transforms Transforms A.cpp B.cpp C.cpp D.cpp E.cpp -DFOO � 14

Compiler Discovered Build System Known Implicit Modules IR Analysis Analysis Transforms Transforms A.cpp B.cpp C.cpp D.cpp E.cpp -DFOO � 15

Compiler Discovered Build System Known Implicit Modules ☹ IR IR Analysis Analysis Transforms Transforms A.cpp B.cpp C.cpp D.cpp E.cpp -DFOO � 16

Compiler Discovered Build System Known Explicit Modules IR IR IR Analysis Analysis Transforms Transforms Transforms A.cpp B.cpp C.cpp D.cpp E.cpp -DFOO � 17

Explicit Clang Modules • Better model: knowing modular dependencies before compiling • Allow more robust and reproducible builds • Faster builds 🏏 • Constraint: users shouldn’t have to specify modular dependencies • Problem: which modules are needed? • Solution: dependency discovery build phase for a build target � 18

• Clang Modules • Dependency Scanning • Fast Dependency Scanning • Dependency Extraction • Future Work � 19

Canonical Dependency Scanning Phase • Preprocess all translation units of a build target • Write out included files into a .d • clang -cc1 -Eonly -MT —dependency-file foo.d foo.c • How fast is the preprocessor? � 20

Clang and LLVM sources: preprocessing time on an 18-Core iMac Pro 60 Preprocessing Time 45 Time (seconds) 30 15 0 6 12 18 24 30 36 Number Of Workers 21 �

Clang and LLVM sources: the 12 workers scenario 60 Preprocessing Time 45 Time (seconds) 30 15 0 6 12 18 24 30 36 Number Of Workers � 22

Clang and LLVM sources: the 12 workers scenario 60 Preprocessing Time 45 31 seconds to preprocess Time (seconds) 31 seconds to preprocess 31 seconds to preprocess 30 15 👏 Not fast enough for every build! 0 6 12 18 24 30 36 Number Of Workers � 23

• Clang Modules • Dependency Scanning • Fast Dependency Scanning • Dependency Extraction • Future Work � 24

What Does The Preprocessor Do? #ifndef HEADER_FILE #define HEADER_FILE Lex tokens… #include “Compiler.h” Evaluate #ifndef & #define // Clang is an awesome tool! Lex more tokens… class Clang: public Compiler { public: Include “Compiler.h” void buildAllCode(); #ifndef NDEBUG Lex more tokens… void dump(); Lex even more tokens 😬 #endif }; #endif � 25

Reducing Preprocessor Workload #ifndef HEADER_FILE #define HEADER_FILE #include “Compiler.h” // Clang is an awesome tool! class Clang: public Compiler { Dependencies aren’t affected by these tokens 31 seconds to preprocess public: 31 seconds to preprocess void buildAllCode(); #ifndef NDEBUG void dump(); #endif }; #endif � 26

Reducing Preprocessor Workload #ifndef HEADER_FILE #define HEADER_FILE #include “Compiler.h” // Clang is an awesome tool! class Clang: public Compiler { Dependencies aren’t affected by these tokens 31 seconds to preprocess public: 31 seconds to preprocess void buildAllCode(); #ifndefdNDEBUG void dump(); #endif }; #endif � 27

Source Minimization #ifndef HEADER_FILE #define HEADER_FILE #include “Compiler.h” #endif � 28

Source Minimization 💢 Keep directives that may affect dependency list #ifndef HEADER_FILE Strip everything else #define HEADER_FILE #include “Compiler.h” Context free: source reused in any compilation #endif � 29

Clang and LLVM sources: 30% faster preprocessing 60 Preprocessing Time Minimized Source Preprocessing Time 45 Time (seconds) 30 15 0 6 12 18 24 30 36 Number Of Workers � 30

Problem: Clang Invocations Preprocess Preprocess Parallel invocations do redundant work #include “Test.h” Read the same file twice #include “Test.h” Minimize Source Minimize the same file twice Minimize Source Preprocess Preprocess � 31

Introducing clang-scan-deps • Library and command line tool for dependency scanning • Tool currently accepts compilation database and emits dependencies • Runs preprocessor invocations in parallel • Efficient: Reads and minimizes a source file only once • one shared FileSystem with shared minimized file cache • one shared FileManager � 32

Minimized File Cache • Maps from file name to cache entry • Shared by worker threads: lock required access the StringMap • High lock contention for many threads 5 One StringMap Time (seconds) 3.75 2.5 1.25 0 18 24 30 36 Number Of Workers 33 �

Optimizing Minimized File Cache • Solution: Array of StringMap addressed by hash of file name 5 One StringMap Time (seconds) Nine StringMaps 3.75 2.5 1.25 0 18 24 30 36 Number Of Workers 34 �

Preprocessor Block Skipping When this #ifdef is not taken… #ifdef NOT_TAKEN // Important comment The tokens inside it are lexed… #include “LexMeNot.h” Until the #elif is found #elif Took up to 10-15% of time in our profiles #include “IAmLexed.h” #endif � 35

Optimizing Preprocessor Block Skipping When this #ifdef is not taken… #ifdef NOT_TAKEN // Important comment #include “LexMeNot.h” Skip to #elif : add offset to Lexer’s pointer #elif Offset computed when minimizing file #include “IAmLexed.h” #endif � 36

Clang and LLVM sources: 5-10x faster dependency scanning 40 Preprocessing Time scan-deps Time 30 Time (seconds) 20 10 0 6 12 18 24 30 36 Number Of Workers � 37

Things We Aren’t Going To Support #define AT_IMPORT @import AT_IMPORT Foundation; #define WHY(X) _##X ("clang module import X") WHY(Pragma); ➡ We want to disallow this behavior in Clang � 38

Modular Dependencies • clang-scan-deps builds implicit modules with minimized files • For now still uses old implicit module build machinery • Dependencies are extracted from the fast implicit build � 39

• Clang Modules • Dependency Scanning • Fast Dependency Scanning • Dependency Extraction • Future Work � 40

Dependency Extraction IR.min Analysis.min Transforms.min A.cpp.min B.cpp.min C.cpp.min D.cpp.min � 41

Dependency Extraction IR.min Analysis.min Transforms.min A.cpp.min B.cpp.min C.cpp.min D.cpp.min � 42

Dependency Extraction IR.min Analysis.min Transforms.min A.cpp.min B.cpp.min C.cpp.min D.cpp.min � 43

Dependency Extraction build LLVM_Transforms.pcm: cxx_explicit_module llvm/include/llvm/module.modulemap | LLVM_IR.pcm LLVM_Analysis.pcm std.pcm module_id = LLVM_Transforms moduledeps = -fmodule-file=LLVM_Config_Config.pcm -fmodule-file=std.pcm -fmodule-file=LLVM_IR.pcm -fmodule-file=LLVM_Analysis.pcm args = builds/release/bin/clang++ -cc1 -fmodules ...

Initial Results - Scanning • Faster than modules -Eonly and -Eonly, but slower than scan-deps • Building modules takes time ☹ Modules -Eonly -Eonly 200 Modules scan-deps scan-deps Scan Time (seconds) 150 100 50 0 6 12 18 24 30 36 Number Of Workers � 45

Initial Results - Scanning • Faster than modules -Eonly and -Eonly, but slower than scan-deps Modules -Eonly -Eonly 60 Modules scan-deps scan-deps Scan Time (seconds) 45 30 15 0 6 12 18 24 30 36 Number Of Workers � 46

Initial Results - Scanning Modules -Eonly -Eonly 22 Modules scan-deps The cost of building modules during scanning scan-deps Scan Time (seconds) 16.5 11 5.5 0 6 12 18 24 30 36 Number Of Workers � 47