bazel
play

Bazel { fast, correct } choose two Klaus Aehlig August 1920, 2017 - PowerPoint PPT Presentation

Oct 06, 2023 •461 likes •1.75k views

Bazel How Bazel Works Extending Bazel Summary Bazel { fast, correct } choose two Klaus Aehlig August 1920, 2017 Bazel How Bazel Works Extending Bazel Summary Bazel What is Bazel? Bazel How Bazel Works Extending Bazel Summary

  1. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld We look at the target :helloworld , in package // , in file BUILD command target pkg file system

  2. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c //lib:hello Two declared dependencies command target pkg file system

  3. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c //lib:hello Two declared dependencies . . . and implicit dependency on the C tool chain command target (not drawn in this diagram) pkg file system

  4. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c lib/ BUILD //lib //lib:hello Two declared dependencies, one in a different package Note: We construct dependency graph over package boundaries! (no recursive calling) command target pkg file system

  5. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello We discover glob expressions command target pkg file system glob

  6. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c hello.h We discover glob expressions, and read the directory. command target pkg file system glob

  7. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h The rules tell us, which artifacts to build. command target pkg file system glob artifact

  8. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD helloworld.pic.o helloworld helloworld.c lib/ BUILD hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h file system artifact

  9. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h command target pkg file system glob artifact

  10. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h command target pkg file system glob artifact

  11. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h foo.c command target pkg file system glob artifact

  12. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h foo.c command target pkg file system glob artifact

  13. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h foo.c command target pkg file system glob artifact

  14. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h foo.c command target pkg file system glob artifact

  15. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h lib/foo.pic.o foo.c command target pkg file system glob artifact

  16. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h lib/foo.pic.o foo.c command target pkg file system glob artifact

  17. Bazel How Bazel Works Extending Bazel Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h lib/foo.pic.o foo.c command target pkg file system glob artifact

  18. Bazel How Bazel Works Extending Bazel Summary Actions

  19. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building

  20. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time

  21. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions

  22. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed

  23. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself

  24. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel

  25. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel • so, no .done foo targets, • and only reading declared inputs

  26. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel

  27. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes”

  28. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes” • isolated environment • only declared inputs/tools present • only declared outputs copied out

  29. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes” • isolated environment • only declared inputs/tools present • only declared outputs copied out • depending on OS, different approaches (none, temp dir, chroot , . . . )

  30. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes”

  31. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes” • bonus: remote execution

  32. Bazel How Bazel Works Extending Bazel Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes” • bonus: remote execution ⇒ enables shared caches. (Several close-by engineers working on the same code base!)

  33. Bazel How Bazel Works Extending Bazel Summary Skylark

  34. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules

  35. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • specialized rules with knowledge about certain languages cc library , cc binary , java library , java binary , . . .

  36. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • specialized rules with knowledge about certain languages cc library , cc binary , java library , java binary , . . . • generic ones, in particular genrule → just specify a shell command (with $@ , $< , . . . ) (basically the only rule available in a Makefile )

  37. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules

  38. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale

  39. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark

  40. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark • Python-like language (familiar syntax) • but restricted to a simple core without global state, complicated feature, . . . � deterministic, hermetic evaluation

  41. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark

  42. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark • To get a feeling for the language, let’s do an example

  43. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark • To get a feeling for the language, let’s do an example . . . and step by step develop rules for L A T EX

  44. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark • To get a feeling for the language, let’s do an example . . . and step by step develop rules for L A T EX • typeset pdf files from textual description ( *.tex files)

  45. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark • To get a feeling for the language, let’s do an example . . . and step by step develop rules for L A T EX • typeset pdf files from textual description ( *.tex files) • the *.tex files can pull in other files ( .sty , images, diagrams, \input other .tex -files)

  46. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark • To get a feeling for the language, let’s do an example . . . and step by step develop rules for L A T EX • typeset pdf files from textual description ( *.tex files) • the *.tex files can pull in other files ( .sty , images, diagrams, \input other .tex -files) • pdflatex main.tex && ...

  47. Bazel How Bazel Works Extending Bazel Summary Skylark • Bazel has built-in rules • but adding specialized rule for every language doesn’t scale � need ways to expend BUILD language: Skylark • To get a feeling for the language, let’s do an example . . . and step by step develop rules for L A T EX

  48. Bazel How Bazel Works Extending Bazel Summary Macros

  49. Bazel How Bazel Works Extending Bazel Summary Macros • First approach

  50. Bazel How Bazel Works Extending Bazel Summary Macros • First approach • latex -rule is given by an entry point and a list of source files

  51. Bazel How Bazel Works Extending Bazel Summary Macros • First approach • latex -rule is given by an entry point and a list of source files • have a script to typeset this (tmpdir, correct number of pdflatex runs, . . . )

  52. Bazel How Bazel Works Extending Bazel Summary Macros • First approach (entry + files; script)

  53. Bazel How Bazel Works Extending Bazel Summary Macros • First approach (entry + files; script) � write a macro in rules/latex/latex.bzl def latex(name="", main="", srcs=[]): run = str(Label("//rules/latex:runlatex.sh")) native.genrule( name = name + " pdf", srcs = srcs, cmd = ("sh $(location " + run +") $@" + " $(location " + main + ") $(SRCS)", outs = [name + ".pdf"], tools = [run], )

  54. Bazel How Bazel Works Extending Bazel Summary Macros • First approach (entry + files; script) � write a macro in rules/latex/latex.bzl def latex(name="", main="", srcs=[]): ... native.genrule(...)

  55. Bazel How Bazel Works Extending Bazel Summary Macros • First approach (entry + files; script) � write a macro in rules/latex/latex.bzl def latex(name="", main="", srcs=[]): ... native.genrule(...) • can be loaded in BUILD files

  56. Bazel How Bazel Works Extending Bazel Summary Macros • First approach (entry + files; script) � write a macro in rules/latex/latex.bzl def latex(name="", main="", srcs=[]): ... native.genrule(...) • can be loaded in BUILD files load("//rules/latex/latex.bzl", "latex")

  57. Bazel How Bazel Works Extending Bazel Summary Macros • First approach (entry + files; script) � write a macro in rules/latex/latex.bzl def latex(name="", main="", srcs=[]): ... native.genrule(...) • can be loaded in BUILD files load("//rules/latex/latex.bzl", "latex") latex( name = "slides", main = "main.tex", srcs = ["diagram.ps"], )

  58. Bazel How Bazel Works Extending Bazel Summary Macros • First approach (entry + files; script) � write a macro in rules/latex/latex.bzl def latex(name="", main="", srcs=[]): ... native.genrule(...) • can be loaded in BUILD files load("//rules/latex/latex.bzl", "latex") latex( name = "slides", main = "main.tex", srcs = ["diagram.ps"], ) � central maintenance; convenience-targets ( xpdf , pdfnup , . . . )

  59. Bazel How Bazel Works Extending Bazel Summary File Groups

  60. Bazel How Bazel Works Extending Bazel Summary File Groups • Start thinking in groups of files

  61. Bazel How Bazel Works Extending Bazel Summary File Groups • Start thinking in groups of files “That slide with all the diagrams belonging to it”

  62. Bazel How Bazel Works Extending Bazel Summary File Groups • Start thinking in groups of files

  63. Bazel How Bazel Works Extending Bazel Summary File Groups • Start thinking in groups of files • Built-in rule: filegroup filegroup(name = "foosection", srcs = ["foosection.tex", ":diagram"]) ... filegroup( name = "barchapter", srcs = ["barchapter.tex", ":foosection", ...])

  64. Bazel How Bazel Works Extending Bazel Summary File Groups • Start thinking in groups of files • Built-in rule: filegroup filegroup(name = "foosection", srcs = ["foosection.tex", ":diagram"]) ... filegroup( name = "barchapter", srcs = ["barchapter.tex", ":foosection", ...]) • Gives a label to a set of files (with traversal order) � single maintenance point

  65. Bazel How Bazel Works Extending Bazel Summary File Groups • Start thinking in groups of files • Built-in rule: filegroup filegroup(name = "foosection", srcs = ["foosection.tex", ":diagram"]) ... filegroup( name = "barchapter", srcs = ["barchapter.tex", ":foosection", ...]) • Gives a label to a set of files (with traversal order) � single maintenance point • Can be nested, inserting the entries (but implemented in a memory-efficient way!)

  66. Bazel How Bazel Works Extending Bazel Summary Rules

More recommend