bazel
play

Bazel { fast, correct } choose two Klaus Aehlig February 45, 2017 - PowerPoint PPT Presentation

Aug 24, 2023 •719 likes •1.96k views

Bazel How Bazel Works Open-Sourcing Summary Bazel { fast, correct } choose two Klaus Aehlig February 45, 2017 Bazel How Bazel Works Open-Sourcing Summary Bazel What is Bazel? Bazel How Bazel Works Open-Sourcing Summary Bazel

  1. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c //lib:hello Two declared dependencies command target pkg file system

  2. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c //lib:hello Two declared dependencies command . . . and implicit dependency on the C tool chain target pkg (not drawn in this diagram) file system

  3. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c lib/ BUILD //lib //lib:hello Two declared dependencies, one in a different package Note: We construct dependency graph over package boundaries! command (no recursive calling) target pkg file system

  4. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello We discover glob expressions command target pkg file system glob

  5. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c hello.h We discover glob expressions, and read the directory. command target pkg file system glob

  6. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h The rules tell us, which artifacts to build. command target pkg file system glob artefact

  7. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Dependencies BUILD helloworld.pic.o helloworld helloworld.c lib/ BUILD hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h file system artefact

  8. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Dependencies BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h command target pkg file system glob artefact

  9. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h command target pkg file system glob artefact

  10. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h foo.c command target pkg file system glob artefact

  11. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h foo.c command target pkg file system glob artefact

  12. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h foo.c command target pkg file system glob artefact

  13. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h foo.c command target pkg file system glob artefact

  14. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h lib/foo.pic.o foo.c command target pkg file system glob artefact

  15. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h lib/foo.pic.o foo.c command target pkg file system glob artefact

  16. Bazel How Bazel Works Open-Sourcing Summary Example cont’d: Adding a File BUILD // //:helloworld build //:helloworld helloworld.pic.o helloworld helloworld.c glob(["*.h"]) lib/ glob(["*.c"]) BUILD //lib //lib:hello hello.c lib/hello.pic.o lib/libhello.{a,so} hello.h lib/foo.pic.o foo.c command target pkg file system glob artefact

  17. Bazel How Bazel Works Open-Sourcing Summary Actions

  18. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building

  19. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time

  20. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions

  21. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed

  22. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself

  23. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel

  24. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel • so, no .done foo targets, • and only reading declared inputs

  25. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel

  26. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes”

  27. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes” • isolated environment • only declared inputs/tools present • only declared outputs copied out

  28. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes” • isolated environment • only declared inputs/tools present • only declared outputs copied out • depending on OS, different approaches ( chroot , temp dir, . . . )

  29. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes”

  30. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes” • bonus: remote execution

  31. Bazel How Bazel Works Open-Sourcing Summary Actions • action do the actual work of building . . . and hence take the most time � particularly interesting to avoid unnecessary actions • dependency graph shows if prerequisites changed • caching of input/output-relation itself ! requires all inputs/outputs to be known to bazel � facilitate correct I/O by running actions in “sandboxes” • bonus: remote execution ⇒ enables shared caches. (All engineers working on the same code base!)

  32. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel

  33. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules

  34. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • specialized rules with knowledge about certain languages cc library, cc binary, java library, java binary, . . .

  35. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • specialized rules with knowledge about certain languages cc library, cc binary, java library, java binary, . . . • generic ones, in particular genrule

  36. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • specialized rules with knowledge about certain languages cc library, cc binary, java library, java binary, . . . • generic ones, in particular genrule → just specify a shell command (with $@ , $< , . . . )

  37. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • specialized rules with knowledge about certain languages cc library, cc binary, java library, java binary, . . . • generic ones, in particular genrule → just specify a shell command (with $@ , $< , . . . ) (basically the only rule available in a Makefile )

  38. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules

  39. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale

  40. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language

  41. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark

  42. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • Python-like language (familiar syntax)

  43. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • Python-like language (familiar syntax) • but restricted to a simple core without global state, complicated features, . . .

  44. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • Python-like language (familiar syntax) • but restricted to a simple core without global state, complicated features, . . . � deterministic, hermetic evaluation

  45. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark

  46. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • simple case: can compose it from existing rules

  47. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • simple case: can compose it from existing rules “that sh -script with these params; always create 5 targets . . . ”

  48. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • simple case: can compose it from existing rules � macros

  49. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • simple case: can compose it from existing rules � macros def mylang(name="", param="default", srcs=[]): script = str(Label("//rules/mylang:bld.sh")) native.genrule( name = name + " out", tools = [script], cmd = "env . . . $(location " + script + ")" + " . . . $@ $(SRCS)", . . . ) native. . . .

  50. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • simple case: can compose it from existing rules � macros

  51. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • simple case: can compose it from existing rules � macros • all extensions are loaded in BUILD files load("// . . . .bzl", "mylang")

  52. Bazel How Bazel Works Open-Sourcing Summary Extending Bazel • Bazel has built-in rules • but adding specialized rules for every language doesn’t scale � need ways to extend BUILD language: Skylark • simple case: can compose it from existing rules � macros • all extensions are loaded in BUILD files load("// . . . .bzl", "mylang") • not so simple case: rules freely specify actions, argument declaration, . . .

  53. Bazel How Bazel Works Open-Sourcing Summary The Task of Open-Sourcing Bazel

  54. Bazel How Bazel Works Open-Sourcing Summary The Task of Open-Sourcing Bazel Bazel became open-source only after years of internal use

  55. Bazel How Bazel Works Open-Sourcing Summary The Task of Open-Sourcing Bazel Bazel became open-source only after years of internal use . . . on a single repository. (large, but just one)

  56. Bazel How Bazel Works Open-Sourcing Summary The Task of Open-Sourcing Bazel Bazel became open-source only after years of internal use . . . on a single repository. (large, but just one) • lot of dependencies, including Google-specific ones

  57. Bazel How Bazel Works Open-Sourcing Summary The Task of Open-Sourcing Bazel Bazel became open-source only after years of internal use . . . on a single repository. (large, but just one) • lot of dependencies, including Google-specific ones “We have those libs anyway, so let’s just use them.”

  58. Bazel How Bazel Works Open-Sourcing Summary The Task of Open-Sourcing Bazel Bazel became open-source only after years of internal use . . . on a single repository. (large, but just one) • lot of dependencies, including Google-specific ones

  59. Bazel How Bazel Works Open-Sourcing Summary The Task of Open-Sourcing Bazel Bazel became open-source only after years of internal use . . . on a single repository. (large, but just one) • lot of dependencies, including Google-specific ones • focus on the “Google languages” (and that built in)

  60. Bazel How Bazel Works Open-Sourcing Summary The Task of Open-Sourcing Bazel Bazel became open-source only after years of internal use . . . on a single repository. (large, but just one) • lot of dependencies, including Google-specific ones • focus on the “Google languages” (and that built in) • no stable interfaces

More recommend