Parallelizing Quickref Didier Verna EPITA / LRDE - PowerPoint PPT Presentation

Toolchain Experiments Solution Perspectives Parallelizing Quickref Didier Verna EPITA / LRDE didier@lrde.epita.fr lrde/~didier @didierverna didier.verna in/didierverna Conclusion Introduction e ELS 2019 E

Toolchain Experiments Solution Perspectives Conclusion Thank you! Any questions? Parallelizing Quickref / ELS 2019 – Didier Verna 2/16 Conclusion Introduction ◮ Quickref parallelized ◮ Performance improvement: 4x

3/16 over 1700+ libraries Experiments Solution Perspectives Introduction Parallelizing Quickref / ELS 2019 – Didier Verna (Quicklisp) libraries Toolchain or local builds depending on the conditions Conclusion Introduction ◮ Reference manuals for CL ◮ quickref.common-lisp.net ◮ Originally: sequential loop ◮ 1h30 – 7h ◮ Parallelism worth investigating

Toolchain Experiments Solution Perspectives Plan Toolchain Experiments Parallel Solution Discussion and Perspectives Parallelizing Quickref / ELS 2019 – Didier Verna 4/16 Conclusion Introduction

Toolchain Experiments Solution Perspectives Plan Toolchain Experiments Parallel Solution Discussion and Perspectives Parallelizing Quickref / ELS 2019 – Didier Verna 5/16 Conclusion Introduction

6/16 Declt Parallelizing Quickref / ELS 2019 – Didier Verna Important remarks: foo.html Toolchain foo.texi Makeinfo foo/ Quicklisp Toolchain Perspectives Solution Experiments Conclusion Introduction ◮ Declt works by introspection ◮ Compilation / loading (of dependencies) may be required ◮ Avoid loading 1700+ libraries in the same Lisp image! ◮ Run Declt in external processes ◮ Makeinfo is a Perl/C script ◮ Ditto

Toolchain Experiments Solution Perspectives Plan Toolchain Experiments Parallel Solution Discussion and Perspectives Parallelizing Quickref / ELS 2019 – Didier Verna 7/16 Conclusion Introduction

8/16 1. All libraries already compiled (1h 27m) Experiments Solution Perspectives Scenarios Parallelizing Quickref / ELS 2019 – Didier Verna 3. Nothing compiled, local compilation cache (7h 01m) 2. Nothing compiled, global compilation cache (1h 51m) Toolchain Conclusion Introduction ◮ Quickref Options ◮ Libraries and Update ◮ full / installed-only ◮ download / compilation may occur ◮ Compilation cache Policy ◮ global / local ◮ global cache may cause problems ◮ Scenarios

9/16 1 3 Scenario Texinfo Html 1 52% 48% 2 60% Toolchain 40% 3 90% 10% Parallelizing Quickref / ELS 2019 – Didier Verna Time Distribution Perspectives Solution Experiments 2 Conclusion Introduction Loading Makeinfo Compilation 4 % 10 % 16 % Declt Loading 10 % Makeinfo 3 % Loading 40 % 1 % Makeinfo 48 % Declt 48 % Declt Compilation 41 % 79 %

Toolchain Experiments Solution Perspectives Plan Toolchain Experiments Parallel Solution Discussion and Perspectives Parallelizing Quickref / ELS 2019 – Didier Verna 10/16 Conclusion Introduction

11/16 Declt Parallelizing Quickref / ELS 2019 – Didier Verna HTML Files Makeinfo Makeinfo Toolchain Texinfo Files Makeinfo Declt Declt Library Pool Parallel Solution Perspectives Solution Experiments Conclusion Introduction ◮ Usable in all scenarios ◮ Dependency graph management cheap ◮ Scenario 2 ◮ Best results: 4 Declt threads / 4 Makeinfo threads ◮ Total time: 28m 17s (25% of sequential time)

12/16 3 Parallelizing Quickref / ELS 2019 – Didier Verna 6 5 4 3 1 2 Done Library Pool 4 Toolchain 2 1 6 5 Dependency Graph Library Pool Management Perspectives Solution Experiments Conclusion Introduction

12/16 Library Pool Parallelizing Quickref / ELS 2019 – Didier Verna 6 5 4 3 1 2 Done 2 1 3 Toolchain 4 2 1 6 5 Dependency Graph Library Pool Management Perspectives Solution Experiments Conclusion Introduction

12/16 Library Pool Parallelizing Quickref / ELS 2019 – Didier Verna 6 5 4 3 1 2 Done 3 1 3 Toolchain 4 2 1 6 5 Dependency Graph Library Pool Management Perspectives Solution Experiments Conclusion Introduction

12/16 Library Pool Parallelizing Quickref / ELS 2019 – Didier Verna 6 5 4 3 1 2 Done 4 3 Toolchain 4 2 1 6 5 Dependency Graph Library Pool Management Perspectives Solution Experiments Conclusion Introduction

12/16 Library Pool Parallelizing Quickref / ELS 2019 – Didier Verna 6 5 4 3 1 2 Done 5 3 Toolchain 4 2 1 6 5 Dependency Graph Library Pool Management Perspectives Solution Experiments Conclusion Introduction

12/16 Library Pool Parallelizing Quickref / ELS 2019 – Didier Verna 6 5 4 3 1 2 Done 6 3 Toolchain 4 2 1 6 5 Dependency Graph Library Pool Management Perspectives Solution Experiments Conclusion Introduction

12/16 3 Parallelizing Quickref / ELS 2019 – Didier Verna 6 5 4 3 1 2 Done Library Pool 4 Toolchain 2 1 6 5 Dependency Graph Library Pool Management Perspectives Solution Experiments Conclusion Introduction

Toolchain Experiments Solution Perspectives Plan Toolchain Experiments Parallel Solution Discussion and Perspectives Parallelizing Quickref / ELS 2019 – Didier Verna 14/16 Conclusion Introduction

15/16 Toolchain Experiments Solution Perspectives Discussion and Perspectives Parallelizing Quickref / ELS 2019 – Didier Verna Conclusion Introduction ◮ Alternative Algorithms ◮ Dependency Management Issues ◮ Based on static information provided by Quicklisp ◮ Fragile, not always correct ◮ CPU vs. I/O Consumption ◮ 4x is a bit disappointing ◮ Open the Declt and Makeinfo black boxes ◮ SJF-like Scheduling ◮ Very diffjcult to fjgure out where complexity comes from ◮ Collect timings and use them in next run ◮ SSD!

Toolchain Experiments Solution Perspectives Acknowledgments Parallelizing Quickref / ELS 2019 – Didier Verna 16/16 Conclusion Introduction ◮ Initial code base: Antoine Martin ◮ Author index and parallel algorithm #4: Antoine Hacquard ◮ Hosting (code & website): CLF / common-lisp.net