Visual Debugger for Jupyter Notebooks: Myth or Reality? Elizaveta - PowerPoint PPT Presentation

Visual Debugger for Jupyter Notebooks: Myth or Reality? Elizaveta Shashkova 
 EuroPython 2019

About Me • Software Developer at JetBrains, PyCharm IDE • Debugger and Data Science tools • @lisa_shashkova � 2

Visual Debugger � 3

Jupyter Notebooks • Popular scientific tool • File is a sequence of cells � 4

Jupyter Notebooks Debug • Logging with print statements • Command-line debugger ipdb � 5

Jupyter Notebooks Debug � 6

Myth or Reality?

Myth or Reality?

Contents • Python files debugging • Jupyter breakpoints • Debugger communication • Jupyter visual debugger � 9

Tracing Function def tracefunc(frame, event, arg): 1 print(frame.f_lineno, event) 2 return tracefunc 3 4 5 sys.settrace(tracefunc) 6 � 10

Tracing Function 1 def greet_neighbors(): 2 planets = ["Mars", "Venus"] 3 for p in planets: 4 print(f"Hi {p}!") 5 return len(planets) 6 7 8 sys.settrace(tracefunc) 9 greet_neighbors() � 11

Tracing Function 1 call 1 def greet_neighbors(): 2 planets = ["Mars", "Venus"] 3 for p in planets: 4 print(f"Hi {p}!") 5 return len(planets) 6 7 8 sys.settrace(tracefunc) 9 greet_neighbors() � 12

Tracing Function 1 call 1 def greet_neighbors(): 2 line 2 planets = ["Mars", "Venus"] 3 for p in planets: 4 print(f"Hi {p}!") 5 return len(planets) 6 7 8 sys.settrace(tracefunc) 9 greet_neighbors() � 13

Tracing Function 1 call 1 def greet_neighbors(): 2 line 2 planets = ["Mars", "Venus"] 3 line 3 for p in planets: 4 line 4 print(f"Hi {p}!") Hi Mars! 5 return len(planets) 6 7 8 sys.settrace(tracefunc) 9 greet_neighbors() � 14

Tracing Function 1 call 1 def greet_neighbors(): 2 line 2 planets = ["Mars", "Venus"] 3 line 3 for p in planets: 4 line 4 print(f"Hi {p}!") Hi Mars! 5 return len(planets) 3 line 6 4 line 7 Hi Venus! 8 sys.settrace(tracefunc) 9 greet_neighbors() � 15

Tracing Function 1 call 1 def greet_neighbors(): 2 line 2 planets = ["Mars", "Venus"] 3 line 3 for p in planets: 4 line 4 print(f"Hi {p}!") Hi Mars! 5 return len(planets) 3 line 6 4 line 7 Hi Venus! 8 sys.settrace(tracefunc) 5 line 9 greet_neighbors() 5 return � 16

Breakpoint • frame.f_lineno - current line number • frame.f_code.co_filename - current file name � 17

Breakpoint • frame.f_lineno - current line number • frame.f_code.co_filename - current file name • Equals to breakpoint’s file and line -> suspend program! � 18

Contents • Python files debugging • Jupyter breakpoints • Debugger communication • Jupyter visual debugger � 19

Contents • Python files debugging • Jupyter breakpoints • Debugger communication • Jupyter visual debugger � 20

Cells Execution IPython kernel IDE Front-end � 21

Cells Execution IPython kernel code IDE Front-end � 22

Cells Execution IPython kernel IDE code execution Front-end � 23

Cells Execution IPython kernel IDE Front-end result � 24

Cells Execution • Kernel generates a unique name for each cell • <ipython-input-5-11faed10a894> • File name of a generated 
 IPython kernel code object code execution � 25

Jupyter Breakpoints • Python files: (filename, line number) -> unique location � 26

Jupyter Breakpoints • Python files: (filename, line number) -> unique location • Jupyter Notebooks? � 27

Jupyter Breakpoints • Python files: (filename, line number) -> unique location • Jupyter Notebooks: • generated cell name • line inside code object � 28

Source Mapping IDE IPython kernel cell MyNotebook.ipynb source code generated <code object> � 29

Source Mapping IDE IPython kernel cell MyNotebook.ipynb source code generated cell id <code object> � 30

Source Mapping IDE IPython kernel cell MyNotebook.ipynb source code ? generated cell id <code object> � 31

Source Mapping • Tracking cells execution in the IDE � 32

Source Mapping • Tracking cells execution in the IDE • Silent cell execution in IPython kernel � 33

Debug Cell Execution <cell source code> � 34

Debug Cell Execution patch name generation <cell source code> - silent mode � 35

Debug Cell Execution patch name generation cell id <cell source code> - silent mode � 36

Jupyter Tracing Function • frame.f_code.co_filename - generated name � 37

Jupyter Tracing Function • frame.f_code.co_filename - generated name • Map: generated name -> cell id � 38

Jupyter Tracing Function • frame.f_code.co_filename - generated name • Map: generated name -> cell id • Send message to the IDE � 39

Jupyter Tracing Function • frame.f_code.co_filename - generated name • Map: generated name -> cell id • Send message to the IDE � 40

Contents • Python files debugging • Jupyter breakpoints • Debugger communication • Jupyter visual debugger � 41

Contents • Python files debugging • Jupyter breakpoints • Debugger communication • Jupyter visual debugger � 42

Debug Communication IPython kernel IDE Front-end “Add breakpoint in a cell 3, line 2” � 43

Debug Communication IPython kernel IDE Front-end “Add breakpoint in a cell 3, line 2” � 44

Debug Communication • Additional connection • Reuse Jupyter channels � 45

Jupyter Messaging IPython kernel Kernel 
 Front-end proxy � 46

Jupyter Messaging IPython kernel Kernel 
 Front-end proxy � 47

Debug Communication • Additional connection • Reuse Jupyter channels � 48

Debug Communication • Additional connection • Reuse Jupyter channels � 49

Jupyter Messaging Shell IPython kernel Kernel 
 IOPub Front-end proxy stdin � 50

Jupyter Architecture • Event loop in a main thread for execution events • Event loop for output events � 51

Jupyter Architecture IPython kernel IDE code Front-end debug x Blocked � 52

Debug Communication • Additional connection • Reuse Jupyter channels � 53

Debug Communication • Additional connection • Reuse Jupyter channels But ipdb works! � 54

But ipdb Works! � 55

But ipdb Works! • Based on input() • Reuses user input channel � 56

Debug Communication • Additional connection • Reuse Jupyter channels � 57

Contents • Python files debugging • Jupyter breakpoints • Debugger communication • Jupyter visual debugger � 58

Contents • Python files debugging • Jupyter breakpoints • Debugger communication • Jupyter visual debugger � 59

Jupyter Visual Debugger • Jupyter tracing function � 60

Jupyter Visual Debugger • Jupyter tracing function • Mapping between editor and generated code � 61

Jupyter Visual Debugger • Jupyter tracing function • Mapping between editor and generated code • Debugger connection � 62

Live Demo � 63

Live Demo • PyCharm doesn’t convert Jupyter Notebooks to Python files! • On disk it’s still the same JSON file with .ipynb extension � 64

Jupyter Visual Debugger • Jupyter tracing function • Mapping between editor and generated code • Debugger connection � 65

Jupyter Visual Debugger • Implement in your favourite IDE � 66

Jupyter Visual Debugger • Implement in your favourite IDE • Try it in PyCharm Pro! � 67

Jupyter Visual Debugger • Implement in your favourite IDE • Try it in PyCharm Pro! • Questions? @ lisa_shashkova � 68