Go(lang) to Python Europython 2019 Basel, Stefan Baerisch - PowerPoint PPT Presentation

Go(lang) to Python Europython 2019 Basel, Stefan Baerisch stefan@stbaer.com � 1

Why this Talk? Good Qt Bindings Frontend Very Expressive / Productive Python Fast Enough Good For Experiments Good Library Support Good Cross Plattform Support Low “Frustration” Factor Expressive Enough Backend Good Concurrency / Parallelism ??? Good Performance � 2

Nice Things about Go Go has: Very Quick Compiler Lightweight, Managed Go-Routines No Sync/ASync Automatic Memory Management Quite Simple Language (“The Programming language for people that don’t like programming languages”) � 3

Why Go with Python? Not {C++Rust} ? Python Go Cpp / Rust Performance / Concurrency / Ok Good Great Parallelism Language Great Good Great Expressiveness Low Frustration Factor Great Good Ok Low Friction <Your Factor Here> ? Likely Good ? � 4

Some Spoilers - Things work, but are currently far from elegant. - They are, however, quite interesting � 5 Image: Professor Butts and the Self-Operating Napkin (1931), Wikipedia, Public Domain

Wrapping Go - The Basics � 6

Options for Using Go from Python Same Process? Published Api? Questions… Transport Format Transport Mechamisn Python Call External Command goPy Bindings Binary RPC (gRPC / Thrift) … Roll me REST / JSON RPC Own …. Go � 7

Wrapping Go with CGO CGO makes bindings possible… C-Calling Convention Go Calling Convention Go C Fixed-size Stacks Go Segmented Stacks CGO Threads Go Routines, scheduled by runtime CGO has some drawbacks… Slower Build Times More Complex Builds Significant Overhead When calling Go from C Restrictions in Sharing Data � 8 https://golang.org/cmd/cgo/ https://dave.cheney.net/2016/01/18/cgo-is-not-go

High Level Architecture Define Extention Types and Wrappers for CGO C Python API C/C++ Representation Cython C / C++ Support of Value-based Export C API. Convert Code parameters and results CGO Parameters and Result Manage Lifetime of Values Go Support Code Exported Go Data Go Idiomatic Go Code we want to use in Python � 9

Build Process C / C++ Support Static Libary Step 1 CGO File Go Build Header Files Go Files Static Libary Header Files Step 2 python / C Python Module setup.py Cython File 10 �

Wrapping a Simple Function � 11

Simple Function - Goals Go Code we want to use func Add(v, v2 int) int {} CGO Provide C API // export cgo_Add func cgo_Add(cgo_v1, cgo_v2 C.int) C.int {} cdef extern from "cgo_lib/cgo_lib.h" : Cython Wrap C API int cgo_Add(int p0, int p1) cpdef int add(int v1, int v2): Python Use Functionality � 12

Simple Function - Go Code Function name, exported if it starts with capital letter Parameters. name(s), then type Result, can be multiple values, similar to tuples � 13

Simple Function - CGO make available in C lib C parameter / result types Type conversation Actual work Build Library Name of C Lib / Header Build the Package � 14

Excerpt from the Header File CGO generated C Header Our Function � 15

Setup.py for Cython Our C Library Build Command � 16

Simple Function - Cython Declare Function Wrapper Function Using it in Python � 17

Wrapping User Defined Types � 18

User Defined Types in Go and Python Python Go Classes / Extention Types User Defined Types Methods Exported Methods Constructors / Factoryfunctinos Functions Destructor (__del__, __dealloc___,…) Garbage Collection Exceptions Error Result Values <ignored for now> Interfaces, Channels 19 �

Wrapping User Types - Go Code Object Pointer / Strings Reference Methods � 20

Wrapping User Types - References How to use Data References in Python? Not Possible: Direct Possible: Using a Proxy Sharing of Go Data Cython Cython OK NOT OK Proxy Data Data Keep the Go Garbage Collector away Jobs of the from the data shared with Python Proxy Translate the Go Pointer / Value into something we can share with Python � 21

Pointer ID Mapping Go Data Cython Instance Management 1 Member 1 Cython Wrapper Member 2 Method … Method Cython Wrapper 1 Go Data Member Member Method Method 22 �

Wrapping User Types - References Id <=> Pointer Mapping Functions Bookkeeping Package Level Variable � 23

Wrapping User Types - CGO “Constructor” Convert Parameters / Results “Destructor” Go Function Call � 24

Wrapping User Types - CGO Convert Parameters Convert Parameters / Results Convert Results Get Object � 25

Wrapping User Types - Headers CGO generated Headers Cython Function Declaration � 26

User Types - Cython Constructor Manage Objects Only Member - ID of the underlying Go objects Convert Parameters Request new or register existing object Register Object � 27

Cython Method Get Name as C String from Go, via CGO � 28

Python Again � 29

Wrapping Maps, Lists, etc. as Results and Parameters Also: Errors � 30

Complex Parameters/Results - Go Golang Error Return Objects Return Slice of Strings Return Map � 31

Complex Results - How? How to express slices/list or maps/dicts on the way from Go to Python (and back)? Go Cgo / C Cython / Python int, uint… (unsigned) int int string *char str user type object id extension type slice ? list map ? dict error ? exception � 32

Complex Results - Idea C++ has std::vector, std::map Cython can wrap C++ Go Cgo / C Cython Python int, uint… (unsigned) int (unsign int ed) int string *char <=> str user type object id <=> extension type slice *void to std::vector, helper functions <=> list map *void to std::map, helper functions <=> dict errors output parameter <=> exception Output parameters are ok for errors, for now � 33

Complex Results - C API � 34

Complex Results - C++ � 35

Complex Results -CGO � 36

Complex Results - Cython � 37

Errors -CGO � 38

Errors- Cython � 39

Complex Results & Errors - Python � 40

Wrapping Callbacks � 41

Wrapping Callbacks - Go Code How to have some callbacks? Callback: Int (age) to bool � 42

Wrapping Callbacks - Thougths We want Go to call Python and use the result We need to express the callback in our C/CGO layer C callpacks are function pointers… Cython can build C-functions that call Python Code So, we can have a C function pointer with Python Code but CGO cannot call function pointers CGO can call a c function that calls a function pointer � 43

Callbacks - Rube Goldberg Approach Cython Method Wrapper Python Callback passed to assigns Python callback to calls calls Cython C Callback passed to is C Function Pointer passed to CGO Version of Go Method defines and call captures in Closure calls C Function to Call Local Go Func calls Function Pointer passed to Go Method with Call Back 44 �

C Helper Functions for Callbacks � 45

Wrapping Callbacks - CGO � 46

Wrapping Callbacks - Cython � 47

Python � 48

Performance, Effort, Summary � 49

Effort and Gain Wrapping Golang Golang ~ 130 Lines CGO ~ 220 Lines C Header ~30 Lines CPP Support Code ~50 Lines Cython ~200 Lines For this (simple) example, the amount of wrapper code is significant Code is repetitive, leads itself to automation � 50

A Simple (Accessor) Benchmark � 51

Go Bindings vs. Python Performance Go Python 1493ms 619ms 1360ms 256ms 3209ms 443ms 93ms � 52

Performance of Method Calls Go Python � 53

Wrapping Go - Lessons Learned Things work Many Go Feature can be expressed in Python Cython works and is a pleasure to use CGO works Some boilerplate, but nothing to bad Improvements possible (Errors, Interfaces, Channels?) Performance is a Overhead to call into Go (runtime) challenge Mapping from Pointers to IDs Looking into alternatives (C++ Msg Queue?) � 54

The End � 55