HIV HIVE: : Sca Scalable, , Cross Platf tform Graph Analyti - PowerPoint PPT Presentation

HIV HIVE: : Sca Scalable, , Cross Platf tform Graph Analyti tics cs Fr Fram amework in in Pyt Pytho hon Vincent Cavé - Intel Stanley Seibert - Anaconda FOSDEM 2020

Outline Ou § What is HIVE? § Architecture § Interfaces § Extensibility § Summary HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 2

HI HIVE: A A Bridge ge Betwe ween n Graphs hs and nd Data Sci cience nce High Perf DASK Graph Graph Analytics in Python • Libraries Data-science Inter-Operability • High Performance • Python Transparent Orchestration • HIVE Graph Users Data Science Community Driven • Packages Hardware Agnostic • Research Hardware In development, to be open • Community Vendors sourced in 2020 HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 3

On One Indirect ction on to o targe get them all Data • High-Level Graph API Science HIVE APIs • Graph Query API with Numba Ecosystem • Data Inter-Operability • Dynamic Task Graph HIVE DASK Runtime • Orchestrate compute & data • Extensible via plugins Graph Frameworks Graph Algorithm using • SuiteSparse Paradigm & API • Galois • GraphIt Graph Hardware • Gunrock Representation Architectures • … HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 4

HI HIVE F Fra rame mework I rk Interfa face ces Transformers Data Models Graph Algorithms Backends {DF@CPU=>CSR {Louvain, XBLAS, CPU, Graphs:{DF@CPU, @CPU}, … CSR}, … CSR@CPU, …} <orchestrate> HIVE / DASK User API Congratulations, you’ve just built a graph! Louvain(G) HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 5

Al All this time, it was a gra graph ph of pl plugi ugins Data Model Graph User API Algorithms Data Backends Model Graph Data Transformers Algorithms Model Backends Data Graph Model Algorithms Backends Data Model HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 6

Do Doing Gr Grap aph Analyt alytics s With The Help lp of Gr Grap aphs Workflow Task Graphs Data Transformation Graphs Load Data Preprocessing File Format #1 File Format #2 Make Graph Make Graph Table Array Graph Op #1 Graph Op #2 Graph Graph HIVE Graph Op #3 Save Visualize Format #1 Format #2 Orchestrate HW backend selection & data movement Automated data transformers selection HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 7

Ex Extens tensibil ibility ity: Supporting ting New ew Hardwa ware Transformers Data Models Graph Algorithms Backends {DF@CPU=>CSR {Louvain, XBLAS, XPU , CSR@ XPU , … @ XPU }, … CSR}, … HIVE / DASK User API § No functional changes to User API Louvain(G) § New hardware only requires a few plugins § Becomes part of the HIVE runtime toolbox § Mixing between HW architectures is automatically supported HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 8

Ex Extens tensibil ibility ity: Supporting ting a new new User er API Transformers Data Models Graph Algorithms Backends {TC, XBLAS, CPU, CSR}, … HIVE / DASK § Extend the User API User API § Provide at least one implementation TC(G) § Becomes part of the HIVE runtime toolbox HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 9

St Stakeholders View Data Scientists Graph Framework Researchers Developers • Unified API for Graph • Python frontend for • Easy integration in Analytics algorithms workflows • Python inter-operability • Increased user base • Easily extensible • State of the art backends • Performance • Performance monitoring • Transparent feedback & optimization orchestration • Increased workflow portability HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 10

HI HIVE: A A Bridge ge Betwe ween n Graphs hs and nd Data Sci cience nce High Perf DASK Graph Libraries Python Questions? HIVE Graph Users Data Science Packages Research Hardware Community Vendors HIVE: Graph Analytics Framework in Python – Vincent Cavé, Stanley Seibert – FOSDEM 2020 11