QueryArrow: Semantically Unified Query and Update of Heterogeneous - PowerPoint PPT Presentation

QueryArrow: Semantically Unified Query and Update of Heterogeneous Data Stores Hao Xu xuhao@renci.org Ben Keller Antoine de Torcy Jason Coposky University of North Carolina at Chapel Hill June 14, 2017 Xu (UNC CH) QueryArrow June 14, 2017 1 / 15

Acknowledgement This research is paritally supported by The National Science Foundation under Grant Number OCI 0940841 ”DataNet Federation Consortium” (Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation) and iRODS Consortium Xu (UNC CH) QueryArrow June 14, 2017 2 / 15

Resources QueryArrow: https://github.com/xu-hao/QueryArrow fs branch QueryArrow Database Plugin: http://github.com/xu-hao/temporary-irods-mod qa3 branch Xu (UNC CH) QueryArrow June 14, 2017 3 / 15

Motivting Applications Big Metadata Challenge: Aggregation: integrating metadata from multiple data stores Policies: for example, metadata access control Unify query and update Semantically specification Heterogeneity: NoSQL, Indexing Improve Testing, Modularity, Robustness Xu (UNC CH) QueryArrow June 14, 2017 4 / 15

QueryArrow Elements QueryArrow is a semantically unified query and update system for heterogeneous metadata store QueryArrow Service (QAS): Register Databases and Execution of QAL QueryArrow Language (QAL): Configuration, QL/DML QueryArrow Plugins (QAP): Mappings between QAL and Databases iRODS QueryArrow Database Plugin Xu (UNC CH) QueryArrow June 14, 2017 5 / 15

QueryArrow Architecture irodsAgent iRODS Core GenQuery Query Update API QueryArrow Database Plugin QAL QueryArrow QAS Postgres QAP Neo4j QAP ElasticSearch QAP SQL Cypher EQL Postgres Neo4j ElasticSearch Xu (UNC CH) QueryArrow June 14, 2017 6 / 15

QAPs Name Description Sum QAP aggregation Translation QAP policy support Cache QAP caching Mutable Map QAP in-memory mutable map Immutable Map QAP in-memory immutable map ElasticSearch QAP interfacing with ElasticSearch Neo4j QAP interfacing with Neo4j PostgreSQL QAP interfacing with Postgres SQLite3 QAP interfacing with SQLite3 CockroachDB QAP interfacing with CockroachDB FileSystem QAP file system Xu (UNC CH) QueryArrow June 14, 2017 7 / 15

QAP Composition Higher-order plugin Cache QAP Translation QAP Sum QAP QAP 1 QAP 2 QAP 3 Data store 1 Data store 2 Data store 3 Xu (UNC CH) QueryArrow June 14, 2017 8 / 15

QAL Syntax t ∶∶= p ∣ v terms ∶∶= P ( t 1 ,..., t n ) a atom c ∶∶= a ∣ insert a ∣ delete a ∣ g c ∣ b ∣ 1 ∣ 0 ∣ c ⊕ c ∣ c ⊗ c command ∶∶= input ? output ? key ? ty pt parameter type R ∶∶= rewrite a c ∣ rewrite insert a c ∣ rewrite delete a c ∣ predicate P ( pt 1 ,..., pt n ) ∣ import ... ∣ export ... configuration Xu (UNC CH) QueryArrow June 14, 2017 9 / 15

QAL Semantics ”store-and-heap” Monad https://github.com/xu-hao/CertifiedQueryArrow Xu (UNC CH) QueryArrow June 14, 2017 10 / 15

QAL Translation A B C A B C P Q . . . . . . . . . . . . . . . . . . P.C :Q :P ”type” : ”P” ”type” : ”Q” ”A” : . . . ”A” : . . . ”B” : . . . ”B” : . . . ”C” : . . . ”C” : . . . Xu (UNC CH) QueryArrow June 14, 2017 11 / 15

QueryArrow Service Local Unix domain socket TCP HTTP Xu (UNC CH) QueryArrow June 14, 2017 12 / 15

QueryArrow Database Plugin Plugs into iRODS, requires no changes to iRODS core except for bug fixes Talks to QueryArrow Service Implements 84 db plugin functions Backward compatibility with GenQuery Tested on 4.2.x: 4.2.0: PostgreSQL QAP passed python tests suite except 2 tests using ssl plugin on Ubuntu 16.04/Centos 7 4.2.1: PostgreSQL QAP passed python tests suite except 2 tests using ssl plugin on Ubuntu 16.04/Centos 7 is running Xu (UNC CH) QueryArrow June 14, 2017 13 / 15

Install Script Run iRODS install script through QueryArrow Xu (UNC CH) QueryArrow June 14, 2017 14 / 15

Demo Only Postgresql Combined Postgresql and Neo4j Xu (UNC CH) QueryArrow June 14, 2017 15 / 15