on the efficient distributed evaluation of sparql queries
play

On the Efficient Distributed Evaluation of SPARQL Queries Damien - PowerPoint PPT Presentation

Apr 06, 2023 •135 likes •1.26k views

On the Efficient Distributed Evaluation of SPARQL Queries Damien Graux Supervisor: Nabil Laya da Co-Supervisor: Pierre Genev` es Funded by: Datalyse Project Universit e Grenoble Alpes, Inria , LIG Tyrex Team < tyrex.inria.fr >

  1. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Protocol and RDF Query Language [G + 13] Considered SPARQL Fragment Basic Graph Pattern (BGP) fragment composed of conjunctions of Triple Patterns (TPs). Triple Pattern (TP) SELECT ?s ?g WHERE { One BGP ?s type Museum ?g type Painter Composed of 3 TPs ?s shows ?g } 5 / 34

  2. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Protocol and RDF Query Language [G + 13] Considered SPARQL Fragment Basic Graph Pattern (BGP) fragment composed of conjunctions of Triple Patterns (TPs). Triple Pattern (TP) SELECT ?s ?g WHERE { One BGP ?s type Museum ?g type Painter Composed of 3 TPs ?s shows ?g } Solutions A candidate solution satisfies a TP when the replacement of the variables of the TP with their value corresponds to a triple that appears in the RDF data. A query solution is a candidate solution that satisfies all the TPs of the query. 5 / 34

  3. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Section 2 Distributed Frameworks 6 / 34

  4. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion MapReduce Strategy The paradigm Parallel processing of massive datasets [DG08] A job has two separate phases: 1 Map phase which takes k/v pairs, performs computations and returns k/v pairs 2 Reduce phase where k/v pairs from the Map are ingested to return a single set of results. Intermediate results sometimes need to be shuffled – exchanged and/or merge-sorted – across the network to be reduced. In brief, MapReduce proposes to not only consider dataset as distributed and fragmented on each machine but also to develop the computation as small blocks (the Map part) which are finally grouped together (the Reduce part). 7 / 34

  5. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Distributed Frameworks Hadoop Framework for distributed systems based on MapReduce It is twofold: a distributed file system (including replication) a MapReduce library Cluster Computing Frameworks Provide an interface with implicit data parallelism and fault-tolerance Offer a set of low-level functions e.g. map, join, collect. . . For instance: PigLatin, Flink, Spark . . . 8 / 34

  6. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Apache Spark[ZCD + 12] Spark in a nutshell Master/Worker(s) Architecture Various file system sources supported e.g. HDFS One of the most active Apache project e.g. 1000+ contributors 2004 July 2016 2010 MapReduce Paper Apache Spark 2.0 Spark Open-Source 2002 2004 2006 2008 2010 2012 2014 2016 2002 2008 May 2014 MapReduce @ Google Hadoop Summit Apache Spark 1.0 2006 Hadoop @ Yahoo! 9 / 34

  7. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Apache Spark[ZCD + 12] Spark in a nutshell Master/Worker(s) Architecture Various file system sources supported e.g. HDFS One of the most active Apache project e.g. 1000+ contributors Resilient Distributed Datasets Distributed object collections Split into partitions stored in RAM or disks Created through deterministic operations Fault-tolerant: automatically re-built 9 / 34

  8. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Section 3 SPARQL Evaluators 10 / 34

  9. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Jumble of Evaluators 4store CouchBaseRDF BitMat YARS Hexastore CliqueSquare RYA Parliament Virtuoso RDF-3X . . . 11 / 34

  10. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Jumble of Evaluators . . . Some Previous Surveys When? Who? What? 2001 Barstow [Bar01] Focuses on open-source solutions; and looks at some of their specificities 2002 Beckett [Bec02] Updates 2003 Beckett [BG03] Focuses on the use of relational database management systems to store rdf datasets 2004 Lee [Lee04] Updates 2012 Faye [FCB12] Lists the various rdf storage approaches mainly used by single-node systems 2015 Kaoudi [KM15] Presents a survey focusing only on rdf in the clouds 11 / 34

  11. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion RDF Storage Strategies rdf Storage Strategies native non-native In-memory On Disks Web APIs DBMS-based Standalone Embedded Schema-Carefree Schema-Aware Triple Table Vertical Partitioning Property Table 12 / 34

  12. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion RDF Storage Strategies rdf Storage Strategies native non-native In-memory On Disks BitMat Web APIs DBMS-based Standalone Embedded Virtuoso RDF-3X Schema-Carefree Schema-Aware Hexastore Triple Table Vertical Partitioning Property Table 3store swStore 12 / 34

  13. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Distributed Evaluation Methods Distributed rdf Storage Methods Federation Key-Value Stores Independent Distributed File System Triple-based Graph-based Horizontal Fragmentation Graph Partitioning Triple Table Vertical Partitioning Property Table 13 / 34

  14. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Distributed Evaluation Methods Distributed rdf Storage Methods Federation Key-Value Stores Independent Distributed File System 4store CouchBaseRDF Triple-based Graph-based Horizontal Fragmentation Graph Partitioning RYA Triple Table Vertical Partitioning Property Table PigSPARQL S2RDF 13 / 34

  15. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Distributed SPARQL Evaluator State-of-the-art Summary Observations 1 Multiple RDF storage strategies 2 Various methods to distribute data and to compute queries 14 / 34

  16. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Distributed SPARQL Evaluator State-of-the-art Summary Observations 1 Multiple RDF storage strategies 2 Various methods to distribute data and to compute queries How to pick an efficient evaluator? 14 / 34

  17. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Distributed SPARQL Evaluator State-of-the-art Summary Observations 1 Multiple RDF storage strategies 2 Various methods to distribute data and to compute queries How to pick an efficient evaluator? Experimental Evaluation! 14 / 34

  18. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Section 4 Multi-Criteria Experimental Ranking 15 / 34

  19. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Experimental Studies When? Who? What? Magkanaraki [MKA + 02] 2002 Reviews solutions dealing with on- tologies Stegmaier [SGD + 09] 2009 Reviews solutions according to several parameters such as their licenses, their architectures and compares them using a scalable test dataset e [CMEF + 13] 2013 Cudr´ Realizes an empirical study of dis- tributed sparql evaluators (na- tive rdf stores and several NoSQL solutions they adapted) 16 / 34

  20. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Popular Benchmarks Name SPARQL Fragment LUBM [GPH05] BGP WatDiv [AH¨ OD14] BGP SP 2 Bench [SHLP09] BGP + FILTER UNION OPTIONAL + Solu- tion Modifiers + ASK BolowgnaB [DEW + 11] BGP + aggregator ( e.g. COUNT ) BSBM [BS09] BGP + FILTER UNION OPTIONAL + So- lution Modifiers + Logical negation + CONSTRUCT DBPSB [MLAN11] Use actually posed queries against dbpedia RBench [Q¨ O15] Generate queries according to considered datasets 17 / 34

  21. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Popular Benchmarks Name SPARQL Fragment LUBM [GPH05] BGP WatDiv [AH¨ OD14] BGP SP 2 Bench [SHLP09] BGP + FILTER UNION OPTIONAL + Solu- tion Modifiers + ASK BolowgnaB [DEW + 11] BGP + aggregator ( e.g. COUNT ) BSBM [BS09] BGP + FILTER UNION OPTIONAL + So- lution Modifiers + Logical negation + CONSTRUCT DBPSB [MLAN11] Use actually posed queries against dbpedia RBench [Q¨ O15] Generate queries according to considered datasets 17 / 34

  22. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 1 – Experimental Comparative Analysis Considered Benchmarks LUBM: generated datasets and 14 queries (Q1-Q14) WatDiv: generated datasets and 20 queries Competitors Selection criteria: OpenSource, Popular or Recent Two types of evaluators: Conventional (with preprocessing): 4store, CumulusRDF, CouchBaseRDF, RYA, CliqueSquare and S2RDF Direct: PigSPARQL 18 / 34

  23. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 1 – Obtained Results We learned: 1 Considering the same dataset, loading times are spread over several magnitude orders 19 / 34

  24. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 1 – Obtained Results With the following RDF datasets: Dataset Number of Triples Original File Size WatDiv1k 109 million 15 GB Lubm1k 134 million 23 GB Lubm10k 1.38 billion 232 GB 4store CliqueSquare RYA S2RDF CouchBaseRDF CumulusRDF 10 5 Time(s) 10 4 10 3 watdiv1k lubm1k lubm10k Figure : Preprocessing Time. 19 / 34

  25. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 1 – Obtained Results We learned: 1 Considering the same dataset, loading times are spread over several magnitude orders 2 For the same query on the same dataset, elapsed times can differ very significantly 19 / 34

  26. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 1 – Obtained Results Q1 4store CliqueSquare CouchBaseRDF CumulusRDF SELECT ?X WHERE { PigSPARQL RYA S2RDF ?X rdf:type ub:GraduateStudent . ?X ub:takesCourse GraduateCourse0 } 10 4 Q2 10 3 SELECT ?X ?Y ?Z WHERE { Time(s) ?X rdf:type ub:GraduateStudent . ?Y rdf:type ub:University . ?Z rdf:type ub:Department . 10 2 ?X ub:memberOf ?Z . ?Z ub:subOrganizationOf ?Y . ?X ub:undergraduateDegreeFrom ?Y 10 1 } 10 0 Q3 Q1 Q2 Q3 SELECT ?X WHERE { ?X rdf:type ub:Publication . Figure : Query Response Time with ?X ub:publicationAuthor AssistantProfessor0 Lubm1k (134 million triples). } 19 / 34

  27. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 1 – Obtained Results We learned: 1 Considering the same dataset, loading times are spread over several magnitude orders 2 For the same query on the same dataset, elapsed times can differ very significantly 3 Even with large datasets, most queries are not harmful per se , i.e. queries that incurr long running times with some implementations still remain in the “comfort zone” for other implementations 19 / 34

  28. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 1 – Obtained Results 102 . 5 101 102 100 C1C2C3F1F2F3F4F5L1L2L3L4L5S1S2S3S4S5S6S7 C1C2C3F1F2F3F4F5L1L2L3L4L5S1S2S3S4S5S6S7 (a) 4store (b) S2RDF 104 103 103 102 101 102 C1 F1F2F3F4F5L1L2L3L4L5S1S2S3S4S5S6S7 C1C2C3F1F2F3F4F5L1L2L3L4L5S1S2S3S4S5S6S7 (c) RYA (d) PigSPARQL Figure : Obtained results with WatDiv1k. 19 / 34

  29. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 1 – Obtained Results We learned: 1 Considering the same dataset, loading times are spread over several magnitude orders 2 For the same query on the same dataset, elapsed times can differ very significantly 3 Even with large datasets, most queries are not harmful per se , i.e. queries that incurr long running times with some implementations still remain in the “comfort zone” for other implementations Ok, but. . . . . . how to rank evaluators? � 19 / 34

  30. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion An extended set of metrics Usual metrics: Time always Disk Footprint only sometimes 20 / 34

  31. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion An extended set of metrics Usual metrics: Time always Disk Footprint only sometimes Our additions: Disk Activity new 20 / 34

  32. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion An extended set of metrics Usual metrics: Time always Disk Footprint only sometimes Our additions: Disk Activity new Network Traffic new 20 / 34

  33. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion An extended set of metrics Usual metrics: Time always Disk Footprint only sometimes Our additions: Disk Activity new Network Traffic new Resources: CPU, RAM, SWAP new 20 / 34

  34. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Multi-Criteria Reading Grid Criteria List Velocity : the fastest possible answers Query Time 21 / 34

  35. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Multi-Criteria Reading Grid Criteria List Velocity : the fastest possible answers Query Time Resiliency : trying to avoid as much as possible to recompute everything when a machine fails Footprint 21 / 34

  36. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Multi-Criteria Reading Grid Criteria List Velocity : the fastest possible answers Query Time Resiliency : trying to avoid as much as possible to recompute everything when a machine fails Footprint Immediacy : evaluating some sparql queries only once Preprocessing Time 21 / 34

  37. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Multi-Criteria Reading Grid Criteria List Velocity : the fastest possible answers Query Time Resiliency : trying to avoid as much as possible to recompute everything when a machine fails Footprint Immediacy : evaluating some sparql queries only once Preprocessing Time Dynamicity : dealing with dynamic data Preprocessing Time & Disk Activity 21 / 34

  38. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Multi-Criteria Reading Grid Criteria List Velocity : the fastest possible answers Query Time Resiliency : trying to avoid as much as possible to recompute everything when a machine fails Footprint Immediacy : evaluating some sparql queries only once Preprocessing Time Dynamicity : dealing with dynamic data Preprocessing Time & Disk Activity Parsimony : minimizing some of the resources CPU, RAM, . . . 21 / 34

  39. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Ranking Velocity Immediacy WatDiv1k Velocity Parsimony Lubm1k Dynamicity Resiliency 22 / 34

  40. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Ranking Velocity Immediacy WatDiv1k Velocity Parsimony Lubm1k 4store Dynamicity Resiliency 22 / 34

  41. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Ranking Velocity Immediacy WatDiv1k Velocity Parsimony Lubm1k 4store PigSPARQL Dynamicity Resiliency 22 / 34

  42. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 2 – Ranking Velocity Immediacy WatDiv1k Velocity Parsimony Lubm1k 4store PigSPARQL S2RDF RYA CumulusRDF CouchBaseRDF CliqueSquare Dynamicity Resiliency 22 / 34

  43. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Section 5 Efficient Distributed SPARQL Evaluation 23 / 34

  44. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 3 – Efficient Distributed SPARQL evaluation We designed: SPARQLGX SDE RDFHive Available from: < https://github.com/tyrex-team > 24 / 34

  45. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 3 – Efficient Distributed SPARQL evaluation These evaluators in nutshells: SPARQLGX a distributed SPARQL evaluator with Apache Spark SDE a direct SPARQL evaluator with Apache Spark RDFHive a direct evaluation of SPARQL with Apache Hive Available from: < https://github.com/tyrex-team > 24 / 34

  46. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Contrib. 3 – Efficient Distributed SPARQL evaluation Considering the reading grid, we have: SPARQLGX velocity, resiliency SDE immediacy, dynamicity, resiliency RDFHive immediacy, dynamicity, resiliency, parsimony Available from: < https://github.com/tyrex-team > 24 / 34

  47. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Details of SPARQLGX 1 Selected storage model 2 SPARQL translation process 3 Optimization strategies 25 / 34

  48. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Vertical Partitioning [Abadi et al. 2007] SPARQLGX Storage Model RDF predicates carry the semantic information, thereby: Limited number of distinct predicates e.g. few hundreds [Gallego et al. 2011] Predicates rarely variable in queries [Gallego et al. 2011] 26 / 34

  49. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Vertical Partitioning [Abadi et al. 2007] SPARQLGX Storage Model RDF predicates carry the semantic information, thereby: Limited number of distinct predicates e.g. few hundreds [Gallego et al. 2011] Predicates rarely variable in queries [Gallego et al. 2011] Vertical Partitioning Splitting by predicate and saving two-column files 26 / 34

  50. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Vertical Partitioning [Abadi et al. 2007] SPARQLGX Storage Model RDF predicates carry the semantic information, thereby: Limited number of distinct predicates e.g. few hundreds [Gallego et al. 2011] Predicates rarely variable in queries [Gallego et al. 2011] Vertical Partitioning Splitting by predicate and saving two-column files Advantages Natural compression and indexing Straightforward implementation 26 / 34

  51. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Vertical Partitioning [Abadi et al. 2007] SPARQLGX Storage Model dataset type.txt Dutch School type Museum Dutch School Museum Dutch School creationDate 2016 Louvre Museum Dutch School use Louvre Rembrandt Painter creationDate.txt Louvre type Museum Dutch School 2016 Hals Painter Rembrandt type Painter Vermeer Painter Hals type Painter Van Dyck Painter use.txt Vermeer type Painter Dutch School Louvre Van Dyck type Painter shows.txt Collection shows Rembrandt Collection Rembrandt mainTopic.txt Dutch School mainTopic Rembrandt Dutch School Rembrandt Dutch School Rembrandt Dutch School shows Rembrandt Dutch School Hals Dutch School shows Hals Dutch School Vermeer Dutch School shows Vermeer Dutch School Van Dyck Dutch School shows Van Dyck 26 / 34

  52. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Translation Process SPARQL → Scala Dealing with one TP . . . textFile to access relevant files filter to keep matching triples 27 / 34

  53. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Translation Process SPARQL → Scala Dealing with one TP . . . textFile to access relevant files filter to keep matching triples ?s type Museum . textFile(“type.txt”) .filter { case(s,o)= > o.equals(“Museum”) } .map { case(s,o)= > s } 27 / 34

  54. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Translation Process SPARQL → Scala Dealing with one TP . . . textFile to access relevant files filter to keep matching triples ?s type Museum . textFile(“type.txt”) .filter { case(s,o)= > o.equals(“Museum”) } .map { case(s,o)= > s } . . . with a conjunction of TPs Translate each TP Join them one by one 27 / 34

  55. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Translation Process SPARQL → Scala ?s type Museum . ?g type Painter . ?s shows ?g 27 / 34

  56. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Translation Process SPARQL → Scala ?s type Museum . tp1=sc.textFile(‘‘type.txt’’) ?g type Painter . .filter { case(s,o)= > o.equals(‘‘Museum’’) } ?s shows ?g .map { case(s,o)= > s } .keyBy { case(s)=>s } 27 / 34

  57. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Translation Process SPARQL → Scala ?s type Museum . tp1=sc.textFile(‘‘type.txt’’) ?g type Painter . .filter { case(s,o)= > o.equals(‘‘Museum’’) } ?s shows ?g .map { case(s,o)= > s } .keyBy { case(s)=>s } tp2=sc.textFile(‘‘type.txt’’) .filter { case(g,o)= > o.equals(‘‘Painter’’) } .map { (g,o)= > g } .keyBy { case(g)=>g } 27 / 34

  58. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Translation Process SPARQL → Scala ?s type Museum . tp1=sc.textFile(‘‘type.txt’’) ?g type Painter . .filter { case(s,o)= > o.equals(‘‘Museum’’) } ?s shows ?g .map { case(s,o)= > s } .keyBy { case(s)=>s } tp2=sc.textFile(‘‘type.txt’’) .filter { case(g,o)= > o.equals(‘‘Painter’’) } .map { (g,o)= > g } .keyBy { case(g)=>g } tp3=sc.textFile(‘‘shows.txt’’) .keyBy { case(s,g)=>(s,g) } 27 / 34

  59. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion SPARQL Translation Process SPARQL → Scala ?s type Museum . tp1=sc.textFile(‘‘type.txt’’) ?g type Painter . .filter { case(s,o)= > o.equals(‘‘Museum’’) } ?s shows ?g .map { case(s,o)= > s } .keyBy { case(s)=>s } tp2=sc.textFile(‘‘type.txt’’) .filter { case(g,o)= > o.equals(‘‘Painter’’) } .map { (g,o)= > g } .keyBy { case(g)=>g } tp3=sc.textFile(‘‘shows.txt’’) .keyBy { case(s,g)=>(s,g) } bgp=tp1.cartesian(tp2).values .keyBy { case(s,g)=>(s,g) } .join(tp3).value 27 / 34

  60. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Join Order SPARQL → Scala To minimize size of intermediate results, we try: 1 Avoiding cartesian product 2 Exploiting statistics on data 28 / 34

  61. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Join Order SPARQL → Scala To minimize size of intermediate results, we try: 1 Avoiding cartesian product 2 Exploiting statistics on data Selectivity Selectivity of an element located at pos is: either its occurrence number at pos if it is a constant or the total number of triples if it is a variable. Selectivity of a TP is the min of its element selectivities. We just sort the TPs of a BGP in ascending order of their selectivities. 28 / 34

  62. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Join Order SPARQL → Scala Initial BGP: ?s type Museum . ?g type Painter . ?s shows ?g 28 / 34

  63. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Join Order SPARQL → Scala Initial BGP: ?s type Museum . ?g type Painter . ?s shows ?g New BGP: ?s shows ?g ?s type Museum . ?g type Painter 28 / 34

  64. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Join Order SPARQL → Scala Initial BGP: Associated Scala code: ?s type Museum . tp1=sc.textFile(‘‘shows.txt’’) ?g type Painter . .keyBy { case(s,g)=>s } ?s shows ?g tp2=sc.textFile(‘‘type.txt’’) .filter { case(s,o)= > o.equals(‘‘Museum’’) } New BGP: .map { case(s,o)= > s } ?s shows ?g .keyBy { case(s)=>s } ?s type Museum . tp3=sc.textFile(‘‘type.txt’’) ?g type Painter .filter { case(s,o)= > o.equals(‘‘Painter’’) } .map { case(g,o)= > g } .keyBy { case(g)=>g } bgp=tp1.join(tp2).values .keyBy { case(s,g)=>(g) } .join(tp3).value 28 / 34

  65. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Direct SPARQL Evaluation 29 / 34

  66. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Direct SPARQL Evaluation SDE (SPARQLGX as Direct Evaluator) Directly considering the initial rdf dataset Designed to evaluate on single query 29 / 34

  67. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Direct SPARQL Evaluation SDE (SPARQLGX as Direct Evaluator) Directly considering the initial rdf dataset Designed to evaluate on single query RDFHive Based on Apache Hive (relational solution on the HDFS) Translation of queries into Hive-QL Offers the possibility of merging relational and rdf datasets 29 / 34

  68. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Direct SPARQL Evaluation 4store CliqueSquare CouchBaseRDF PigSPARQL RDFHive RYA S2RDF SDE SPARQLGX 106 105 104 103 102 1 10 20 30 40 50 60 70 80 90 100 Figure : Tradeoff between preprocessing and query evaluation times (seconds) linear WatDiv. 29 / 34

  69. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Section 6 Conclusion & Perspectives 30 / 34

  70. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Conclusion Summary of Contributions 1 Update comparative Cudr´ e et al. survey Submitted 31 / 34

  71. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Conclusion Summary of Contributions 1 Update comparative Cudr´ e et al. survey Submitted 2 Provide a new reading grid (new set of metrics) Submitted 31 / 34

  72. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Conclusion Summary of Contributions 1 Update comparative Cudr´ e et al. survey Submitted 2 Provide a new reading grid (new set of metrics) Submitted 3 Develop several distributed SPARQL evaluators: Reusability Openly available under the CeCILL license from: < https://github.com/tyrex-team > 31 / 34

  73. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Conclusion Summary of Contributions 1 Update comparative Cudr´ e et al. survey Submitted 2 Provide a new reading grid (new set of metrics) Submitted 3 Develop several distributed SPARQL evaluators: SPARQLGX ISWC 2016 SDE ISWC 2016 RDFHive Reusability Openly available under the CeCILL license from: < https://github.com/tyrex-team > 31 / 34

  74. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Conclusion Velocity Immediacy WatDiv1k Velocity Parsimony Lubm1k S2RDF RYA PigSPARQL CumulusRDF CouchBaseRDF CliqueSquare 4store Dynamicity Resiliency 31 / 34

  75. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion Conclusion Velocity Immediacy WatDiv1k SPARQLGX SDE RDFHive Velocity Parsimony Lubm1k S2RDF RYA PigSPARQL CumulusRDF CouchBaseRDF CliqueSquare 4store Dynamicity Resiliency 31 / 34

  76. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion I – Perspectives: SPARQL Benchmarking Uniform test-suite for dynamicity Short-Term Designing a benchmark for the SPARQL UPDATE fragment Staying up to date Continuous Adding new evaluators Considering other test suites Benchmarking on other clusters Varying the number of nodes Mid-Term Validating our results on larger clusters New kind of limitation? 32 / 34

  77. RDF & SPARQL Distributed Frameworks SPARQL Evaluators Experiments Distributed Evaluation Conclusion II – Perspectives: SPARQL Evaluators Improving our evaluators On going Extending the supported SPARQL fragment Improving the storage models Designing criteria-specific evaluators Mid-Term Implementing a parsimonious and resilient evaluator Developing evaluators in highly dynamic context Storage-adaptative distributed evaluators Long-Term c et al. [A¨ Adapting the idea of Alu¸ OD14] in a distributed context Considering SPARQL query shapes = ⇒ Choosing its storage model dynamically! 33 / 34

Recommend

The Tractability Frontier of Well-designed SPARQL Queries Miguel Romero (University of Oxford)

The Tractability Frontier of Well-designed SPARQL Queries Miguel Romero (University of Oxford)

The Tractability Frontier of Well-designed SPARQL Queries Miguel Romero (University of Oxford) ACM PODS 2018, 12 June, Houston-USA Well-designed SPARQL SPARQL : standard query language for RDF graphs Well-designed SPARQL (Perez, Arenas, Gutierrez

296 views • 29 slides
0 to 60 on SPARQL queries in 50 minutes Ethan Gruber American Numismatic Society

0 to 60 on SPARQL queries in 50 minutes Ethan Gruber American Numismatic Society

0 to 60 on SPARQL queries in 50 minutes Ethan Gruber American Numismatic Society gruber@numismatics.org @ewg118 Some URLs Blog post with list of SPARQL queries in Gist (including slideshow link) http://bit.ly/1PGneCf Online Coins of the

367 views • 35 slides
Towards Efficient Distributed Towards Efficient Distributed Simulation in Modelica using

Towards Efficient Distributed Towards Efficient Distributed Simulation in Modelica using

Towards Efficient Distributed Towards Efficient Distributed Simulation in Modelica using Transmission Line Modeling Martin Sjlund, Peter Fritzson Dept. of Computer and Information Science Robert Braun, Petter Krus Dept. of Management and

716 views • 16 slides
Property-based Testing of SPARQL queries Jess M. Almendros-Jimnez Antonio Becerra-Tern

Property-based Testing of SPARQL queries Jess M. Almendros-Jimnez Antonio Becerra-Tern

Property-based Testing of SPARQL queries Jess M. Almendros-Jimnez Antonio Becerra-Tern University of Almera, SPAIN 16th International Symposium on Database Programming Languages (DBPL 2017), Munich, Germany Type Checking and Testing of

364 views • 21 slides
Lecture 3: SPARQL (1.1) Aidan Hogan aidhog@gmail.com PREVIOUSLY First SPARQL (1.0) Then

Lecture 3: SPARQL (1.1) Aidan Hogan aidhog@gmail.com PREVIOUSLY First SPARQL (1.0) Then

Lecture 3: SPARQL (1.1) Aidan Hogan aidhog@gmail.com PREVIOUSLY First SPARQL (1.0) Then SPARQL 1.1 Covered SPARQL 1.0 http://www.w3.org/TR/rdf-sparql-query/ SPARQL: WHERE clause example Query: What solutions would this query return?

1.17k views • 91 slides
Differential Forms for Target Tracking and Aggregate Queries in Distributed Networks Distributed

Differential Forms for Target Tracking and Aggregate Queries in Distributed Networks Distributed

Differential Forms for Target Tracking and Aggregate Queries in Distributed Networks Distributed Networks Rik Sarkar Jie Gao Stony Brook University 1 Target Tracking with Sensor Networks 2 Target Queries Range queries : # targets within

894 views • 40 slides
Efficient Detection of Empty-Result Queries Gang Luo IBM Watson Research Centre Damon Sotoudeh

Efficient Detection of Empty-Result Queries Gang Luo IBM Watson Research Centre Damon Sotoudeh

Efficient Detection of Empty-Result Queries Gang Luo IBM Watson Research Centre Damon Sotoudeh Agenda Introduction The detection method Related work Future work Conclusion Empty-Result Queries Queries that return

802 views • 27 slides
SPARQL 1.1 Peter Fischer DMQL SPARQL 1.0 limitations Limited graphs operations: How to

SPARQL 1.1 Peter Fischer DMQL SPARQL 1.0 limitations Limited graphs operations: How to

SPARQL 1.1 Peter Fischer DMQL SPARQL 1.0 limitations Limited graphs operations: How to compute connectedness? No updates No aggregates No explicit negation No subqueries Property Paths Motivation RDF is a

478 views • 33 slides
Knowledge Representation VII - IKT507 This sentence is false! SPARQL stands for SPARQL Protocol

Knowledge Representation VII - IKT507 This sentence is false! SPARQL stands for SPARQL Protocol

Knowledge Representation VII - IKT507 This sentence is false! SPARQL stands for SPARQL Protocol And RDF Query Language Jan Pettersen Nytun, UiA 1 S O The Semantic Web Stack P Hierarchy of languages, where each layer exploits and uses

801 views • 19 slides
C-SPARQL: A Continuous Extension of SPARQL Marco Balduini marco.balduini@polimi.it Share,

C-SPARQL: A Continuous Extension of SPARQL Marco Balduini marco.balduini@polimi.it Share,

Stream Reasoning For Linked Data M. Balduini, J-P Calbimonte, O. Corcho, D. Dell'Aglio, E. Della Valle, and J.Z. Pan http://streamreasoning.org/sr4ld2013 C-SPARQL: A Continuous Extension of SPARQL Marco Balduini marco.balduini@polimi.it

528 views • 38 slides
Pratyaastha: An Efficient Elastic Distributed SDN Control Plane Anand Krishnamurthy, Shoban P.

Pratyaastha: An Efficient Elastic Distributed SDN Control Plane Anand Krishnamurthy, Shoban P.

Pratyaastha: An Efficient Elastic Distributed SDN Control Plane Anand Krishnamurthy, Shoban P. Chandrabose and Aaron Gember-Jackobson 1 Motivation Architecture Evaluation Summary SDN Control Plane Operator goals: 1. Better Performance

364 views • 9 slides
C-SPARQL: A Continuous Extension of SPARQL Marco Balduini marco.balduini@polimi.it Share,

C-SPARQL: A Continuous Extension of SPARQL Marco Balduini marco.balduini@polimi.it Share,

Stream Reasoning for Linked Data M. Balduini, J-P Calbimonte, O. Corcho, D. Dell'Aglio, E. Della Valle http://streamreasoning.org/events/sr4ld2014 C-SPARQL: A Continuous Extension of SPARQL Marco Balduini marco.balduini@polimi.it Share,

707 views • 37 slides
RBF: A New Storage Structure for Space- Efficient Queries for Multidimensional Metadata in OSS Yu

RBF: A New Storage Structure for Space- Efficient Queries for Multidimensional Metadata in OSS Yu

RBF: A New Storage Structure for Space- Efficient Queries for Multidimensional Metadata in OSS Yu Hua 1 , Dan Feng 1 , Hong Jiang 2 , Lei Tian 1 1 School of Computer, Huazhong University of Science and Technology, China. 2 Department of Computer,

438 views • 11 slides
Efficient Algorithm for Answering Fact-based Queries Using Relational Data Enriched by

Efficient Algorithm for Answering Fact-based Queries Using Relational Data Enriched by

Efficient Algorithm for Answering Fact-based Queries Using Relational Data Enriched by Context-Based Embeddings Ph.D. Dissertation Abdulaziz (Aziz) Altowayan Computer Science Pace University 12/12/2019 Outlines Motivation Problem and

549 views • 52 slides
Introduction to SPARQL Acknowledgements This presentation is based on the W3C Candidate

Introduction to SPARQL Acknowledgements This presentation is based on the W3C Candidate

Introduction to SPARQL Acknowledgements This presentation is based on the W3C Candidate Recommendation SPARQL Query Language for RDF from http://www.w3.org/TR/rdf-sparql-query/ Some of the material in this presentation is

1.2k views • 76 slides
Queries in PSM The following rules apply to the use of queries: CS 235: 1. Queries

Queries in PSM The following rules apply to the use of queries: CS 235: 1. Queries

Queries in PSM The following rules apply to the use of queries: CS 235: 1. Queries returning a single value can be Introduction to Databases used in assignments 2. Queries returning a single tuple can be used Svetlozar Nestorov with

530 views • 4 slides
Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai

Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai

Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai Sheng Tai, Vatsal Sharan, Peter Bailis Stanford University 1 Motivation: Monitoring production data streams Billions of events / day of mobile app

245 views • 23 slides
-Tree: A Gas-Efficient Structure for Authenticated Range Queries in Blockchain

-Tree: A Gas-Efficient Structure for Authenticated Range Queries in Blockchain

-Tree: A Gas-Efficient Structure for Authenticated Range Queries in Blockchain Ce Zhang, Cheng Xu, Jianliang Xu, Yuzhe Tang, Byron Choi Hong Kong Baptist University, Hong Kong Syracuse University, NY, USA Introduction Source:

693 views • 34 slides
Strong Cryptography from Weak Secrets Building Efficient PKE and IBE from Distributed Passwords

Strong Cryptography from Weak Secrets Building Efficient PKE and IBE from Distributed Passwords

Distributed Cryptography Distributed Password Public-Key Cryptography The Decision-Linear Case Strong Cryptography from Weak Secrets Building Efficient PKE and IBE from Distributed Passwords Xavier Boyen 1 Cline Chevalier 2 Georg Fuchsbauer 3

680 views • 36 slides
Processing Heterogeneous RDF Event Streams with Standing SPARQL Update Mikko Rinne, Haris

Processing Heterogeneous RDF Event Streams with Standing SPARQL Update Mikko Rinne, Haris

Processing Heterogeneous RDF Event Streams with Standing SPARQL Update Mikko Rinne, Haris Abdullah, Seppo Trm, Esko Nuutila http://cse.aalto.fi/instans/ 11.9.2012 Department of Distributed Computer Science and Systems Engineering

462 views • 17 slides
RUBIK: Efficient Threshold Queries on Massive Time Series Eleni Tzirita Zacharatou Thomas

RUBIK: Efficient Threshold Queries on Massive Time Series Eleni Tzirita Zacharatou Thomas

RUBIK: Efficient Threshold Queries on Massive Time Series Eleni Tzirita Zacharatou Thomas Heinis* Farhan Tauheed Anastasia Ailamaki cole Polytechnique *Imperial College London Oracle Labs, Zurich Fdrale de Lausanne

343 views • 15 slides
R-Trees and applications Efficient grid queries in the Dynamo reservoir modelling software

R-Trees and applications Efficient grid queries in the Dynamo reservoir modelling software

R-Trees and applications Efficient grid queries in the Dynamo reservoir modelling software R-Tree Project A.N. Yzelman (UU) Supervisors A.N. Swart (Alten) G.L.G. Sleijpen (UU) Project Owners Eric Haesen (Alten) Hans Molenaar (Shell)

514 views • 28 slides
Optimization of in collaboration with: Parke Godfrey and Jarek Gryz Regular Path Queries in

Optimization of in collaboration with: Parke Godfrey and Jarek Gryz Regular Path Queries in

Optimization of in collaboration with: Parke Godfrey and Jarek Gryz Regular Path Queries in Large Graphs Nikolay Yakovets Optimization of RPQs Scalable &amp; e ffi cient evaluation of regular path queries Evaluation Implementation RPQs

876 views • 35 slides
Query Evaluation Doing what we're told to do. Hoyt Koepke 10/02/06 1 Purpose of the Paper

Query Evaluation Doing what we're told to do. Hoyt Koepke 10/02/06 1 Purpose of the Paper

Query Evaluation Doing what we're told to do. Hoyt Koepke 10/02/06 1 Purpose of the Paper Survey efficient algorithms and software architectures for database query execution. How database systems handle complex queries. Complex means

732 views • 22 slides

More recommend