data systems that are easy to design tune and use
play

Data Systems that are Easy to Design, Tune and Use Stratos Idreos - PowerPoint PPT Presentation

Dec 28, 2022 •370 likes •1.56k views

Data Systems that are Easy to Design, Tune and Use Stratos Idreos applications api/sql algorithms/operators cpu data data data memory hierarchy data system kernel Stratos Idreos design space it all starts with how we store data every

  1. column A 4 4 13 Q1: 9 2 16 select R.A 2 1 4 piece1: A<=7 from R piece1: 7 3 9 where R.A>10 A<=10 1 6 2 and R.A<14 3 7 12 8 9 7 piece2: 7<A<=10 6 8 1 Q2: 13 13 19 piece2: select R.A piece3: 10<A<14 12 12 3 10<A<14 from R 11 11 14 where R.A>7 16 14 11 and R.A<=16 piece3: 19 16 8 A>=14 14 19 6 dynamically/on-the-fly within the select-operator Database Cracking CIDR 2007 Stratos Idreos

  2. column A 4 4 13 Q1: 9 2 16 select R.A 2 1 4 piece1: A<=7 from R piece1: 7 3 9 where R.A>10 A<=10 1 6 2 and R.A<14 3 7 12 8 9 7 piece2: 7<A<=10 6 8 1 Q2: 13 13 19 piece2: select R.A piece3: 10<A<14 12 12 3 10<A<14 from R 11 11 14 where R.A>7 16 14 11 piece4: 14<=A<=16 and R.A<=16 piece3: 19 16 8 piece5: A>16 A>=14 14 19 6 dynamically/on-the-fly within the select-operator Database Cracking CIDR 2007 Stratos Idreos

  3. column A 4 4 13 Q1: 9 2 16 select R.A 2 1 4 piece1: A<=7 from R piece1: 7 3 9 where R.A>10 A<=10 1 6 2 and R.A<14 3 7 12 8 9 7 piece2: 7<A<=10 6 8 1 Q2: 13 13 19 result piece2: select R.A piece3: 10<A<14 12 12 3 10<A<14 from R 11 11 14 where R.A>7 16 14 11 piece4: 14<=A<=16 and R.A<=16 piece3: 19 16 8 piece5: A>16 A>=14 14 19 6 dynamically/on-the-fly within the select-operator Database Cracking CIDR 2007 Stratos Idreos

  4. the more we crack, the more we learn column A 4 4 13 Q1: 9 2 16 select R.A 2 1 4 piece1: A<=7 from R piece1: 7 3 9 where R.A>10 A<=10 1 6 2 and R.A<14 3 7 12 8 9 7 piece2: 7<A<=10 6 8 1 Q2: 13 13 19 result piece2: select R.A piece3: 10<A<14 12 12 3 10<A<14 from R 11 11 14 where R.A>7 16 14 11 piece4: 14<=A<=16 and R.A<=16 piece3: 19 16 8 piece5: A>16 A>=14 14 19 6 dynamically/on-the-fly within the select-operator Database Cracking CIDR 2007 Stratos Idreos

  5. Database Cracking CIDR 2007 Stratos Idreos

  6. select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  7. select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  8. select [15,55] 40 50 30 60 10 20 Database Cracking CIDR 2007 Stratos Idreos

  9. select [15,55] 40 50 30 60 10 20 select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  10. select [15,55] 40 50 30 60 10 20 select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  11. touch at most two pieces at a time pieces become smaller and smaller select [15,55] 40 50 30 60 10 20 select [15,55] Database Cracking CIDR 2007 Stratos Idreos

  12. continuous adaptation set-up Scan 100K random selections 1000 random selectivity random value ranges 100 in a 10 million integer column Response time (secs) 10 Crack 1 0.1 Full Index 0.01 0.001 1 10 100 1000 10000 100000 Query sequence (x1000) Database Cracking CIDR 2007 Stratos Idreos

  13. continuous adaptation set-up Scan 100K random selections 1000 random selectivity random value ranges 100 in a 10 million integer column Response time (secs) 10 Crack 1 almost no 0.1 initialization overhead Full Index 0.01 0.001 1 10 100 1000 10000 100000 Query sequence (x1000) Database Cracking CIDR 2007 Stratos Idreos

  14. continuous adaptation set-up Scan 100K random selections 1000 random selectivity random value ranges 100 in a 10 million integer column Response time (secs) 10 Crack 1 almost no 0.1 initialization overhead Full Index 0.01 continuous improvement 0.001 1 10 100 1000 10000 100000 Query sequence (x1000) Database Cracking CIDR 2007 Stratos Idreos

  15. continuous adaptation set-up Scan 100K random selections 1000 random selectivity random value ranges 100 in a 10 million integer column Response time (secs) 10 Crack 1 almost no 0.1 initialization overhead Full Index 0.01 continuous improvement 0.001 1 10 100 1000 10000 100000 Query sequence (x1000) Database Cracking CIDR 2007 Stratos Idreos

  16. continuous adaptation set-up 200 Full Index 100 10K random selections Cumulative average response time (secs) selectivity 10% random value ranges 10 in a 30 million integer column Scan 1 0.1 0.01 Crack 0.004 0.001 1 10 100 1000 10000 Query sequence Database Cracking CIDR 2007 Stratos Idreos

  17. continuous adaptation set-up 200 Full Index 100 10K random selections Cumulative average response time (secs) selectivity 10% random value ranges 10 in a 30 million integer column Scan 1 0.1 0.01 Crack 0.004 0.001 1 10 100 1000 10000 Query sequence Database Cracking CIDR 2007 Stratos Idreos

  18. continuous adaptation set-up 200 Full Index 100 10K random selections Cumulative average response time (secs) selectivity 10% random value ranges 10 in a 30 million integer column Scan 1 0.1 0.01 Crack 10K queries later, 0.004 Full Index still has not 0.001 amortized the initialization costs 1 10 100 1000 10000 Query sequence Database Cracking CIDR 2007 Stratos Idreos

  19. table1 A Stratos Idreos

  20. table1 A B C D ... ... Stratos Idreos

  21. select R.A from R where R.A>10 and R.A<14 table1 A B C D ... ... Stratos Idreos

  22. select R.A from R where R.A>10 and R.A<14 select max(R.A),max(R.B),max(S.A),max(S.B) from R,S where v1 <R.C<v2 and v3 <R.D<v4 and v5 <R.E<v6 and k1 <S.C<k2 and k3 <S.D<k4 and k5 <S.E<k6 and R.F = S.F table1 A B C D ... ... Stratos Idreos

  23. select R.A from R where R.A>10 and R.A<14 select max(R.A),max(R.B),max(S.A),max(S.B) from R,S where v1 <R.C<v2 and v3 <R.D<v4 and v5 <R.E<v6 and k1 <S.C<k2 and k3 <S.D<k4 and k5 <S.E<k6 and R.F = S.F table1 updates A B C D joins ... concurrency control ... ... Stratos Idreos

  24. cracking databases basics (CIDR07) >1 storage- updates >1 columns columns restrictions (SIGMOD07) (SIGMOD09) (SIGMOD09) benchmarking robustness algorithms robustne (TPCTC10) (PVLDB12) (PVLDB11) concurrency control multi-cores time-series (PVLDB12) (SIGMOD15) (SIGMOD14) adaptive storage (SIGMOD14) b-trees hadoop (HP Labs) (Yale/Saarland)

  25. cracking tangram base data as queries arrive... table 1 A B C D table 2 A B C D Stratos Idreos

  26. cracking tangram base data as queries arrive... table 1 table 1 A B C D A B C D table 2 table 2 A B C D A B C D Stratos Idreos

  27. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D table 2 table 2 A B C D A B C D Stratos Idreos

  28. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing table 2 table 2 A B C D A B C D Stratos Idreos

  29. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation table 2 table 2 A B C D A B C D Stratos Idreos

  30. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D x partial indexing continuous adaptation storage adaptation table 2 table 2 x A B C D A B C D x Stratos Idreos

  31. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation table 2 table 2 A B C D A B C D Stratos Idreos

  32. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction table 2 table 2 A B C D A B C D Stratos Idreos

  33. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 A B C D A B C D Stratos Idreos

  34. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 A B C D A B C D Stratos Idreos

  35. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 sort in caches A B C D A B C D Stratos Idreos

  36. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 sort in caches A B C D A B C D crack joins Stratos Idreos

  37. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 sort in caches A B C D A B C D q1 crack joins q2 lightweight locking Stratos Idreos

  38. cracking tangram base data as queries arrive... table 1 table 1 partial materialization A B C D A B C D partial indexing continuous adaptation storage adaptation no tuple reconstruction adaptive alignment table 2 table 2 sort in caches A B C D A B C D query crack joins random lightweight locking stochastic cracking Stratos Idreos

  39. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 250 200 150 100 70 0 5 10 15 20 25 30 Query sequence Sideways Cracking, SIGMOD 09 Stratos Idreos

  40. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 100 70 0 5 10 15 20 25 30 Query sequence Sideways Cracking, SIGMOD 09 Stratos Idreos

  41. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 100 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  42. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 MonetDB with 100 sideways cracking 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  43. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 MonetDB with 100 sideways cracking 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  44. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 MonetDB with 100 sideways cracking 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  45. MonetDB Sel. Crack MySQL Presorted Sid. Crack Presorted 10000 Response time (milli secs) 1000 764 TPC-H Query 15 330 420 300 normal MonetDB 250 200 selection cracking 150 MonetDB with 100 sideways cracking 70 presorted MonetDB 0 5 10 15 20 25 30 preparation cost Query sequence 3-14 minutes Sideways Cracking, SIGMOD 09 Stratos Idreos

  46. cracking on Skyserver (4TB) (Sloan Digital Sky Survey, www.sdss.org) cracking answers 160.000 queries while full indexing is still half way creating one index Stochastic Cracking, PVLDB 12 Stratos Idreos

Recommend

7 Habits of highly effective woodworkers Workshop tune-up/makeover Workbench tune-up

7 Habits of highly effective woodworkers Workshop tune-up/makeover Workbench tune-up

FWW references 7 Habits of highly effective woodworkers Workshop tune-up/makeover Workbench tune-up Storage of tools Tune-up of squares Handplane tune-up Sharpening- bevel angles Cabinet scrapers

477 views • 26 slides
Tune In &amp; Tune Up San Joaquin Valley, CA Helping Immigrant Families Build Financial

Tune In &amp; Tune Up San Joaquin Valley, CA Helping Immigrant Families Build Financial

Tune In &amp; Tune Up San Joaquin Valley, CA Helping Immigrant Families Build Financial Capability Webinar August 25 th , 2016 San Joaquin Valley, CA Epicenter of pollution and poverty Population of 4 million people Largest rural

509 views • 15 slides
API Design made easy with the GraphQL Whiteboard Jrgen Albert Data In Motion Consulting GmbH

API Design made easy with the GraphQL Whiteboard Jrgen Albert Data In Motion Consulting GmbH

API Design made easy with the GraphQL Whiteboard Jrgen Albert Data In Motion Consulting GmbH 1 About Us Founded in 2010 Located in Jena/Thuringia - Germany Consulting, Independen RnD, Development, Training Assisted

346 views • 15 slides
To tune or not to tune Thomas Pasquier tfjmp@cs.ubc.ca https://tfjmp.org The team - Ayat Fekry

To tune or not to tune Thomas Pasquier tfjmp@cs.ubc.ca https://tfjmp.org The team - Ayat Fekry

To tune or not to tune Thomas Pasquier tfjmp@cs.ubc.ca https://tfjmp.org The team - Ayat Fekry , PhD student - Lucian Carata , Senior Research Associate - Andrew Rice , Professor - Andy Hopper , Professor 2 About me - Assistant Professor

626 views • 37 slides
Computer Science 240 Principles of Software Design Goals of Software Design Create systems

Computer Science 240 Principles of Software Design Goals of Software Design Create systems

Computer Science 240 Principles of Software Design Goals of Software Design Create systems that Work Easy as possible to understand, debug, and maintain Hold up well under changes Have reusable components Design is

470 views • 20 slides
Easy-to-Use Easy-to-Install Easy on the Budget orecx.com Easy-to-Use

Easy-to-Use Easy-to-Install Easy on the Budget orecx.com Easy-to-Use

Easy-to-Use Easy-to-Install Easy on the Budget orecx.com Easy-to-Use Easy-to-Install Easy on the Budget OrecX Call Recording The issue is simple: You want to comply with regulatory requirements and improve customer service by

204 views • 9 slides
PRESENTATION OF DATA Data summarization Is the organization of data in a way for easy

PRESENTATION OF DATA Data summarization Is the organization of data in a way for easy

Tabular presentation PRESENTATION OF DATA Data summarization Is the organization of data in a way for easy understanding. It is the first step of data interpretation (analysis). Consists of the following steps: 1) Data entering. 2)

932 views • 71 slides
ELECTRONIC MEDICAL RECORDS FOR EMERGENCIES / PROJECT BUENDIA WORKS EVERYWHERE EXTREMELY

ELECTRONIC MEDICAL RECORDS FOR EMERGENCIES / PROJECT BUENDIA WORKS EVERYWHERE EXTREMELY

Fabian Tamp @capnfabs ELECTRONIC MEDICAL RECORDS FOR EMERGENCIES / PROJECT BUENDIA WORKS EVERYWHERE EXTREMELY ADAPTABLE Photos: MSF PORTABLE EASY TO ANALYSE EASY TO BACK UP EASY TO ACCESS HISTORICAL DATA BETTER DESIGN FOR DECISION SUPPORT

631 views • 15 slides
Snake orbit effect on the spin tune in RHIC M. Bai, V. Ptitsyn, T. Roser Spin tune versus snake

Snake orbit effect on the spin tune in RHIC M. Bai, V. Ptitsyn, T. Roser Spin tune versus snake

Snake orbit effect on the spin tune in RHIC M. Bai, V. Ptitsyn, T. Roser Spin tune versus snake orbit angle Making use of the depolarization by 0.75 Example of measurements done in Blue at 250 GeV (second order) spin resonance: - vertical

381 views • 4 slides
Augmented Cognition Augmented Cognition systems use real-time cognitive state data to adapt

Augmented Cognition Augmented Cognition systems use real-time cognitive state data to adapt

Platform-Based Design of Augmented Cognition Systems Latosha Marshall &amp; Colby Raley ENSE623 Fall 2004 Design &amp; implementation of Augmented Cognition systems: Modular design can make it possible Platform-based design makes it feasible

185 views • 15 slides
Design Space Optimization of Embedded Memory Design Space Optimization of Embedded Memory Systems

Design Space Optimization of Embedded Memory Design Space Optimization of Embedded Memory Systems

Design Space Optimization of Embedded Memory Design Space Optimization of Embedded Memory Systems via Data Remapping Systems via Data Remapping Krishna V. Palem, Rodric M. Rabbah, Vincent J. Mooney III, Pinar Korkmaz and Kiran Puttaswamy Center

775 views • 38 slides
Data Driven Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and

Data Driven Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and

Data Driven Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and Design of Algorithms Classic algo design: solve a worst case instance. Easy domains, have optimal poly time algos. E.g., sorting, shortest

768 views • 43 slides
TS 83 DORMA DORMA TS 83 Easy-action Door Closer Easy-action door closer Data and features TS

TS 83 DORMA DORMA TS 83 Easy-action Door Closer Easy-action door closer Data and features TS

Easy-action Door Closer TS 83 DORMA DORMA TS 83 Easy-action Door Closer Easy-action door closer Data and features TS 83 with thinking backcheck Variable Spring EN EN closing force strength 2 6 7 Easy to fix and even

178 views • 6 slides
Data Movement Instructions Systems Design &amp; Programming CMPE 310 Intel Assembly Data

Data Movement Instructions Systems Design &amp; Programming CMPE 310 Intel Assembly Data

Data Movement Instructions Systems Design &amp; Programming CMPE 310 Intel Assembly Data Movement Instruction: mov (covered already) push, pop lea (mov and offset) lds, les, lfs, lgs, lss movs, lods, stos ins, outs

406 views • 11 slides
Learning Faster from Easy Data II Wouter Koolen Tim van Erven Aim of the Workshop

Learning Faster from Easy Data II Wouter Koolen Tim van Erven Aim of the Workshop

Learning Faster from Easy Data II Wouter Koolen Tim van Erven Aim of the Workshop Minimax analysis gives robust algorithms But in common easy cases these are overly conservative Large gap between performance predicted by

347 views • 32 slides
Design Innovation in Complex Systems Design: Integrating Design Thinking and Systems Thinking

Design Innovation in Complex Systems Design: Integrating Design Thinking and Systems Thinking

ASME International Design Engineering Technical Conference, Design Theory and Methodology (DTM) Conference, Presentation and Extended Abstract, 52938, August 2020. Design Innovation in Complex Systems Design: Integrating Design Thinking and

646 views • 5 slides
Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and Design of

Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and Design of

Sample Complexity for Data Driven Algorithm Design Maria-Florina (Nina) Balcan Carnegie Mellon University Analysis and Design of Algorithms Classic algo design: solve a worst case instance. Easy domains, have optimal poly time algos.

993 views • 43 slides
Scaling Data Analytics Jan Vitek Challenges How do we program big data? What are the

Scaling Data Analytics Jan Vitek Challenges How do we program big data? What are the

Scaling Data Analytics Jan Vitek Challenges How do we program big data? What are the tools? What are the abstractions? How do we debug, visualize, tune big data? Some big data infrastructures Hadoop MapReduce X10 RHIPE Pig

471 views • 22 slides
OS Essentials Systems Design &amp; Programming CMPE 310 Processes and Tasks What comprises the

OS Essentials Systems Design &amp; Programming CMPE 310 Processes and Tasks What comprises the

OS Essentials Systems Design &amp; Programming CMPE 310 Processes and Tasks What comprises the state of a running program (a process or task)? DRAM Address bus Microprocessor OS code Control and data special caches Data bus code/data

449 views • 17 slides
bPro-H22 1. Good design 2. Narrow housing 3. Easy to clean 4. Robust smooth housing 5.

bPro-H22 1. Good design 2. Narrow housing 3. Easy to clean 4. Robust smooth housing 5.

bPro-H22 1. Good design 2. Narrow housing 3. Easy to clean 4. Robust smooth housing 5. Easy printer loading-&gt; USP !! 6. Installation friendly-&gt; USP !! 7. WLAN integration-&gt; USP !! 8. Standard printer types 9. Water

319 views • 11 slides
make experiment WMT 2010 workflow management Goals JHU Submission WMT 2010 Running translation

make experiment WMT 2010 workflow management Goals JHU Submission WMT 2010 Running translation

make experiment WMT 2010 workflow management Goals JHU Submission WMT 2010 Running translation pipeline should be easy Easy to understand Easy to configure Easy to monitor Easy to run All results must be reproducible Original Data

343 views • 7 slides
CS411 Pick application Conceptual design domain Database Systems What data do I What data do

CS411 Pick application Conceptual design domain Database Systems What data do I What data do

Steps in building a DB application CS411 Pick application Conceptual design domain Database Systems What data do I What data do I need for my need for my application application 02: The Entity-Relationship Model How can I How can I

399 views • 7 slides
Design and performance evaluation of NUMA-aware RDMA-based end-to-end data transfer systems Yufei

Design and performance evaluation of NUMA-aware RDMA-based end-to-end data transfer systems Yufei

Design and performance evaluation of NUMA-aware RDMA-based end-to-end data transfer systems Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas G. Robertazzi Presented by Zach Yannes Introduction Need to transfer large amounts of data long

384 views • 27 slides
Smartpods PULSE The Software is easy to use and provides relevant data to improve health and

Smartpods PULSE The Software is easy to use and provides relevant data to improve health and

Smartpods PULSE The Software is easy to use and provides relevant data to improve health and facility management. 3 Easy Steps For New Users Step 1 Type of Movement Step 1 Viewing Video Step 1 Login Process Step 2 Height Setting Step

381 views • 8 slides

More recommend