database cracking
play

Database cracking Stratos Idreos, Martin Kersten and Stefan Manegold - PowerPoint PPT Presentation

May 23, 2023 •318 likes •627 views

Database cracking Stratos Idreos, Martin Kersten and Stefan Manegold CWI Amsterdam, The Netherlands The open problem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Dynamic

  1. Database cracking Stratos Idreos, Martin Kersten and Stefan Manegold CWI Amsterdam, The Netherlands

  2. The open problem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Dynamic environments What kind of indices should be used, when and on which data? Database experts or special tools monitor the system More difficult in databases with huge datasets

  3. Database cracking - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - We explore self-organization Each query triggers physical re-organization of the database We designed and implemented a DBMS using database cracking We work on top of MonetDB, a column oriented database system

  4. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 8 6 2 12 13 4 15

  5. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 3 4 8 2 6 2 12 13 4 15

  6. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 3 4 8 2 6 8 2 6 12 13 4 15

  7. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  8. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select A>5 and A<10 17 3 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  9. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>5 and A<10 17 3 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  10. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>5 and A<10 select A>2 and A<15 17 3 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  11. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>5 and A<10 select A>2 and A<15 17 3 2 <= 5 3 4 8 2 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  12. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 2 17 3 <= 5 3 4 4 8 2 3 6 8 > 5 2 6 12 12 13 15 >= 10 4 17 15 13

  13. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 2 17 3 <= 5 3 4 4 8 2 3 6 8 8 > 5 2 6 6 12 12 13 15 >= 10 4 17 15 13

  14. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 2 17 3 <= 5 3 4 4 8 2 3 6 8 8 > 5 2 6 6 12 12 12 13 13 15 >= 10 4 17 15 13

  15. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 <= 2 2 17 3 3 4 4 > 2 8 2 3 6 8 8 > 5 > 5 2 6 6 12 12 12 >= 10 13 13 15 >= 10 17 4 17 >= 15 15 13 15

  16. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data access for future queries select A>2 and A<15 select A>5 and A<10 <= 2 2 17 3 3 4 4 > 2 8 2 3 6 8 8 > 5 > 5 2 6 6 12 12 12 >= 10 13 13 15 >= 10 17 4 17 >= 15 15 13 15

  17. Cracking a database - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Improve data The more we access for crack the more future queries we learn select A>2 and A<15 select A>5 and A<10 <= 2 2 17 3 3 4 4 > 2 8 2 3 6 8 8 > 5 > 5 2 6 6 12 12 12 >= 10 13 13 15 >= 10 17 4 17 >= 15 15 13 15

  18. Cracking algorithms - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - There are two types of cracking algorithms Split a piece in Split a piece in two new pieces three new pieces

  19. Design - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The first time a range query is posed on an attribute A , a cracking DBMS makes a copy of column A, called the cracker column of A A cracker column is continuously physically re-organized based on queries that need to touch attribute such as the result is in a contiguous space For each cracker column, there is a cracker index

  20. The cracker select operator - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  The simple select operator: 1. Scans the column 2. Return a new column that contains qualifying values  The crackers select operator: 1. Searches the cracker index 2. Physically re-organizes pieces found 3. Update the cracker index 4. Return a slice of the cracker column as result  More steps but faster because we analyze less data

  21. Testing the select operator - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  22. Research and opportunities for cracking - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  Optimization  optimal piece size / granularity / index (avl) depth  Exploit cracking for join queries, aggregate queries etc.  Concurrency issues  Cracking histograms  Distributed cracking  A priori cracking  ...

  23. Research and opportunities for cracking - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  Optimization  optimal piece size / granularity / index (avl) depth  Exploit cracking for join queries, aggregate queries etc.  Concurrency issues  Cracking histograms  Distributed cracking  A priori cracking THANK YOU!  ...

  24. Cracking vs Indices and Sorting - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -  How cracking compares to a sorting strategy?  Sort data upfront and then use binary search  For a sorting strategy we have to make an investment upfront  Sorting needs prior knowledge of query workload  Similar arguments stand for indices

  25. Impact on query plan - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select R.c from R where 5 < R.a < 10 and 9 < R.b < 20 Rb1 Ra1 algebra.select(Ra, 5, 10) algebra.select(Rb, 9, 20) Ra2 algebra.OIDintersect(Ra1, Rb1) algebra.fetch(Ra2, Rc)

  26. Impact on query plan - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select R.c from R where 5 < R.a < 10 and 9 < R.b < 20 Rb1 Ra1 crackers.select(Rb, 9, 20) crackers.select(Ra, 5, 10) algebra.select(Ra, 5, 10) algebra.select(Rb, 9, 20) Ra2 algebra.OIDintersect(Ra1, Rb1) algebra.fetch(Ra2, Rc)

  27. Impact on query plan - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select R.c from R where 5 < R.a < 10 and 9 < R.b < 20 Rb1 Ra1 crackers.select(Rb, 9, 20) crackers.select(Ra, 5, 10) algebra.select(Ra, 5, 10) algebra.select(Rb, 9, 20) Becomes Ra2 algebra.OIDintersect(Ra1, Rb1) expensive algebra.fetch(Ra2, Rc)

  28. Impact on query plan - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - select R.c from R where 5 < R.a < 10 and 9 < R.b < 20 Ra1 crackers.select(Ra, 5, 10) algebra.select(Ra, 5, 10) Ra2 algebra.joinselect(Ra1, Rb,9,20) algebra.OIDintersect(Ra1, Rb1) algebra.fetch(Ra2, Rc)

  29. Scalability - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  30. TPC-H query 6 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Recommend

DATABASE CRACKING: Fancy Scan, not Poor Mans Sort! Hardware Folks Cracking Folks Don Holger

DATABASE CRACKING: Fancy Scan, not Poor Mans Sort! Hardware Folks Cracking Folks Don Holger

DATABASE CRACKING: Fancy Scan, not Poor Mans Sort! Hardware Folks Cracking Folks Don Holger Pirk Eleni Petraki Strato Idreos Stefan Manegold Martin Kersten EVALUATING RANGE PREDICATES COMPLEXITY ON PAPER Scanning: O(n) Sorting:

910 views • 43 slides
Database Cracking September 7, 2016 CSE 662 - Database Languages &amp; Runtimes 1 Row Stores

Database Cracking September 7, 2016 CSE 662 - Database Languages &amp; Runtimes 1 Row Stores

Database Cracking September 7, 2016 CSE 662 - Database Languages &amp; Runtimes 1 Row Stores A1 B1 C1 D1 A2 B2 C2 D2 A3 B3 C3 D3 A4 B4 C4 D4 Traditional DB: Lay out data on disk in rows CSE 662 - Database Languages &amp;

753 views • 41 slides
Database Cracking Languages and Runtimes for Big Data CSE 662 - Database Languages &amp; Runtimes

Database Cracking Languages and Runtimes for Big Data CSE 662 - Database Languages &amp; Runtimes

Database Cracking Languages and Runtimes for Big Data CSE 662 - Database Languages &amp; Runtimes 1 Row Stores A1 B1 C1 D1 A2 B2 C2 D2 A3 B3 C3 D3 A4 B4 C4 D4 Traditional DB: Lay out data on disk in rows CSE 662 - Database

544 views • 41 slides
Project Seeds Languages &amp; Runtimes for Big Data Reminder Homework 1: Database Cracking

Project Seeds Languages &amp; Runtimes for Big Data Reminder Homework 1: Database Cracking

Project Seeds Languages &amp; Runtimes for Big Data Reminder Homework 1: Database Cracking Read the paper (linked from the course page) Submit 2 discussion points (strength and weakness of the work) or make a counterargument to

588 views • 38 slides
Cracking the code Cracking the code Houston, we have a problem. Consanguineous Marriage Tay

Cracking the code Cracking the code Houston, we have a problem. Consanguineous Marriage Tay

Cracking the code Cracking the code Houston, we have a problem. Consanguineous Marriage Tay Brain T2-Hyperintensity Sachs Stem Pelizaeus- B12 Metabolism U-Fibres Merzbacher Hypomyelination T1-Hypointensity CSF Salla LEUKODYSTROPHY

918 views • 62 slides
IAPF TRUSTEE NETWORK EVENT CRACKING THE DC CODE: VALUE FOR MONEY WELCOME Cracking the DC Code:

IAPF TRUSTEE NETWORK EVENT CRACKING THE DC CODE: VALUE FOR MONEY WELCOME Cracking the DC Code:

IAPF TRUSTEE NETWORK EVENT CRACKING THE DC CODE: VALUE FOR MONEY WELCOME Cracking the DC Code: Value for Money Rickard Mills Council Member, IAPF THANK YOU SPONSOR Cracking the DC Code: Value for Money HOUSEKEEPING Cracking the DC Code:

1.04k views • 54 slides
twenty six diagonal cracking shear f c http://www.bam.de concrete construction:

twenty six diagonal cracking shear f c http://www.bam.de concrete construction:

A RCHITECTURAL S TRUCTURES : Concrete in Compression F ORM, B EHAVIOR, AND D ESIGN ARCH 331 crushing D R. A NNE N ICHOLS vertical cracking S PRING 2018 tension lecture twenty six diagonal cracking shear f c

137 views • 3 slides
Cracking Wireless Ryan Curtin LUG@GT Ryan Curtin Cracking Wireless - p. 1 Goals By the end of

Cracking Wireless Ryan Curtin LUG@GT Ryan Curtin Cracking Wireless - p. 1 Goals By the end of

Cracking Wireless Ryan Curtin LUG@GT Ryan Curtin Cracking Wireless - p. 1 Goals By the end of this presentation (if you stay awake), you will: Goals Setting Up Checking Injection Understand the different types of wireless keys

447 views • 13 slides
Merging Whats Cracked, Cracking Whats Merged Adaptive Indexing in Main-Memory Column-Stores

Merging Whats Cracked, Cracking Whats Merged Adaptive Indexing in Main-Memory Column-Stores

Merging Whats Cracked, Cracking Whats Merged Adaptive Indexing in Main-Memory Column-Stores Stratos Idreos, Stefan Manegold, (CWI) Harumi Kuno, Goetz Graefe (HP Labs) PVLDB 2011, 4(9) Physical design problem Database systems perform

1.24k views • 122 slides
Designing for durability and strength Cracking &amp; durability Local studies on corrosion &amp;

Designing for durability and strength Cracking &amp; durability Local studies on corrosion &amp;

Designing for durability and strength Cracking &amp; durability Local studies on corrosion &amp; cracking Other aspects Presented by: Professor Mark Alexander Concrete Materials and Structural Integrity Research Unit (CoMSIRU)

718 views • 29 slides
PicoCrack: The Art of Efficient Cracking FPGAs (Field Programmable Gate Arrays) allow custom

PicoCrack: The Art of Efficient Cracking FPGAs (Field Programmable Gate Arrays) allow custom

PicoCrack: The Art of Efficient Cracking FPGAs (Field Programmable Gate Arrays) allow custom silicon to be implemented easily. The result is a chip that can be built specifically for cracking passwords. This presentation focuses on uncovering

606 views • 47 slides
Distributed GPU password cracking Alexander Kasabov &amp; Jochem van Kerkwijk System and

Distributed GPU password cracking Alexander Kasabov &amp; Jochem van Kerkwijk System and

Distributed GPU password cracking Alexander Kasabov &amp; Jochem van Kerkwijk System and Network Engineering { akasabov | jkerkwijk } @os3.nl February 2, 2011 Outline Introduction Password cracking Graphics processing unit Distributed

501 views • 16 slides
Physical and Metallurgical Analysis Aids in Understanding Recovery Boiler Tube Cracking

Physical and Metallurgical Analysis Aids in Understanding Recovery Boiler Tube Cracking

Physical and Metallurgical Analysis Aids in Understanding Recovery Boiler Tube Cracking November, 2014 PdM Technology Infrared Thermography Gas Find IR Example of LNG Leak Safety Minute Economizer Tube Cracking Assessment

447 views • 18 slides
ramp. Photo 2. Looking along the guardrail. Cracking has protruded through asphalt patch,

ramp. Photo 2. Looking along the guardrail. Cracking has protruded through asphalt patch,

SHOP SLIDE Photo 1. Cracking across the footpath at the end of the Hwy 2 off ramp. Photo 2. Looking along the guardrail. Cracking has protruded through asphalt patch, with a drop depression in the asphalt and significant bow in the

247 views • 10 slides
Seamless Asphalt Pavement Preservation Featuring Types of asphalt failure Alligator cracking

Seamless Asphalt Pavement Preservation Featuring Types of asphalt failure Alligator cracking

36 th Annual Utah Asphalt Conference Seamless Asphalt Pavement Preservation Featuring Types of asphalt failure Alligator cracking Joint cracking Raveling Rutting Potholes Typical Repair Approaches Sawcut, removal, and

445 views • 27 slides
Database Utilities 10/17/2007 DC/Win Database Utilities Opening Database Utilities From File on

Database Utilities 10/17/2007 DC/Win Database Utilities Opening Database Utilities From File on

Database Utilities 10/17/2007 DC/Win Database Utilities Opening Database Utilities From File on the Menu Bar DC/Win Database Utilities Options Vary Depending on Database Two Types of Databases Sequel (SQL) Database Access Database

460 views • 6 slides
September 28, 2016 1 Cracking Drupal Security concepts and pitfalls Peter Wolanin Moshe

September 28, 2016 1 Cracking Drupal Security concepts and pitfalls Peter Wolanin Moshe

September 28, 2016 1 Cracking Drupal Security concepts and pitfalls Peter Wolanin Moshe Weitzman Track: PHP https://events.drupal.org/dublin2016/sessions/cracking-drupal Special thanks to Klaus Purer for creating the original talk and slides

683 views • 39 slides
City of Red LodgeProposed Special Improvement District April 12, 2018 Great West Engineering

City of Red LodgeProposed Special Improvement District April 12, 2018 Great West Engineering

City of Red LodgeProposed Special Improvement District April 12, 2018 Great West Engineering Dorsey &amp; Whitney LLP 1 Existing Conditions Existing pavement experiencing: block cracking, large transverse cracking, alligator cracking, and

342 views • 15 slides
Chloride-Induced Stress Corrosion Cracking Tests and Example Aging Management Program Darrell S.

Chloride-Induced Stress Corrosion Cracking Tests and Example Aging Management Program Darrell S.

Chloride-Induced Stress Corrosion Cracking Tests and Example Aging Management Program Darrell S. Dunn NRC/NMSS/SFST Public Meeting with Nuclear Energy Institute on Chloride Induced Stress Corrosion Cracking Regulatory Issue Resolution Protocol

527 views • 26 slides
Password Cracking Sam Martin and Mark Tokutomi CS466/566: Computer Security April 22, 2012 Sam

Password Cracking Sam Martin and Mark Tokutomi CS466/566: Computer Security April 22, 2012 Sam

Introduction Lets Attack A Time-Memory Trade-Off The LanManager Hash Physical Attacks Password Cracking Sam Martin and Mark Tokutomi CS466/566: Computer Security April 22, 2012 Sam Martin and Mark Tokutomi Password Cracking

1.49k views • 76 slides
1 Factors that affect cracking Our goal Age Eliminate cracking in bridge decks Deck

1 Factors that affect cracking Our goal Age Eliminate cracking in bridge decks Deck

Outline Overview of LC-HPC Specifications and their application Program Background and Where we stand Introduction to Specifications 9 th Annual Meeting - Construction of Crack-Free Concrete Bridge Decks July 19, 2011 Kansas City,

479 views • 8 slides
Advanced Database CS 525: Organization? Advanced Database =Database Implementation

Advanced Database CS 525: Organization? Advanced Database =Database Implementation

Advanced Database CS 525: Organization? Advanced Database =Database Implementation Organization =How to implement a database system and have fun doing it ;-) 01: Introduction Boris Glavic Slides: adapted from a course taught by

199 views • 7 slides
DATABASE SYSTEMS Database programming in a web environment Database System Course AGENDA FOR

DATABASE SYSTEMS Database programming in a web environment Database System Course AGENDA FOR

DATABASE SYSTEMS Database programming in a web environment Database System Course AGENDA FOR TODAY The final project Advanced Mysql Database programming Recap: DB servers in the web Web programming architecture HTTP on a need-to-know basis.

688 views • 54 slides
The Cracking of the Cipher Challenge 10 steps to 10,000 www.simonsingh.net

The Cracking of the Cipher Challenge 10 steps to 10,000 www.simonsingh.net

The Cracking of the Cipher Challenge 10 steps to 10,000 www.simonsingh.net www.simonsingh.net The most incomprehensible thing about the universe is that it is comprehensible. Albert Einstein We are 13.7 billion light-years We

1.2k views • 22 slides

More recommend