buffer pools
play

Buffer Pools Lecture # 05 Database Systems Andy Pavlo AP AP - PowerPoint PPT Presentation

Dec 22, 2023 •10 likes •770 views

Buffer Pools Lecture # 05 Database Systems Andy Pavlo AP AP Computer Science 15-445/15-645 Carnegie Mellon Univ. Fall 2018 2 UPCO M IN G DATABASE EVEN TS Relational AI Talk Wednesday Sep 12 th @ 4:00pm GHC 8102 MapD Talk

  1. Buffer Pools Lecture # 05 Database Systems Andy Pavlo AP AP Computer Science 15-445/15-645 Carnegie Mellon Univ. Fall 2018

  2. 2 UPCO M IN G DATABASE EVEN TS Relational AI Talk → Wednesday Sep 12 th @ 4:00pm → GHC 8102 MapD Talk → Thursday Sept 20 th @ 12pm → CIC 4 th Floor CMU 15-445/645 (Fall 2018)

  3. 3 DATABASE STO RAGE Problem #1: How the DBMS represents the database in files on disk. Problem #2: How the DBMS manages its memory and move data back-and-forth from disk. CMU 15-445/645 (Fall 2018)

  4. 4 DATABASE STO RAGE Spatial Control: → Where to write pages on disk. → The goal is to keep pages that are used together often as physically close together as possible on disk. Temporal Control: → When to read pages into memory, and when to write them to disk. → The goal is minimize the number of stalls from having to read data from disk. CMU 15-445/645 (Fall 2018)

  5. 5 DISK- O RIEN TED DBM S Get page #2 Execution Engine Buffer Pool Directory Header 2 Pointer to page #2 Memory Database File Directory Header Header Header Header Header … 1 2 3 4 5 Pages Disk CMU 15-445/645 (Fall 2018)

  6. 6 TO DAY'S AGEN DA Buffer Pool Manager Replacement Policies Allocation Policies Other Memory Pools CMU 15-445/645 (Fall 2018)

  7. 7 BUFFER PO O L O RGAN IZATIO N Buffer Memory region organized as an array Pool of fixed-size pages. frame1 page1 An array entry is called a frame. frame2 page3 frame3 When the DBMS requests a page, an exact copy is placed into one of these frame4 frames. page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  8. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 per page: → Dirty Flag frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  9. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 per page: → Dirty Flag frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  10. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 per page: → Dirty Flag frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  11. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 page2 per page: → Dirty Flag frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  12. 8 BUFFER PO O L M ETA- DATA Page Buffer The page table keeps track of pages Table Pool that are currently in memory. page1 frame1 page1 page3 frame2 page3 Also maintains additional meta-data frame3 page2 per page: → Dirty Flag page2 frame4 → Pin/Reference Counter page1 page2 page3 page4 On-Disk File CMU 15-445/645 (Fall 2018)

  13. 9 LO CKS VS. LATCH ES Locks: → Protects the database's logical contents from other transactions. → Held for transaction duration. → Need to be able to rollback changes. Latches: → Protects the critical sections of the DBMS's internal data structure from other threads. → Held for operation duration. → Do not need to be able to rollback changes. CMU 15-445/645 (Fall 2018)

  14. 10 PAGE TABLE VS. PAGE DIRECTO RY The page directory is the mapping from page ids to page locations in the database files. → All changes must be recorded on disk to allow the DBMS to find on restart. The page table is the mapping from page ids to a copy of the page in buffer pool frames. → This is an in-memory data structure that does not need to be stored on disk. CMU 15-445/645 (Fall 2018)

  15. 11 M ULTIPLE BUFFER PO O LS The DBMS does not always have a single buffer pool for the entire system. → Multiple buffer pool instances → Per-database buffer pool → Per-page type buffer pool Helps reduce latch contention and improve locality. CMU 15-445/645 (Fall 2018)

  16. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. Q1 page0 → Sequential Scans page1 → Index Scans page2 Buffer Pool page3 page4 page5 CMU 15-445/645 (Fall 2018)

  17. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. Q1 page0 → Sequential Scans page1 → Index Scans page2 Buffer Pool page3 page0 page4 page5 CMU 15-445/645 (Fall 2018)

  18. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans Q1 page2 Buffer Pool page3 page0 page4 page1 page5 CMU 15-445/645 (Fall 2018)

  19. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans Q1 page2 Buffer Pool page3 page0 page4 page1 page5 CMU 15-445/645 (Fall 2018)

  20. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans Q1 page2 Buffer Pool page3 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  21. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans page2 Q1 Buffer Pool page3 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  22. 12 PRE- FETCH IN G Disk Pages The DBMS can also prefetch pages based on a query plan. page0 → Sequential Scans page1 → Index Scans page2 Buffer Pool page3 page3 page4 page4 Q1 page5 page5 CMU 15-445/645 (Fall 2018)

  23. 13 PRE- FETCH IN G index-page0 Disk Pages index-page1 index-page4 index-page0 index-page2 index-page3 index-page5 index-page6 index-page1 index-page2 Buffer Pool index-page3 index-page4 index-page5 CMU 15-445/645 (Fall 2018)

  24. 13 PRE- FETCH IN G index-page0 Disk Pages Q1 index-page1 index-page4 index-page0 index-page2 index-page3 index-page5 index-page6 index-page1 index-page2 Buffer Pool index-page3 index-page0 index-page4 index-page5 CMU 15-445/645 (Fall 2018)

  25. 13 PRE- FETCH IN G index-page0 Disk Pages index-page1 index-page4 index-page0 index-page2 index-page3 index-page5 index-page6 index-page1 Q1 index-page2 Buffer Pool index-page3 index-page0 index-page4 index-page1 index-page5 CMU 15-445/645 (Fall 2018)

  26. 13 PRE- FETCH IN G index-page0 Disk Pages index-page1 index-page4 index-page0 index-page2 index-page3 index-page5 index-page6 index-page1 Q1 index-page2 Buffer Pool index-page3 index-page0 index-page4 index-page1 index-page5 CMU 15-445/645 (Fall 2018)

  27. 14 SCAN SH ARIN G Queries are able to reuse data retrieved from storage or operator computations. → This is different from result caching. Allow multiple queries to attach to a single cursor that scans a table. → Queries do not have to be exactly the same. → Can also share intermediate results. CMU 15-445/645 (Fall 2018)

  28. 15 SCAN SH ARIN G If a query starts a scan and if there one already doing this, then the DBMS will attach to the second query's cursor. → The DBMS keeps track of where the second query joined with the first so that it can finish the scan when it reaches the end of the data structure. Fully supported in IBM DB2 and MSSQL. Oracle only supports cursor sharing for identical queries. CMU 15-445/645 (Fall 2018)

  29. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A Q1 page0 page1 page2 Buffer Pool page3 page4 page5 CMU 15-445/645 (Fall 2018)

  30. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A Q1 page0 page1 page2 Buffer Pool page3 page0 page4 page5 CMU 15-445/645 (Fall 2018)

  31. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 page1 page2 Q1 Buffer Pool page3 page0 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  32. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 page1 page2 Buffer Pool Q1 page3 page0 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  33. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 page1 page2 Buffer Pool Q1 page3 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  34. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A Q2 page0 Q2 SELECT AVG (val) FROM A page1 page2 Buffer Pool Q1 page3 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  35. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 Q2 SELECT AVG (val) FROM A page1 page2 Buffer Pool Q1 page3 Q2 page3 page4 page1 page5 page2 CMU 15-445/645 (Fall 2018)

  36. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A page0 Q2 SELECT AVG (val) FROM A page1 page2 Buffer Pool page3 page3 page4 page4 Q2 Q1 page5 page5 CMU 15-445/645 (Fall 2018)

  37. 16 SCAN SH ARIN G Disk Pages Q1 SELECT SUM (val) FROM A Q2 page0 Q2 SELECT AVG (val) FROM A page1 page2 Buffer Pool page3 page3 page4 page4 page5 page5 CMU 15-445/645 (Fall 2018)

Recommend

Buffer Manager Each relation is can be mapped onto many files (each file containing data from one

Buffer Manager Each relation is can be mapped onto many files (each file containing data from one

Buffer Management The database buffer is the mediator between the basic file system and the tuple-oriented file system. The buffer managers task is to make the pages addressable in main memory and to coordinate the writing of pages to disk

1.12k views • 8 slides
Introduction Buffer Overflows Buffer overflows were the most common form of security

Introduction Buffer Overflows Buffer overflows were the most common form of security

Introduction Buffer Overflows Buffer overflows were the most common form of security vulnerability in the 90s. Buffer overflows dominate in the area of remote network penetration vulnerabilities. CS 465 They represent one of the

651 views • 4 slides
Buffer Overflows with Content 2 A Process Stack Buffer Overflow Common Techniques employed

Buffer Overflows with Content 2 A Process Stack Buffer Overflow Common Techniques employed

1 Buffer Overflows with Content 2 A Process Stack Buffer Overflow Common Techniques employed in buffer overflow exploits to create backdoors Execution of additional network services via the INETD daemon The addition of new users

497 views • 30 slides
External buffer Raslan Darawsheh Mellanox External buffer First was introduced by Olivier

External buffer Raslan Darawsheh Mellanox External buffer First was introduced by Olivier

x External buffer Raslan Darawsheh Mellanox External buffer First was introduced by Olivier in his presentation in 2016. The support for External buffer has been there since DPDK 18.05 2 External buffer: Contd 3 External

544 views • 12 slides
Lab 2: Buffer Overflows Fengwei Zhang SUSTech CS 315 Computer Security 1 Buffer Overflows

Lab 2: Buffer Overflows Fengwei Zhang SUSTech CS 315 Computer Security 1 Buffer Overflows

Lab 2: Buffer Overflows Fengwei Zhang SUSTech CS 315 Computer Security 1 Buffer Overflows One of the most common vulnerabilities in software Programming languages commonly associated with buffer overflows including C and C++

1.14k views • 17 slides
Buffer Overflows and Defenses CS 419: Computer Security Lecture 10 and 11 Buffer Overflow a

Buffer Overflows and Defenses CS 419: Computer Security Lecture 10 and 11 Buffer Overflow a

Buffer Overflows and Defenses CS 419: Computer Security Lecture 10 and 11 Buffer Overflow a very common attack mechanism from 1988 Morris Worm to Code Red, Slammer, Sasser and many others prevention techniques known still of major concern

873 views • 53 slides
TinyOS Determine when Fill message Specify Pass buffer message buffer Network Communication

TinyOS Determine when Fill message Specify Pass buffer message buffer Network Communication

Inter-Node Communication General idea: Sender: TinyOS Determine when Fill message Specify Pass buffer message buffer Network Communication buffer with data Recipients to OS can be reused Computer Network Receiver:

176 views • 3 slides
Week 03 Lectures PostgreSQL Buffer Manager 1/95 PostgreSQL buffer manager: provides a shared

Week 03 Lectures PostgreSQL Buffer Manager 1/95 PostgreSQL buffer manager: provides a shared

Week 03 Lectures PostgreSQL Buffer Manager 1/95 PostgreSQL buffer manager: provides a shared pool of memory buffers for all backends all access methods get data from disk via buffer manager Buffers are located in a large region of shared

505 views • 25 slides
Riparian Buffer or Riparian Forest Buffer Equivalency Demonstration and Offsetting Water

Riparian Buffer or Riparian Forest Buffer Equivalency Demonstration and Offsetting Water

Implementation of Act 162 of 2014 Riparian Buffer or Riparian Forest Buffer Equivalency Demonstration and Offsetting Water Resources Advisory Committee August 12, 2015 Tom Wolf, Governor John Quigley, Secretary Agenda 1. Overview of Act 162

530 views • 37 slides
Chapter 10 Buffer Overflow Buffer Overflow Common attack mechanism first wide use by

Chapter 10 Buffer Overflow Buffer Overflow Common attack mechanism first wide use by

Chapter 10 Buffer Overflow Buffer Overflow Common attack mechanism first wide use by the Morris Worm in 1988 Prevention techniques known NX bit, stack canaries, ASLR Still of major concern Recent examples:

354 views • 21 slides
Patent Pools, Litigation and Innovation Jay Pil Choi (UNSW, Michigan State) Heiko Gerlach (U of

Patent Pools, Litigation and Innovation Jay Pil Choi (UNSW, Michigan State) Heiko Gerlach (U of

Patent Pools, Litigation and Innovation Jay Pil Choi (UNSW, Michigan State) Heiko Gerlach (U of Queensland) 2nd ATE Symposium at UNSW Sydney December 15, 2014 Patent Pools, Litigation and Innovation 1 Patent Pools and Antitrust patent

730 views • 44 slides
Buffer Trees Lars Arge. The Buffer Tree: A New Technique for Optimal I/O Algorithms . In

Buffer Trees Lars Arge. The Buffer Tree: A New Technique for Optimal I/O Algorithms . In

Buffer Trees Lars Arge. The Buffer Tree: A New Technique for Optimal I/O Algorithms . In Proceedings of Fourth Workshop on Algorithms and Data Structures (WADS), Lecture Notes in Computer Science Vol. 955, Springer-Verlag, 1995, 334-345. 1

518 views • 27 slides
AKC Education Webinar Series: Social Media Resources Buffer: What Is It? Buffer is a software

AKC Education Webinar Series: Social Media Resources Buffer: What Is It? Buffer is a software

AKC Education Webinar Series: Social Media Resources Buffer: What Is It? Buffer is a software application for the web and mobile, designed to manage accounts in social networks, by providing the means for a user to schedule posts to Twitter,

506 views • 35 slides
More Vulnerabilities (buffer overreads, format string, integer overflow, heap overflows) Chester

More Vulnerabilities (buffer overreads, format string, integer overflow, heap overflows) Chester

More Vulnerabilities (buffer overreads, format string, integer overflow, heap overflows) Chester Rebeiro Indian Institute of Technology Madras Buffer Overreads 2 Buffer Overread Example 3 Buffer Overread Example len read from command line

906 views • 76 slides
Buffer Software Security overflows and other memory safety vulnerabilities Buffer overflow

Buffer Software Security overflows and other memory safety vulnerabilities Buffer overflow

Last time We continued By launching our 1st section: Buffer Software Security overflows and other memory safety vulnerabilities Buffer overflow fundamentals This time We will finish up By looking at Buffer Overflow overflows

1.33k views • 103 slides
k + -buffer: Fragment Synchronized k-buffer I3D 2014 Andreas A. Vasilakis & Ioannis Fudos {

k + -buffer: Fragment Synchronized k-buffer I3D 2014 Andreas A. Vasilakis & Ioannis Fudos {

k + -buffer: Fragment Synchronized k-buffer I3D 2014 Andreas A. Vasilakis & Ioannis Fudos { abasilak,fudos } @cs.uoi.gr Department of Computer Science & Engineering, University of Ioannina, Greece 16 March 2014 Introduction

782 views • 44 slides
Database Systems IIB: DBMS-Implementation Chapter 5: The Buffer Cache Prof. Dr. Stefan Brass

Database Systems IIB: DBMS-Implementation Chapter 5: The Buffer Cache Prof. Dr. Stefan Brass

Storage Hierarchy, Buffer Manager Disk/Buffer Performance in Oracle Database Systems IIB: DBMS-Implementation Chapter 5: The Buffer Cache Prof. Dr. Stefan Brass Martin-Luther-Universit at Halle-Wittenberg Wintersemester 2019/20

623 views • 50 slides
ASSEMBLY AND INSTALLATION INSTRUCTIONS S.R. SMITH HELIX2 SLIDES ARE MANUFACTURED FOR INSTALLATION

ASSEMBLY AND INSTALLATION INSTRUCTIONS S.R. SMITH HELIX2 SLIDES ARE MANUFACTURED FOR INSTALLATION

ASSEMBLY AND INSTALLATION INSTRUCTIONS S.R. SMITH HELIX2 SLIDES ARE MANUFACTURED FOR INSTALLATION AND USE ON RESIDENTIAL INGROUND SWIMMING POOLS ONLY. THE HELIX2 IS NEVER TO BE INSTALLED AND USED ON ABOVEGROUND POOLS, ONGROUND POOLS,

389 views • 20 slides
Delta Pointers: Buffer Overflow Checks Without the Checks Tadde us Kroes & Koen Koning

Delta Pointers: Buffer Overflow Checks Without the Checks Tadde us Kroes & Koen Koning

Delta Pointers: Buffer Overflow Checks Without the Checks Tadde us Kroes & Koen Koning Erik van der Kouwe Herbert Bos Cristiano Giuffrida June 19, 2018 Preview buffer[10] secret 2 Preview buffer[10] secret buffer[5] 2 Preview

934 views • 74 slides
IT 4500 : Information Security Secure Programming Dr Joe Francom Buffer Overflow What it is? A

IT 4500 : Information Security Secure Programming Dr Joe Francom Buffer Overflow What it is? A

IT 4500 : Information Security Secure Programming Dr Joe Francom Buffer Overflow What it is? A buffer overflow occurs when a program or process tries to store more data in a buffer (temporary data storage area) than it was intended to hold.

221 views • 3 slides
Smashing the Buffer Smashing the Buffer Miroslav tampar Miroslav tampar (mstampar@zsis.hr )

Smashing the Buffer Smashing the Buffer Miroslav tampar Miroslav tampar (mstampar@zsis.hr )

Smashing the Buffer Smashing the Buffer Miroslav tampar Miroslav tampar (mstampar@zsis.hr ) (mstampar@zsis.hr ) Summary BSidesVienna 2014, Vienna (Austria) November 22nd, 2014 2 Buffer overflow (a.k.a.)

462 views • 35 slides
Buffer Overflow CS461/ECE422 Spring 2012 Reading Material

Buffer Overflow CS461/ECE422 Spring 2012 Reading Material

Buffer Overflow CS461/ECE422 Spring 2012 Reading Material Based on Chapter 11 of the text Outline Buffer Overflow Stack Buffer Overflow Shell

783 views • 28 slides
Understanding Investment Pools Presented By: Susan K. Anderson Valley View Consulting, LLC November

Understanding Investment Pools Presented By: Susan K. Anderson Valley View Consulting, LLC November

Understanding Investment Pools Presented By: Susan K. Anderson Valley View Consulting, LLC November 7, 2019 Wh What at ar are Loc Local Gov Governmen rnment In Investment Pools? ools? (L (LGIP) P) A Local Government Investment Pool is an

565 views • 30 slides
Shared buffer laboratory 2 implements a shared buffer Process loop Ke yboard wait for

Shared buffer laboratory 2 implements a shared buffer Process loop Ke yboard wait for

slide 1 gaius Shared buffer laboratory 2 implements a shared buffer Process loop Ke yboard wait for keyboard int port get character put character into buffer end Read (char *ch) shared by the application process Read (char *ch); shared

398 views • 28 slides

More recommend