building better benchmarks
play

Building Better Benchmarks PGCon 2020 . . . . . . . . . . - PowerPoint PPT Presentation

Nov 24, 2023 •220 likes •472 views

Building Better Benchmarks PGCon 2020 . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . . About me Employed by 2ndQuadrant PostgreSQL

  1. Building Better Benchmarks PGCon 2020 . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  2. About me … • Employed by 2ndQuadrant • PostgreSQL Contributor since 2005 • Director at United States PostgreSQL Association since 2011 • Portland PostgreSQL Users Group . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  3. Overview • What is benchmarking about. • What benchmarks are out there? • Issues with benchmarking. • Where do we want to go now? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  4. Reasons for benchmarking • Competition • Performance • Costs • Self­assessment • How well am I utilizing the hardware? • How does this patch affect the system? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  5. Competitive Benchmarking Taking a look at just a of couple of industry standards bodies… . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  6. https://www.spec.org/jAppServer2004/results/res2007q3/ Benchmark Publications Only a couple of published Postgres benchmarks: • Two SPECjAppServer2004 results in 2007 • No TPC publications for PostgreSQL . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  7. http://www.spec.org/ SPEC SPEC is a non­profit organization that establishes, maintains and endorses standardized benchmarks and tools to evaluate performance and energy efficiency for the newest generation of computing systems. • develops suites of benchmarks intended to measure computer performance • benchmarks suites are available to the public for a fee covering development and administrative costs • publishes news and benchmark results at . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  8. http://www.tpc.org/ TPC Transaction Processing Performance Council performs two major activity: 1. creating benchmarks 2. creating a process for reviewing and monitoring those benchmarks . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  9. TPC Benchmarks Many TPC benchmarks are not trivial: • Complete benchmarking kits generally not provided • Database design • Data generation • Transaction profiles • Auditing • Publication . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  10. Summarizing Competitive Benchmarking • Well­defined workloads • Expensive • Non­trivial effort to implement • Non­trivial effort to execute • Not intended for individuals . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  11. Needs for Self-assessment Important data in addition to benchmark metrics: • System statistics: • processor utilization • i/o throughput • etc. • Software profiling: • annotated source code • call graphs • flame charts • etc. . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  12. http://www.vldb.org/pvldb/vol7/p277-difallah.pdf OLTPBench In many cases, researchers and developers are limited to a small number of workloads to evaluate the performance characteristics of their work. This is due to the lack of a universal benchmarking infrastructure, and to the difficulty of gaining access to real data and workloads. This results in lots of unnecessary engineering efforts and makes the performance evaluation results difficult to compare. . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  13. Adapting benchmarks At least 12 open source projects already exist: • http://oltpbenchmark.com/ • https://github.com/tvondra/pg_tpch • https://github.com/Percona-Lab/sysbench-tpcc • https://sourceforge.net/projects/benchmarksql/ • http://jtpcc.sourceforge.net/ • https://www2.infor.uva.es/~diego/tpcc-uva.php • https://github.com/pivotalguru/TPC-DS • https://oss.oracle.com/projects/olt/ • https://www.hammerdb.com/ • https://github.com/2ndQuadrant/pg-tpch • http://osdldbt.sourceforge.net/ • https://github.com/jopereira/java-tpcw . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  14. http://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp Potential Licensing Issues While subject to change, EULA may not be agreeable. Excerpt from a previous TPC EULA: 4. Restrictions. The following restrictions apply to all use of the Materials by You. a. General: You may not: … iii. submit or contribute the Materials, or any part thereof, to any open source software project or as part of any third­party software or project without the express written consent of the TPC chair. . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  15. https://ycsb.site Quick Mention of Other Open Source Database Workloads Want to discuss more in unconference session? • Yahoo Cloud Serving Benchmark ­ key­value store • LDBC ­ graph databases http://ldbcouncil.org/ . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  16. Summarizing Needs for Self-Assessment Benchmarking kits are more helpful if they: • Collect system statistics • Profile the system • Unencumbered licensing . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  17. Where should we go from here? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  18. Develop Our Own Database Workloads Focusing on just two generate types of workload: • Online transaction processing (OLTP) • Multi­tier client­server • Electronic data processing, wholesaler supplier managing orders, brokerage firm executing customer transactions • Business Intelligence (BI) • Single system • Business oriented ah­doc queries, big data systems star­schema . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  19. Summary of Workload Architectures • OLTP ­ Complex: • multi­tier: develop driver, develop or use existing connection pooler • Scalability of driver • Server vs client side transaction logic • BI ­ Simple: • Single­tier • Scripts to execute queries and streams of queries. • Maybe we can outline a test in an unconference session? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  20. Is there a current need for competitive benchmarking? • Lots to consider: • Fair playing field • Enforcing fair play • etc. • Further discussion in an unconference session? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  21. A little fun: Compare a Java vs. C implementation of a TPC­C derived workload. . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  22. My hardware HP ZBook Studio G5 Mobile Workstation: • Intel(R) Core(TM) i7­8750H CPU @ 2.20GHz (6 cores / 12 threads) • 64 GB RAM • HP EX950 M.2 NVMe 2TB • Any hardware donations available? . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  23. Test Parameters OLTPBench vs DBT­2: • Scale factor: 1 warehouse • 10 terminals each with a database connection • No keying or thinking time • 6 Cores (SMT disabled) . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

  24. Results OLTPBench: DBT­2: • Metric 21,580.49 • Metric 28,838.20 new­order TPM new­order TPM • Processor cores 80% ­ • Processor cores 60% 90% utilized utilized • driver ­ 1 java process • driver ­ 76% processor 100% processor utilization utilization • i/o 99% utilized • i/o 97% utilized • 10 postgres backends 30% • 10 postgres backends 30% processor utilization each processor utilization each . . . . . . . . . . . . . . . . . . . . https://www. 2ndQuadrant.com . . . . . . . . . . . . . . . . . . . .

Recommend

BENCHMARKS TOPIC SUMMARY Scott Adams, Dilbert BENCHMARKS The Investment Process and how BM fits

BENCHMARKS TOPIC SUMMARY Scott Adams, Dilbert BENCHMARKS The Investment Process and how BM fits

BENCHMARKS TOPIC SUMMARY Scott Adams, Dilbert BENCHMARKS The Investment Process and how BM fits BM definition and why is it important? Uses for BMs Different types Who are the relevant parties? Properties of Ideal BM Examples Problems in

584 views • 22 slides
AGENDA 1. Review of Metro Hartford Future 2. Vision and Benchmarks 3. Goals 4. Key

AGENDA 1. Review of Metro Hartford Future 2. Vision and Benchmarks 3. Goals 4. Key

AGENDA 1. Review of Metro Hartford Future 2. Vision and Benchmarks 3. Goals 4. Key Indicators 5. Proposed Strategies 6. Strategy Refinement and Prioritization 2 What are We Building More Than a Plan Vision Situational

950 views • 55 slides
Benchmarks Online Testing Data District Benchmarks English/Language Arts and Math

Benchmarks Online Testing Data District Benchmarks English/Language Arts and Math

Benchmarks Online Testing Data District Benchmarks English/Language Arts and Math Benchmark A: Sept. 16-25 Benchmark B: Jan. 25-Feb. 5 Benchmark C: June 6-17 All ELA/Math benchmarks will be delivered online and in LinkIt!

266 views • 14 slides
Demystifying Benchmarks How to Use Them to Better Evaluate Databases Peter Friedenbach,

Demystifying Benchmarks How to Use Them to Better Evaluate Databases Peter Friedenbach,

Flexible transactional scale for the connected world. Demystifying Benchmarks How to Use Them to Better Evaluate Databases Peter Friedenbach, Performance Architect | Clustrix Demystifying Benchmarks Outline A Brief History of Database

591 views • 20 slides
Establishing Realistic Investment Earnings Benchmarks What is a Benchmark? A benchmark is a

Establishing Realistic Investment Earnings Benchmarks What is a Benchmark? A benchmark is a

Establishing Realistic Investment Earnings Benchmarks What is a Benchmark? A benchmark is a standard reference point that is used to compare the results of similar investments over time. Institutional investors use benchmarks to analyze

543 views • 39 slides
Multicore OS Benchmarks: We Can Do Better Ihor Kuz* , Zachary Anderson, Pravin Shinde, Timothy

Multicore OS Benchmarks: We Can Do Better Ihor Kuz* , Zachary Anderson, Pravin Shinde, Timothy

Multicore OS Benchmarks: We Can Do Better Ihor Kuz* , Zachary Anderson, Pravin Shinde, Timothy Roscoe Systems Group, ETH Zurich * NICTA Australia 1 Multicore OS benchmarks do not evaluate performance isolation between independent apps. 2

214 views • 10 slides
Building Blocks for Value-Based Payment N A H D O 2 0 1 9 H E A L T H C A R E D A T A S U M M

Building Blocks for Value-Based Payment N A H D O 2 0 1 9 H E A L T H C A R E D A T A S U M M

Medicare Price Benchmarks: Building Blocks for Value-Based Payment N A H D O 2 0 1 9 H E A L T H C A R E D A T A S U M M I T N O V E M B E R 6 , 2 0 1 9 L I T T L E R O C K , A R C H A P I N W H I T E S E N I O R P O L I C Y R E S

464 views • 29 slides
and Benchmarks May 24, 2018 Panelists Katy Miller Regional Coordinator Jasmine Hayes Deputy

and Benchmarks May 24, 2018 Panelists Katy Miller Regional Coordinator Jasmine Hayes Deputy

Federal Criteria and Benchmarks for Achieving the Goal of Ending Youth Homelessness Part 1: : Understanding the Crit iteria and Benchmarks May 24, 2018 Panelists Katy Miller Regional Coordinator Jasmine Hayes Deputy Director Katie

597 views • 32 slides
FOUNDA TION FINANCE COMMITTEE September 23, 2016 ULF Performance vs. Peers and Benchmarks 2

FOUNDA TION FINANCE COMMITTEE September 23, 2016 ULF Performance vs. Peers and Benchmarks 2

FOUNDA TION FINANCE COMMITTEE September 23, 2016 ULF Performance vs. Peers and Benchmarks 2 ULF Performance vs. Selected Benchmarks 3 Erroneous Comparisons UK 46% increase between 2006 and 2015 UK moved $150M of hospital funds

336 views • 30 slides
The HPC Challenge Benchmarks and the PMaC project Certificates of relevance for benchmarks

The HPC Challenge Benchmarks and the PMaC project Certificates of relevance for benchmarks

The HPC Challenge Benchmarks and the PMaC project Certificates of relevance for benchmarks Certificates of relevance for benchmarks Do they cover a useful performance space? Do they cover a useful performance space? Do they

525 views • 4 slides
Ev Evaluation Benchmarks and Learning Criteria fo for Di Discou ourse-Aw Aware Sente ntence

Ev Evaluation Benchmarks and Learning Criteria fo for Di Discou ourse-Aw Aware Sente ntence

Ev Evaluation Benchmarks and Learning Criteria fo for Di Discou ourse-Aw Aware Sente ntence Represent ntations Mingda Chen Joint work with Zewei Chu and Kevin Gimpel Prior work on evaluation benchmarks Focus on capabilities of

650 views • 39 slides
Development of AI has been driven by benchmarks and datasets. Computer Vision: (Russakovsky et

Development of AI has been driven by benchmarks and datasets. Computer Vision: (Russakovsky et

Adversarial NLI: A New Benchmark for Natural Language Understanding Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, Douwe Kiela UNC Chapel Hill & Facebook AI Research 1 Development of AI has been driven by benchmarks and

851 views • 45 slides
Lessons from the Tevatron and QCD/SM benchmarks for the LHC Re-discovering the SM at the

Lessons from the Tevatron and QCD/SM benchmarks for the LHC Re-discovering the SM at the

Lessons from the Tevatron and QCD/SM benchmarks for the LHC Re-discovering the SM at the LHC Joey Huston Michigan State University LISHEP 2006 Albert and I also make up the experimental CDF group at Durham. Tevatron by this

931 views • 82 slides
BUILDING ON FIRM FOUNDATIONS DELIVERING A SUSTAINABLE FUTURE ENHANCING OUTCOMES KEY FEATURES

BUILDING ON FIRM FOUNDATIONS DELIVERING A SUSTAINABLE FUTURE ENHANCING OUTCOMES KEY FEATURES

KUMBA IRON ORE LIMITED AUDITED ANNUAL RESULTS FOR THE YEAR ENDED 31 DECEMBER 2017 BUILDING ON FIRM FOUNDATIONS DELIVERING A SUSTAINABLE FUTURE ENHANCING OUTCOMES KEY FEATURES Material improvement IN ALL KEY SAFETY BENCHMARKS AND NO FATAL

610 views • 40 slides
Construction of Realistic Gate Construction of Realistic Gate Sizing Benchmarks Sizing

Construction of Realistic Gate Construction of Realistic Gate Sizing Benchmarks Sizing

Construction of Realistic Gate Construction of Realistic Gate Sizing Benchmarks Sizing Benchmarks With Known Optimal Solutions With Known Optimal Solutions Andrew B. Kahng, Seokhyeong Kang VLSI CAD LABORATORY, UC San Diego International

501 views • 26 slides
WCET Tool Challenge 2014 Outline 1. Objectives of the challenge 2. Benchmarks and problems 3.

WCET Tool Challenge 2014 Outline 1. Objectives of the challenge 2. Benchmarks and problems 3.

WCET Tool Challenge 2014 Outline 1. Objectives of the challenge 2. Benchmarks and problems 3. Participants 4. State of progress / next? Objectives a set of benchmarks and problems to: o compare the properties of different WCET tools

355 views • 20 slides
NPFL103: Information Retrieval (5) Ranking, Complete search system, Evaluation, Benchmarks Pavel

NPFL103: Information Retrieval (5) Ranking, Complete search system, Evaluation, Benchmarks Pavel

Ranking Complete search system Evaluation Benchmarks NPFL103: Information Retrieval (5) Ranking, Complete search system, Evaluation, Benchmarks Pavel Pecina Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics

995 views • 63 slides
What is SAT-Race? Small SAT-Competition Only industrial category benchmarks (no

What is SAT-Race? Small SAT-Competition Only industrial category benchmarks (no

What is SAT-Race? Small SAT-Competition Only industrial category benchmarks (no handcrafted or random) Short run-times (15 minutes timeout per instance) Mixture of satisfiable / unsatisfiable instances (thus not suitable for

355 views • 23 slides
Memcached Install, Overview & Benchmarks What is Memcached? Really, its Memcache-d (d as

Memcached Install, Overview & Benchmarks What is Memcached? Really, its Memcache-d (d as

Memcached Install, Overview & Benchmarks What is Memcached? Really, its Memcache-d (d as in daemon) TCP/IP service that allows you to store stuff in RAM, somewhat persistently Basically, a persistent hash table Super awesome

342 views • 13 slides
UQ Benchmark Problems for Multiphysics Modeling Maarten Arnst March 31, 2014 SIAMUQ UQ

UQ Benchmark Problems for Multiphysics Modeling Maarten Arnst March 31, 2014 SIAMUQ UQ

SIAMUQ UQ Challenge Benchmarks UQ Benchmark Problems for Multiphysics Modeling Maarten Arnst March 31, 2014 SIAMUQ UQ Challenge Benchmarks 1 / 25 Motivation Previous presentation at USNCCM2013 UQ Challenge Benchmarks:

751 views • 25 slides
Scrambling and Descrambling SMT-LIB Benchmarks Tjark Weber Uppsala University, Sweden SMT 2016

Scrambling and Descrambling SMT-LIB Benchmarks Tjark Weber Uppsala University, Sweden SMT 2016

Scrambling and Descrambling SMT-LIB Benchmarks Tjark Weber Uppsala University, Sweden SMT 2016 Coimbra, Portugal Tjark Weber Scrambling and Descrambling . . . 1 / 16 Motivation The benchmarks used in the SMT Competition are known in

843 views • 19 slides
Someone actually benchmarks public pension administration Founded in 1992 Started with

Someone actually benchmarks public pension administration Founded in 1992 Started with

Someone actually benchmarks public pension administration Founded in 1992 Started with investment management in Canada and the United States The current service includes: comprehensive benchmarking a network for questions

148 views • 10 slides
A PERFORMANCE COMPARISON USING HPC BENCHMARKS: WINDOWS HPC SERVER 2008 AND RED HAT ENTERPRISE

A PERFORMANCE COMPARISON USING HPC BENCHMARKS: WINDOWS HPC SERVER 2008 AND RED HAT ENTERPRISE

A PERFORMANCE COMPARISON USING HPC BENCHMARKS: WINDOWS HPC SERVER 2008 AND RED HAT ENTERPRISE LINUX 5 R. Henschel, S. Teige, H. Li, J. Doleschal , M. S. Mueller October 2010 Contents HPC at Indiana University FutureGrid

281 views • 27 slides
Information Visualization Computational benchmarks? domain situation who are the

Information Visualization Computational benchmarks? domain situation who are the

How to evaluate a visualization: So many methods, how to pick? Nested model: Four levels of visualization design Information Visualization Computational benchmarks? domain situation who are the target users? quant: system

186 views • 3 slides

More recommend