yes the size matters
play

Yes! The size matters! PostgreSQL implementation on a large - PowerPoint PPT Presentation

Sep 09, 2022 •579 likes •1.65k views

Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Yes! The size matters! PostgreSQL implementation on a large operational database,. . . Thomas BOUSSEKEY (@ThomasBoussekey) DECATHLON, retail IT

  1. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Size considerations Volume of Records Physical E-commerce POSLOG Length 12 kB 8 kB POSLOG per Tx 1 7 Days opened per year 325 365 Tx per year (M) 683.96 6.71 Poslog per year (M) 683.96 47.01 Tx = Transaction Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 28 / 95

  2. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Database size estimation for EUROPE only Transactions (M) DB Size (GB) Yearly growth (%) End 2017 80 350 2300 End 2018 480 2.000 570 End 2019 1.000 5.000 250 End 2020 1.600 8.000 160 Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 29 / 95

  3. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Industrialisation Implementing PostgreSQL puppet Brick Derivated from puppet forge Modified for multiple instance purposes. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 30 / 95

  4. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps PUPPET PGSQL Brick structure classes: dkt_postgresql: instances: cash: instance_port: 5432 pg_conf: [ config_options @ postgresql.conf ] pg_hba: [ hba_entries @ pg_hba.conf ] databases: [ database_config ] Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 31 / 95

  5. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 32 / 95

  6. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 33 / 95

  7. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 34 / 95

  8. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Risk management Spread data in multiple tables to limit risks, locking . . . Possible solutions: Design the application to store data for each pos in a different table. Partitionning manually Using extensions pg_partman pg_pathman Both extensions were easy to install and configure Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 35 / 95

  9. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Decision When we had to take a decision, pg_partman was chosen: Extension maturity (IN 2016 Q2): pg_partman was already “mature” pg_pathman was in version 0.4 had many stars on GITHUB “almost” compliant with our cloud DB managed service Project was up-to-date (regarding Pg releases) Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 36 / 95

  10. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Breaking news PostgreSQL10 Partitioning is now fully integrated into PostgreSQL 10! Native partitioning feature Link to PostgreSQL WIKI Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 37 / 95

  11. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Partitioning Build a partition key for point of sales 3 columns to define a Point of sale Type of TIER TIER number TIER sub-number Merge these 3 columns into a single column Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 38 / 95

  12. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Partitioning Build a partition key for point of sales 3 columns to define a Point of sale Type of TIER TIER number TIER sub-number Merge these 3 columns into a single column Huge interest for database statistics Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 38 / 95

  13. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Breaking news PostgreSQL 10 Cross-column Statistics feature is now AVAILABLE Link to PostgreSQL WIKI Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 39 / 95

  14. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Type of value for the partitioning key INTEGER format preserved, for: search efficiency ID partitioning VARCHAR might be: more difficult to handle less performant for searches and joining Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 40 / 95

  15. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Range (Exact-ID) partitionning Need to size partitioning range to 10 (minimum set by pg_partman extension) Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 41 / 95

  16. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Range (Exact-ID) partitionning Need to size partitioning range to 10 (minimum set by pg_partman extension) NEED Get a ‘human readable identifier‘ Create a ‘bigint‘, grouping the 3 fields and a final "0" Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 41 / 95

  17. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Range (Exact-ID) partitionning Need to size partitioning range to 10 (minimum set by pg_partman extension) NEED Get a ‘human readable identifier‘ Create a ‘bigint‘, grouping the 3 fields and a final "0" Example: 7 00118 00118 0 for: Type of tier: 7 , 2 digits (7, 48, 50) Tier number: 00118 , 5 possible digits Sub-tier number: 00118 , 5 possible digits Plus , the trailing 0 , for range partitioning Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 41 / 95

  18. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Build a partition key for daily retention Create 2 partitioned tables named transaction_live and poslog_live to store records and ease time-window searches. These partitions have a 30 day retention time. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 42 / 95

  19. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps All partitions Table Name partitioning Data purge specifity transaction BU NEVER none transaction_live day > 2 weeks none poslog BU NEVER XML poslog poslog_live day > 1 week none BU = Business Unit Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 43 / 95

  20. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps SQL commands Integer range partitioning SELECT partman.create_parent( p_parent_table := 'posdata.transaction', p_control := 'id_business_unit', p_type := 'id', p_interval := '10', p_premake := 1); Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 44 / 95

  21. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Timestamp daily partitioning SELECT partman.create_parent( p_parent_table := 'posdata.transaction_live', p_control := 'creation_date', p_type := 'time', p_interval := 'daily'); Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 45 / 95

  22. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Additionnal tooling For adding: point of sales new partitions for new days Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 46 / 95

  23. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 47 / 95

  24. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 48 / 95

  25. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 49 / 95

  26. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Updating configuration for pg_partman instances: cash: pg_conf: [ .../... ] max_locks_per_transaction: 512 max_pred_locks_per_transaction: 256 shared_preload_libraries: [ ... ] ,pg_partman_bgw databases: posdata: database_name: posdata extensions: pg_partman: ensure: present schema: partman [ .../... ] Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 50 / 95 [ .../... ]

  27. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Create partitions for new business units: On modification on the parameter table business_unit Fire the trigger business_unit_fct If the partitions doesn’t exist, they are created Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 51 / 95

  28. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Version1 GO PROD Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 52 / 95

  29. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 53 / 95

  30. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 54 / 95

  31. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 55 / 95

  32. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Why do we encounter IOWait , 6 months after live on production ? IOWAIT strong increase discoverd on Friday JUNE 23rd morning on Production . . Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 56 / 95 WHAT MAY HAPPEN THIS WEEK-END ???

  33. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Searching a ROOT-cause! Here are your 3 new friends: pg_buffercache pg_stat_statements EXPLAIN Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 57 / 95

  34. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Working with PG_BUFFERCACHE SELECT c.relname as "Name", count(*) as "buffers", round(count(*)*8.0/1024,2) AS Buffer_MB, round(count(*)*100.0/( SELECT count(*) FROM pg_buffercache),2) as "%" FROM pg_buffercache b INNER JOIN pg_class c ON b.relfilenode = pg_relation_filenode(c.oid) WHERE b.reldatabase IN (0, ( SELECT oid FROM pg_database WHERE datname = (current_database()))) GROUP BY c.relname ORDER BY 2 DESC LIMIT 50; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 58 / 95

  35. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Clue #1 Small amount of heap partitioned POSLOG were found into the BUFFERS Can be improved by index creation Which query needs to be indexed? Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 59 / 95

  36. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Working with PG_STAT_STATEMENTS Having a good sample Do not investigate on N-1 version queries Avoid DBA & Dev queries A small tip On Friday afternoon, Run a SELECT pg_stat_statements_reset(); On Monday morning, “Snapshot” pg_stat_statements table Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 60 / 95

  37. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Query used to focus on IOWait SELECT left (pss.query, 120), pss.calls, pss.total_time, pss.mean_time, pss.stddev_time, pss.shared_blks_hit, pss.shared_blks_dirtied from pg_stat_statements pss where pss.dbid = ( SELECT oid from pg_database where datname = current_database()) order by pss.total_time desc limit 80; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 61 / 95

  38. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps CULPRIT identified: SELECT COALESCE (max(p.version), 0) FROM posdata.poslog_p700346003460 p WHERE p.id_business_unit= 700346003460 AND p.id_transaction= 1234; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 62 / 95

  39. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Execution plan EXPLAIN ( ANALYZE , BUFFERS) SELECT COALESCE (max(p.version), 0 WHERE p.id_business_unit= 700346003460 AND p.id_transaction= Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 63 / 95

  40. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Execution plan EXPLAIN ( ANALYZE , BUFFERS) SELECT COALESCE (max(p.version), 0 WHERE p.id_business_unit= 700346003460 AND p.id_transaction= Aggregate (cost=8797.93..8797.94 rows=1 width=4) (actual time=67.086..67.087 Buffers: shared hit=**5504** -> **Seq Scan** on poslog_p700346003460 p (cost=0.00..8797.92 Filter: ((id_business_unit = '700346003460'::bigint) AND (id_transaction Rows Removed by Filter: 219595 Buffers: shared hit=5504 Planning time: 0.321 ms Execution time: **67.162 ms** Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 63 / 95

  41. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps How to improve? How to index it efficiently? B-Tree index BRIN index Introduced in PostgreSQL 9.5 Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 64 / 95

  42. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps BRIN index Definition BRIN stands for Block Range Index, default range is 1MB. BRIN is designed for handling very large tables in which certain columns have some natural correlation with their physical location within the table. Store MIN and MAX values for the related block. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 65 / 95

  43. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Test it before execution plan, once the BRIN index created: Aggregate (cost=16.03..16.04 rows=1 width=4) (actual time=0.116..0.1(10 Buffers: shared hit=**20** -> **Bitmap Heap Scan** on poslog_p700346003460 p (cost=12.01..16.03 Recheck Cond: ((id_business_unit = '700346003460'::bigint) AND Buffers: shared hit=20 -> **Bitmap Index Scan** on poslog_p700346003460_brin_idx01 Index Cond: ((id_business_unit = '700346003460'::bigint) AND Buffers: shared hit=20 Planning time: 0.406 ms Execution time: **0.198 ms** Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 66 / 95

  44. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Compare the 2 kinds of index Size (kB) Blocks read Heap 736.000 5.504 B-Tree INDEX 8.738 6 BRIN INDEX 48 20 BRIN index needs less IO on INSERT. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 67 / 95

  45. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Industrialize index creation CREATE OR REPLACE FUNCTION add_brin_index_on_poslog_tables() DECLARE live_part_table RECORD; BEGIN RAISE NOTICE 'Creating BRIN indexes on poslog partitions FOR live_part_table IN SELECT tab.relname as "tab_relname" RAISE NOTICE 'Updating constraint for table % ...', EXECUTE 'CREATE INDEX IF NOT EXISTS ' || quote_ident(live_part_table.tab_relname) --- CREATE INDEX CONCURRENTLY IF NOT EXISTS poslog_p700346003460_brin_idx01 END LOOP ; RAISE NOTICE 'Done creating BRIN indexes on poslog partitions RETURN 0; END ; $$ LANGUAGE plpgsql; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 68 / 95

  46. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Other thing, I’ve learned at this point PL/pgsql LANGUAGE doesn’t allow: VACUUM inside a transaction block. CREATE INDEX CONCURRENTLY inside a transaction block. COMMIT / ROLLBACK inside a transaction block. Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 69 / 95

  47. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Oooopps!!! new IOWait peak several weeks after !!! RTFM! Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 70 / 95

  48. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Oooopps!!! new IOWait peak several weeks after !!! RTFM! BRIN index maintenance: Tuples on a new heap page are not summarized into the BRIN index automatically Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 70 / 95

  49. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Oooopps!!! new IOWait peak several weeks after !!! RTFM! BRIN index maintenance: Tuples on a new heap page are not summarized into the BRIN index automatically 2 possible ways to update the BRIN index: Using procedure: brin_summarize_new_values(regclass) Perform a VACUUM on the table. a a . https://www.postgresql.org/docs/9.5/static/brin-intro.html Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 70 / 95

  50. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 71 / 95

  51. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Solution CREATE OR REPLACE FUNCTION refresh_brin_index_on_poslog_tables() DECLARE live_part_table RECORD; BEGIN FOR live_part_table IN SELECT tab.relname as "tab_relname" RAISE NOTICE 'Refreshing BRIN index for table % ...', quote_ident(live_part_table.tab_relname); EXECUTE 'SELECT brin_summarize_new_values(''' || quote_ident(live_part_table.ind_relname) END LOOP ; RAISE NOTICE 'Done refreshing BRIN indexes on poslog partitions RETURN 0; END ; $$ LANGUAGE plpgsql; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 72 / 95

  52. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Emergency DDL update Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 73 / 95

  53. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Possible solutions 1 Create a new table for ReceiptNumber 2 Add a new column for ReceiptNumber into Poslog table Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 74 / 95

  54. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Impact analysis Tested on integration, on ~300k rows Size impact (MB) + Column + Table Delta (%) Poslog HEAP 13.75 0 Poslog INDEX 0 0 Receipt HEAP 0 31.79 Receipt INDEX 0 7.69 Total size 13.75 39.48 +287 Insert 75k rows (IO) 60,526 62,124 +3 Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 75 / 95

  55. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Time impact (ms) + Column + Table Delta (%) Creation time 40 114162 Time for insertion 99236 0 Vacuum time 30981 0 Total 130257 114162 +14/-14 Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 76 / 95

  56. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Improve XML storage Try on JSONB, 7% space reduced Offers full text search improvement Quick look over XMLTABLE 2 feature on PostgreSQL 10 2. https://blog.2ndquadrant.com/xmltable-intro/ Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 77 / 95

  57. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 78 / 95

  58. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Cloud Image from [Chris Potter](www.seywut.com/Chris). Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 79 / 95

  59. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Technology State of the art 2016Q1 2017Q3 Location onPremise Cloud OS RHEL 7 CentOS 7 Pg version 9.5 10 OR Postgres_XL Config mgmt Puppet V4 Foreman Terraform + foreman SQL scripts Flyway Flyway Deployment JENKINS JENKINS Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 80 / 95

  60. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps New hosting solution Leave onPremise datacenters to go to the cloud. Improve industrialization Start contenerization Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 81 / 95

  61. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Terraform Infrastructure as code Ability to perform operations on multiple hosting solutions cloud computing onPremise datacenters Interact with FOREMAN & PUPPET Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 82 / 95

  62. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Postgres_XL Key drivers Critical size estimated to 1 database for 200 databases. Needing 7 different databases for production (regarding DB size) Adding complex platform management to spread and search data accross the platforms Scalable ALTER NODE data_5 WITH ( TYPE = 'datanode'); We can define the sharding structure for EACH table ALTER TABLE disttab ADD NODE (data_5); Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 83 / 95

  63. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 84 / 95

  64. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Features Fully ACID Open Source Cluster-wide Consistency Multi-tenant Security PostgreSQL-based Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 85 / 95

  65. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Pros No new engine Only a few system tables & columns difference Table location strategy DISTRIBUTED: for large table, align distribution startegy for joins REPLICATED: for parameter tables SELECT xc_node_id, count(*) FROM transaction GROUP BY xc_node_id; Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 86 / 95

  66. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Pros No new engine Only a few system tables & columns difference Table location strategy DISTRIBUTED: for large table, align distribution startegy for joins REPLICATED: for parameter tables SELECT xc_node_id, count(*) FROM transaction GROUP BY xc_node_id; Cons Prevent cross datanodes queries Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 86 / 95

  67. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Poslog storage Stored in raw mode No specific indexing on its content Only field to contain personal information Can be stored into a cloud file storage solution with encryption, to comply with GDPR regulation Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 87 / 95

  68. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Other new functionnalities / tools Improved monitoring TICK ELK Use containers for JAVA application New flow solution Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 88 / 95

  69. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Thanks for your attention! Any question? Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 89 / 95

  70. Introduction: The context Building Version 1 Version1 GO PROD Emergency DDL update Next steps Created with opensource tools: markdown pandoc laTEX beamer and PostgreSQL Thomas BOUSSEKEY (@ThomasBoussekey) Yes! The size matters! 90 / 95

Recommend

Step Size Matters in Deep Learning Kamil Nar Shankar Sastry Neural Information Processing

Step Size Matters in Deep Learning Kamil Nar Shankar Sastry Neural Information Processing

Step Size Matters in Deep Learning Kamil Nar Shankar Sastry Neural Information Processing Systems December 4, 2018 Nar & Sastry Step Size Matters 1 Gradient Descent: Effect of Step Size Example min x R ( x 2 + 1)( x 1) 2 ( x

753 views • 18 slides
Size Matters How Loading is Losing You Players W3C Workshop on Web Games - 27-28 June 2019;

Size Matters How Loading is Losing You Players W3C Workshop on Web Games - 27-28 June 2019;

Size Matters How Loading is Losing You Players W3C Workshop on Web Games - 27-28 June 2019; Redmond, WA, USA Kasper Mol Game Tech Engineer @ Poki Poki - Web Gaming Platform 30M 200+ 28 Languages Monthly Active Game Developers Users SIZE

624 views • 14 slides
I . Preliminaries: practical matters I . Preliminaries: practical matters A. Office

I . Preliminaries: practical matters I . Preliminaries: practical matters A. Office

Krol Lear Knig Lear Krol Lear King Lear King Lear King Lear W. Shakespeare Shakespeare Shakespeare Wieslaw Walkuski Jerzy Czerniawski Rafal Olbinski theater; 1992; size B1 theater; 1981; size A1 theater; 2000; size B1 Polish Poster

677 views • 10 slides
Qtopia for KDE developers 07 | 07 That Size-and-matters-thingy Remember that PC you had 5 years

Qtopia for KDE developers 07 | 07 That Size-and-matters-thingy Remember that PC you had 5 years

Qtopia for KDE developers 07 | 07 That Size-and-matters-thingy Remember that PC you had 5 years ago? That hardware is in your phone now 2 Screens :) 3 So what is it? Qtopia contains... Qtopia Core a.k.a Qt/embedded - same Qt API as

325 views • 16 slides
Hardware Operators for Pairing-Based Cryptography Part I: Because size matters Jean-Luc

Hardware Operators for Pairing-Based Cryptography Part I: Because size matters Jean-Luc

Hardware Operators for Pairing-Based Cryptography Part I: Because size matters Jean-Luc Beuchat Laboratory of Cryptography and Informantion Security University of Tsukuba, Japan jeanluc.beuchat@gmail.com Joint work with: enaire, LIP,

1.73k views • 139 slides
2016 UPDATE ON CLASS SIZE, OVERCROWDING, AND CAPITAL PLAN: P .S. 39, DISTRICT 15 AND CITYWIDE

2016 UPDATE ON CLASS SIZE, OVERCROWDING, AND CAPITAL PLAN: P .S. 39, DISTRICT 15 AND CITYWIDE

2016 UPDATE ON CLASS SIZE, OVERCROWDING, AND CAPITAL PLAN: P .S. 39, DISTRICT 15 AND CITYWIDE Leonie Haimson for P.S. 39K PTA Class Size Matters March 31, 2016 Class Size reduction: a proven reform Lowering class size is only of only four

432 views • 24 slides
Size Matters, If You Control Your Junk Clifford S. Asness AQR Capital Management LLC Andrea

Size Matters, If You Control Your Junk Clifford S. Asness AQR Capital Management LLC Andrea

Size Matters, If You Control Your Junk Clifford S. Asness AQR Capital Management LLC Andrea Frazzini AQR Capital Management LLC, NYU Ronen Israel AQR Capital Management LLC, NYU Tobias Moskowitz AQR Capital Management, University of Chicago,

441 views • 31 slides
URBAN FRIENDLY BIOMASS @ SCALE Size Matters Scott Layne, CEM Nexterra System Corp.

URBAN FRIENDLY BIOMASS @ SCALE Size Matters Scott Layne, CEM Nexterra System Corp.

Advanced Energy Conference October 2011 URBAN FRIENDLY BIOMASS @ SCALE Size Matters Scott Layne, CEM Nexterra System Corp. slayne@nexterra.ca US Renewables Demand The EIA Annual Energy Outlook for 2010 projects that the demand

157 views • 15 slides
When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really

When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really

When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters When (Low ) Pow er Really Matters Yogesh K. Ramadass, AnanthaGroup Microsystems Technology Laboratory Outline Outline Outline Outline

507 views • 36 slides
Lab 2 discussion Last Time Debugging Its a science use experiments to refine

Lab 2 discussion Last Time Debugging Its a science use experiments to refine

size= 64 correct= 0 size= 73 correct= 0 Shuying Liang size= 107 correct= 1 size= 107 correct= 0 size= 108 correct= 0 size= 108 correct= 1 Please do not handin a .doc file, a .zip file, a .tar file, size= 111 correct= 1 size=

430 views • 7 slides
The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions! joint work with

The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions! joint work with

The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions! joint work with Yachong Yang and Weijie Su Hua Wang The Wharton School, University of Pennsylvania June 2, 2020 Hua Wang (Wharton) The Price of Competition June

1.06k views • 66 slides
Measuring What Matters Quality, Impact and Measuring Social Value Philip Angier, Angier Griffin

Measuring What Matters Quality, Impact and Measuring Social Value Philip Angier, Angier Griffin

Measuring What Matters Quality, Impact and Measuring Social Value Philip Angier, Angier Griffin Measuring what matters Introductions: Name Organisation Where youre from first part of postcode Size approx annual

583 views • 27 slides
Titus 1:5-9 It matters who you are as a leader and it matters what you do as a leader. God is

Titus 1:5-9 It matters who you are as a leader and it matters what you do as a leader. God is

Leadership Matters (Verb) I Titus 1:5-9 It matters who you are as a leader and it matters what you do as a leader. God is primarily interested in who you are and then what you do (Daniel Akin). The Pastoral Crucible I.

600 views • 23 slides
Lumber Size Lumber Size Control Control Studies Studies Lumber Size Control Lumber Size

Lumber Size Lumber Size Control Control Studies Studies Lumber Size Control Lumber Size

Lumber Size Lumber Size Control Control Studies Studies Lumber Size Control Lumber Size Control Troubleshoot machine center performance Locate & identify problems Stabilize & improve sawing accuracy Reduce

686 views • 16 slides
S et the Bar Low. Be a WINNER every time. Public Power Matters Public Power Matters Innovation

S et the Bar Low. Be a WINNER every time. Public Power Matters Public Power Matters Innovation

S et the Bar Low. Be a WINNER every time. Public Power Matters Public Power Matters Innovation Workforce Advocacy Y our S tory Rates Matter Matters Matters Matters Matters Its the Rates, S tupid! Innovation Matters We must

280 views • 26 slides
FUNCTION MERGING by SEQUENCE ALIGNMENT DOES SIZE MATTER? DOES SIZE MATTER? S E Y DOES SIZE

FUNCTION MERGING by SEQUENCE ALIGNMENT DOES SIZE MATTER? DOES SIZE MATTER? S E Y DOES SIZE

Rodrigo Rocha http://rcor.me University of Edinburgh FUNCTION MERGING by SEQUENCE ALIGNMENT DOES SIZE MATTER? DOES SIZE MATTER? S E Y DOES SIZE MATTER? S E Y ... EMBEDDED SYSTEMS SAVINGS IN CODE Smaller Memories 1 0 1 1 0 1

1.89k views • 177 slides
+ + Technical tax update Various issues covered FRS 102 tax matters Practical matters

+ + Technical tax update Various issues covered FRS 102 tax matters Practical matters

+ + Technical tax update Various issues covered FRS 102 tax matters Practical matters re Pay and file tax deadline 2014 Research and development tax credits 1 + FRS 102 Tax matters The technical side Chapter 29 FRS 102

285 views • 26 slides
Sample Size Power, Sample Size, and the FDR How many observations do we need? Depends on

Sample Size Power, Sample Size, and the FDR How many observations do we need? Depends on

Sample Size Power, Sample Size, and the FDR How many observations do we need? Depends on Peter Dalgaard Design Standard error of measurements Department of Biostatistics Effect size University of Copenhagen How sure you

411 views • 3 slides
SAMPLE SIZE IN TRIAXIAL LOADS How sample size affects the frictional behavior Photo by H.

SAMPLE SIZE IN TRIAXIAL LOADS How sample size affects the frictional behavior Photo by H.

SAMPLE SIZE IN TRIAXIAL LOADS How sample size affects the frictional behavior Photo by H. Roshan et al. The rate of brittle-ductile transition varies based on STUDY sample size OVERVIEW The sample size influences the angle of the

276 views • 9 slides
Size based indicators rjan stman Dept. Aquatic Resources, regrund SLU What fish size

Size based indicators rjan stman Dept. Aquatic Resources, regrund SLU What fish size

Size based indicators rjan stman Dept. Aquatic Resources, regrund SLU What fish size indicate? Diet size dependend the ecological function of a stock/community depends on size distributions High mortality or selective

298 views • 15 slides
Estimating web size and search engine index size Near-duplicate document detection Size of the

Estimating web size and search engine index size Near-duplicate document detection Size of the

CS490W Web Search (I I ) Luo Si Department of Computer Science Purdue University Modified Slides from Manning, C., Raghavan, P. and Schtze, H. Todays topics Estimating web size and search engine index size Near-duplicate document

193 views • 16 slides
20 January 2017 1 Purpose Where Every Child Matters, Every Staff Matters Parents to know

20 January 2017 1 Purpose Where Every Child Matters, Every Staff Matters Parents to know

Where Every Child Matters, Every Staff Matters Meet-the-Parents Session 20 January 2017 1 Purpose Where Every Child Matters, Every Staff Matters Parents to know their childs teachers Teachers to share expectations and class

481 views • 27 slides
PROCUREMENT IN ANGOLA Why it Matters and How to Achieve it? 1. WHY IT MATTERS PUBLIC

PROCUREMENT IN ANGOLA Why it Matters and How to Achieve it? 1. WHY IT MATTERS PUBLIC

TRANSPARENCY IN INFRASTRUCTURE PROCUREMENT IN ANGOLA Why it Matters and How to Achieve it? 1. WHY IT MATTERS PUBLIC INVESTMENT INSTITUTIONS MATTER IMF has identified 15 institutions that are of vital importance to strengthen efficiency of

710 views • 12 slides
Inf nfluenz uenza Vi a Virol ologi ogic S Sur urveillan ance Rig Right Size Size Pr

Inf nfluenz uenza Vi a Virol ologi ogic S Sur urveillan ance Rig Right Size Size Pr

Inf nfluenz uenza Vi a Virol ologi ogic S Sur urveillan ance Rig Right Size Size Pr Proje ject Launc aunch July ly 20 2013 13 1 Todays Discussion Outline General Influenza Surveillance Right Size Rationale Roadmap

453 views • 42 slides

More recommend