Summary of etcd benchmarks Exploring etcd as a potential back-end for - PowerPoint PPT Presentation

Summary of etcd benchmarks Exploring etcd as a potential back-end for the O 2 Configuration module Pascal Boeschoten 1 / 18

Outline Objective ● Test setup ● Results ● Next steps ● Notes ● 2 / 18

Objective ● Evaluate the performance of etcd as a store for configuration parameters – Using representative workloads – In “single-server” and “cluster” modes – With multiple parameter data structures 3 / 18

Test setup – Workload ● At start of run, thousands of processes retrieve parameters – Large burst of GET requests from clients – High load on server(s) for a short period Metric Value Number of client processes 7k to 70k Parameters per client process 100 Average size per parameter 100 bytes Total configuration data volume 700 MB Key workload estimations 4 / 18

Test setup – Parameter organization ● The structure of the data will vary per client process, thus various approaches were explored Name Description Query Fragmented Separate key-value pairs One GET per parameter Blob Key-value pairs stored in single blob One GET for whole blob Flat Key-value pairs grouped in one Recursive GET directory Tree Key-value pairs grouped in nested Recursive GET directories, structured like a binary tree, with five parameters per directory 5 / 18

Results – Single server ● Fragmented structure: low performance ● Others: much better Time to pull parameters, fragmented structure Time to pull parameters, varying structures 1 server, 43 client hosts, 16 processes per client 1 server, 42 client hosts 12 16 14 Time 10 12 8 Blob 10 Seconds Seconds Flat 6 8 Tree 4 6 4 2 2 0 100 ppc, 200 ppc, 0 0 200000 400000 600000 800000 7 Mp 14 Mp Total number of parameters Parameters per process, total parameters 6 / 18

Results – Cluster ● With higher server counts, performance quickly improves to acceptable levels (except for fragmented ) Time to pull parameters, various data organizations and cluster sizes 70k client processes across 42 hosts 16 Fragmented: Blob 14 13.7 s Flat 12 Tree (other results omitted 10 for chart readability...) 8 Seconds 6 4 2 0 7 Mp, 7 Mp, 7 Mp, 7 Mp, 7 Mp, 14 Mp, 14 Mp, 14 Mp, 14 Mp, 14 Mp, 1 srv 3 srv 5 srv 7 srv 9 srv 1 srv 3 srv 5 srv 7 srv 9 srv Millions of parameters, server count 7 / 18

Results – Cluster ● Roughly linear relationship between server count and performance → good! Time to pull 7M parameters as a function of 1/servers 70k client processes across 42 hosts 8 Blob 7 Flat 6 Tree Blob (avg/proc) 5 Flat (avg/proc) Seconds 4 Tree (avg/proc) 3 2 1 0 0 0.2 0.4 0.6 0.8 1 1.2 1 / servers 8 / 18

Results – JSON overhead ● Significant JSON response size overhead for flat and tree – Tree overhead is “exponential” ● ● A newer (not yet stable) etcd API uses Google Protocol Buffers for serialization – The binary format might reduce overhead Parameter Usable size JSON response Overhead structure (bytes) size (bytes) 100 blob 10000 10892 8.92 % 100 flat 10000 18098 81.0 % 100 tree 10000 21214 112 % 200 blob 20000 21796 8.92 % 200 flat 20000 36198 81.0 % 200 tree 20000 44214 121 % 9 / 18

Results – Timeouts ● Connection timeouts occur when server is overloaded – Harmful to performance with 1 server Of tested structures, – Practically disappear with >=3 servers 100 blob is “best case”, 200 tree “worst case” (not counting fragmented ) Frequency of amount of timeouts per process Frequency of amount of timeouts per process 7M parameters in "200 tree" structure, 42 clients, 9 servers, 42 clients, 70k processes, 70k processes, 10s timeout 250ms timeout 80000 50000 1 server 100 Blob 3 servers 40000 200 Tree 60000 5 servers Frequency Frequency 30000 7 servers 40000 9 servers 20000 20000 10000 0 0 0 1 2 3 0 1 2 3 4 5 Timeouts per process Timeouts per process 10 / 18

Results – Resource usage ● Not ideal, but OK – Large spikes – Overall underutilization Etcd server resource usage with "200 tree" structure 9 servers, 42 client hosts, 70k processes, 14M parameters 100 1000 CPU usage 80 800 Tx MB CPU usage (%) Rx MB Megabytes 60 600 40 400 20 200 0 0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 Seconds 11 / 18

Results – Resource usage (2) ● Spiking apparent even during very short runs ● Cause unknown ● But is tuning necessary? “Good enough” already? Etcd server resource usage with "200 blob" structure 9 servers, 42 client hosts, 70k processes, 14M parameters 100 1200 CPU usage 1000 80 Tx MB CPU usage (%) Rx MB 800 Megabytes 60 600 40 400 20 200 0 0 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Seconds 12 / 18

Results – Resource usage (3) ● Resource usage with fragmented structure: – Similar CPU, much lower transmission. Etcd server resource usage with "50 fragmented" structure 9 servers, 42 client hosts, 70k processes, 3.5M parameters CPU usage Tx MB Rx MB 100 50 90 45 80 40 70 35 CPU usage (%) 60 30 Megabytes 50 25 40 20 30 15 20 10 10 5 0 0 0 1 2 3 4 5 6 7 Time (seconds) 13 / 18

Next steps ● Investigate a structure in between fragmented and tree – Multiple recursive GETs per process ● Multiple etcd instances per physical server – May improve resource utilization ● Compare with other back-ends – Consul, Zookeeper? 14 / 18

Notes ● More details available attached to JIRA ticket: https://alice.its.cern.ch/jira/browse/OCONF-3 15 / 18

Reference slides 16 / 18

Machine specifications Hostname Specifications aidrefpc001/45 2x E5520, 8 threads, 2.26 GHz aidrefsrv10 2x E5-2650 v2, 16 threads, 2.6 GHz aido2qc10/13 2x E5-2640 v3, 16 threads, 2.6 GHz aido2qc40/43 aido2web1 17 / 18

JSON snippet Formatted snippet of a “200 tree” JSON response { "action":"get", "node":{ "key":"/test/tree100", "dir":true, "nodes":[ { "key":"/test/tree100/key3", "value":"value00...03", "modifiedIndex":103, "createdIndex":103 }, { "key":"/test/tree100/key4", "value":"value00...04", "modifiedIndex":104, "createdIndex":104 }, { "key":"/test/tree100/dirA", "dir":true, "nodes":[ { "key":"/test/tree100/dirA/key8", "value":"value00...08", "modifiedIndex":78, "createdIndex":78 }, ... 18 / 18