Hidden Scalability Gotchas Gotchas Hidden Scalability in Memcached - - PowerPoint PPT Presentation

▶
hidden scalability gotchas gotchas hidden scalability in
SMART_READER_LITE
LIVE PREVIEW

Hidden Scalability Gotchas Gotchas Hidden Scalability in Memcached - - PowerPoint PPT Presentation

Jan 21, 2023 384 likes •763 views

Hidden Scalability Gotchas Gotchas Hidden Scalability in Memcached Memcached and Friends and Friends in Neil Gunther Gunther , , Performance Dynamics Performance Dynamics Neil Shanti Subramanyam , , Oracle Corp Oracle Corp oration oration

slide-1
SLIDE 1

Hidden Scalability Hidden Scalability Gotchas Gotchas in in Memcached Memcached and Friends and Friends

Neil Neil Gunther Gunther, , Performance Dynamics Performance Dynamics Shanti Subramanyam Shanti Subramanyam, , Oracle Corp Oracle Corporation

  • ration

Stefan Stefan Parvu Parvu, , Oracle Finland Oracle Finland

Velocity 2010 Web Performance and Operations Conference

slide-2
SLIDE 2

Velocity 2010, June 24 Velocity 2010, June 24 2 2

Scalability Scalability

slide-3
SLIDE 3

Velocity 2010, June 24 Velocity 2010, June 24 3 3

Memcached Memcached scale out scale out

  • Tier of older servers
  • Mostly single CPU
  • Single threading ok
slide-4
SLIDE 4

Velocity 2010, June 24 Velocity 2010, June 24 4 4

Scalability Strategies Scalability Strategies

  • Qualitative scalability

– Scale up, e.g., big SMP servers – Scale out,e.g, many cheap servers (Unis)

  • Quantitative scalability

– What this talk is about – Need controlled measurements – Need numbers to see cost-benefit

slide-5
SLIDE 5

Velocity 2010, June 24 Velocity 2010, June 24 5 5

Been Bad for Web 2.0 Been Bad for Web 2.0 Lately Lately

slide-6
SLIDE 6

Velocity 2010, June 24 Velocity 2010, June 24 6 6

Capacity Planning Capacity Planning

  • You know you need it

– The planning bit, especially – Data ain’t information – Info is hidden in the data

  • Just like finance, you need a model

Metrics + Models == Information

slide-7
SLIDE 7

Controlled Controlled Measurements Measurements

slide-8
SLIDE 8

Velocity 2010, June 24 Velocity 2010, June 24 8 8

Why Controlled Measurements? Why Controlled Measurements?

Trying to predict scalability by looking at time series data is like trying to predict the stock mkt by watching the DJX ticker

slide-9
SLIDE 9

Velocity 2010, June 24 Velocity 2010, June 24 9 9

Bad Throughput Bad Throughput Measurements Measurements

Need x-axis to be load (N) defined in terms of processes or users Need throughput measured in steady state (which this isn’t)

slide-10
SLIDE 10

Velocity 2010, June 24 Velocity 2010, June 24 10 10

Average Throughput in Time Average Throughput in Time

This is what steady state looks like as function of time. It corresponds to ONE throughput load point (N).

slide-11
SLIDE 11

Velocity 2010, June 24 Velocity 2010, June 24 11 11

Controlled MCD Tests Controlled MCD Tests

Load Drivers 2 Sun Fire X4170 2 sockets, 64 GB

SUT

Memcached Sun Fire X4170 2 sockets, 64 GB 10 Gbe Switch

slide-12
SLIDE 12

Velocity 2010, June 24 Velocity 2010, June 24 12 12

Memcached Memcached scaling is thread limited scaling is thread limited

slide-13
SLIDE 13

Velocity 2010, June 24 Velocity 2010, June 24 13 13

Better on SPARC Better on SPARC Multicore Multicore

slide-14
SLIDE 14

Quantifying Scalability Quantifying Scalability

Universal Universal Scalability Law Scalability Law USL USL

slide-15
SLIDE 15

Velocity 2010, June 24 Velocity 2010, June 24 15 15

  • 1. Equal Bang for The Buck
  • 1. Equal Bang for The Buck

Capacity Load

Ideal parallelism

slide-16
SLIDE 16

Velocity 2010, June 24 Velocity 2010, June 24 16 16

  • 2. Cost of Sharing Resources
  • 2. Cost of Sharing Resources

Capacity Load

slide-17
SLIDE 17

Velocity 2010, June 24 Velocity 2010, June 24 17 17

  • 3. Resource Limitation
  • 3. Resource Limitation

Capacity Load

Amdahl’s law

slide-18
SLIDE 18

Velocity 2010, June 24 Velocity 2010, June 24 18 18

  • 4. Degradation Negative Return
  • 4. Degradation Negative Return

Load Capacity

slide-19
SLIDE 19

Velocity 2010, June 24 Velocity 2010, June 24 19 19

Universal Scalability Law (USL) Universal Scalability Law (USL)

C(N) = N 1+ (N 1) + N(N 1)

Concurrency α = 0, β = 0 Contention α > 0, β = 0 Coherency α > 0, β > 0

slide-20
SLIDE 20

Velocity 2010, June 24 Velocity 2010, June 24 20 20

USL regression in Excel USL regression in Excel

A miracle happens …

slide-21
SLIDE 21

Memcached Memcached Scalability Scalability

Quantitative USL Analysis Quantitative USL Analysis

slide-22
SLIDE 22

Velocity 2010, June 24 Velocity 2010, June 24 22 22

Scalability of Scalability of mcd mcd 1.2.8 1.2.8

Nmax = 7 α = 0.0255, β = 0.0210

USL curve (not Excel)

slide-23
SLIDE 23

Velocity 2010, June 24 Velocity 2010, June 24 23 23

Scalability of Scalability of mcd mcd 1.4.1 1.4.1

Nmax = 6 α = 0.0821, β = 0.0207

USL curve (not Excel)

slide-24
SLIDE 24

Velocity 2010, June 24 Velocity 2010, June 24 24 24

Scalability of Scalability of mcd mcd 1.4.5 1.4.5

Nmax = 6 α = 0.0988, β = 0.0209

USL curve (not Excel)

slide-25
SLIDE 25

Velocity 2010, June 24 Velocity 2010, June 24 25 25

Scalability of SPARC version Scalability of SPARC version

Nmax = 22 α = 0.0041, β = 0.00197 α = 0, β = 0.000434

USL curves (not Excel)

slide-26
SLIDE 26

Velocity 2010, June 24 Velocity 2010, June 24 26 26

USL projected scalability USL projected scalability

α = 0.0041, β = 0.00197 Nmax = 22 Nmax = 48 α = 0, β = 0.000434

USL curves (not Excel)

slide-27
SLIDE 27

Velocity 2010, June 24 Velocity 2010, June 24 27 27

Parameter interpretation Parameter interpretation

  • Why α ~ 0

– Cache further partitioned – Single lock replaced by multiple locks

  • Why β > 0?

– Is it in mcd code? – Could it be in O/S, H/W, …?

slide-28
SLIDE 28

Scaling Among Friends Scaling Among Friends

Scalability as a function of Scalability as a function of virtual virtual users users ( (“ “friends friends” ”) not threads ) not threads

slide-29
SLIDE 29

Velocity 2010, June 24 Velocity 2010, June 24 29 29

JAppServer JAppServer USL Analysis USL Analysis

N = 700 users α = 0.00001486 β = 6.7E-9 N = 1200 users α = 0 β = 2.4E-7

USL curves (not Excel)

slide-30
SLIDE 30

Velocity 2010, June 24 Velocity 2010, June 24 30 30

Scalability on Amazon EC2 Scalability on Amazon EC2

Nmax = 22 α = 0.038988298 β = 0.001432176

USL curve (not Excel)

slide-31
SLIDE 31

Memcached Gotchas Memcached Gotchas

slide-32
SLIDE 32

Velocity 2010, June 24 Velocity 2010, June 24 32 32

Just throw more hardware at Just throw more hardware at it! it!

slide-33
SLIDE 33

Velocity 2010, June 24 Velocity 2010, June 24 33 33

Old scaling rules will be broken Old scaling rules will be broken

  • Current scale-out strategy relies on using
  • lder cheap hardware
  • Older hardware is often single CPU

– Single-threadedness of mcd is ok

  • Newer hardware will be multicore

– New hardware is faster with lots of cores – But mcd won’t be able to utilize all cores – Multiple mcd instances are mgmt headache

slide-34
SLIDE 34

Velocity 2010, June 24 Velocity 2010, June 24 34 34

Single threading can wreck you Single threading can wreck you

slide-35
SLIDE 35

Velocity 2010, June 24 Velocity 2010, June 24 35 35

Summary Summary

  • Current mcd versions are thread limited

– OK for older uniprocessor servers – Not OK for deployment on new multicores – Reason: unused processor capacity costs money

  • Controlled measurements

– Not time-series data from prod (but maybe can work) – Steady state throughput (or pick small prod window)

  • Quantify scalability

– Metrics + Models == Information – Goal is to reduce contention (α) and coherency (β) – Nmax in mcd: Increased from 6 to 48 threads

slide-36
SLIDE 36

Velocity 2010, June 24 Velocity 2010, June 24 36 36

Resources Resources

  • Neil
  • perfdynamics.blogspot.com
  • twitter.com/DrQz
  • www.perfdynamics.com/books.html
  • www.perfdynamics.com/Manifesto/USLscalability.html
  • Shanti

– perfwork.wordpress.com – twitter.com/shantiS

  • Stefan

– www.systemdatarecorder.org – twitter.com/sperformance