Coor dinat ed Omission in NoSQ L Dat abase Benchmar king St ef f en - PowerPoint PPT Presentation

Coor dinat ed Omission in NoSQ L Dat abase Benchmar king St ef f en Fr iedr ich, W o lf r am W inger at h, No r ber t R it t er U niver sit y o f Hambur g Depar t ment o f Inf o r mat ics Dat abases and Inf o r mat io n Syst ems M ar ch 6, 2 0 1 7

NoSQ L Per f or mance Evaluat ion ? Co o per et al.: Benchmar king Clo ud Ser ving Syst ems w it h YCSB, So CC‘1 0 , ACM , 2 0 1 0 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2

YCSBs load gener at ion _targetOpsTickNanos = (long) (1 000 000 000 / target) long overallStartTime = System.nanoTime(); while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); long endTime = System.nanoTime(); _measurements.measure("READ", (int)( (endTime - startTime) / 1000)); _opsdone++; long deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } } Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 3

YCSBs load gener at ion _targetOpsTickNanos = (long) (1 000 000 000 / targetThroughput) long overallStartTime = System.nanoTime(); while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); long endTime = System.nanoTime(); _measurements.measure("READ", (int)( (endTime - startTime) / 1000)); _opsdone++; long deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } } Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 4

YCSBs load gener at ion _targetOpsTickNanos = (long) (1 000 000 000 / targetThroughput) long overallStartTime = System.nanoTime(); while (_opsdone < _opcount) { long startTime = System.nanoTime(); W hat if Status status = _db.read( table, key, fields, result ); long endTime = System.nanoTime(); lat ency > _ t ar get OpsTickNanos ? _measurements.measure("READ", (int)( (endTime - startTime) / 1000)); now > deadline ? _opsdone++; long deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } } Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 5

The Coor dinat ed Omission Pr oblem “a conspir acy w e’r e all a par t of ” Gil Tene, Azul Syst ems: Ho w NOT t o M easur e Lat ency, Q Co n, 2 0 1 3 - 2 0 1 6 infoq.com/presentations/latency-response-time Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 6

The Coor dinat ed Omission Pr oblem E xample I Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 7

The Coor dinat ed Omission Pr oblem E xample II Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 8

The Coor dinat ed Omission Pr oblem E xample III Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 9

The Coor dinat ed Omission Pr oblem E xample III Dat abase is able t o inf luence t he r eq uest r at e ! = > coor dinat ed omission of r elevant measur ement s Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 0

The Coor dinat ed Omission Pr oblem E xample The R esult s: Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 1

The Coor dinat ed Omission Pr oblem R eal W o r ld GC E E vent ent Char t r t 80 -1 2 0 seco nds GC pauses @ Net f lix Cassandr a Clust er 2 0 1 2 ht t p:/ / t echblog.net f lix.com/ 2 0 1 3/ 0 5 / gar bage-collect ion-visualizat ion.ht ml Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 2

The Coor dinat ed Omission Pr oblem R eal W o r ld GC E E vent ent Char t r t Af t er f ixing t he pr o blem: max 5 seco nds GC pauses ht t p:/ / t echblog.net f lix.com/ 2 0 1 3/ 0 5 / gar bage-collect ion-visualizat ion.ht ml Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 3

Closed VS. Open Syst em M odel Clo sed Syst em M o del Open Syst em M o del Schr ö der et al. Open Ver sus Clo sed: A Caut io nar y Tale, 2 0 0 6 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 4

Closed VS. Open Syst em M odel Schr ö der et al. sur veyed syst em models in var ious w eb r elat ed w or kload gener at or s: • “M ost of t hese gener at or s/ benchmar ks assume a closed syst em model” • “For many of t hese w or kload gener at or s, it w as q uit e dif f icult t o f igur e out w hich syst em model w as being assumed” • “Pr inciple (i): For a given load, mean r esponse t imes ar e signif icant ly low er in closed syst ems t han in open syst ems.” Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 5

Coor dinat ed Omission Cor r ect ion since YCSB Ver sio n 0 .2 .0 R C 1 , June 2 0 1 5 while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); = > int ended measur ement int er val long endTime = System.nanoTime(); _measurements.measure("INTENDED_READ", (int)( (endTime - _deadline) / 1000)); _opsdone++; _deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); … while((now = System.nanoTime()) < deadline) { LockSupport.parkNanos( deadline – now ); } } Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 6

Coor dinat ed Omission Cor r ect ion since YCSB Ver sio n 0 .2 .0 R C 1 , June 2 0 1 5 while (_opsdone < _opcount) { long startTime = System.nanoTime(); Status status = _db.read( table, key, fields, result ); = > int ended measur ement int er val long endTime = System.nanoTime(); _measurements.measure("INTENDED_READ", (int)( (endTime - _deadline) / 1000)); _opsdone++; _deadline = overallStartTime + _opsdone * _targetOpsTickNanos; long now = System.nanoTime(); while((now = System.nanoTime()) < deadline) { = > but st ill inf luences t he r eq uest r at e ! LockSupport.parkNanos( deadline – now ); } } Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 7

Scalable NoSQ L- Benchmar king nosqlmark.informatik.uni-hamburg.de Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 8

Scalable NoSQ L- Benchmar king nosqlmark.informatik.uni-hamburg.de • Scaling YCSB compat ible w or kloads t o mult iple benchmar king nodes = > Aut omat ically aggr egat ed r esult s • Compat ible t o YCSB dat abase int er f ace layer Closed and Open Syst em M odel • Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 1 9

Coor dinat ed Omission Avoidance in No SQ LM ar k implicit val ec = context.system.dispatchers.lookup("blocking-io-dispatcher") case DoOperation => { val operation = workload.nextOperation val startTime = System.nanoTime val future = Future { sendRequest(operation) asynchr onous load } future.onComplete { gener at ion ! case Success(status) => { val endTime = System.nanoTime measurementActor ! Measure(operation.name, (endTime - startTime) / 1000) } case Failure(ex) => { log.error(ex, "Error occured during operation {}", operation.name) } } … Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 0

Coor dinat ed Omission Validat ion w it h Single-node inconsist ent key-value St or e Or iginally developed t o validat e consit ency measur ement appr oaches • consist ent single-node backend • mult i-node behaviour • t unable st aleness • globaly consist ent logf ile github.com/steffenfriedrich/SickStore W inger at h, Fr iedr ich, Gesser , R it t er : W ho W at ches t he W at chmen? BTW 2 0 1 5 Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 1

Coor dinat ed Omission Validat ion w it h Single-node inconsist ent key-value St or e New Feat ur e: Simulat ion of maximum t hr oughput and dat abase hiccups 1 . comput e t heor et ical w ait ing t ime T i of r eq uest i in t he dat abse syst em 2 . calling client t hr ead has t o sleep f or T i Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 2

Exper iment al Validat ion on: : Sick SickSt or e Benchmar k: 90 0 0 0 o ps, t ar get = 1 0 0 0 o ps/ sec, SickSt o r e: 1 seco nd hiccup, max t hr o ughput = 1 2 5 0 o ps/ sec, 80 % o f max t hr o ughput YCSB NoSQ LM ar k YCSB Int ended AVG .: 2 ms 2 9 ms 1 80 ms Fr iedr ich, W inger at h, Rit t er | Coor dinat ed Omission in NoSQ L Dat abase Benchmar king 2 3