Using TCP/IP Traffic shaping to achieve iSCSI service predictability - PowerPoint PPT Presentation

Using TCP/IP Traffic shaping to achieve iSCSI service predictability Paper presentation Jarle Bjørgeengen University of Oslo / USIT November 11, 2010

Outline About resource sharing in storage devices Lab setup / job setup Experiment illustrating the problem One half of the solution: the throttle Live demo The throttle Part two of the solution: the controller How the controller works Conclusion and future work

General problem of sharing resources Centralized storage pool SAN QoS bridge QoS bridge QoS bridge QoS bridge QoS bridge Consumers Virtual disks Shared physical resources Free competition causes unpredictable I/O performance for any given consumer.

Lab setup Striped logical volumes. 64KB Argus stripe size b1 TCP across 10 disks Connections HP SC10 iSCSI target 10 x 36GB 10k (iet) /dev/iscsi_0 bm lv_b2 iqn.iscsilab:perc_b2 b2 lv_b3 iqn.iscsilab:perc_b3 b3 b lv_b4 iqn.iscsilab:perc_b4 b4 lv_b5 iqn.iscsilab:perc_b5 vg_perc b5 TCP/IP Layer Block Layer Block Layer

Is read response time affected by write activity ? bm lv_b2 Random read b2 rate=256kB/s lv_b3 Seq write b3 full speed b lv_b4 Seq write b4 full speed lv_b5 Seq write b5 full speed

The Answer is yes Long response times adversely affect application service availability. 120 No interference 1 thread (1 machines) 3 threads (3 machines) 100 12 threads (3 machines) 80 Wait time (ms) 60 40 20 0 0 100 200 300 400 500 Time (s)

Throttling method Timeline Timeline without delay with delay Initiator Target Initiator Target SYN SYN Throttling delay SYN+ACK SYN+ACK ACK ACK Write Write ACK ACK Write Time ACK Write Write ACK ACK Write Write ACK ACK Write ACK

Relation between packet delay and average rate 40 80 Time to read 200MB of data (s) Time to write 200MB of data (s) 30 60 20 40 10 20 0 0 0 0.6 1.6 2.6 3.6 4.6 5.6 6.6 7.6 8.6 9.6 0 0.6 1.6 2.6 3.6 4.6 5.6 6.6 7.6 8.6 9.6 Introduced delay (ms) Introduced delay (ms) Write rate 15 MB/s - 2.5 MB/s Read rate 22 MB/s - 5 MB/s

Managing consumers Need to operate on sets of consumers (throttlable={10.0.0.243,10.0.0.244}) Ipset: One rule to match them all ✞ ☎ ipset -N $throttlable ipmap --network 10.0.0.0/24 ipset -A $throttlable 10.0.0.243 ipset -A $throttlable 10.0.0.244 iptables --match-set $throttlable dst -j MARK --set-mark $mark ✝ ✆ The mark is a step in the range of available packet delays

Live demonstration Manual throttling and QoS specification An automatic QoS policy and automated throttling

Dynamic throttling decision Figure: Block diagram of a PID controller. Created by SilverStar(at)en.wikipedia. Licensed under the terms of Creative Commons Attribution 2.5 Generic.

Modified PID function Ud = Kp × Td × e k − e k − 1 Up = Kp × e k Start Ts Ui = Ui k − 1 + Ts × Kp × e k Ti Calculate Up,Ui,Ud Uk = Up+Ui+Ud 0 < Ui < Ukmax Y 0 < Uk < Ukmax Y mark = int(ceil(Uk)) N N Stop Uk=0 Ui < 0 Y Ui=0 Uk < 0 Y N N Ui > Ukmax Y Ui=Ui k-1 Uk > Ukmax Y Uk=Uk max

The completely automated approach ISCSIMAP /proc/diskstats PDATA /proc/net/iet/sessions /proc/net/iet/volumes Saturation indicators Create Read Read Read Read CMEM lvs Run set_maintaner.pl perf_maintainer.pl Read Throttle values Read output pid_reg.pl perf_server.pl Create & maintain members Spawn($resource) gnuplot Throttles IP-sets pid_threads Dependency Legend: Files Shared memory Processes Command

Impact The packet delay throttle is very efficient Solves the throttling need completely for iSCSI (likely other TCP based storage networks too) The modified PID controller is consistently keeping response time low in spite of rapidly changing load interference. The concept is widely applicable

Impact The packet delay throttle is very efficient Solves the throttling need completely for iSCSI (likely other TCP based storage networks too) The modified PID controller is consistently keeping response time low in spite of rapidly changing load interference. The concept is widely applicable

Impact The packet delay throttle is very efficient Solves the throttling need completely for iSCSI (likely other TCP based storage networks too) The modified PID controller is consistently keeping response time low in spite of rapidly changing load interference. The concept is widely applicable

Future work QoS bridge QoS bridge QoS bridge iSCSI disk array QoS bridge Ethernet sw. QoS bridge QoS bridge Consumers Array specific plugin SNMPGET Resource/consumer maps Virtual disk latencies Packet delay throttle with other algorithms PID controller with other throttles

Thanks for the attention !

Overhead Negligeble overhead introduced by TC filters Differences measured 20 times t-test 99% confidence shows 0.4% / 1.7 % • overhead for read/write (worst case)

Is response time improved by throttling ? 50000 Small job average wait time (Left) 80 Interference aggregated throughput (Right). Throttling period with 4.6 ms delay Throttling period with 9.6 ms delay Aggregated interference (kB/s) 40000 60 Wait time (ms) 30000 40 20000 20 10000 0 0 0 100 200 300 400 500 Time (s)

Automatically controlled wait time 100 No regulation 20 ms treshold 15 ms threshold 10 ms threshold 80 Average wait time (ms) 60 40 20 0 0 100 200 300 400 Time (s)

The throttled rates 50000 No regulation 20 ms threshold (smoothed) 15 ms threshold (smoothed) 40000 10 ms threshold (smoothed) Aggregate Write (kB/s) 30000 20000 10000 0 0 100 200 300 400 500 Time (s)

Exposing the throttling value 50 vg_aic read wait time with automatic regulation, thresh=15ms Packet delay introduced to writers 40000 Aggregated write rate 40 30000 30 (kB/s) (ms) 20000 20 10000 10 0 0 0 50 100 150 200 Time (s)

Effect of the packet delay throttle: Reads 20000 b2 b3 b4 b5 15000 Read (kB/s) 10000 5000 0 0 50 100 150 200 250 300 Time (s)

Effect of the packet delay throttle: Writes 20000 b2 b3 b4 b5 15000 Write (kB/s) 10000 5000 0 0 50 100 150 200 250 300 Time (s)