Part 6. Confidence Interval Min Chen School of Computer Science and - PowerPoint PPT Presentation

Introduction to NS-2 Part 6. Confidence Interval Min Chen School of Computer Science and Engineering Seoul National University 1

Outline  Definitions  Normal Distribution  Confidence Interval  Central Limit Theorem  Error-Bar  An Example 2

Definitions  Population  The set of elements we are inteseted in  Sample  A subset of the Population  Population Mean μ  Always stands for the everage value = ∑ X i  N  Sample Mean X  The Mean for the Sample subset 3

Definitions (2)  Variance 2 = ∑  X i − 2  N  Sample Variance 2 = ∑  X i − X  2 S N  Standard Deviation : =   2  Sample Standard Deviation : S =  S 2 4

Normal Distribution  A continuous probability distribution that often gives a good descriptio of data that cluster around the mean  For 2  X ~ N  ,   Probability Density Function(PDF) 2 − x − 1 2  x = 2  2 e  2   Cumulative Distribution Function(CDF) 2 − t 1 2 dt  2  ∫ e  x = 5

Normal Distribution : PDF 6

Normal Distribution : CDF 7

Confidence Interval  Confidence Interval (CI) is a particular kind of interval estimate of a population parameter. Instead of estimating the reliability of a single value, an interval likely to include the parameter is given. Thus, CIs are used to indicate the reliability of an estimate.  If Probability{c1<= µ <=c2} = 1- α  Then we call (c1,c2) Confidence Interval  1-α is the Confidence Coefficient 8

Central Limit Theorem  Central Limit Theorem (CLT) states conditions under which the mean of a sufficiently large number of independent random variables, each with finite mean and variance, will be approximatedly normally distributed.  This distribution is X ~ N  ,   M  Where M is the Sample Size 9

Central Limit Theorem  According to CLT, we can calculate the Confidence Interval by a given Confidence Coefficient  Define Z value: Z =  X −  n   Frequently used Z:  Z(90%) = 1.645  Z(95%) = 1.96  Z(99%) = 2.33 10

Calculate Confidence Interval  With the value of Z, the confidence interval can be calculated as follow: X − Z × S X  Z × S    M ,   M  Where X denotes mean, S denotes sample variance and M denotes the sample size 11

Error-Bar  Error Bar is used on graphs to indicate the error, or uncertainty in a reported measurement.  In simulation result figures, error bar represents the confidence of the simulation  The error bar in the figure are the variance of confidence intervals for each mean value in the simulation work Z × S  M 12

Example: Network Topology Throughput in 5s ~ 10s TCP: Throughput in 5s ~ 10s TCP: 1Mb 10ms 1Mb 10ms Random: [3,4] s n1 n6 0.0 s n2 n4 n5 n7 Each simulation will 0.0 s be repeated for 30 times n3 n8 with different seeds UDP: UDP: 10Mb 1ms UDP: UDP: 10Mb 1ms Controlable Data Rate Controlable Data Rate Controlable Data Rate Controlable Data Rate 100 ~ 500 step 100 100 ~ 500 100 ~ 500 step 100 100 ~ 500 13

OTCL Script if {$argc != 2} { puts "Usage: ns errorbar.tcl rate_ no_" exit } #=================================== # Simulation parameters setup #=================================== set par1 [lindex $argv 0] ;# date rate for the interference flow set par2 [lindex $argv 1] ;# the number of the experiments set val(stop) 12.0 ;# time of simulation end #=================================== # Initialization #=================================== #Create a ns simulator set ns [new Simulator] #Open the NS trace file set tracefile [open out$par1-$par2.tr w] $ns trace-all $tracefile #Open the NAM trace file set namfile [open out.nam w] $ns namtrace-all $namfile 14

OTCL Script (2) #=================================== # Nodes Definition #=================================== #Create 8 nodes set n0 [$ns node] set n1 [$ns node] ... set n7 [$ns node] #=================================== # Links Definition #=================================== #Createlinks between nodes $ns duplex-link $n0 $n3 10.0Mb 1ms DropTail $ns queue-limit $n0 $n3 50 ... $ns duplex-link $n3 $n4 1.0Mb 10ms DropTail $ns queue-limit $n3 $n4 50 #Give node position (for NAM) $ns duplex-link-op $n0 $n3 orient right-down ... $ns duplex-link-op $n4 $n7 orient right-down $ns duplex-link-op $n3 $n4 orient right 15

OTCL Script (3) #=================================== # Agents Definition #=================================== #Setup a TCP connection set tcp0 [new Agent/TCP] $ns attach-agent $n0 $tcp0 set sink3 [new Agent/TCPSink] $ns attach-agent $n5 $sink3 $ns connect $tcp0 $sink3 #Setup a TCP connection set tcp1 [new Agent/TCP] $ns attach-agent $n1 $tcp1 set sink4 [new Agent/TCPSink] $ns attach-agent $n6 $sink4 $ns connect $tcp1 $sink4 #Setup a UDP connection set udp [new Agent/UDP] $ns attach-agent $n2 $udp set null [new Agent/Null] $ns attach-agent $n7 $null 16 $ns connect $udp $null

OTCL Script (4) #=================================== # Applications Definition #=================================== #Setup a FTP Application over TCP connection set ftp0 [new Application/FTP] $ftp0 attach-agent $tcp0 $ftp0 set type_ FTP set ftp1 [new Application/FTP] $ftp1 attach-agent $tcp1 $ftp1 set type_ FTP set traffic [new Application/Traffic/Exponential] $traffic set packetSize_ 1000 $traffic set burst_time_ 0.5 $traffic set idle_time_ 0 $traffic set rate_ [expr $par1*1000] $traffic attach-agent $udp 17

OTCL Script (5) #=================================== # Traffic Running Time #=================================== set rng [new RNG] $rng seed 0 set RVstart [new RandomVariable/Uniform] $RVstart set min_ 3 $RVstart set max_ 4 $RVstart use-rng $rng set startT [expr [$RVstart value]] puts "startT $startT sec" $ns at 0.0 "$ftp1 start" $ns at 0.0 "$traffic start" $ns at $startT "$ftp0 start" $ns at 11.0 "$ftp0 stop" $ns at 11.5 "$ftp1 stop" $ns at 11.5 "$traffic stop" 18

OTCL Script (6) #=================================== # Termination #=================================== #Define a 'finish' procedure proc finish {} { global ns tracefile namfile $ns flush-trace close $tracefile close $namfile # exec nam out.nam & exit 0 } $ns at $val(stop) "$ns nam-end-wireless $val(stop)" $ns at $val(stop) "finish" $ns at $val(stop) "puts \"done\" ; $ns halt" $ns run 19

AWK for Throughput BEGIN { Init = 0; startT = 0; endT = 0; } { Action = $1; time= $2; From = $3; to = $4; Type = $5; pktsize = $6; flow_id = $8; src = $9; dst = $10; seq_no = $11; packet_id = $12; if ( action == "r" && type == "tcp" && time>=5.0 $$ time <=10.0 && from == 4 && to == 5 ) { if(init == 0) { start_time = time; Init =1; } pkt_byte_sum += pktsize; endT = time; } } END{ time=endT-startT; throughput=pkt_byte_sum*8/time/1000; printf("%f\n",throughput); } 20

AWK for ErrorBar with 95% CI BEGIN { FS="\t"; } { ln++; throughput[ln]=$1; sum=sum+throughput[ln]; } END{ average=sum/ln; for(i=1;i<=ln;i++) { gap=throughput[ln]-average; sum2=sum2+gap*gap; } variance=sqrt(sum2/(ln-1)); confidence_interval=1.96*variance/sqrt(ln); printf("%f %f %f\n",t,average,confidence_interval); } 21

The result file  The use of parameters awk -v t=100 -f errorbar.awk result100 >> errorbar.txt  Three columns  1 : the data rate  2 : the mean throughput  3 : the confidence interval variance 22

GPL script set title "Impact of Data Rate on the Throughput" set xlabel "On-Off Flow Rate Setting (kbps)" set ylabel "Flow Throughput (kbps)" set xrange [0:600] set terminal png set output "throughput.png" plot "errorbar.txt" using 1:2:3 with errorbars 1, "errorbar.txt" with lines 1 23

Perl Script in Linux for($i=100; $i<=500; $i=$i+100) { for($j=1; $j<=30; $j++) { system("ns errorbar.tcl $i $j"); $f1="out$i-$j.tr"; $f2="result$i"; system("awk -f throughput.awk $f1 >> $f2"); print "\n"; } print "\n"; } for($i=100; $i<=500; $i=$i+100) { $f1="result$i"; $f2="errorbar.txt"; system("awk -v t=$i -f errorbar.awk $f1 >> $f2"); print "\n"; } system("gnuplot errorbar.gpl"); 24

To run perl script in Linux  Grant the privilege to the script file chmod 777 errorbar.pl  Run the script perl errorbar.pl 25

Results: 26