On the Impact of Flow Monitoring Configuration Petr Velan et al. velan@ics.muni.cz Institute of Computer Science, Masaryk University April 20, 2020
Flow Monitoring Recapitulation Network Flow Monitoring Network Flow Monitoring Used for monitoring of large networks Scales better than DPI Supported by network equipment (NetFlow and IPFIX protocols) Use of Flow Monitoring Data Network management Network planning (using long term statistics) Network debugging Security Incident handling Policy verification Attack detection Anomaly detection P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 2 / 19
Flow Monitoring Recapitulation Flow Record Creation A B Time Individual packet Sending side Receiving side P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 3 / 19
Flow Monitoring Recapitulation Flow Record Creation Active Timeout A B Time Individual packet Sending side Receiving side P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 3 / 19
Flow Monitoring Recapitulation Flow Record Creation Active Timeout Active Timeout Active Timeout A B Time Individual packet Sending side Receiving side P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 3 / 19
Flow Monitoring Recapitulation Flow Record Creation Active Timeout Active Timeout Active Timeout A B Time Individual packet Sending side Active Timeout Active Timeout Receiving side A B P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 3 / 19
Flow Monitoring Recapitulation Flow Record Creation Active Timeout Active Timeout Active Timeout A B Time Individual packet Sending side Active Timeout Active Timeout Receiving side A B Active Timeout Inactive Timeout Inactive Timeout A B P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 3 / 19
Flow Monitoring Recapitulation Flow Record Creation Flow Expiration Conditions Active timeout Inactive timeout Protocol specific reasons (e.g. end of TCP connection) Resource restrictions (e.g. limited flow cache size) Exporter shutdown P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 4 / 19
The Important Lesson Configuration of flow monitoring is essential! If you are publishing results based on flow data: Always include description of flow monitoring configuration. P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 5 / 19
Flow Expiration Configuration Impact Flow Expiration Configuration Impact What is affected by flow expiration timeouts? Flow export Larger inactive timeout causes flows to be cached longer which increases computing and memory requirements Smaller timeouts increase number of generated flow records, which increases computing and export bandwidth requirements Flow collection Larger number of flow records increases computing and storage space requirements Flow analysis Larger number of flow records increases computing requirements Different number of flow records with different properties influences analysis results P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 6 / 19
Flow Expiration Configuration Impact Number of Created Flows How large is impact of flow expiration timeouts on flow creation? To find out, we: Selected datasets The CAIDA Anonymized Internet Traces 2015 Dataset (1.1 billion of packets) A Realistic Cyber Defense Dataset (CSE-CIC-IDS2018) (3.6 million of packets) Computed flows using a range of different timeouts We were interested only in number of flows Python tool – unsuitable for the CAIDA dataset (too slow, large memory consumption) C++ tool, fast computation the flow records Analysed the results P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 7 / 19
Analysis of Flow Expiration Timeouts Impact Impact of the Inactive Timeout (CAIDA Dataset, TCP) 0.07 # of fl ows 4x107 Interpacket gap 0.06 frequency Interpacket gap frequency 3.5x107 0.05 # of fl ows 3x107 0.04 2.5x107 0.03 2x107 0.02 1.5x107 0.00 10 20 30 40 50 60 70 80 90 100 110 120 130 Interpacket gap / inactive timeout length (seconds) P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 8 / 19
Analysis of Flow Expiration Timeouts Impact Impact of the Inactive Timeout (CAIDA Dataset, TCP) Changing inactive timeout setting: 30 s -> 10 s causes an increase of almost 26% flows records 60 s -> 30 s causes an increase of almost 16% flows records 60 s -> 10 s causes an increase of almost 44% flows records A 45 second interpacket gap is quite common. Number of generated flows increases by 1.2% for 46 s and 45 s inactive timeout setting. P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 9 / 19
Analysis of Flow Expiration Timeouts Impact Impact of the Active Timeout (CSE-CIC Dataset, TCP) 0.175 750000 # of fl ows Connection 0.150 700000 length frequency Connection length frequency 0.125 650000 0.100 # of fl ows 600000 0.075 550000 0.050 500000 0.025 0.000 450000 10 20 30 40 50 60 70 80 90 100 Connection / Active timeout length (seconds) P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 10 / 19
Analysis of Flow Expiration Timeouts Impact Impact of the Active Timeout (CSE-CIC Dataset, TCP) The impact of active timeout is more complicated to evaluate Correlation between active timeout and connection length is weaker Number of multiples of active timeout that can fit into a connection length is also important (e.g. 10 s active timeout) Interpacket gaps influence the result as well Decreasing the active timeout from 300 seconds to 120 seconds increases the number of flow records only by 3% (for this dataset). P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 11 / 19
Analysis of Flow Expiration Timeouts Impact Impact of the Combination of Both Timeouts (CAIDA Dataset, TCP) 2.4*10 7 120 2.3*10 7 100 2.2*10 7 Inactive timeout 2.1*10 7 80 2.0*10 7 60 1.9*10 7 1.8*10 7 40 1.7*10 7 20 1.6*10 7 50 100 150 200 250 300 Active timeout P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 12 / 19
Analysis of Flow Expiration Timeouts Impact Impact of the Combination of Both Timeouts (CAIDA Dataset, TCP) Following can be derived from analysing the timeouts: Specifics of used transport protocols such as timeouts and common connection lengths (e.g. HTTP keepalive) can be observed as faster changes in the number of flows (colour changes) Different protocols (UDP, TCP, ICMP) and networks (datasets) behave differently Decreasing the active timeout from 300 seconds to 120 seconds and the inactive timeout from 30 seconds to 10 seconds increases the number of flow records by 26% (for this dataset). P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 13 / 19
Analysis of Flow Expiration Timeouts Impact Impact on Flow Data Analysis The magnitude of impact of flow timeouts depends on a type of analysis, for example: Port scan detection will not be affected because port scans always generate only short flow records Covert dictionary attack can run slowly from multiple attackers to avoid detection. Their detection might be be affected by flow timeout settings DDoS attacks detection such as Slowloris detection will be affected the most P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 14 / 19
Analysis of Flow Expiration Timeouts Impact Impact on a Slowloris Attack Detection There is a Slowloris attack in the CSE-CIC dataset, which we analysed: Successful attack establishes a TCP connection, sends first part of HTTP header and continues to send a small part of request header every 100 seconds (first after 78 s). The server responds with Bad Request response after approximately 2470 seconds. When the attack is successful, attacker does not get response for initial part of GET request and closes connection after 108 seconds In the most severe case, TCP connection cannot be established and attacker gives up after sending tree SYN packet in 3 seconds. P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 15 / 19
Analysis of Flow Expiration Timeouts Impact Impact on a Slowloris Attack Detection 9.0*10 4 120 100s interpacket gap 8.0*10 4 100 7.0*10 4 108s failed HTTP GET 78s request chunk 80 Inactive timeout 6.0*10 4 54s fraction 5.0*10 4 60 4.0*10 4 40 3.0*10 4 20 2.0*10 4 1.0*10 4 0 0 20 40 60 80 100 120 140 160 180 200 220 240 Active timeout P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 16 / 19
Analysis of Flow Expiration Timeouts Impact Impact on a Slowloris Attack Detection Caution is required when relying on flow data for Slowloris detection Large enough timeouts should be used Preprocessing using flow aggregation might be needed When using machine learning, ensure that flow expiration conditions remain the same throughout the whole process P. Velan · On the Impact of Flow Monitoring Configuration · April 20, 2020 17 / 19
Recommend
More recommend
