Traffic Measurement Lothar Braun Outline Why do we need to measure - PowerPoint PPT Presentation

Chair for Network Architectures and Services Department of Informatics Technische Universität München Traffic Measurement Lothar Braun

Outline  Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab? 2

Outline  Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab? 3

Why do You Measure in the Lab?  Task: Setup your lab environment  It doesn’t work!  What do you do to find the problem? 4

Why do We Need to Measure in the Internet?  Problems get worse in large-scale networks  Openness  No/little access control for new Internet participants  Free deployment of new applications and services  Heterogeneity  Different technologies at lower layers  Different protocols and services on top of IP  different requirements / different failures  Consequences for network operators  Little control of utilization of network resources  possible misuse: hackers, attacks, spam  Little knowledge about applications and services  unpredictable traffic  Traffic measurements can help to understand our network 5

Applications of Traffic Measurements  What information is useful for network operators?  Network monitoring  Fault detection  Connectivity and routing  Performance measurements  Link capacity and utilization  Quality of Service parameters (delay, jitter, throughput)  Accounting and charging  Traffic volume per customer or peering AS Map of the MWN (December 2010)  Network security  Worm and attack detection 6

Outline  Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab? 7

Active Measurements  Methodology  Probe packets exchanged between two end-systems  Measurement of packet loss, one-way delay, round-trip times, packet interarrival times  Analysis  Complete packet loss  link down, invalid route, router defect  Partial packet loss  available bandwidth , level of congestion  Delay = propagation time + buffer time  distance , filling level of buffers  Interarrival times of packet pairs/trains  path capacity  Pros and cons  Does not require access to internal network components  Intrusive  existing traffic is disturbed  No information about existing traffic 8

Passive Measurements (= Focus of this Lab)  Methodology  Observation of existing traffic using monitoring probes in the network  Measurement of traffic volume, traffic composition, packet interarrival times  Different levels of granularity: packet-level, flow-level, link-level  Analysis  Measurement of network utilization for accounting and traffic engineering  Measurement of quality-of-service parameters (e.g., throughput, delay)  Detection of failures, traffic anomalies, flooding attacks and scans  Traffic characterization with deep packet inspection  Pros and cons  Non-intrusive  existing traffic is not disturbed  Installation of monitoring probes at appropriate locations in the network 9

Outline  Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurement  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab? 10

Packet-Level Measurements  Gather information about individual packets :  Observation time  Location (interface and direction)  Packet header information (link layer, IP header, transport header)  Packet payload Application ports 31 0 16 Source Port Destination Port Sequence Number Acknowledgement Connection U A P R S F 4 bit TCP 6 bit Window R C S S Y I header Information unused G K H T N N length Checksum Urgent Pointer Options Application GET /index.html HTTP/1.1 ... payload 11

Using Packet-Level Measurements  You already used packet-level measurements  For understanding the SCTP traffic in the SCTP lab You can look at all the packets You can look into the packets You can identify problems with the packets You can look into the application protocol 12

Signature Detection in Packets  Some applications try to look for patterns in the payload  Application identification  Search for attacks like worm or botnet traffic  Application detection  Idea: Identify an application by looking for specified protocol fields  Example: HTTP • GET /index.html HTTP/1.1  Malicious traffic detection  Idea: Security analysts know how malicious traffic looks like  Signature detection systems try to find such patterns  These systems are often called Intrusion Detection Systems 13

Example: IRC-based botnets  Internet Relay Chat (IRC)  Real-time chat  Users join a chat room  Also used by botnets to control bot clients  Important properties  Every user must choose a nickname  Each nickname must be unique on a network  Bot clients need to generate nicknames  Intrusion Detection Systems can look for such automatically generated nicks 14

Signature Detection: Snort  Snort is an Intrusion Detection System  Operation  Capture network packets like wireshark  Perform signature matching on packets according to signature database  Signature database must be configured by the users alert tcp $HOME_NET any -> $EXTERNAL_NET any (flow:established,to_server; flowbits:isset,is_proto_irc; content:"USER XP-"; pcre:"/USER XP-[A-z0-9]{4,8} \* 0 \:.*/" ; msg:"E4[rb] ET TROJAN Likely Bot Username in IRC (XP-..)" ; classtype:trojan-activity; reference:url,doc.emergingthreats.net/2008123; reference:url,www.emergingthreats.net/cgi- bin/cvsweb.cgi/sigs/VIRUS/TROJAN_IRC_Bots; sid:2008123; rev:4;) 15

Using Packet-Level Measurements  Also useful: tcpdump  It will give you network and transport layer information about the traffic  Important information:  Packet loss  Not a problem on my desktop system, but … 16

Packet-Level Measurements in Large Networks  … can be a big problem for larger systems  Example: X-Win from MWN to DFN (Internet access) • 10 Gigabit Ethernet link • Connects more than 80.000 systems to the Internet  Problem  Processing of all packets requires a lot of resources  Packet loss will occur if you do not have these resources 17

Packet Selection  Most applications require only specific packets to be observed  Packet selection  Capture only the interesting subset of packets that you can handle  Packet Filtering  Select packets with properties X and Y  Goal: Select “interesting traffic” that helps to identify problems  Example: tcpdump -i eth0 tcp and port 80 • Useful if you are interested in HTTP traffic on port 80  Packet Sampling  Systematic or probabilistic sampling algorithms  Goal: Infer statistics of overall traffic from sampled packets  Example: Select each packet with probability of 10 % 18

Flow-Level Measurements  Some applications do not require packet-level information  Other information can be more important  How much traffic is in my network?  How much UDP traffic is there?  Is there some client that sends a lot of mail traffic? • E.g. a spammer in my network?  Answering these questions does not require packet data  Instead: It requires information about who communicates with whom  Solution: Generate and analyze flow data 19

Using Flow Data  Flows are often stored for a long time period  Used for understanding and visualizing traffic Source: NfSen- Homepage 20

Network Flows  Definition of a flow  Set of packets common properties called flow keys (often IP-quintuple)  Observed at one point in the network (e.g., router interface) Host A Host B  Measured properties  Observation period (timestamp of first and last packet)  Number of bytes and packets belonging to the flow  Flow record = {( flow keys ), (measured properties)}  {( Host A, port a, Host B, port b, TCP ), (200 packets, 5000 bytes, 11:00, 11:05)}  Flow duration  Start: first packet with new flow key values  End: after timeout or based on signaling (e.g., TCP FIN) 21

Flow Duration  Flow expiration after timeout  Inactive timeout  maximum gap between two consecutive packets  Active timeout  maximum flow duration since flow start 22

Flow Measurement Deployment  Flows are typically generated at a network router  Analysis does not happen on the router  Instead: Data is transmitted to a flow collecting machine Flow-Collector Flow Data (e.g. encoded in IPFIX) Internal Network Internal Network Internet Internet 23

Outline  Why do we need to measure traffic in the Internet?  Active measurement vs. passive measurements  Passive measurement: Packet-level vs. flow-level  Protocols for transporting measurement data  What are you going to do in the Lab? 24