goProbe: A Scalable Distributed Network Monitoring Solution Christian Decker Lennart Elsen Fabian Kohn Roger Wattenhofer
Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks
Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks Scalability
? ? ? Debugging/Operations Reporting
Acquisition of Traffic Data Packet Capture Storage
Acquisition of Traffic Data Grouping Information Reduction Packet Capture Storage
NetFlow NetFlow Packet Field N Length Source IP Network Packet aggregation by set of Destination IP Field N Type Next Layer Protocol … shared attributes IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type Network packet headers & Destination Port Field 1 Length … packet counters Field 1 Type Count Packet Size Number of Packets Meta Info … Sampling Interval Expiry time System Uptime TTL Sequence # Interface Name … NetFlow Version
NetFlow NetFlow Packet Field N Length Source IP Network Destination IP NetFlow Field N Type Exporter Next Layer Protocol … IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type NetFlow Destination Port Exporter Field 1 Length … Field 1 Type Count Network A Packet Size Number of Packets Meta Info … Sampling Interval System Uptime TTL Sequence # Interface Name … NetFlow Version Network B NetFlow Collector
NetFlow NetFlow Packet Field N Length Source IP Network Destination IP NetFlow Field N Type Exporter Next Layer Protocol … IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type NetFlow Destination Port Exporter Field 1 Length … Field 1 Type Count Network A Packet Size Number of Packets Meta Info … Sampling Interval System Uptime TTL Sequence # Interface Name … NetFlow Version Network B NetFlow Collector
Current Network Monitoring System Analysts Request Traffic d Metadata Aggregated Formatted Results Results Query Flow Exporter Tool Data DB nProbe FastBit Queries Single Host
Challenges Capturing Process Query nProbe Tool FastBit
Challenges Capturing Process Immense memory footprint Query nProbe Tool FastBit
Challenges Capturing Process One process per capture interface Query nProbe nProbe nProbe Tool FastBit
Challenges Storage Backend Query nProbe nProbe nProbe Tool FastBit
Challenges Storage Backend Inefficient memory management Query nProbe nProbe nProbe Tool FastBit
Challenges Storage Backend No data compression Query nProbe nProbe nProbe Tool FastBit
Challenges Storage Backend Long query execution times Query nProbe nProbe nProbe Tool FastBit
Challenges y t i l i b a l a c S r o o P Query nProbe nProbe nProbe Tool FastBit
Reduced Flow Format IP Packets Packets Bytes Bytes Src IP Dst IP Src Port Dst Port Protocol Rcvd Sent Rcvd Sent Shared Attributes Counters
Reduced Flow Format Deep Packet Inspection Appl. Src Port Dst Port Layer Protocol Shared Attributes Counters
Reduced Flow Format Deep Packet Inspection Appl. Src Port Dst Port Layer Protocol Flow in goProbe Source Port Aggregation ✗ Appl. Layer Dst Port Protocol Stored Flow
Collection of Flow Information — goProbe Written in Google Go One capture routine per interface goProbe Packet capture using modified libpcap Database flush in regular intervals
goProbe – Concept (Multiple Interfaces) Flow Table Interface Data Channel Timer Data Aggregation Prepare … DB Local Database
Recommend
More recommend