goprobe a scalable distributed network monitoring solution
play

goProbe: A Scalable Distributed Network Monitoring Solution - PowerPoint PPT Presentation

goProbe: A Scalable Distributed Network Monitoring Solution Christian Decker Lennart Elsen Fabian Kohn Roger Wattenhofer Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks Goal


  1. goProbe: A Scalable Distributed Network Monitoring Solution Christian Decker Lennart Elsen Fabian Kohn Roger Wattenhofer

  2. Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks

  3. Goal Enable quick and efficient retrieval of key pieces of information about traffic patterns in global networks Scalability

  4. ? ? ? Debugging/Operations Reporting

  5. Acquisition of Traffic Data Packet Capture Storage

  6. Acquisition of Traffic Data Grouping Information Reduction Packet Capture Storage

  7. NetFlow NetFlow Packet Field N Length Source IP Network Packet aggregation by set of Destination IP Field N Type Next Layer Protocol … shared attributes IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type Network packet headers & Destination Port Field 1 Length … packet counters Field 1 Type Count Packet Size Number of Packets Meta Info … Sampling Interval Expiry time System Uptime TTL Sequence # Interface Name … NetFlow Version

  8. NetFlow NetFlow Packet Field N Length Source IP Network Destination IP NetFlow Field N Type Exporter Next Layer Protocol … IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type NetFlow Destination Port Exporter Field 1 Length … Field 1 Type Count Network A Packet Size Number of Packets Meta Info … Sampling Interval System Uptime TTL Sequence # Interface Name … NetFlow Version Network B NetFlow Collector

  9. NetFlow NetFlow Packet Field N Length Source IP Network Destination IP NetFlow Field N Type Exporter Next Layer Protocol … IPv4/6 Next Hop … … Field 2 Length Transport Source Port Field 2 Type NetFlow Destination Port Exporter Field 1 Length … Field 1 Type Count Network A Packet Size Number of Packets Meta Info … Sampling Interval System Uptime TTL Sequence # Interface Name … NetFlow Version Network B NetFlow Collector

  10. Current Network Monitoring System Analysts Request Traffic d Metadata Aggregated Formatted Results Results Query Flow Exporter Tool Data DB nProbe FastBit Queries Single Host

  11. Challenges Capturing Process Query nProbe Tool FastBit

  12. Challenges Capturing Process Immense memory footprint Query nProbe Tool FastBit

  13. Challenges Capturing Process One process per capture interface Query nProbe nProbe nProbe Tool FastBit

  14. Challenges Storage Backend Query nProbe nProbe nProbe Tool FastBit

  15. Challenges Storage Backend Inefficient memory management Query nProbe nProbe nProbe Tool FastBit

  16. Challenges Storage Backend No data compression Query nProbe nProbe nProbe Tool FastBit

  17. Challenges Storage Backend Long query execution times Query nProbe nProbe nProbe Tool FastBit

  18. Challenges y t i l i b a l a c S r o o P Query nProbe nProbe nProbe Tool FastBit

  19. Reduced Flow Format IP Packets Packets Bytes Bytes Src IP Dst IP Src Port Dst Port Protocol Rcvd Sent Rcvd Sent Shared Attributes Counters

  20. Reduced Flow Format Deep Packet Inspection Appl. Src Port Dst Port Layer Protocol Shared Attributes Counters

  21. Reduced Flow Format Deep Packet Inspection Appl. Src Port Dst Port Layer Protocol Flow in goProbe Source Port Aggregation ✗ Appl. Layer Dst Port Protocol Stored Flow

  22. Collection of Flow Information — goProbe Written in Google Go One capture routine per interface goProbe Packet capture using modified libpcap Database flush in regular intervals

  23. goProbe – Concept (Multiple Interfaces) Flow Table Interface Data Channel Timer Data Aggregation Prepare … DB Local Database

Recommend


More recommend