Cloud-based Log Analysis and Visualization RMLL 2010, Bordeaux, - PowerPoint PPT Presentation

Cloud-based Log Analysis and Visualization RMLL 2010, Bordeaux, France My syslog mobile-166 Ra fg ael Marty - @zrlram Tuesday, July 6, 2010

Ra fg ael (Ra fg y) Marty • Founder @ • Chief Security Strategist and Product Manager @ Splunk • Manager Solutions @ ArcSight • Intrusion Detection Research @ IBM Research • IT Security Consultant @ PriceWaterhouse Coopers Applied Security Visualization Publisher: Addison Wesley (August, 2008) ISBN: 0321510100 Logging as a Service (c) by Ra fg ael Marty 2 Tuesday, July 6, 2010

Agenda •Do it Yourself •Introduction • AfterGlow •Visualization • Google Visualization API •InfoViz Process •Visualization Use-Cases •Visualization Tools •Visualization Resources •The Cloud •Loggly Logging as a Service (c) by Ra fg ael Marty 3 Tuesday, July 6, 2010

Open Your Eyes Logging as a Service (c) by Ra fg ael Marty 4 Tuesday, July 6, 2010

Security Is About Seeing Logging as a Service (c) by Ra fg ael Marty 5 Tuesday, July 6, 2010

Goals - Learn how you can - use visualization to help solve security problems - leverage the cloud to build security visualization tools Logging as a Service (c) by Ra fg ael Marty 6 Tuesday, July 6, 2010

Information Visualization? A picture is worth a thousand log records. Inspire Explore and Discover Answer a Increase Pose a New Communicate Support Question Question Efficiency Information Decisions Logging as a Service (c) by Ra fg ael Marty 7 Tuesday, July 6, 2010

Visualization and The Cloud 8 Tuesday, July 6, 2010

InfoViz Process Process Visualize Collect • Visualization Tools • large-scale data collection • Your parsers • and Libraries • and processing • Standard formats Logging as a Service (c) by Ra fg ael Marty 9 Tuesday, July 6, 2010

Collect 10 Tuesday, July 6, 2010

Log Management • Log Collection and Centralization • Log Storage • Log Filtering • Log Aggregation • Log Search and Extraction • Log Retention and Archiving Logging as a Service (c) by Ra fg ael Marty 11 Tuesday, July 6, 2010

Process 12 Tuesday, July 6, 2010

Standard Formats • Multiple formats Oct 13 20:00:43.874401 rule 193/0(match): block in on xl0: 212.251.89.126.3859 >: S 1818630320:1818630320(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) Oct 13 20:00:43 fwbox local4:warn|warning fw07 %PIX-4-106023: Deny tcp src internet: 212.251.89.126/3859 dst 212.254.110.98/135 by access-group "internet_access_in" Oct 13 20:00:43 fwbox kernel: DROPPED IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:0f:cc: 81:40:94:08:00 SRC=212.251.89.126 DST=212.254.110.98 LEN=576 TOS=0x00 PREC=0x00 TTL=255 ID=8624 PROTO=TCP SPT=3859 DPT=135 LEN=556 • Log Standards ‣ SDEE ‣ WELF ‣ CEE (cee.mitre.org) ‣ CBE ‣ XDAS ‣ IDMEF Logging as a Service (c) by Ra fg ael Marty 13 Tuesday, July 6, 2010

Normalization • Parsers “To analyze or separate (input, for example) into more easily processed components.” (answers.com) • Generate a common output format for vis-tools (e.g., CSV) • For example /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})/g ‣ Regex ‣ http://secviz.org/content/parser-exchange Logging as a Service (c) by Ra fg ael Marty 14 Tuesday, July 6, 2010

Visualize 15 Tuesday, July 6, 2010

Choose Your Poison Logging as a Service (c) by Ra fg ael Marty 16 Tuesday, July 6, 2010

Reporting vs. Visualization • Reporting Libraries • Visualization Libraries - HighCharts - TheJIT - Flot - Graphael - Google Chart API - Protovis - Open Flash Chart - ProcessingJS - Flare JavaScript vs. Flash vs. XYZ Logging as a Service (c) by Ra fg ael Marty 17 Tuesday, July 6, 2010

HighCharts • Click-Through • On load - near real-time updates • AJAX data input via JSON • Zoom http://www.highcharts.com/ Logging as a Service (c) by Ra fg ael Marty 18 Tuesday, July 6, 2010

Google Visualization API http://code.google.com/apis/visualization/interactive_charts.html • JavaScript • Based on DataTables() • Many graphs • Playground - http://code.google.com/apis/ajax/playground Logging as a Service (c) by Ra fg ael Marty 19 Tuesday, July 6, 2010

ProtoVis • JavaScript based visualization library • Charting • Treemaps • BoxPlots • Parallel Coordinates • etc. http://vis.stanford.edu/protovis/ Logging as a Service (c) by Ra fg ael Marty 20 Tuesday, July 6, 2010

TheJIT http://thejit.org/ • JavaScript InfoVis Toolkit • Interactive • Link Graphs Logging as a Service (c) by Ra fg ael Marty 21 Tuesday, July 6, 2010

Processing •Visualization library •Java based •Interactive (event handling) •Number of libraries to - draw in OpenGL - read XML files - write PDF files •Processing JS - JavaScript http://processingjs.org/ - HTML 5 Canvas http://processing.org/ - Web IDE Logging as a Service (c) by Ra fg ael Marty 22 Tuesday, July 6, 2010

Building Your Own 23 Tuesday, July 6, 2010

Build Your Own AfterGlow Loggly Regexes Google Vis Logging as a Service (c) by Ra fg ael Marty 24 Tuesday, July 6, 2010

Data Collection in the Cloud 25 Tuesday, July 6, 2010

The (public) Cloud What it is Types • multi-tenancy • SaaS - Software • PaaS - Platform • elastic • IaaS - Infrastructure • “infinite” resources Benefits • pay as you go • No installation • self provisioning • No elaborate configurations It’s not • No maintenance • private data center • Great scalability • virtualization • 7x24 availability Logging as a Service (c) by Ra fg ael Marty 26 Tuesday, July 6, 2010

LaaS - Logging as a Service • All your data in one place • Loggly manages your data (index, store, archive, etc.) • Extremely fast search across all your data • Data source agnostic (no parsers) • Data management • access control • data segregation • data overview and summaries • API access Logging as a Service (c) by Ra fg ael Marty 27 Tuesday, July 6, 2010

Loggly Architecture Loggly Data Sources Clients user interface My syslog mobile-166 Data collection API Data access Proxies Distributed indexing and Indexers and Search Machines processing Distributed data store Logging as a Service (c) by Ra fg ael Marty 28 Tuesday, July 6, 2010

Loggly APIs • URL format: http://wiki.loggly.com/api-documentation http:// < subdomain > .loggly.com/api/< resource > • RESTful API HTTP Based - Access through: /api/< resource > • GET - read • POST - create - JSON, XML, JSONP output • PUT - update • Authentication • DELETE - delete - Basic auth - oAuth syslog to: http:// loggly. loggly.com/api/ search /?q=error User: guest / Password: loggly logs.loggly.com:514 Logging as a Service (c) by Ra fg ael Marty 29 Tuesday, July 6, 2010

Search http://[domain].loggly.com/api/search?q=404 { "data": [ { "indexed": "2010-07-03T17:17:38.909Z", "ip": "75.101.249.172", " text ": " Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au] [|domain] (DF) ", "inputname": "logglyweb", "timestamp": "2010-07-03 10:17:38" }, { "indexed": "2010-07-03T17:17:37.879Z", "ip": "75.101.249.172", " text ": " Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au] [|domain] (DF) ", "inputname": "logglyapp", "timestamp": "2010-07-03 10:17:37" }, ... Logging as a Service (c) by Ra fg ael Marty 30 Tuesday, July 6, 2010

Parser Oct 13 20:00:38.018152 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 62.2.32.250.53: 34388 [1au][|domain] (DF) Raw Oct 13 20:00:38.115862 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 192.134.0.49.53: 49962 [1au][|domain] (DF) Oct 13 20:00:38.157238 rule 57/0(match): pass in on xl1: 195.141.69.45.1030 > 194.25.2.133.53: 14434 [1au][|domain] (DF) (.*) rule ([-\d]+\/\d+)\(.*?\): (pass|block) (in|out) on (\w+): (\d+\.\d+\.\d+\.\d+)\.?(\d*) [<>] Regex / Parser (\d+\.\d+\.\d+\.\d+)\.?(\d*): (.*) Oct 13 20:00:38.018152,57/0,match,pass,in,xl1,195.141.69.45,1030,62.2.32.250,53,34388 [1au][|domain] (DF) Normalized Oct 13 20:00:38.115862,57/0,match,pass,in,xl1,195.141.69.45,1030,192.134.0.49,53,49962 [1au][|domain] (DF) (CSV) Oct 13 20:00:38.157238,57/0,match,pass,in,xl1,195.141.69.45,1030,194.25.2.133,53,14434 [1au][|domain] (DF) Logging as a Service (c) by Ra fg ael Marty 31 Tuesday, July 6, 2010

Visualize AfterGlow Parser Grapher CSV file Graph file digraph structs { graph [label="AfterGlow 1.5.8", fontsize=8]; node [shape=ellipse, style=filled, Configuration fontsize=10, width=1, height=1, fixedsize=true]; edge [len=1.6]; color.source =“green” if ($fields[0] ne “d”) "aaelenes" -> "Printing Resume" ; cluster.target =regex_replace("(\\d\+)\\.")."/8" "abbe" -> "Information Encryption" ; threshold.event =5 "aanna" -> "Patent Access" ; "aatharuv" -> "Ping" ; size.target =$fields[1] } http://afterglow.sf.net Logging as a Service (c) by Ra fg ael Marty 32 Tuesday, July 6, 2010

AfterGlow Cloud Loggly Grapher JSON CSV DOT Graph Logging as a Service (c) by Ra fg ael Marty 33 Tuesday, July 6, 2010