Elastic S tack for S ecurity Monitoring 1 in a Nutshell 2019 Pass the S ALT Workshop
Overview 2 Introduction to Elastic S tack Beats Logstash Elasticsearch Kibana Elastic S tack Alerting and S ecurity
Introductory Workshop! 3 • This is an introductory workshop • Y ou probably won’ t hear/ see a lot of new things if you have: • Used Elastic S tack in the past; • Took the Elastic training… ; • Followed S ANS S EC455, S EC555, FOR572, etc.; • If you are stuck, please do not suffer in silence!
Workshop VM 4 • ais_workshop_xubuntu-18.04.2-desktop-amd64 • VMware Workstation, Player, or Fusion sorry! • Y ou can try VirtualBox too, but you are on your own with that… • 8 GB RAM • 30-50 GB disk space • Keyboard layout: EN-US !!! • Workshop VM (Ubuntu) user/ pass: user / Workshop1234% • Normally, it should not require password for login and sudo
About David 5 • Managing partner at Alzette Information S ecurity (@ AlzetteInfoS ec) • Network penetration testing, security architectures, security monitoring, incident response • Instructor at S ANS Institute: FOR572 • BS ides Luxembourg organizer https:/ / bsideslux.lu • Twitter: @ DavidS zili • E-mail: david.szili@ alzetteinfosec.com • Blog: http:/ / j umpespj ump.blogspot.com
About Eva 6 • Managing partner at Alzette Information S ecurity (@ AlzetteInfoS ec) • Web application penetration testing, source code review, security monitoring • CyberWayFinder • BS ides Luxembourg organizer https:/ / bsideslux.lu • Twitter: @ EvaS zilagyiS ec • E-mail: eva.szilagyi@ alzetteinfosec.com • Blog: http:/ / j umpespj ump.blogspot.com
Introduction to Elastic S tack 7 2019 Pass the S ALT Workshop
About Elastic S tack 8 What is Elastic Stack? Why Elastic Stack? • 4 main components: • (Free) Open S ource S oftware • Elasticsearch • Distributed, real-time search and • Logstash analytics (very scalable) • Kibana • Parsing and data enrichment • Beats • Large Community • And several other smaller • InfoS ec Proj ects built around it: components • S ecurity Onion • Elastic S tack Features (X-Pack) • Moloch (Elasticsearch) • APM (Application Performance • S OF-ELK Monitoring) • S ELKS • HELK • ROCK NS M
Elastic S tack History 9 2019: Core security 2015: "Release 2017: Elastic features (TLS , Early 2000s: Bonanza“ , Cloud RBAC) are S hay Banon’s Beats, Elastic Enterprise free, S IEM, Recipe App Cloud (AWS ) (ECE) EndGame 2012: 2016: Elastic 2018: Open Elasticsearch S tack 5.0 source X-Pack, Inc. New Y ork S tock Exchange S ource: https:/ / www.elastic.co/ about/ history-of-elasticsearch
Elastic S tack (Very) High-Level Overview 10 • Beats : single-purpose data shippers • Logstash : server-side data processing pipeline • Elasticsearch : distributed search and analytics engine • Kibana : visualization and dashboards S ee also: https:/ / www.elastic.co/ assets/ blt2614227bb99b9878/ architecture-best-practices.pdf
Beats 11 2019 Pass the S ALT Workshop
Beats: Lightweight Data S hippers 12 • Lightweight log agents • Written in Go • Can send to Logstash or directly to Elasticsearch • Beats Family: • Filebeat • Winlogbeat • Auditbeat • Packetbeat • Heartbeat • Metricbeat • Functionbeat • Etc. S ee also: https:/ / www.elastic.co/ guide/ en/ beats/ libbeat/ current/ index.html
Beats Configuration Examples 13 Winlogbeat Filebeat
Beats Hands-On 14 2019 Pass the S ALT Workshop
Logstash 15 2019 Pass the S ALT Workshop
Logstash Overview 16 • LOTS AND LOTS of plugins! • Input : tcp, udp, syslog , beats , jdbc , kafka, rabbitmq, file , exec, cloudwatch, etc. • Filter : csv , json , xml , kv , grok , date , mutate , split, useragent, ruby, drop , etc. • Output : elasticsearch , graphite, nagios, kafka, rabbitmq, radis, file, email , irc, etc. • Easy to learn and use Input Filter Output S ee also: https:/ / www.elastic.co/ guide/ en/ logstash/ current/ index.html
Input Plugin Examples 17 i nput { Plugin Description s t di n { beats Events from Elastic Beats } } cloudwatch Events from AWS CloudWatch file S treams events from files i nput { be a t s { jdbc Events from JDBC data por t => 5044 } kafka Reads events from Kafka } rabbitmq Pulls events from RabbitMQ i nput { s3 Events from files in S 3 s ys l og { por t => 5514 snmp Polls devices using S NMP } syslog Reads syslog messages } S ee also: https:/ / www.elastic.co/ guide/ en/ logstash/ current/ input-plugins.html
Filter Plugin Examples 18 Plugin Description Plugin Description cidr Check IP against net blocks kv Parses key-value pairs csv mutate Parses CS V data into fields Performs mutations on fields date Parses dates from fields ruby Executes Ruby code dissect Extracts unstructured data split S plits multi-line messages drop Drops all events translate Replaces field contents elasticsearch Gets data from Elasticsearch truncate Truncates fields geoip Geo info about an IP urldecode Decodes URL-encoded fields grok Parses unstructured data useragent Parses user agent strings json Parses JS ON data xml Parses XML data S ee also: https:/ / www.elastic.co/ guide/ en/ logstash/ current/ filter-plugins.html
Filters - The Easy S tuff 19 JSON CSV f i l t e r { f i l t e r { . . . . . . c s v { j s on { c ol um ns => [ " t s " , " ui d" , " i d. or i g_h" , s our c e => " m e s s a ge “ " i d. or i g_p" , " i d. r e s p_h" , " i d. r e s p_p" , " pr ot o" , } " s e r vi c e " , " dur a t i on" , " or i g_byt e s " , . . . " r e s p_byt e s c onn_s t a t e " , " l oc a l _or i g" , " l oc a l _r e s p" , m ut a t e { " m i s s e d_byt e s " , " hi s t or y" , " or i g_pkt s " , " or i g_i p_byt e s " , r e m ove _f i e l d => [ " m e s s a ge " ] " r e s p_pkt s " , " r e s p_i p_byt e s " , " t unne l _pa r e nt s " ] s e pa r a t or => " “ } } } . . . m ut a t e { r e m ove _f i e l d => [ " m e s s a ge " ] } }
Filters - RegExp vs. Grok, Dissect (1) 20 RegExp Dissect • S tring-based split operation • (? <![0-9])(? :(? :25[0-5]| 2[0- 4][0-9]| [0-1]? [0- • Very fast 9]{1,2})[.](? :25[0-5]| 2[0-4][0- Grok 9]| [0-1]? [0-9]{1,2})[.](? :25[0- • % {IPV4:source_ip} 5]| 2[0-4][0-9]| [0-1]? [0- • Pre-cooked RegExp patterns 9]{1,2})[.](? :25[0-5]| 2[0-4][0- 9]| [0-1]? [0-9]{1,2}))(? ![0-9]) • Custom Patterns: • (? <queue_id>[0-9A-F]{10,11}) Grok Debuggers: • Heroku App: http:/ / grokdebug.herokuapp.com • S ource: https:/ / github.com/ nickethier/ grokdebug • Docker: https:/ / hub.docker.com/ r/ fdrouet/ grokdebug • Kibana / Dev Tools / Grok Debugger
Filters - RegExp vs. Grok, Dissect (2) 21 dissect grok f i l t e r { f i l t e r { . . . . . . di s s e c t { gr ok { m a ppi ng => { m a t c h => { " m e s s a ge " => " % {t s } % {+t s } " m e s s a ge " => % {+t s } % {s r c } % {pr og}[ % {pi d}] : % {m s g}“ " % {SYSLOGTI M ESTAM P: s ys l og_t i m e s t a m p} } % {SYSLOGHOST: s ys l og_hos t na m e } } % {DATA: s ys l og_pr ogr a m }( ? : \ [ % {POSI NT: s ys l og_pi d}\ ] ) ? : . . . % {GREEDYDATA: s ys l og_m e s s a ge }" } #" m e s s a ge " => " % {SYSLOGBASE2} % {GREEDYDATA: m e s s a ge }“ } } . . . }
Filters - Enrichment Examples 22 ruby geoip f i l t e r { f i l t e r { . . . . . . i f [ pr ogr a m ] == " br o_dns " { i f [ r e s p_h_r out a bl e ] == " t r ue " { r uby { ge oi p { c ode => " e ve nt . s e t ( ' que r y_l e ngt h' , s our c e => " i d. r e s p_h“ e ve nt . ge t ( ' que r y' ) . l e ngt h) “ t a r ge t => " ge oi p“ } de f a ul t _da t a ba s e _t ype => " Ci t y“ } } . . . ge oi p { } s our c e => " i d. r e s p_h“ t a r ge t => " ge oi p“ de f a ul t _da t a ba s e _t ype => " ASN“ } } . . . }
Output Plugin Examples 23 Plugin Description out put { csv Writes events to disk in CS V s t dout { elasticsearch S tores logs in Elasticsearch c ode c => r ubyde bug } email S ends email to an address } exec Runs a command file Writes events to files out put { graphite Writes metrics to Graphite e l a s t i c s e a r c h { hos t s => [ " l oc a l hos t : 9200" ] kafka Writes events to Kafka } rabbitmq Pushes events to RabbitMQ } redis S ends events to Redis S ee also: https:/ / www.elastic.co/ guide/ en/ logstash/ current/ output-plugins.html
Recommend
More recommend