Monitor with the Stack Philipp Krenn @xeraa 1 Infrastructure | - PowerPoint PPT Presentation

Monitor with the Stack Philipp Krenn ����� @xeraa 1

Infrastructure | Developer Advocate 2

Disclaimer This is not a training https://www.elastic.co/training 3

Who Is Using Elasticsearch Logstash and Kibana Beats 4

5

6

7

Starting Point https://github.com/xeraa/mongodb-monitoring 8

USB Sticks 9

Box Vagrant Ansible Provisioner 10

Credentials vagrant & vagrant 11

SSH $ ssh vagrant@127.0.0.1 -p 2222 -o PreferredAuthentications=password Windows: http://www.putty.org 12

Ansible $ cd /elastic-stack/ $ ls 13

14

REST $ curl -XGET -u "elastic:changeme" http://localhost:9200/ 15

16

Login http://localhost:5601 elastic & changeme 17

18

Filebeat 19

Filebeat Modules 20

System Dashboards 21

22

MongoDB Logs /var/log/mongodb/mongod.log 23

/etc/filebeat/filebeat.yml filebeat.prospectors: - input_type: log paths: - /var/log/mongodb/mongod.log document_type: mongodb 24

PS: Multiline Logs 25

/etc/filebeat/filebeat.yml filebeat.prospectors: - input_type: log paths: - /var/log/java-app/*.log document_type: java multiline.pattern: '^[[:space:]]' multiline.negate: false multiline.match: after 26

Test $ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20 27

Kibana Discover Limit Kibana view to the mongodb type 28

29

Grok Patterns https://github.com/logstash-plugins/logstash-patterns- core/blob/master/patterns/grok-patterns 30

MongoDB Patterns https://github.com/logstash-plugins/logstash-patterns- core/blob/master/patterns/mongodb 31

Building Patterns Grokdebug in Kibana 5.5+ (X-Pack Basic) https://grokdebug.herokuapp.com 32

33

/etc/filebeat/filebeat.yml output.elasticsearch: hosts: ["localhost:9200"] username: "{{ elastic_user }}" password: "{{ elastic_password }}" pipeline: "mongodb_log" 34

Console Pattern in a single line PUT _ingest/pipeline/mongodb_log { "description": "Ingest pipeline for MongoDB logs", "processors": [ { "grok": { "field": "message", "patterns": [ "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{MONGO3_SEVERITY:mongodb.severity} %{SPACE}%{MONGO3_COMPONENT:mongodb.component}%{SPACE} (?:[%{DATA:mongodb.context}])?%{SPACE}%{GREEDYDATA:message}" ] } } ] } 35

Filebeat Restart $ sudo service filebeat restart 36

Test $ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20 37

Refresh Field List Management → Index Patterns → filebeat-* → Refresh field list 38

Visualize mongodb.component of log events 39

40

41

Fetch the Slow Log 42

/etc/logstash/conf.d/00-mongodb- input.conf input { mongodb { uri => 'mongodb://127.0.0.1:27017/POCDB' placeholder_db_dir => '/var/local/logstash-mongodb/' collection => 'system.profile' batch_size => 500 generateId => true } } 43

/etc/logstash/conf.d/20-elasticsearch- output.conf output { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "mongodb-%{+YYYY.MM.dd}" document_type => "slowlog" user => "{{ elastic_user }}" password => "{{ elastic_password }}" } } 44

Logstash Restart $ sudo service logstash restart 45

Debug Logstash $ less /var/log/logstash/logstash-plain.log 46

Doesn't Work Getting data from system collections https://github.com/phutchins/logstash-input-mongodb/issues/8 47

Extension Idea Build an exec filter to run slow queries against MongoDB's .explain() 48

Metricbeat 49

Metricbeat System 50

Metricbeat Service 51

/etc/metricbeat/metricbeat.yml - module: mongodb metricsets: ["dbstats", "status"] hosts: ["localhost:27017"] 52

Metricbeat Restart $ sudo service metricbeat restart 53

Test $ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20 54

55

Visual Builder mongodb.status.network.in.bytes vs mongodb.status.network.out.bytes 56

57

Packetbeat 58

Protocols 59

Flows Application layer: Unsupported / encrypted (TLS) protocols IP / TCP / UDP Number of packets & bytes Retransmissions Temporal flow 60

/etc/packetbeat/packetbeat.yml packetbeat.protocols.mongodb: ports: [27017] 61

Packetbeat Restart $ sudo service packetbeat restart 62

Test $ java -jar /opt/pocdriver/bin/POCDriver.jar -k 20 -i 10 -u 10 -b 20 63

64

Heartbeat 65

Heartbeat ICMP, TCP, HTTP, HTTPS 66

/etc/heartbeat/heartbeat.yml heartbeat.monitors: - type: tcp hosts: ["127.0.0.1:27017"] schedule: '@every 10s' 67

Heartbeat Restart $ sudo service heartbeat restart 68

Test $ sudo service mongod stop $ sudo service mongod start 69

Visualize Up or down and TCP response times 70

71

72

Dashboard Combining visualizations 73

74

Winlogbeat 75

libbeat https://github.com/elastic/beats/tree/master/generate/beat 76

77

X-Pack Security Monitoring Graph Reporting Alerting Machine Learning 78

X-Pack Basic 79

Conclusion 80

81

82

83

Thanks! Questions? Philipp Krenn ����� @xeraa 84