Event Logging System for Collaborative Communities Divyansh Sharma - PowerPoint PPT Presentation

Event Logging System for Collaborative Communities Divyansh Sharma Kartik Verma Rahul Raj Vishrut Jetly July 4, 2018 Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 1 / 20

Outline Storing Logs in Elasticsearch 6 Motivation 1 Event Log REST API 7 Technologies Used 2 User Analytics 8 Work Flow 3 Problems Solved 9 Capturing the Events 4 10 Future Scope Creating the Log 5 Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 2 / 20

Motivation Motivation What is Event Logging ? Logging or storing the activities done by user of the system e.g. login, view a community, view article, edit article etc. Adding an Event Logging System which can be easily integrated with the current system without affecting it. Adding an Event Logging System which can be easily extended to capture new events whenever they are added. Providing information For recommendation System. Providing User Analytics Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 3 / 20

Technologies Used Technologies Used Django Middleware ELK stack C3.js Django REST framework Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 4 / 20

Technologies Used Technologies Used Django Middleware C3.js Django REST Framework Every activity that gen- C3.js is used to im- erates a http request to plement high quality, Django REST frame- the server first passes interactive charts for work is a powerful through these middle- visualisation. Charts and flexible toolkit for wares which attach and built using this library building Web APIs. Its extract some informa- can be rendered in real a way to expose parts tion from the request time. of the application to e.g. session id, authen- third-parties tication tokens etc. Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 5 / 20

Technologies Used Technologies Used ELK Stack ELK is acronym for three open source projects Elasticsearch Logstash Kibana Logstash receives the data from a file or through an http request, applies filters, and then Elasticsearch stores the data, which can be later retrieved using Elasticsearch API. Kibana is used for visualisations and analytics. Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 6 / 20

Work Flow Work Flow Diagram Figure: Work Flow Diagram Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 7 / 20

Capturing the Events Capturing the Events Figure: Django Middleware Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 8 / 20

Capturing the Events Capturing the Events The request after passing through all the middlewares finally moves through the eventlog middleware which captures all the incoming information from the request. This is a reason why the eventlog module is kept below all the middlewares. Primary Benefit of using middleware is that it keeps the eventlog system as a completely separate module or app which can be reused in django projects. Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 9 / 20

Creating the Log Creating the Log Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 10 / 20

Creating the Log Important Points The size of the bucket holding logs is not fixed. The function which processes the dict present in bucket runs in a seperate daemon thread. Logs can be stored either in file or in elasticsearch but here we are showing it only with elasticsearch because storing logs in a file are not useful. Logs are passed to ELK in JSON format. Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 11 / 20

Storing Logs in Elasticsearch Storing Logs in Elasticsearch Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 12 / 20

Storing Logs in Elasticsearch Important Points Elasticsearch, Logstash and Kibana runs on separate ports inside a docker. The name of the index which stores the logs is logs but can be changed in settings. Kibana is available only for the Super Admin users. Elasticsearch uses DSL query syntax built over Apache lucene Query Syntax parser for searching. Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 13 / 20

Event Log REST API Event Log REST API Event log System provides a feature rich REST based API which is a wrapper over the Elasticsearch search API. This API is used for Recommendation System and for User Analytics Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 14 / 20

Event Log REST API Features of REST API The eventlog REST API is a feature rich API. The notable ones are: Pagination Sorting(both in ascending and descending) Custom Fields Filters based searching Time Range Based searching Aggregations or group by Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 15 / 20

User Analytics User Analytics User Analytics provides an overview of the activities of the users. Some of the Analytics added are: Trending articles in a community or group (based on logs from 10 days) Most viewed articles in community Users most viewed articles Users own articles in various states Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 16 / 20

User Analytics Some Snapshots Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 17 / 20

Problems Solved Problems Solved Added the Event Logging System without affecting the existing system in any way using middleware. Added a System which can be easily modified to add new events for capture. Provided a feature rich API for use by recommendation system and for user analytics. Added User Analytics to the current System. Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 18 / 20

Future Scope Future Scope Adding browser events i.e. events that are generated in front end, for instance Window resize events Scroll events These events may provide some useful information about human behaviour Adding Video related events Adding more User Analytics Add real time analytics whenever logs are updated Storing logs in cloud storage. Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 19 / 20

Future Scope Thank You Divyansh, Kartik, Rahul, Vishrut Event Logging System July 4, 2018 20 / 20