Clustered Logging with mod_log_spread Theo Schlossnagle - PowerPoint PPT Presentation

Clustered Logging with mod_log_spread Theo Schlossnagle <jesus@omniti.com>

Theo Schlossnagle Pricipal @ OmniTI open - source developer mod_backhand W ackamole daiquiri OpenSSH/SecurID Spread etc. closed - source developer Ecelerity MTA Portcullis Anti - Spam The Speaker 2

Agenda Understanding the Problem Space A Survey of Technologies Implementing Clustered Logging Understanding New Possibilities 3

Understanding the Problem Space

The Purpose of Logging Journalling the fact that a transaction has taken place. Correlating a series of transactions into a session. An audit trail. Forensics. Activity analysis to understand current trends and predict the future. 5

Logs are reliable. Events are logged in the order they occur. They can be partitioned by date. They can be multiplexed and demultiplexed on Basic demand. Expectations 6

Introducing Clustering Clustering: several machines acting together to provide a single servic e Sessions may now be composed of a series transactions that occur on di ff erent machines. Ordering is “ harder ” and more important. 7

A Survey of Technologies

Traditional Logging Logs written locally on web servers space must be allocated Consolidation happens periodically crashes will result in missing data aggregators must preserve chronology real - time metrics cannot be calculated Monitors must run against log servers monitors must tail log fi les requires resources on the log servers 9

Traditional Approach: Logging in its infancy

Active Network Logging Logs written directly to log servers UDP is unreliable and thus not useful TCP is a point - to - point protocol Two log server mean double tra ffi c Add a monitor and that ’ s triple! Real - time metrics are possible monitors must tail log fi les still ( or publishers must send directly to the monitors... yuck! ) 11

Network Approach: Adolescent Logging

Passive Network Logging Logs constructed from sni ff ed tra ffi c The players no longer matter W eb servers can be added easily Drops logs! When tested head - to - head with active logging frameworks we see loss Missing logs is unacceptable 13

Passive Logging: A lapse in judgement

mod_log_spread Logging Logs are published over Spread E ffi cient reliable network multicast Preserves global ordering of logs Multiple subscribers at no cost well... almost zero Extends well beyond Apache All logging ( enterprise wide ) can be utilize this publish/subscribe messaging bus 15

mod_log_spread: Mature Logging

With clustered logging we get: instant aggregation ordering publish/subscribe model multiple subscribers multiple subscribers multiple subscribers... 17

Data “ feeds ” Write them to disk Understand load - balanced click streams Real - time analysis: Multiple popular pages concurrent sessions Subscriber Who ’ s online? 18 Magic

Implementing Clustered Logging

Spread Apache 1.3 or 2.0 mod_log_spread spreadlogd A spread client API for your favorite language: Perl, Python, C Okay, so Java, Ruby, PHP , etc. show me! 20

Get and install Spread http://www.spread.org/ A simple /etc/spread.conf: DebugFlags = { EXIT CONFIGURATION } EventLogFile = /var/log/spread/mainlog EventTimeStamp Spread_Segment 10.225.209.255:4913 { # order matters admin - va - 1 10.225.209.68 # staging server www - va - 1 10.225.209.71 www - va - 2 10.225.209.72 www - va - 3 10.225.209.73 samwise 10.225.209.240 # logging machines gollum 10.225.209.241 # monitoring machine } 21

Get and install mod_log_spread http://www.backhand.org/ A simple httpd.conf: LoadModule log_spread_module libexec/mod_log_spread.so AddModule mod_log_spread.c #AddModule mod_log_con fi g.c SpreadDaemon 4913 LogFormat “% h % l % u % t \”% r \” % >s % b ” common <VirtualHost coolsiteip:80> CustomLog $ coolsite common </VirtualHost> <VirtualHost slicksiteip:80> CustomLog $ slicksite common </VirtualHost> 22

V erify by hand that it is working... ; /opt/spread/bin/spuser - s 4913 User: connected to 4913 with private group #user#admin - va - 1 User> j coolsite ============================ Received REGULAR membership for group coolsite with 2 members, where I am member 1: #user#admin - va - 1 grp id is 182571332 1092928408 2 Due to the JOIN of #user#admin - va - 1 User> ============================ received RELIABLE message from #ap25454#admin - va - 1, of type 1, ( endian 0 ) to 1 groups ( 182 bytes ) : 68.55.183.91 - - [ 30/Oct/2004:11:48:51 - 0400 ] “ GET / ~ jesus/ HTTP/1.1 ” 200 57940 “-” “ Mozilla/5.0 ( Macintosh; U; PPC Mac OS X; en ) AppleW ebKit/125.5 ( KHTML, like Gecko ) Safari/125.9 ” ... ... ... 23

Get spreadlogd and install it: http://www.backhand.org/mod_log_spread/ A simple /etc/spreadlogd.conf: Bu ff erSize = 65536 Spread { Port = 4913 Log { RewriteTimestamp = CommonLogFormat Group = “ coolsite ” File = /data/logs/apache/coolsite/common_log } Log { RewriteTimestamp = CommonLogFormat Group = “ slicksite ” File = /data/logs/apache/slicksite/combined_log } } 24

Spreadlogd Super advanced Kung - Fu Bu ff erSize = 65536 PerlLib /opt/spreadlogd/custom PerlUse mylogger Spread { Port = 4913 Log { RewriteTimestamp = CommonLogFormat Group = “ coolsite ” PerlLog mylogger::log File = /data/logs/apache/coolsite/common_log } Log { RewriteTimestamp = CommonLogFormat Group = “ slicksite ” File = /data/logs/apache/slicksite/combined_log } } 25

Spreadlogd Super advanced Kung - Fu ( continued ) package mylogger; use DBI; our $ dbh; our $ sth; sub log ($$$) { my $ sender = shift; my $ group = shift; my $ message = shift; my ($ user, $ host ) = ($ sender = ~ /# ([^ # ] + ) # ([^ # ] + ) / ) ; chomp ($ message ) ; $ dbh || = DBI - >connect (” DBI:mysql:database=weblogs ” , “ logger ” , “” , { RaiseError => 0 }) ; warn “ DBI - >connect failed. ” unless ($ dbh ) ; if ($ dbh ) { $ sth || = $ dbh - >prepare ( q { INSERT INTO logs ( host, group, timestamp, data ) VALUES ( ?,?,NOW () ,? )}) ; $ sth - >execute ($ host, $ group, $ message ) ; } } 26

Understanding New Possibilities

Logs are now streaming in real time Real - time metrics per server hit rates ( tra ffi c ) per server hits by response code relative error serving rate per server document size metrics detect unexpected bugs do to anomalous tra ffi c T rack deeper data user habits length of visit online All this happens passively!!! 28

On community sites and social networking sites it is vital to understand who is online and what they are looking at. Having that in real - time a ff ord unprecedented opportunity. Why not expose the information to a user ’ s peers as a value add. Who’s Online? 29

The concept: Skiplist Index on: username url,hitdate hitdate Single thread, event driven Receive messages from Spread: parse username, url, hitdate delete from skiplist by username O ( lg n ) insert into skiplist O ( lg n ) pop the end of the skiplist of any hitdate > 30 minutes O ( 1 ) Receive client requests Cardinality query is O ( 1 ) , single write () Users on a url query is O ( lg ) , O ( 1 ) to fi ll out iovec, single writev () 30

The result: 6 hours worth of coding and testing 800 lines of C code ( server ) use libspread and libskiplist 40 lines of perl ( client module for web app ) On commodity hardware ($ 2k box ) ~ 80,000 inserts/second more hits than we ’ ll ever see! ~ 100,000,000 counts/second -- not including write () ~ 500,000 users for urls/second -- not including writev () Add a bit more e ff ort and expose this data over SOAP or SNMP or something clever and you can integrate it seamlessly into your overall architecture. 31

Stupid Pet Tricks 32

Credit Where Credit’s Due

The John Hopkins University The Center for Networking and Distributed Systems OmniTI Computer Consulting The Authors and Contributors of Spread: Yair Amir, Michal Miskin-Amir, Jonathan Stanton, Christin Nita-Rotaru, Theo Schlossnagle, Dan Schoenblum, John Schultz, Ryan Caudy, Ben Laurie, Daniel Rall, Marc Zyngier The Authors of mod_log_spread and Tools: George Schlossnagle, Theo Schlossnagle, Jonathan Stanton, Yair Amir Questions?