Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick Debugging Tricks with Apache HTTP Server Introduction 2.4 What kinds of issues encountered Using tools to look inside the Jeff Trawick web server Looking from http://emptyhammock.com/ the outside trawick@emptyhammock.com What if you build the code differently April 7, 2014 Compare with httpd 2.2 References and further reading
Get these slides... Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick Introduction What kinds of issues encountered http://emptyhammock.com/projects/info/slides.html Using tools to look inside the web server Looking from the outside What if you build the code differently Compare with httpd 2.2 References and further reading
Table of Contents Debugging Tricks with Apache HTTP 1 Introduction Server 2.4 Jeff Trawick 2 What kinds of issues encountered Introduction What kinds of 3 Using tools to look inside the web server issues encountered Using tools to 4 Looking from the outside look inside the web server Looking from 5 What if you build the code differently the outside What if you build the code 6 Compare with httpd 2.2 differently Compare with httpd 2.2 7 References and further reading References and further reading
Introduction — Who am I? Debugging Tricks with Apache HTTP Server 2.4 I’ve worked at Jeff Trawick several large corporations, for over two decades Introduction my own one-person company , Emptyhammock, for the last What kinds of two years issues encountered I’ve worked on Using tools to look inside the several products which were primarily based on or web server otherwise included Apache HTTP Server Looking from the outside lower-level networking products What if you web applications build the code differently I’ve debugged many customer and user problems over the Compare with years. httpd 2.2 References and further reading
Introduction — What will we attempt to cover? Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick Introduction What kinds of Touch on all the basics. issues encountered Describe all the new httpd 2.4 debugging features. Using tools to look inside the Summarize the techniques which are different with httpd web server 2.2. Looking from the outside What if you build the code differently Compare with httpd 2.2 References and further reading
What kinds of issues encountered Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick High CPU Crash Introduction High memory Hang of server What kinds of High consumption of other issues Stall of individual requests encountered pooled resources Termination Using tools to look inside the Incorrect output - wrong web server Bad response time transformation Looking from Limited concurrency the outside Incorrect output - What if you without problem missing/bad protocol build the code differently symptoms element Compare with httpd 2.2 References and further reading
Non-problems Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick Introduction What kinds of issues Validate behavior of new software/configuration encountered Understand steady-state behavior for baseline when Using tools to look inside the something is wrong web server Looking from the outside What if you build the code differently Compare with httpd 2.2 References and further reading
Using tools to look inside the web server Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick Logging (the information itself, the timestamp, information Introduction about other processing at about the same time) What kinds of issues OS-level tools (view use of resources, whether discrete encountered items like files or continuous like CPU) Using tools to look inside the CPU-, code-level tools (determine what code is running web server frequently, what is running for the request, analyze Looking from the outside memory references, walk through the processing of a What if you request, etc.) build the code differently Compare with httpd 2.2 References and further reading
Careful with logging! Debugging Tricks with Apache HTTP Server 2.4 As you increase the level of logging, you increase the chances Jeff Trawick that private data will be logged. Introduction Passwords, session keys, etc. What kinds of issues encountered Modules/log configurations of particular interest: Using tools to look inside the mod dumpio, mod log config when configured to log web server certain request or response header fields Looking from the outside mod log forensic What if you build the code http (the built-in module) when configured at higher trace differently levels Compare with httpd 2.2 References and further reading
Logging Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick Introduction What kinds of issues encountered Error log Using tools to look inside the web server Looking from the outside What if you build the code differently Compare with httpd 2.2 References and further reading
Error log records Debugging Tricks with Apache HTTP Server 2.4 Configurable content Jeff Trawick Fields dropped when information is unavailable Introduction Third-party modules can implement their own fields What kinds of issues encountered Typical message: Using tools to look inside the [Sun Oct 28 13:37:27.676386 2012] [:error] web server [pid 14340:tid 140625844377344] [client 127.0.0.1:50837] Looking from the outside mod_wsgi (pid=14340): Target WSGI script What if you '/home/trawick/myhg/apache/documents/AC20 build the code differently 12EU/lookup.wsgi' does not contain WSGI Compare with application 'application'. httpd 2.2 References and further reading
Hiding error log fields The ErrorLogFormat directive can limit which fields are logged, but you could implement post-processing Debugging to remove fields as appropriate for what you are debugging. Tricks with Apache HTTP My own silly mechanism: Server 2.4 Jeff Trawick $ grep Accept-Ranges logs/error_log [Thu Apr 03 07:26:49.605322 2014] [http:trace4] [pid 13680:tid 140130244732672] http_filters.c(837): $ grep Accept-Ranges logs/error_log | nots.pl Introduction [http:trace4] [pid 13680:tid 140130244732672] http_filters.c(837): [client 192.168.1.207:60141] $ grep Accept-Ranges logs/error_log | nots.pl | nomodlevel.pl What kinds of issues [pid 13680:tid 140130244732672] http_filters.c(837): [client 192.168.1.207:60141] Accept-Ranges: encountered $ grep Accept-Ranges logs/error_log | nots.pl | nomodlevel.pl | nopidtid.pl http_filters.c(837): [client 192.168.1.207:60141] Accept-Ranges: bytes Using tools to $ grep Accept-Ranges logs/error_log | nots.pl | nomodlevel.pl | nopidtid.pl | noclient.pl look inside the http_filters.c(837): Accept-Ranges: bytes web server Looking from the outside What if you build the code differently Compare with httpd 2.2 References and further reading
Detailed logging only for specified client IP Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick LogLevel info <If "%{REMOTE_ADDR} =~ /127.0.0/"> Introduction LogLevel trace8 What kinds of issues </If> encountered Using tools to look inside the web server Only works once request processing has reached a certain Looking from point. Connection-level issues which occur before that the outside point won’t be logged. What if you build the code Prior to 2.4.4, this expression needed to be placed inside a differently Location container to be effective. Compare with httpd 2.2 References and further reading
Detailed logging for problematic requests Debugging Tricks with Apache HTTP Server 2.4 Jeff Trawick Introduction What kinds of LogLevel info issues encountered <Location /problem/> Using tools to LogLevel trace8 look inside the web server </Location> Looking from the outside What if you build the code differently Compare with httpd 2.2 References and further reading
(Mostly) HTTP layer logging at different levels Debugging [core:trace5] Request received from client: GET / HTTP/1.1 Tricks with [http:trace4] Headers received from client: Apache HTTP [http:trace4] Connection: keep-alive Server 2.4 [http:trace4] Cache-Control: max-age=0 Jeff Trawick [http:trace4] User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KH... [http:trace4] Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Introduction [http:trace4] Accept-Encoding: gzip,deflate,sdch [http:trace4] Accept-Language: en-US,en;q=0.8 What kinds of [http:trace4] Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 issues [http:trace4] If-None-Match: \\"2d-4b1922bade1c0\\" encountered [http:trace4] If-Modified-Since: Sat, 12 Nov 2011 23:41:03 GMT Using tools to [http:trace3] Response sent with status 304, headers: look inside the [http:trace5] Date: Tue, 06 Nov 2012 12:18:57 GMT web server [http:trace5] Server: Apache/2.4.4-dev (Unix) OpenSSL/1.0.0e mod_wsgi/3.4 Python... Looking from the outside What if you build the code differently Compare with httpd 2.2 References and further reading
Recommend
More recommend
