HTTP Module

The HTTP module can be used to analyze web (HTTP 1.0 [8] and HTTP 1.1 [5]) traffic from dumpfiles. The module can be run by passing in the -xhttp[P] option to tcptrace , where P represents the HTTP port number. By default, the module looks for web traffic in the TCP well known port 80. If your dumpfile has web traffic in port number P (not 80), you may pass it in as P in the command line as shown above.

The http module implicitly has the effect of the -e option (refer Section 8.3), and extracts data found from individual connections to data files of the form X2Y_contents.dat. Note that since the HTTP module needs the data from HTTP connections, it is important that the packet contents are fully captured in dumpfiles. With tcpdump for example, you need to ensure that an appropriate snaplen (with the -s option) value is chosen.

When the HTTP module is invoked as in

Beluga:/Users/mani> tcptrace -n -xhttp severus.dmp.gz

we get the following output.

mod_http: Capturing HTTP traffic (port 80)
1 arg remaining, starting with '/Users/mani/dmpfiles/standard/severus.dmp.gz'
Ostermann's tcptrace -- version 6.4.6 -- Tue Jul 1, 2003

4810 packets seen, 4810 TCP packets traced
elapsed wallclock time: 0:00:00.840086, 5725 pkts/sec analyzed
trace file elapsed time: 0:04:06.838094
TCP connection info:
  1: 103.196.209.147:15414 - 151.12.146.176:80 (a2b)   1601> 2671<
  2: 88.56.8.130:35457 - 199.19.215.115:80 (c2d)          1>    1<
  3: . . .
  . . .
 19: 247.139.32.218:2349 - 80.169.50.43:80 (ak2al)       10>   11<  (complete)
 20: 181.246.188.179:4482 - 133.114.250.132:80 (am2an)   26>   34<  (reset)  
Http module output:
103.196.209.147:15414 ==> 151.12.146.176:80 (a2b)
  Server Syn Time:              <the epoch>        (0.000)
  Client Syn Time:              <the epoch>        (0.000)
  Server Fin Time:      Thu May  1 14:33:34.998156 2003 (1051814014.998)
  Client Fin Time:      Thu May  1 14:33:35.734937 2003 (1051814015.735)
No additional information available, beginning of connection (SYNs) were not found in trace file.
88.56.8.130:35457 ==> 199.19.215.115:80 (c2d)
. . .
. . .
88.56.8.130:35458 ==> 21.28.79.90:80 (i2j)
  Server Syn Time:      Thu May  1 14:29:41.695579 2003 (1051813781.696)
  Client Syn Time:      Thu May  1 14:29:41.657188 2003 (1051813781.657)
  Server Fin Time:      Thu May  1 14:30:47.963562 2003 (1051813847.964)
  Client Fin Time:      Thu May  1 14:30:47.924972 2003 (1051813847.925)
    GET /main.asp HTTP/1.0
	Response Code:       200 (OK)
	Request Length:      438
	Reply Length:        30730
	Content Length:      30447
	Content Type  :      text/html
	Time request sent:   Thu May  1 14:29:41.700691 2003 (1051813781.701)
	Time reply started:  Thu May  1 14:29:45.189720 2003 (1051813785.190)
	Time reply ACKed:    Thu May  1 14:29:46.394593 2003 (1051813786.395)
	Elapsed time:  3489 ms (request to first byte sent)
	Elapsed time:  4694 ms (request to content ACKed)
    GET /poll-include.asp HTTP/1.0
	Response Code:       200 (OK)
	Request Length:      392
	Reply Length:        2038
	Content Length:      1836
	Content Type  :      text/html
	Time request sent:   Thu May  1 14:29:46.023979 2003 (1051813786.024)
	Time reply started:  Thu May  1 14:29:46.300471 2003 (1051813786.300)
	Time reply ACKed:    Thu May  1 14:29:49.254442 2003 (1051813789.254)
	Elapsed time:  276 ms (request to first byte sent)
	Elapsed time:  3230 ms (request to content ACKed)
    POST /poll-include.asp HTTP/1.0
	Response Code:       302 (Found)
	Request Length:      496
	Reply Length:        376
	Content Length:      137
	Content Type  :      text/html
	Time request sent:   Thu May  1 14:29:48.771677 2003 (1051813788.772)
	Time reply started:  Thu May  1 14:29:49.160714 2003 (1051813789.161)
	Time reply ACKed:    Thu May  1 14:29:50.939836 2003 (1051813790.940)
	Elapsed time:  389 ms (request to first byte sent)
	Elapsed time:  2168 ms (request to content ACKed)
    GET /poll-include.asp HTTP/1.0
        . . .
    GET /img/color.gif HTTP/1.0
        . . .
. . .
. . .

First, we see the regular output of tcptrace listing the 20 connections traced in the dumpfile, which is followed by the http module output. For the first connection labeled a2b, the module lists the time the FIN segments were received from the client and server. Since the SYN segments opening the connections were not captured in the dumpfile, the connection is incomplete, and hence, the module does not report any detailed HTTP information. The connection labeled i2j was complete however, and we see the times the SYN and FIN segments were received from the client and server respectively. This is followed by various HTTP requests and responses seen in the connection : GET /main.asp, GET /poll-include.asp, POST /poll-include.asp, ... etc., requesting (GET) or submitting (POST) files main.asp and poll-include.asp.

Let us see the information reported as part of the GET /main.asp HTTP request.

Besides listing HTTP information for connections as specified above, the module also generates the http.times file that lists for all the complete connections found, the time the connection was open, the times when the request/responses were received. The http.times file looks similar to the following :

conn 88.56.8.130:35458 21.28.79.90:80 i2j 2117 5 34953 5
reqrep 88.56.8.130:35458 21.28.79.90:80 i2j 1051813781.701 1051813785.190 
  1051813786.395 438 30730 200 GET /main.asp HTTP/1.0 text/html
reqrep 88.56.8.130:35458 21.28.79.90:80 i2j 1051813786.024 1051813786.300 
  1051813789.254 392 2038 200 GET /poll-include.asp HTTP/1.0 text/html
reqrep . . .
reqrep . . .
reqrep . . .
conn 231.65.90.63:63018 151.12.146.176:80 o2p 562 1 8558 1

The first line (beginning with conn) denotes the opening of a HTTP connection and lists the client and server endpoints (IP and port # : 88.56.8.130:35458, 21.28.79.90:80), the connection label assigned to the connection (i2j), the total request length in bytes (2117) found as the length of the file storing the contents of data from the client to server, the total number of requests found (5), the total reply length in bytes (34953) found as the length of the file storing the contents of data from the server to the client, and the total number of responses found (5). The first reqrep line shown above denotes the first request/response seen as part of the first connection. This line lists first the client and server endpoints and the connection label (i2j) assigned. The following fields list the timestamps when the request was sent (1051813781.701), when the first byte of the response is seen (1051813785.190), and when the response was ACKed ( 1051813786.395); the length of the request (438) and the response (30730); the response code (200) and the method it stands for (GET), the content requested (/main.asp HTTP/1.0) and the content type (text/html).

The http module also generates graphs for every client found in the dumpfile, plotting information on every web connection generated by the client.

A sample graph generated for all the web connections initiated by client 88.56.8.130, is shown in Figure 9.9.

Figure 9.9: HTTP Module Plot #1
Image http-1

Each of the long lines in the figure represent a web connection initiated by the client and their length represents the lifetime of the connection. The y-axis labeled URL doesn't mean anything specific, and is just an offset that begins from 1000 and is incremented by a constant value by the module for every web connection found in the dumpfile.

The ticks drawn below the lines represent the times when non-zero data segments were received from the server.

If we zoom into the beginning of the bottom-most connection shown in Figure 9.9, we get Figure 9.10.

Figure 9.10: HTTP Module Plot #2
Image http-2

The Clnt SYN and Serv SYN ticks on the line represent the times when the SYN segments were seen from the client and server respectively. Similarly Clnt FIN and Serv FIN found towards the end of the line represent the times when the FIN segments were seen from the client and server respectively.

In Figure 9.11, we zoom into the information printed on top of the connection lines. Each such small line segment represents a request-response seen in the connection. We can see in this Figure, the requests for main.asp, poll-include.asp, etc. The left diamond adjacent to the label /main.asp HTTP/1.0 represents the time when the request was seen. The length of the small line segment found towards the right represents the time elapsed receiving the response, with the left and right arrows representing the times when the first and last bytes of the response were received. The diamond on the right represents the time the response was ACKed, with the text field 30447 representing the length of the response.

Figure 9.11: HTTP Module Plot #3

Image http-3


Super-User 2003-08-29