PHP : Web Automation - [11.7] Debugging the Raw HTTP Exchange

11.7.1 Problem

You want to analyze the HTTP request a browser makes to your server and the corresponding HTTP response. For example, your server doesn't supply the expected response to a particular request so you want to see exactly what the components of the request are.

11.7.2 Solution

For simple requests, connect to the web server with telnet and type in the request headers:

% telnet www.example.com 80
Trying 10.1.1.1...
Connected to www.example.com.
Escape character is '^]'.
GET / HTTP/1.0
Host: www.example.com

HTTP/1.1 200 OK
Date: Sat, 17 Aug 2002 06:10:19 GMT
Server: Apache/1.3.26 (Unix) PHP/4.2.2 mod_ssl/2.8.9 OpenSSL/0.9.6d
X-Powered-By: PHP/4.2.2
Connection: close
Content-Type: text/html

// ... the page body ...

11.7.3 Discussion

When you type in request headers, the web server doesn't know that it's just you typing and not a web browser submitting a request. However, some web servers have timeouts on how long they'll wait for a request, so it can be useful to pretype the request and then just paste it into telnet. The first line of the request contains the request method (GET ), a space and the path of the file you want (/), and then a space and the protocol you're using (HTTP/1.0). The next line, the Host header, tells the server which virtual host to use if many are sharing the same IP address. A blank line tells the server that the request is over; it then spits back its response: first headers, then a blank line, and then the body of the response.

Pasting text into telnet can get tedious, and it's even harder to make requests with the POST method that way. If you make a request with HTTP_Request , you can retrieve the response headers and the response body with the getResponseHeader( ) and getResponseBody( ) methods:

require 'HTTP/Request.php';

$r = new HTTP_Request('http://www.example.com/submit.php');
$r->setMethod(HTTP_REQUEST_METHOD_POST);
$r->addPostData('monkey','uncle');
$r->sendRequest();

$response_headers = $r->getResponseHeader();
$response_body    = $r->getResponseBody();

To retrieve a specific response header, pass the header name to getResponseHeader( ). Without an argument, getResponseHeader( ) returns an array containing all the response headers. HTTP_Request doesn't save the outgoing request in a variable, but you can reconstruct it by calling the private _buildRequest( ) method:

require 'HTTP/Request.php';

$r = new HTTP_Request('http://www.example.com/submit.php');
$r->setMethod(HTTP_REQUEST_METHOD_POST);
$r->addPostData('monkey','uncle');

print $r->_buildRequest();

The request that's printed is:

POST /submit.php HTTP/1.1
User-Agent: PEAR HTTP_Request class ( http://pear.php.net/ )
Content-Type: application/x-www-form-urlencoded
Connection: close
Host: www.example.com
Content-Length: 12

monkey=uncle

With cURL, to include response headers in the output from curl_exec( ), set the CURLOPT_HEADER option:

$c = curl_init('http://www.example.com/submit.php');
curl_setopt($c, CURLOPT_HEADER, 1);
curl_setopt($c, CURLOPT_POST, 1);
curl_setopt($c, CURLOPT_POSTFIELDS, 'monkey=uncle&rhino=aunt');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
$response_headers_and_page = curl_exec($c);
curl_close($c);

To write the response headers directly to a file, open a file handle with fopen( ) and set CURLOPT_WRITEHEADER to that file handle:

$fh = fopen('/tmp/curl-response-headers.txt','w') or die($php_errormsg);
$c = curl_init('http://www.example.com/submit.php');
curl_setopt($c, CURLOPT_POST, 1);
curl_setopt($c, CURLOPT_POSTFIELDS, 'monkey=uncle&rhino=aunt');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_WRITEHEADER, $fh);
$page = curl_exec($c);
curl_close($c);
fclose($fh) or die($php_errormsg);

The cURL module's CURLOPT_VERBOSE option causes curl_exec( ) and curl_close( ) to print out debugging information to standard error, including the contents of the request:

$c = curl_init('http://www.example.com/submit.php');
curl_setopt($c, CURLOPT_VERBOSE, 1);
curl_setopt($c, CURLOPT_POST, 1);
curl_setopt($c, CURLOPT_POSTFIELDS, 'monkey=uncle&rhino=aunt');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
$page = curl_exec($c);
curl_close($c);

This prints:

* Connected to www.example.com (10.1.1.1)
> POST /submit.php HTTP/1.1
Host: www.example.com
Pragma: no-cache
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Content-Length: 23
Content-Type: application/x-www-form-urlencoded

monkey=uncle&rhino=aunt* Connection #0 left intact
* Closing connection #0

Because cURL prints the debugging information to standard error and not standard output, it can't be captured with output buffering, as Section 10.11 does with print_r( ). You can, however, open a file handle for writing and set CURLOUT_STDERR to that file handle to divert the debugging information to a file:

$fh = fopen('/tmp/curl.out','w') or die($php_errormsg);
$c = curl_init('http://www.example.com/submit.php');
curl_setopt($c, CURLOPT_VERBOSE, 1);
curl_setopt($c, CURLOPT_POST, 1);
curl_setopt($c, CURLOPT_POSTFIELDS, 'monkey=uncle&rhino=aunt');
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_STDERR, $fh);
$page = curl_exec($c);
curl_close($c);
fclose($fh) or die($php_errormsg);