Jakarta HttpClient: reading response data from a get method - getResponseBody() gives you a byte array, but don't forget that there is a getResponseBodyAsStream() which gives you back an InputStream...

Here's an example of how to set system properties from within Java code. The following is setting up the HttpClient to log all debug messages, but similar things apply to any other system properties etc. that you care to use:

System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.SimpleLog");
System.setProperty("org.apache.commons.logging.simplelog.showdatetime", "true");
System.setProperty("org.apache.commons.logging.simplelog.log.httpclient.wire", "debug");
System.setProperty("org.apache.commons.logging.simplelog.log.org.apache.commons.httpclient", "debug");

Here's some sample code and notes for creating a Jakarta Commons Httpclient making use of the timeout mechanism:

Notes:
1. main() executes the static execute() method of TimeoutController, passing it an instance of the GetWithTimeout which implements Runnable, and a timeout parameter in millisecs
2. run() in GetWithTimeout does the standard stuff to do an Http GET. Note that I have included a Thread.sleep() call for testing so I could force it to timeout.
3. If the timeout expires, two things happen. The worker thread gets interrupted, causing an InterruptedException in the run() method - at which point it exits. Also, a TimeoutController.TimeoutException is thrown in main(), to inform the caller of what's happened.

I haven't figured out what would happen if it really did timeout before the releaseConnection() method was called. Does that mean the socket will just hang around until it finally gets timed out by the OS? (7mins default socket T/O on WinNT apparently).

Sample Code:

/**
* sample with Timeout
*/
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.*;
import org.apache.commons.httpclient.util.*;
import java.io.*;

public class JakartaHttpClientTimeoutTest {

public static void main(String[] args) {
try {
TimeoutController.execute(new GetWithTimeout(),30000L); // i.e. 30 seconds
}
catch (TimeoutController.TimeoutException to) {
System.out.println("Timeout popped!");
}
}
}

class GetWithTimeout implements Runnable {

public void run() {

String url = "http://www.jeannot.uklinux.net/";

// Create an instance of HttpClient.
HttpClient client = new HttpClient();

// Create a method instance.
HttpMethod method = new GetMethod(url);

// Execute the method.
int statusCode = -1;
int attempt = 0;
// We will retry up to 3 times.
while (statusCode == -1 && attempt < 3) {
try {
// execute the method.
statusCode = client.executeMethod(method);
} catch (HttpRecoverableException e) {
System.err.println(
"A recoverable exception occurred, retrying." +
e.getMessage());
} catch (IOException e) {
System.err.println("Failed to download file.");
e.printStackTrace();
System.exit(-1);
}
}
// Check that we didn't run out of retries.
if (statusCode == -1) {
System.err.println("Failed to recover from exception.");
System.exit(-2);
}

// Read the response body.
byte[] responseBody = method.getResponseBody();

// Release the connection.
method.releaseConnection();

try {
Thread.sleep(15000L);
}
catch (InterruptedException e) {
System.exit(-3);
}

// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
System.err.println(new String(responseBody));
}
}

Initial notes on Jakarta Commons HttpClient
When using this package, it also requires the Jakarta commons-logging.jar ...
Tried the sample code which just does a simple run thru of the basic process:
- create an HttpClient instance
- create a method instance (the GetMethod if you're retrieving data from an HTTP server)
- tell the HttpClient to execute the method
- read the response (all of it!)
- release the connection (don't forget this bit)
- then do whatever you need to do with the returned data
The bit that allows you to have the method timing out is based round the org.apache.commons.httpclient.util.TimeoutController which allows you to run a query in a separate thread and wait for a timeout. I haven't tried this bit yet...

All the above information is contained in the supporting documentation on the Jakarta site. It helps me make sense of it if I precis it in this weblog though!

The last link added is the JavaDoc for the Http client in Jakarta Commons.
Main page is http://jakarta.apache.org/commons/httpclient/

The stuff I was on about a few days ago re. a non-blocking HTTP connection... the option where you try to start and kill off Threads on the fly was no good (methods deprecated), the www.logicamente.com stuff looked interesting but in fact the outbound query doesn't get interrupted because the response timeout (InterruptedIOException) gets caught by the underlying I/O routines and ignored... so that doesn't help.

HOWEVER: I have just heard that there might be a solution in Jakarta Commons:
http://jakarta.apache.org/commons/httpclient/apidocs/index.html

Debugging Java: use the "jdb" program provided with the SDK. This allows you to step thru a method, set breakpoints etc.
C:\java> jdb
> stop in ClassName.main
> run ClassName params
> next
> etc. etc.

Further to the previous stuff about HttpURLConnection and needing to use Thread.stop() to kill threads which got no response (even though they are blocked), I have tried something else. The ThreadDeath solution didn't work really well and it is much harder to understand what is going on. And it's not recommended.

However there is some sample code at http://www.logicamente.com/sockets.html which implements a non-blocking HTTP connection and a timeout mechanism, based around overriding the basic classes from Sun. I have tried it out with my testing and it seems to work fine. Definitely worth a try...