Customizing User Agent for URLConnections in Java

When using URLConnection in Java to retrieve web content, setting a custom user agent is often necessary for accurate website crawling and user simulation. However, the default Java runtime appends its own identifier to the user agent string, which may not be desirable in certain scenarios.

The Query

A Java developer sought assistance in setting a user agent without the "Java/1.5.0_19" suffix appended by the runtime. The user provided a code snippet illustrating how they attempted to set the user agent using setRequestProperty().

The Solution

Fortunately, in Java 1.6.30 and later, setting the user agent through setRequestProperty("User-Agent", "") works flawlessly, without any additional Java runtime information being added. To demonstrate this, the developer used netcat to listen for incoming HTTP requests and observed that the custom user agent was successfully sent without the Java suffix.

Full Implementation

import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;

public class CustomUserAgent {

    public static void main(String[] args) throws IOException {
        URL url = new URL("http://localhost:8080/foobar");
        URLConnection hc = url.openConnection();
        hc.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2");

        System.out.println(hc.getContentType());
    }

}

By utilizing this approach, developers can effectively set user agents for their URLConnections in Java without any unwanted Java-specific additions, ensuring the desired behavior for their web scraping or other communication tasks.