How to set a referrer in java?

Get Started
1

Hi,
I am currently coding a robot in one of my class. The bot crawls websites, and colect data, for an artificial intelligence project.

I would like to set a referrer, when the robot crawls websites, so that webmasters doesnt think that their sites are beeing attacked.

The referrer would be a page where it would be explained what the project is, and a place where webmaster could enter their site, to remove themselves from my crawling list.

Anyone knows how to set a referrer in Java? I have been reading the Java API for a couple hours without luck.

Thanks all in advance!

2

I’m not sure what you are doing but the HttpURLConnection object lets you set headers, you can get a HttpURLConnection by something like this:


URL url = new URL("http://www....");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
3

I want webmasters think that my robot is coming from a specific url, so that if they go at that url, they will see why a robot came to their site.

I just checked the HttpURLConnection class, and there is no method to set an header?

Thanks,
Logan

4 

connection.addRequestProperty("REFERER", "http://www.mydomain.com");

Its actually a method of URLConnection.

5

Does it work?
I’m building a similar project… what ends up in the server logs?

6

Look into a product called HTTPClient. If I remember correctly, you will have to set a Runtime Property to get the Referrer to work properly.

Best of Luck.

BTW, in the future, it might be best to start a new thread and then point to the thread in question if you have a similar question.

7

In case of Java applications, we need to send REFERER as below :

httpUrlConnection.setRequestProperty(“REFERER”, “ID”);

In case of .NET applications, we need to send REFERER as below :

webClient.Headers.Set(“REFERER”, “ID”);