SitePoint.com

redghost · Dec 6, 2004, 17:14

Hey I have this little site http://flipsource.net and notice Alexa (IA Archiver) on my webstat. Anyone know what the bot is from?

boccio · Dec 7, 2004, 00:33

Surprise, surprise... it comes from Alexa

Alexa uses google results for it's primary search, but crawls with ia_archiver mainly for archive.org purposes...

Alio · Dec 11, 2004, 21:08

I personally like to block it to save bandwidth. When you've got thousands of pages, it really does matter.

wildcard · Dec 12, 2004, 18:14

Quote:

Originally Posted by Alio

I personally like to block it to save bandwidth. When you've got thousands of pages, it really does matter.

You can simply put a robots.txt in there as http://www.archive.org/about/faqs.php#2 says but generally I'm happy to have it as I like to visit my site back in time.

Dec 6, 2004, 17:14	#1
redghost SitePoint Enthusiast Join Date: Oct 2003 Location: Canada Posts: 45	Alexa (IA Archiver) Hey I have this little site http://flipsource.net and notice Alexa (IA Archiver) on my webstat. Anyone know what the bot is from?

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Dec 7, 2004, 00:33	#2
boccio ko pročita magarac :) Join Date: Oct 2003 Location: belgrade Posts: 359	Surprise, surprise... it comes from Alexa Alexa uses google results for it's primary search, but crawls with ia_archiver mainly for archive.org purposes...

Dec 11, 2004, 21:08	#3
Alio SitePoint Member Join Date: Dec 2004 Location: Istanbul Posts: 12	I personally like to block it to save bandwidth. When you've got thousands of pages, it really does matter.