Which is the best robust website spidering software around?

I intend to create a web spider. It needs to crawl webpages at specified intervals. I’m looking for suggestions from you, Please let me know which are all the best robust background spider softwares that are available ? I can use that and then build on top of it.

It can be coded in any programming language.