Web spiders or XML sitemap generator to extract links

I need to collect a list of all URLs from a particular website, in total around 3000. I was thinking of using a XML sitemap generator but I need one which supports logins as the website disables all links unless logged in.

Ideally a free tool would be great or anything up to $50

Any ideas ?

Which OS are you using, @AndyUK? I’m pretty sure gURLchecker does what you want, although I’ve never used it on a site which requires authentication. It’s only available for Linux, though, AFAIK.

The OS doesn’t really matter, I can fire up an instance of pretty much anything.

I’ll give that tool a try but I don’t think it’ll support site logins.

If you open the “new project” dialogue, there is an option for authentication:

Isn’t that for HTTP authentication rather than a PHP based login ?

Good question - I’ve never used that feature.

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.