Check if a website has changed its content

I would like to know if there is a way to know if a website has changed its content. I use java to parse websites, but as there are thousands of products, I would like to instead parsing every single URL, parse only the ones that have been changed.

There is or was a very good free application that was configurable to do just about everything as far as downloading, checking, etc. Unfortunately I cannot remember the name. It was something like Xanadu and had some sort of skull as an avatar.

Can anyone else remember?

I also think it was written for Windows only…

I can’t think of any reliable method that wouldn’t periodically require crawling the current page(s) and comparing the previous data. Are you allowed to scrap this data? If so perhaps it might be best to discuss alternative methods with the business entity itself. Perhaps there is an API available or you could work them to build a more a more efficient method of supplying the data.

Here we go, marvellous how a good sleep can stir ancient memories.

http://home.snafu.de/tilman/xenulink.html

It is not only a link checker but configurable to generate numerous reports.

3 Likes

Thanks so much! It seems a bit difficult to understand but I will try to make it run and see how I could use it with my code.

1 Like

Just stumbled across this topic which may be of interest…

Detect file changes

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.