I want to scrape a vbulletin message listing with PHP, any ideas/scripts?

PHP
1

I want to write a script that can log into a vbulletin forum (ive already got permission to do this if I keep the load small) and scrape the content from one of the listing pages to look at the code for certain keywords etc.

I’ve got my code working as far as the scraping and searching content etc goes, and on a public page, it works fine.

But because the vbulletin forum page requires users to log in before they can see the message list content, I need to make the php login to vbulletin with my user account and handle the session etc.

Anyone done this? can point me in the right direction?

Again, the forum owner has given me permission so long as it doesnt cause him any work and I dont need access to his server etc. (his words)

2

You can do this with curl
Make sure you use cookie file. Then it’s a two step process: 1) just make a post request to vbulletin’s login form using your username/password
that would save the login cookie to the cookie file
2) now request a page with regular GET request but use the same cookie file in curl’s config
So you will get the page that logged in user would see. then just parse/scrape that page
http://us2.php.net/curl