Developing a blog feed aggregator

[webchalkboard]

Hi,

I have the task of developing a blog feed aggregator. We have a website where the members have their own personal blogs. I need to design a script that gets their blog entries and puts in our database every couple of hours.

I've pretty much managed to do this and it works for basic wordpress style rss feeds, however what happens if the feed is in a different format? I had kind of assumed that RSS was RSS and one parser would suit all, but after doing a little testing this is not turning out to be the case.

Can anyone tell me how many different type of RSS / XML feeds I need to account and code for?

Is this the best way in your opinions of creating a feed aggregator site?

Any thoughts or opinions appreciated, i'm not sure if i'm barking up the wrong tree completely with this one.

Thanks,
Tom

[jgstevenson]

Maybe instead of developing your own rss parser you can use a class that is already out there. It might deal with this problem:

Try this one: http://pear.php.net/package/XML_Feed_Parser
or
http://pear.php.net/package/XML_RSS

[webchalkboard]

Brilliant well done, I thought there might be something handy like that out there already.

Cheers,
Tom

[webchalkboard]

Hmm well it doesn't seem to work! I've tried that XML_Feed_Parser on a couple of feeds and it just complains saying:

PHP Warning: DOMDocument::loadXML(): Start tag expected, '<' not found in Entity, line: 1 in /usr/share/pear/XML/Feed/Parser.php on line 90
PHP Fatal error: Uncaught XML_Feed_Parser_Exception: Invalid input: this is not valid XML in /var/www/cron/feedmonster2.php on line 18
#0 /var/www/cron/feedmonster2.php(18): XML_Feed_Parser->__construct('rss.xml', true, false, true)
#1 {main}
thrown in /usr/share/pear/XML/Feed/Parser.php on line 101

I've had a play around with http://pear.php.net/manual/en/package.xml.xml-rss.php

And it looked very promising, but then didn't like this feed:

www. affiliateprogramadvice. com/blog/rss.xml
(I've messed up the URL to stop it being a link)

I'm guessing it's because it's not in the normal format for RSS... But it's very annoying. Can anyone suggest a parser that will work with ALL blog RSS formats?

I'm trying to create a blog feed aggregator.

Thanks,
Tom

[jgstevenson]

sorry, i thought those would have/should have worked. I know there are some issues with certain rss feeds that are not standard, are homemade.

It looks like the link www. affiliateprogramadvice. com/blog/rss.xml you gave is an Atom feed and not RSS. Do you need to explicitly tell your parser to parse Atom vs. RSS?

[jgstevenson]

Maybe this one will work:
http://www.phpbuilder.com/columns/ia...n20060103.php3

[mangrova]

Try these too

http://magpierss.sourceforge.net/
http://ajaxian.com/archives/build-an...g-ajax-and-php
http://www.phpclasses.org/browse/package/3724.html

They might help!

[webchalkboard]

Cool thanks, will try out those leads now. At the moment i'm using some kind of custom built thing I got off a friend, but whatever approach I take it doesn't sem to be able to deal with the Atom feed. I should have thought it ought to auto detect it, after all I won't necessarily know what kind of feed it is before I parse it. I thought the parser would be able to recognise the file format and parse accordingly.

I'll try those other suggestions.

Thanks,
Tom

[jgstevenson]

In the header of the feed it will tell you what kind it is, and you just have to parse that out to detect if it is RSS or Atom, then send it to the right parser.

[Kat]

What did you end up going with?

[webchalkboard]

I used something called Simplepie http://simplepie.org/ Their website seems to be down at the moment but I expect it will come back online at some point.

Very easy to use, just include the class, create an object then call the various functions. It also puts the data you have parsed in a much simpler format than the previous methods I've used.

Tom