creating a temporary file?

[strugglingon]

I am writing a program that reads through a website grabbing the content of each page and then following links and grabbing the contents of those pages and putting them all in to one linear file.

any ideas on how best to store the contents of the content grabber so it can move on to the next page?

I was thinking that I could do it in a temp file but I am not really sure how I would go about implementing it.

Any ideas?

[icky_bu]

Uhm, you would certainly have to work in some regular expressions I'd think that all that content would add up like mad.

Btw, what would you be using the content for?
Please be carefull with copyright issues.

[Grubilo]

strugglingon, the best way to store this content will depend on the things you want to do with this content

But... for sure I can tell you that storing all data in one file is not effecient, better use separate files for each node/domain you are processing. This can make a benefit in time when you will be reading content locally.

For more advanced scheme I would recommend to use a database fro indexing urls and mapping files to each database record.
Good luck

[stereofrog]

I am writing a program that reads through a website grabbing the content of each page and then following links and grabbing the contents of those pages and putting them all in to one linear file.

You should be able to use wget for this

[strugglingon]

Uhm, you would certainly have to work in some regular expressions I'd think that all that content would add up like mad.

Btw, what would you be using the content for?
Please be carefull with copyright issues.

The idea is for it to be used to searialise the pages of online help manuals so that the user can then have the aibility to print the whole thing as a book rather than going through each page and printing individually.