Scraping Script

[BuLLDoG.Name]

Is their any examples of scraping script on the net.

[HarryR]

A scraping script is just something which connects to a webserver, pretends to perform an action or just gets a page, and interprets the resulting HTML.

There's no generic "scraping script", as everything is different.

[spikeZ]

lol, though I might see you in here Bulldog!
Here are a few links for you to browse.....

http://www.daniweb.com/code/snippet293.html
http://www.devnewz.com/devnewz-3-200...eInternet.html
http://codingforums.com/archive/index.php?t-36563.html

Should set you on the right path

[r937]

i would hardly call any scraping script "the right path"

[spikeZ]

i would hardly call any scraping script "the right path"

true....

[HarryR]

i would hardly call any scraping script "the right path"

REST FTW

But for those situations where you cant get it from anything other than scraping?

[r937]

But for those situations where you cant get it from anything other than scraping?

for those situations, resist the urge to steal

because that's what screen scraping usually is -- theft of copyright material

[kromey]

for those situations, resist the urge to steal

because that's what screen scraping usually is -- theft of copyright material

There's countless legitimate uses of scrapers - I've written more than a dozen here at work in the last year, none of which are stealing copyrighted material and all of which were the only option available for the purposes we needed. Most of them are scraping content from our own servers (e.g. pulling in dashboard data from a myriad different monitoring utilities that don't provide alternative access means such as SOAP or RSS); the ones that reach out across the internet I took great pains to make as "friendly" as possible - they connect directly to what they need and nothing more, and all of them implement local caching so that at most I'm only sucking down the remote page once per hour (most are cached for a full 24 hours).

[r937]

i was very careful to insert the word "usually" in my statement

bulldog's previous thread was about large databases for sale on the web, for example lyric databases over 500,000+ and recipe database and so on, and how do people get the data for those databases... and i answered "they scrape them from other sites" ... and then he started this new thread

[TheRedDevil]

There's countless legitimate uses of scrapers - I've written more than a dozen here at work in the last year, none of which are stealing copyrighted material and all of which were the only option available for the purposes we needed. Most of them are scraping content from our own servers (e.g. pulling in dashboard data from a myriad different monitoring utilities that don't provide alternative access means such as SOAP or RSS); the ones that reach out across the internet I took great pains to make as "friendly" as possible - they connect directly to what they need and nothing more, and all of them implement local caching so that at most I'm only sucking down the remote page once per hour (most are cached for a full 24 hours).

If you are "scraping" your own sites, then you would be better off writing a simple api instead. You would in most cases finish the code faster, and it would be a lot more effective.

[kromey]

i was very careful to insert the word "usually" in my statement

bulldog's previous thread was about large databases for sale on the web, for example lyric databases over 500,000+ and recipe database and so on, and how do people get the data for those databases... and i answered "they scrape them from other sites" ... and then he started this new thread

Well, now that we've got this context around this thread, it is sounding pretty shady.

[r937]

Off Topic:

REST FTW

Restricted Environmental Stimulation Technique in Fort Worth?

[HarryR]

Off Topic:


Restricted Environmental Stimulation Technique in Fort Worth?

For The Win.

i was very careful to insert the word "usually" in my statement

bulldog's previous thread was about large databases for sale on the web, for example lyric databases over 500,000+ and recipe database and so on, and how do people get the data for those databases... and i answered "they scrape them from other sites" ... and then he started this new thread

So in this specific instance it's copyright plagurism.

[spikeZ]

For The Win.

I thought you mean "For The Wicked" as in No Rest For The Wicked.....

So in this specific instance it's copyright plagurism.

"Possibly copyright plagiarism......"

[r937]

REST?

plagiarism

[HarryR]

REST?

Yeah, or any other type of structured data access method (soap/xml-rpc/rss) that the site explicitly provides (e.g. they want you to use or syndicate their content/site).