SitePoint Sponsor |
|
User Tag List
Results 1 to 9 of 9
Hybrid View
-
Aug 17, 2004, 03:13 #1
- Join Date
- Apr 2004
- Location
- Denmark
- Posts
- 1,215
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
How to protect against "website grabbing/download"?
Hi all,
How do I protect my website from beeing grabbed/downloaded? I think Mozilla support this function.
/Kenneth...
-
Aug 17, 2004, 05:07 #2
- Join Date
- Mar 2004
- Location
- Gloucester UK
- Posts
- 57
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
Originally Posted by wide
There is nothing to stop someone taking a copy of your site, if that is what you mean.
Obscure Javascript and Flash can make it more difficult to spider all the required Urls, but other than that, you cannot stop it without preventing users in general from accessing your site.
-
Aug 17, 2004, 05:20 #3
- Join Date
- Apr 2004
- Location
- Denmark
- Posts
- 1,215
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
no no, that is not what I mean. You can use programs to download/grab a COMPLETE website (all pages including images etc). I know there is a way to prevent it (by using .htaccess) but I dont remember where I read it
...
-
Aug 17, 2004, 07:56 #4
- Join Date
- May 2003
- Location
- Washington, DC
- Posts
- 10,653
- Mentioned
- 4 Post(s)
- Tagged
- 0 Thread(s)
No, not really. For someone to view the site, one essentially needs to be able to download the entire thing. One could ban any website grabbing software which declared itself in the browser type header. Then again, most of it allows one to masquerade as interenet explorer. If it is so important that it cannot be shared, it probably should not be on the public internet, no?
WWB
-
Aug 17, 2004, 08:22 #5
- Join Date
- Apr 2004
- Location
- Denmark
- Posts
- 1,215
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
If it is so important that it cannot be shared, it probably should not be on the public internet, no?"
That is not the issue ... I have +10.000 pictures posted at atleast as many pages. If someone grab the entire site it will cost alot of bandwidth... and I dont want some moron to copy my presious content so easy.
I know some people here at SPF use htaccess to prevent this, I just cant find the threads....
-
Aug 17, 2004, 10:10 #6
- Join Date
- Mar 2004
- Location
- Gloucester UK
- Posts
- 57
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
Originally Posted by wide
That may be enough.
Then you could use server-side software to limit the amount of bandwidth you will serve to to a single IP address. Of course a devious grabber could use multiple addresses, but that's unlikely.
Of course it would also stop legitimate users from browsing the whole site.
-
Aug 17, 2004, 10:37 #7
- Join Date
- Apr 2004
- Location
- Denmark
- Posts
- 1,215
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
Originally Posted by geebee2
- Actualy I want Google to index my pictures, I did optimized for that :P
"Then you could use server-side software to limit the amount of bandwidth you will serve to to a single IP address. Of course a devious grabber could use multiple addresses, but that's unlikely."
- That would be a possibility, but then I will get a problem with SE bots.
I thanks for your suggestions, but I know there is another way (something about listing 50-100 lines of code into .htaccess). I will keep searching....
-
Aug 23, 2004, 10:04 #8
- Join Date
- Aug 2004
- Location
- New York, NY, USA
- Posts
- 50
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
Hi Kenneth,
Putting exclusions in your robots.txt is a good step, although I suspect that site ripping software wouldn't pay much attention to that.
What you are looking for sounds like a Bot trap.
Basically, place a link on your pages (one that is hidden from legitimate users). Site ripping software will follow every link that it finds to download the content. If someone/something follows your hidden link then they're probably up to no good (and you can opt to block them).
A useful site I found a while back was: How to build a bot trap
Does this jog any memories?
Regards,
Mark
-
Aug 23, 2004, 12:37 #9
- Join Date
- Apr 2004
- Location
- Denmark
- Posts
- 1,215
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
Great solution mrobinson.
I will add it later this week...
Bookmarks