Stop harvest

If i have a html pages a lot of them and i do not want people to just download them. Is there anyway i can stop it with some code in the HTML or something.

Can i add something in the robot txt or will that stop Google as well?

 

if you mean "i dont want people to spider my whole site and steal everything", that can be done, by having javascript fill in the page after it loads.. i haven't found any way yet to run javascript from the commandline / a script.

but if you want to display HTML to people, people can rip it out of their browser page by page..

 

You mean with a download program? They can do it anyway right? No one page per page i am not scared its loads of pages.

 

@Rene

You can with PHP assuming you have some vital components installed.

@Yellowberry.org - I wouldn't bother, nothing is unrippable when it comes to HTML, anything you try will inevitably be doomed to failure.

 

Most likely, whatever you try to do to stop being from downloading would just be a pest. It would just affect the people who are less web savy, whilst the people who actually want to go in and download the site would be able to just bypass it straight away.

As an example, think of the right click blockers. They are easy to bypass, and people wanting to save the images will still do so with ease.

 

You can try to disallow crawling for certain downloader programs/bots who respect robots.txt. If bot ignores robots.txt, it is useless (in that case you can try to block downloading with .htaccess by user-agent).

For example you can write this to robots.txt to prevent crawling by most popular downloaders (and to save some bandwidth):

User-agent: DISCo Pump, Wget, WebZIP, Teleport, TeleportPro, Teleport Pro, WebSnake, Offline Explorer, Web-By-Mail
Disallow: /

Code (markup):

For longer versions of robots.txt see this thread.

 

ahh thats good but google will still be ok?

 

Yes.

Whatever agents aren't in that list will be fine.

 

Oki pretty cool, what does all this things crawl for anyway?