Debt Consolidation - Wordpress Themes - HASP - Deaf Topics - IKA Works Laboratory equipment
PDA

View Full Version : Disallowing All But Certain Spiders

digitalpoint
Apr 23rd 2005, 7:31 pm
Quick question for anyone that is more familiar with robots.txt than I am.

Let's say I wanted to only allow a few spiders, but give the ones I do allow a few specific disallows.

Would this work?

User-agent: Googlebot
User-agent: Slurp
User-agent: msnbot
User-agent: askjeeves
Disallow: /images/
Disallow: /purchase.php
Disallow: /go.php

User-agent: *
Disallow: /
fryman
Apr 24th 2005, 1:15 am
I think so, look a lot like the one in the article shere

http://www.search-marketing.info/meta-tags/robots-meta.htm
digitalpoint
Apr 24th 2005, 9:34 am
Yeah, as a FYI, it seems to be working... lame bots are not continuing after reading the robots.txt file, but so far only one of the allowed bots has visited (Yahoo), and it continued spidering after reading the robots.txt file.
fryman
Apr 24th 2005, 10:54 am
You mean the Googlebot got stuck?