All except One - Robots.txt

I was wondering if it was possible to create a robots.txt that'd disallow bots from access all directories except one. It would also allow all access to directory. inside that directory.

Would my robots.txt would?

User-agent: *
Disallow: /
Allow: /images

Code (markup):

Or would it be Disallow:? instead of Disallow: /

Also does anyone have any idea how to do this with .htaccess? I've been having problems with that also.

 

It's possible, I will give you an example with 3 different directories excluded

To exclude all robots from part of the server

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~tieu/

Code (markup):

Hope it helps.

Cheers!

 

Thanks for the reply but it isn't necessarily what I need. I have over 2000 directories in my index, I was wondering how I'd block 1999 of them and leave just one folder allowed.

Would it be Disallow: / or Disallow: /* or Disallow:

 

In this case you can use the following to exclude all files except one:

User-agent: *
Disallow[B][SIZE="3"][COLOR="red"]: /~[/COLOR][/SIZE][/B]tieu/online/

Code (markup):

Cheers!

 

Okay this will work. But how do you know to use ~tieu? not ~index etc?

 

That was just an example.... you put

Disallow: /~whateveryouwant

Cheers!

 

since you want only only one folder to be accessible, try following code:

User-agent: *
Allow: /Folder Name/
Disallow: /

This code will allow crawlers to access only 'folder name' folder and rest of the website would not be accessible. Hope this would help you.

 

User-agent: *
Disallow: /
Allow: /images

will work. The only problem is, you have to have back link to your /images from other website. otherwise decent bots can never get there.

 

How about if I do this, would it allow the folders not to be browseable but the index page is viewable?

User-agent: *
Disallow: /
Allow: /images
Allow: /index.php