Something about robots.txt

I see some blog state that I can disallow the bot to index the wordpress files. Here is how they use:

Disallow: /wp-

Or

Disallow: /wp-*

Or

Disallow: /wp-includes/

Anyone know what's the differences between /wp- and /wp-* ? Which once should I use if I want to block all the subdirectories and files which starting at wp- ?

 

I dont there are as many directories with Wp- as you say. All you have is wp-admin and wp-includes or may be one more.

So please take the pain in writing them down rather than going for experiments.

Cheers!
Mani

 

wp-admin, wp-content and wp-includes are it

Those are the directories you want to disallow in your robots.txt file.

Just bear in mind that if a Web page is password protected, the search engine spider won't be able to get past it. For more information on the robots.txt protocol, go to http://www.robotstxt.org/wc/robots.html

 

I second that.

 

this one block everythink which starts with wp-.

this will block

wp-
wp-gjgjgjgj/ (folder)
wp-kgjkfggfngfng.php (html,asp etc.)

 

Damn - that confuses me. How about this one?

Disallow: /script

doesn't that disallow all the files/folders beneath the folder "script"?

If i was to disallow only the folder "script" then it should be

Disallow: /script/