Help Understanding Robots.txt basic commands

Hi,

I'd like to clean up some of the duplicate content on some of my blogs.

What would be the difference between these three commands in robots.txt

Disallow: /tag/

and

Disallow: /tag/*

and

Disallow: */tag/

Thank you

 

Here you go http://www.robotstxt.org/robotstxt.html

 

* is a regular expression for any number of characters.

Back to the question,

/tag/ and /tag/* are eventually the same. And the code Disallow: /tag/ means that the bots are requested to not index any of the following URLs.

http://yourdomain.com/tag/
http://yourdomain.com/tag/index.php
http://yourdomain.com/tag/category/post1.html
and so on...

 

As Cipals said, /tag/ and /tag/* work the same way, they both restrict all urls starting from tag directory. This command Disallow: /*/tag/ will be used to block all urls that do have tag at the second level on their urls like http://yourdomain.com/abc/tag/pqr.html etc

 

Try using plugin robot.txt..

 

its tag of robots.txt for show no follow link