Disallow Directories shows up in search with the omitted results included

Discussion in 'Google' started by anjanesh, Nov 24, 2008.

0
  1. #1
    Hi

    In my rorbots.txt file I have
    User-agent: *
Disallow: /location/
Disallow: /tags/
Disallow: /category/
    Code (markup):
    Yet in googe.com, site:mydomain.com on clicking on
    If you like, you can repeat the search with the omitted results included.
    it shows some links in domain.com/tags,
    domain.com/location and
    domain.com/category !

    How come it still crawls these directories ?

    Thanks
     
    anjanesh, Nov 24, 2008 IP