Need help with robots.txt

Discussion in 'Google Sitemaps' started by arin2u, May 24, 2007.

  1. #1
    I've my robots.txt created and I need to stop some bad urls from getting indexed in google. I'm not able to find from where google is finding those urls. Those pages are on the sub domain. The urls are something like 'http://xxx.mysite.com/1/2/3/4.php' .

    Suppose I want to block /1/ directory and all its content, how should I block them through robots.txt?

    Thanks
     
    arin2u, May 24, 2007 IP
  2. arin2u

    arin2u Peon

    Messages:
    66
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I've my google sitemap account for the main site. Do I need to add another sitemap for my sub.domain? Because what I find that I won't be getting the url removal option (new feature in google sitemap) for the sub domain?
     
    arin2u, May 25, 2007 IP
  3. MD.45

    MD.45 Peon

    Messages:
    109
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Just add the '1' directory to your list in your robots.txt file, that will stop majority of bots from indexing it.
     
    MD.45, May 25, 2007 IP
  4. totalmasala

    totalmasala Banned

    Messages:
    1,454
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #4
    just it 1 directory like this /1/
     
    totalmasala, May 25, 2007 IP
  5. arin2u

    arin2u Peon

    Messages:
    66
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I've my google sitemap account for the main site. Do I need to add another sitemap for my sub.domain? Because what I find that I won't be getting the url removal option (new feature in google sitemap) for the sub domain?
     
    arin2u, May 26, 2007 IP
  6. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #6
    MD.45 is right, but the robots.txt file must be placed in the root directory of the subdomain, not in the root directory of the main domain.

    Jean-Luc
     
    Jean-Luc, May 26, 2007 IP