Robots.txt Question

Discussion in 'robots.txt' started by samib, Jun 15, 2006.

  1. #1
    Should Subdomains contain their own robots.txt?
     
    samib, Jun 15, 2006 IP
  2. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Definitely yes.

    The robots.txt file of your sudomain.domain.com should be visible at http ://subdomain.domain.com/robots.txt.

    I do not mean that all domains and subdomains need a robots.txt. It is useful only when you want to avoid the visit of robots in some parts of the domain or subdomain.

    Jean-Luc
     
    Jean-Luc, Jun 16, 2006 IP
  3. smile

    smile Peon

    Messages:
    323
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #3
    if we dont have robots.txt in our site, will it cause problems ? ( not getting indexed by google , etc )
     
    smile, Jun 19, 2006 IP
  4. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #4
    If there is no robots.txt, Google and the other search engines will consider they are allowed to visit and index all your pages. robots.txt is only useful to disallow access to some addresses.

    Jean-Luc
     
    Jean-Luc, Jun 19, 2006 IP