Help: Different Robot.txt for different domain

Discussion in 'robots.txt' started by init2, Apr 11, 2009.

0
  1. #1
    Hello All,

    I have verified my website by creating a html file in the root folder

    while browsing i found that below lines

    "Google and other search engines treat http://www.example.com, https://www.example.com, and http://example.com as different sites. If you want to restrict crawling on each of these sites, you can create a separate robots.txt for every version of your site's URL. "

    My Confusion:
    -------------

    1) As my site is resolving same by http:\\example.com and http:\\www.example.com

    so is it necessary to create Robots.txt file for both and if i have to create then as both are same so what should my file content will be ??

    Please help me out to create the robots.txt file.

    Any help will be highly appreciated.

    Regards,
    Init2
     
    init2, Apr 11, 2009 IP
  2. stevejeff

    stevejeff Peon

    Messages:
    187
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Hi,

    Any www or non-www version of a website appears as a different website in the eyes of search engines. So better create separate robots.txt file .

    Thank You
     
    stevejeff, Apr 12, 2009 IP
  3. init2

    init2 Peon

    Messages:
    41
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    ok, thats sounds good but where to place those?? public_html??

    Please provide the content also

    it will be highly appreciated

    Regards,
    Init2
     
    init2, Apr 13, 2009 IP
  4. yenerich

    yenerich Active Member

    Messages:
    697
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    75
    #4
    Yes, in public_html put your robots.txt
     
    yenerich, Apr 16, 2009 IP
  5. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #5
    Quite funny reading all this..okk..if I agree with your points for a min then could you let me know how you will add different robots files in the same root as all three version of the website would be sharing the same root...??
     
    manish.chauhan, Apr 17, 2009 IP
  6. yenerich

    yenerich Active Member

    Messages:
    697
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    75
    #6
    Because usually www.site and site. are the same.
    If you put robots.txt in the root, Google will find it as www.site.com/robots.txt and site.com/robots.txt

    You DO NOT need to add DIFERENT robots.txt to the same site.
    only if you use subdomains (www is not really a different subdomain in most cases). But in that case put the robots.txt in the ROOT of each subdomain.

    ----------------------

    Also, you can go to Google's Webmaster tools and there tell Google if you choose your site was www.site.com or site.com and Google will always show your urls in that way in the Search Engine.

    That show that is not enterily truth that for Google www.site and site are different. Google detects that is the same one.
     
    yenerich, Apr 19, 2009 IP