Confused between xml sitemap and robots due to one issue

Discussion in 'Search Engine Optimization' started by sf_guy, Jan 5, 2009.

  1. #1
    hi everyone,

    the home page of my website is http://www.xyz.com/index.html but everywhere on the website i made it to point at http://www.xyz.com/.even in robots file i wrote Disallow: /index.html. now i am not able to understand that when robot will not go to the home page, how will it crawl it as the domain name is obviously the home page.have i done the right thing:confused:
     
    sf_guy, Jan 5, 2009 IP
  2. poseidon

    poseidon Banned

    Messages:
    4,356
    Likes Received:
    246
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I am not sure I understood what you are asking though, to stop bots crawling sites, use

    Disallow: /
     
    poseidon, Jan 6, 2009 IP