1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Can robots.txt block or allow search engines from specific countries?

Discussion in 'robots.txt' started by agklein, May 20, 2010.

  1. #1
    Hello,

    Is it possible to create a robots.txt file that will exclude all Australian based search engines (ie, google.com.au, au.bing.com, au.yahoo.com, etc) from indexing my website? If so, how would I do that?
    SEMrush
    Conversely, can you create a robots.txt file that ONLY allows Australian based search engines (such as the ones listed above) from indexing a site? And even more specifically, is it possible to create a robots.txt file that allows only search engines from Australia to index just one subdirectory of a site, while the rest of the site is open to search engines from any country?

    Cheers,
    Andy
     
    agklein, May 20, 2010 IP
    SEMrush
  2. 50plus

    50plus Guest

    Messages:
    235
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Unfortunately, the answer to all of your questions is no and to my knowledge - but I could be wrong there - all the SEs that you mention use the same UA for crawling in any country so blocking it would stop them indexing your site alltogether.
     
    50plus, May 21, 2010 IP
  3. agklein

    agklein Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks very much, appreciate the response, although too bad it wasn't the answer I wanted!

    Cheers,
    Andy
     
    agklein, May 23, 2010 IP