Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

how to block all other BOT except Google, MSN, Yahoo

Discussion in 'robots.txt' started by danieloffice, Jan 9, 2008.

0
Page 2 of 2
  1. singh03

    singh03 Member

    Messages:
    62
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #21
    if you want to block other crawler except Google,Yahoo and bing crawler.so please you can use following syntax.
    User-agent: Googlebot
    Disallow:

    User-agent: Yahoo-slurp
    Disallow:
    SEMrush
    User-agent: Msnbot
    Disallow:

    User-agent: *
    Disallow:/
     
    singh03, Dec 13, 2010 IP
    SEMrush
  2. AcevedoAaron

    AcevedoAaron Peon

    Messages:
    49
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #22
    robot.txt are not worked always but if you update them every time with changing security then its works best also some java text can make help batter
     
    AcevedoAaron, Dec 17, 2010 IP
  3. Cincinnati SEO

    Cincinnati SEO Greenhorn

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #23
    Why Would you want to block all those bots. How would this help an SEO Agency like SundaySeo.net
     
    Cincinnati SEO, Jul 10, 2015 IP
  4. moneyroad

    moneyroad Member

    Messages:
    88
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    48
    #24
    There are some search engine which is popular in exact coutry: lets say yandex is most popular in Russia. So if you dont want to be searchable in Russia block yandex bot, it will stop indexing your site, less non targeted users will come, you will save some hosting traffic as well.
     
    moneyroad, Jul 10, 2015 IP
Page 2 of 2