Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Stop ANY search engine indexing my site

Discussion in 'robots.txt' started by crazyryan, Feb 2, 2008.

0
  1. #1
    Hey

    How can I block any search engines from indexing my domain?
    SEMrush
    Thanks! :confused:
     
    crazyryan, Feb 2, 2008 IP
    SEMrush
  2. catanich

    catanich Peon

    Messages:
    1,921
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #2
    You could use the following in the robots.txt file:

    User-agent: *
    Disallow: *

    But you really don't want to do this. What are you trying to do?
     
    catanich, Feb 2, 2008 IP
  3. crazyryan

    crazyryan Well-Known Member

    Messages:
    3,087
    Likes Received:
    165
    Best Answers:
    0
    Trophy Points:
    175
    #3
    I want to block all search engines from indexing my site.
     
    crazyryan, Feb 2, 2008 IP
  4. shockie

    shockie Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    just disallow all crawlers as catanich suggested.
     
    shockie, Feb 3, 2008 IP
  5. thefourthroom

    thefourthroom Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    User-agent: *
    Disallow: /

    Will do the thing
     
    thefourthroom, Apr 4, 2008 IP
  6. ezcat

    ezcat Peon

    Messages:
    233
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Anyone know how to block just one bot and allow all others? I got a picture bot hitting my sites a couple times a day using several IPs
     
    ezcat, Apr 4, 2008 IP
  7. Dreads

    Dreads Well-Known Member

    Messages:
    1,883
    Likes Received:
    24
    Best Answers:
    0
    Trophy Points:
    150
    #7
    Just ip ban them? using php, not very hard to do. Make something like this

    <?php
    $banip = "0.0.0.0";
    $ip = GetHostByName($REMOTE_ADDR);
    if ( $ip == $banip )
    echo "
    <META HTTP-EQUIV=Refresh CONTENT="0; URL=http://www.google.com">
    ";

    ?>

    replace 0.0.0.0 with the ip that you wish to remove :p
     
    Dreads, Apr 4, 2008 IP
  8. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #8
    Use this robots.txt file:

    User-agent: your bot name you want to block
    Disallow: /

    User-agent: *
    Disallow: /your private files
     
    manish.chauhan, Apr 4, 2008 IP