How can i stop the robot from this site?

Discussion in 'Search Engine Optimization' started by skionxb, Oct 15, 2006.

  1. #1
    robot - checking site status - www.websitezone.com
    This site is constanlty sending a robot to my site. I want to block it in ROBOTS.txt

    Does anyone know what is the name of the robot?

    What exsactly do i need to place in my robotx.txt?

    Thanks
     
    skionxb, Oct 15, 2006 IP
  2. Jim_Westergren

    Jim_Westergren Notable Member

    Messages:
    1,882
    Likes Received:
    247
    Best Answers:
    0
    Trophy Points:
    235
    #2
    Better would be to block it in .htaccess as that would really block.

    
    RewriteEngine on
    RewriteCond %{HTTP_REFERER} ^http(s)?://(www\.)?websitezone.com/.*$ [NC]
    RewriteRule .* - [F,L]
    
    Code (markup):
     
    Jim_Westergren, Oct 15, 2006 IP
  3. rxbbx

    rxbbx Peon

    Messages:
    185
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I block them with ip.. also in htaccess
     
    rxbbx, Oct 15, 2006 IP
  4. TuesZ

    TuesZ Well-Known Member

    Messages:
    934
    Likes Received:
    66
    Best Answers:
    0
    Trophy Points:
    163
    #4
    Have you used Google? They should help, also robotstxt dot org is a good site.
     
    TuesZ, Oct 15, 2006 IP
  5. nevetS

    nevetS Evolving Dragon

    Messages:
    2,544
    Likes Received:
    211
    Best Answers:
    0
    Trophy Points:
    135
    #5
    UserAgent INSERTAGENTNAMEHERE
    Disallow: *

    Although it may not follow robots.txt rules.

    .htaccess can be used as above or something like
    Deny from 123.456.789.012
    or
    SetEnvIf User-Agent "USER AGENT NAME" bozos
    Deny from env=bozos
     
    nevetS, Oct 16, 2006 IP