robots.txt gives code 302; I want 200

Discussion in 'Apache' started by bluethumb, Jan 17, 2008.

0
  1. #1
    There is a web crawler which is causing most of the traffic on my server. I corresponded with a technical person who told me that their program ignores robots.txt if it has a response code other than 200. My site has two names for the same machine, call them goodhost.domain and badhost.domain. Goodhost returns a 200 code for robots.txt and badhost returns 302. There is no explicit redirection for this file. In the Apache configuration file, goodhost is the ServerName.

    Is there some way I can get badhost to serve up a 200 code?
     
    bluethumb, Jan 17, 2008 IP
  2. larysmith711

    larysmith711 Notable Member

    Messages:
    3,374
    Likes Received:
    341
    Best Answers:
    0
    Trophy Points:
    215
    #2
    what about banning that IP???
     
    larysmith711, Jan 17, 2008 IP