ErrorDocument 404 redirect PLEASE HELP (blocked robots.txt)

Discussion in 'Apache' started by skionxb, Jul 27, 2006.

  1. #1
    Hello,

    To keep the visitors on the page i created .httaccess and placed

    ErrorDocument 404 redirect http://www.yoursite-name.com/sitemap.php

    It works fine!
    when i type www.yoursite-name.com/blanlabakhdfd

    it goest to http://www.yoursite-name.com/sitemap.php

    I am fine with that. Unfortunately, my IT guy, the company where i work, blocked robots.txt file, he said robots.txt file is not necessary for SEO purposes. he also said that SEearch engine robots will index the site with out it. and added by having robots.txt and accidently put the wrong information in there, you could have serious problems. I told him that you need to put the correct info in there.

    anyways, what happens now when robots.txt is blocked

    when i type www.yoursite-name.com/robotx.txt in browse

    it goest to http://www.yoursite-name.com/sitemap.php

    the question is this:

    Will this CONFUSE the SE robots (spiders)? Will my site still be indexed? Any sugesstions? comments?
     
    skionxb, Jul 27, 2006 IP
  2. NetMidWest

    NetMidWest Peon

    Messages:
    1,677
    Likes Received:
    151
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Until you change the error page in your .htaccess to a relative path, you will not generate a 404 for the robots.txt, but a 302. A 302 is a temporary redirect, and Google will index the resulting content from the redirect under the url it comes in on.
     
    NetMidWest, Jul 27, 2006 IP
  3. sarahk

    sarahk iTamer Staff

    Messages:
    28,851
    Likes Received:
    4,542
    Best Answers:
    123
    Trophy Points:
    665
    #3
    I'd leave a totally blank file there. When the file isn't found the server has to think about what to serve up. An empty file is quicker and more efficient.
     
    sarahk, Jul 27, 2006 IP