Hello, To keep the visitors on the page i created .httaccess and placed ErrorDocument 404 redirect http://www.yoursite-name.com/sitemap.php It works fine! when i type www.yoursite-name.com/blanlabakhdfd it goest to http://www.yoursite-name.com/sitemap.php I am fine with that. Unfortunately, my IT guy, the company where i work, blocked robots.txt file, he said robots.txt file is not necessary for SEO purposes. he also said that SEearch engine robots will index the site with out it. and added by having robots.txt and accidently put the wrong information in there, you could have serious problems. I told him that you need to put the correct info in there. anyways, what happens now when robots.txt is blocked when i type www.yoursite-name.com/robotx.txt in browse it goest to http://www.yoursite-name.com/sitemap.php the question is this: Will this CONFUSE the SE robots (spiders)? Will my site still be indexed? Any sugesstions? comments?
if you dont have a robots.txt the natural reaction from spiders will be to simply spider all of your site
yes, but what if the site redirects to site map? ErrorDocument 404 redirect "http://www.yoursite-name.com/sitemap.php" don't you think that spiders, could get confused? Basically, what happens now, when i type "http://www.yoursite-name.com/robots.txt" it shows me 200ok header Some people told me it's not good!
who told you its not good? if you told me your site this would be a hell of a lot easier so when you go to www.yoursite.com you get redirected to www.yoursite.com/sitemap.php ? *confused* If the robots.txt is there but hidden, then just remove it
ok, here is what happened. The IT guys in my company had bad experience before with robots.txt. Someone put a wrong code by mistake and all of the sudden all the listing dissapered from the google SERPs. Since that time, they decided to block it completely. SO if i upload robots.txt file into the root directory of any site on that server. It's basically doesn't see it. It's been blocked! When you type "http://www.verrado.com/robots.txt"in browser, it automatically redirects me to "http://www.verrado.com/sitemap.php" It happens becuase i implemented 404 redirect in .htaccess file. ErrorDocument 404 "http://www.verrado.com/sitemap.php" . Instead of creating 404 custom page, i just forwarded regular 404 page to the sitemap of the site. Now, one of these ITs says it's not a good idea to do that. Because SE robots might get confused. When they go to the site, the first thing they do is looking for robots.txt. In this case they don't find it, and they are forwarded to the SITE MAP. Now robots might think that the sitemap is a robots.txt and if the do so, they might realize that the robots.txt is too heavy and just refuse to crawl the site further. When i checked the http headers of the the site map, here it is "http://www.delorie.com/web/headers.cgi?url=http%3A%2F%2Fwww.verrado.com%2Fsitemap.php" It shows me that there is no 404 error. INstead is shows 200ok. In my opinion the crawlers will still crawl the site with my .htaccess tweek. But IT's want me to remove ErrorDocument 404 "http://www.verrado.com/sitemap.php" from .htaccess. And i know how beneficial is to have that feature to keep the visitors on the site. They refuse to enable robots.txt file on the server. This is nonsane to me. Why do you need a custom 404 page? Simply put, to increase the number of visitors to your web site. Everybody wants more people to visit their website. On average, 7% of visits to any given web site result in a 404 "not found" error page, according to web trends of popular web sites. If you can lure in most of the visitors that hit your 404 page, then you're increasing your web traffic.