My logs are getting filled with 404 hits looking for the robots.txt file, is it ok if I just upload a blank one?
Yes a blank robots.txt is fine if you don't need to block any spiders and you want to reduce the number of 404 errors in your logs.
That's exactly what I did, so that to have emptier logs. An alternative is to put in the two lines or so, that "allow everything", don't have it handy, but if you search for it, it is everywhere. Anyway, with blank robots.txt, you are just fine.
yes this will work if you are concerned about 404 from robots A nice tool is to use a customized 404 that sends you e-mail based on referrer. Thus you know immediately when users hit a 404 and can react and you can filter out bot's so you also know when undesirables start hammering your site - which is a good time to use a redirect based on referrer to screw them up a bit - love to send them to nonexisting (dynamic) pages with fake e-mails etc.... Expat
Our investigations show that without a robots.txt file MSN will not index you! We added a blank file and voila a listing appeared within a matter of hours. Vitally important to have one in place from that point of view then...
Our initial findings and the article at this URL don't agree with you on this one, check out.... http://www.articlealley.com/article_6548_6.html Knew I'd read about it somewhere...
I can get any site indexed at MSN in 24 hours, doesn't mater at all if it has robots file or not. Don't believe anything you read...
Enigma121, I would like to add my own experience to Fryman's. I have sites at the top of MSN searches for slightly worthwhile (but not difficult) search terms. These sites have no robots.txt files. Fryman, I also have sites doing well with blank robots.txt files. I hope that helps.
It might have been pure co-incidence in our case, as I added the robots.txt file at around the same time I started to build backlinks. If that's the case then the timing in our case was unfortunate.
1.Many people write articles just for content with twisted so called facts. When somebody makes a ridiculous claim that an engine won't index your website for not using robots.txt, its bogus! 2.I would totally agree with Fryman on this one. You don't need robots.txt at all for a regular website to function well in a search engine index.
According to the robots.txt standard, a blank robots.txt file is treated the same as no robots.txt file (except you won't see an error 404 in your logs in case that worries you). I've just never seen the point - in the time it takes you to create and upload an empty robots.txt file, you can create one that says: User-agent: * Disallow: Code (markup): and then there's no question at all...
I have 3 yahoo stores. I cannot upload a robots.txt file to any of them. I cannot have any file in the root domain actually (its a pain in the butt) that yahoo doesn't create. I am indexed by MSN just fine. Robots.txt does not factor into it at all. I was just browsing the log files of a new site I made. I noticed a ton of errors for the robots.txt file; and the favicon.ico file as well. I just made them and uploaded them, just to have a smaller error log.