is a blank robots file ok?

[fryman]

My logs are getting filled with 404 hits looking for the robots.txt file, is it ok if I just upload a blank one?

[kdb003]

i dont think 404s on robots.txt files are really a problem
i think you will be ok either way

[fryman]

Yeah, I just don't want my log filled with this error

[magellan]

Yes a blank robots.txt is fine if you don't need to block any spiders and you want to reduce the number of 404 errors in your logs.

[subseo]

That's exactly what I did, so that to have emptier logs. An alternative is to put in the two lines or so, that "allow everything", don't have it handy, but if you search for it, it is everywhere. Anyway, with blank robots.txt, you are just fine.

[johnt]

That's exactly what I did, so that to have emptier logs. An alternative is to put in the two lines or so, that "allow everything", don't have it handy, but if you search for it, it is everywhere. Anyway, with blank robots.txt, you are just fine.

User-agent: *
Disallow:

Will tell every bot that they can read everything

John

[expat]

yes this will work if you are concerned about 404 from robots

A nice tool is to use a customized 404 that sends you e-mail based on referrer.
Thus you know immediately when users hit a 404 and can react and you can filter out bot's so you also know when undesirables start hammering your site - which is a good time to use a redirect based on referrer to screw them up a bit - love to send them to nonexisting (dynamic) pages with fake e-mails etc....


Expat

[exam]

My logs are getting filled with 404 hits looking for the robots.txt file, is it ok if I just upload a blank one?
Yes see here (http://forums.digitalpoint.com/showpost.php?p=220210&postcount=19)

[Enigma121]

Our investigations show that without a robots.txt file MSN will not index you!

We added a blank file and voila a listing appeared within a matter of hours. Vitally important to have one in place from that point of view then...

[fryman]

A robots file doesn't have anything to do with MSN not indexing you

[Enigma121]

Our initial findings and the article at this URL don't agree with you on this one, check out....

http://www.articlealley.com/article_6548_6.html

Knew I'd read about it somewhere...

[fryman]

I can get any site indexed at MSN in 24 hours, doesn't mater at all if it has robots file or not.

Don't believe anything you read...

[explorer]

Enigma121, I would like to add my own experience to Fryman's. I have sites at the top of MSN searches for slightly worthwhile (but not difficult) search terms. These sites have no robots.txt files.

Fryman, I also have sites doing well with blank robots.txt files.

I hope that helps.

[Enigma121]

It might have been pure co-incidence in our case, as I added the robots.txt file at around the same time I started to build backlinks. If that's the case then the timing in our case was unfortunate.

[Willy]

Don't believe anything you read...

Hehe, I sure hope you meant "don't believe everything you read"... ;)

[fryman]

yeah, but I couldn't edit my post anymore

[INV]

1.Many people write articles just for content with twisted so called facts. When somebody makes a ridiculous claim that an engine won't index your website for not using robots.txt, its bogus!

2.I would totally agree with Fryman on this one. You don't need robots.txt at all for a regular website to function well in a search engine index.

[Shoemoney]

in my experience a blank robots.txt file is fine. I do not have experience with not having one.

[minstrel]

According to the robots.txt standard, a blank robots.txt file is treated the same as no robots.txt file (except you won't see an error 404 in your logs in case that worries you).

I've just never seen the point - in the time it takes you to create and upload an empty robots.txt file, you can create one that says:

User-agent: *
Disallow:

and then there's no question at all...

[lorien1973]

I have 3 yahoo stores. I cannot upload a robots.txt file to any of them. I cannot have any file in the root domain actually (its a pain in the butt) that yahoo doesn't create.

I am indexed by MSN just fine. Robots.txt does not factor into it at all.

I was just browsing the log files of a new site I made. I noticed a ton of errors for the robots.txt file; and the favicon.ico file as well. I just made them and uploaded them, just to have a smaller error log.

[minstrel]

You're correct in that you don't NEED a robots.txt file at all unless you want to EXCLUDE (Disallow) access by spiders to certain files or subdirectories. I have one that is used for that purpose on pretty much every site.

My point was that if you're going to have one at all, why not do it right? An empty one accomplishes no more than not having one.

[lorien1973]

My point was that if you're going to have one at all, why not do it right? An empty one accomplishes no more than not having one.
Maybe people have robotstxtophobia - which is a fear of typing anything into a file called robots.txt :P

[minstrel]

There are treatments for that... PM me for fee quotes :D

[carowan]

I cannot have any file in the root domain actually (its a pain in the butt) that yahoo doesn't create.


That is exactly why I am in the process of moving from Yahoo stores/Merchant Services, more control for half the price.

[lorien1973]

Where are you moving too? xfactormm seemed to be the only alternative that would let us keep our current pagenames.

Other than cost, I haven't seen a good reason to leave yahoo as of yet, and I'm too comfortable there to leave anyways.

[exam]

My point was that if you're going to have one at all, why not do it right?According to the standard (and Google) a blank robots.txt *is* right.
An empty one accomplishes no more than not having one.
Umm are you not listening? I accomplishes at least 2 things.
- Fewer 404 errors in the logs.
- Saved bandwidth (your 404 error doc might be 1-2k and a blank robots.txt will be 0k) (Ok, it's hardly anything, but why not run a tight ship)

Does that clear things up? :D

[minstrel]

Are YOU not listening, exam?

if you're going to have one at all, why not do it right?
in the time it takes you to create and upload an empty robots.txt file, you can create one that says:

User-agent: *
Disallow:

Does THAT clear things up?

[exam]

Does THAT clear things up?
Not in the slightest :D

Don't get all worked up Minnie. Whether you put nothing or User-agent: *
Disallow: in your robots.txt file, it will do the exact same thing and is completely correct according to the standard. Why not choose the simplest solution? That way you save having to look up the syntax and you don't risk mistyping something and it even saves a couple of bytes of bandwidth.

Also you haven't responded to :DAn empty one accomplishes no more than not having one.

[LittleG]

Hi Guys,

I have recently installed a PHPbb forum.
I used some keyword analyser and realized that words like, class, post, moderator, user names come as the most used words on the site.
Can i write certain words in the robots.txt file which would be ignored from the crawlers?:confused:

Cheers ;) ,
LittleG

[minstrel]

No. You can only exclude files and directories

[LittleG]

is there any solution to this problem?

[minstrel]

You can exclude certain phpBB directories from indexing -- add this to robots.txt and make sure the robots.txt file is in the ROOT directory of your site:

User-agent: *
Disallow: /phpbb/admin/
Disallow: /phpbb/db/
Disallow: /phpbb/images/
Disallow: /phpbb/includes/
Disallow: /phpbb/language/
Disallow: /phpbb/profile.php
Disallow: /phpbb/groupcp.php
Disallow: /phpbb/memberlist.php
Disallow: /phpbb/login.php
Disallow: /phpbb/modcp.php
Disallow: /phpbb/posting.php
Disallow: /phpbb/privmsg.php
Disallow: /phpbb/search.php

If your forum is in the root directory, admit the /phpbb part above. If the forum is in a directory with a different name, make the appropriate corrections.

Then download and apply Parts 1 and 2 of the following phpBB MOD:

## MOD Title: Able2Know.com Search Engine Optimization 2.0.0
## MOD Author: Craven de Kere (N/A) http://www.Able2Know.com
## MOD Description: The definitive guide to phpBB SEO

[LittleG]

I downloaded the Able2Know.com Search Engine Optimization 2.0.0.txt 2 days ago.
It makes too many changes which i don't really want to do.
I am actually interested in the 4th.
4) Remove listing of moderators on index.php (AbelaJohnB)
but i can't really find this mod.
I will be really happy if You can help!
Cheers ;) ,
LittleG

[minstrel]

LittleG - doing that isn't going to help your forum SEO much. You NEED part 1 - you do NOT need any of the mod_rewrite stuff.

See attached file -- in there you will see this, which should remove the moderators listing for index.php:

#
#
#-----[ OPEN ]------------------------------------------
#

index.php

#
#-----[ FIND ]------------------------------------------
#

//
// Obtain list of moderators of each forum
// First users, then groups ... broken into two queries
//

#
#-----[ BEFORE, ADD ]------------------------------------
#

/*/

#
#-----[ FIND ]------------------------------------------
#

//
// Find which forums are visible for this user
//

#
#-----[ BEFORE, ADD ]------------------------------------
#

/*/

#
#-----[ FIND ]------------------------------------------
#

'L_MODERATOR' => $lang['Moderators'],

#
#-----[ REPLACE WITH ]------------------------------------
#

// 'L_MODERATOR' => $lang['Moderators'],

#
#-----[ FIND ]------------------------------------------
#

if ( count($forum_moderators[$forum_id]) > 0 )
{
$l_moderators = ( count($forum_moderators[$forum_id]) == 1 ) ? $lang['Moderator'] : $lang['Moderators'];
$moderator_list = implode(', ', $forum_moderators[$forum_id]);
}
else
{
$l_moderators = ' ';
$moderator_list = ' ';
}

#
#-----[ REPLACE WITH ]------------------------------------
#

/*
if ( count($forum_moderators[$forum_id]) > 0 )
{
$l_moderators = ( count($forum_moderators[$forum_id]) == 1 ) ? $lang['Moderator'] : $lang['Moderators'];
$moderator_list = implode(', ', $forum_moderators[$forum_id]);
}
else
{
$l_moderators = ' ';
$moderator_list = ' ';
}
*/

#
#-----[ FIND ]------------------------------------------
#

'MODERATORS' => $moderator_list,

#
#-----[ REPLACE WITH ]------------------------------------
#

// 'MODERATORS' => $moderator_list,

#
#-----[ FIND ]------------------------------------------
#

'L_MODERATOR' => $l_moderators,

#
#-----[ REPLACE WITH ]------------------------------------
#

// 'L_MODERATOR' => $l_moderators,

This basically comments out the part that lists moderators.

[melaniejk]

Hi.
I've had a site online for years and I don't have a robot.txt file.

I was just given a free site review and was told that:

No robot.txt file in root / not good


Do I really need to add one? Will it change anything by doing it now?
I don't want to risk messing anything up.
My site ranks very well in MSN already. I'm listed in all the SE.

I do get those errors in my logs but I ignore them.


Thank you.

[exam]

Go ahead and create a blank one. It won't hurt anything and you'll get fewer 404's in the logs.

[minstrel]

1. you don't need one unless you wish to prevent certain files or directories from being indexed, or if you want to TRY to block certain spiders

2. you CAN upload a blank one but it's vey easy to create the default one and, IMO, makes more sense that way. Just create a normal ASCII text file as follows (you can even cut and paste this one):

User-agent: *
Disallow:

Save this file as robots.txt and upload it to the root of your website.

[melaniejk]

Ok, ASCII text file?

I know I have heard of ASCII, but how do I create one.

I would have just opened a new file in Notepad and saved it as robot.txt

What do I have to do to save it as ASCII ?

I think there is an option on my FTP software that lets me upload files as ASCII or Binary. It's default is on Binary. Do I just upload the file I create in Notepad with the ASCII option checked ?

Sorry, I'm such a newbie with many of these things...

[minstrel]

It just means plain text, aka ANSI text.

If you use Notepad, I believe recent versions at least will default to saving the file as Unicode. Sometimes that can cause problems so use the drop-down box under Encoding and select ANSI.

If you use a different text editor, just save as any one opf ASCII, ANSI, Plain Text...

[melaniejk]

Thank you, Minstrel.

My Notepad default is ANSI.

So, it should be ok.


Thanks again.
:)

[minstrel]

This is a new computer with a new Windows XP installation. I'm almost certain that my old XP machine also had ANSI as the default but this one always defaults to Unicode. No idea why... :confused: