is a blank robots file ok?

You're correct in that you don't NEED a robots.txt file at all unless you want to EXCLUDE (Disallow) access by spiders to certain files or subdirectories. I have one that is used for that purpose on pretty much every site.

My point was that if you're going to have one at all, why not do it right? An empty one accomplishes no more than not having one.

 

Maybe people have robotstxtophobia - which is a fear of typing anything into a file called robots.txt

 

There are treatments for that... PM me for fee quotes

 

That is exactly why I am in the process of moving from Yahoo stores/Merchant Services, more control for half the price.

 

Where are you moving too? xfactormm seemed to be the only alternative that would let us keep our current pagenames.

Other than cost, I haven't seen a good reason to leave yahoo as of yet, and I'm too comfortable there to leave anyways.

 

According to the standard (and Google) a blank robots.txt *is* right.

Umm are you not listening? I accomplishes at least 2 things.
- Fewer 404 errors in the logs.
- Saved bandwidth (your 404 error doc might be 1-2k and a blank robots.txt will be 0k) (Ok, it's hardly anything, but why not run a tight ship)

Does that clear things up?

 

Are YOU not listening, exam?

Does THAT clear things up?

 

Not in the slightest

Don't get all worked up Minnie. Whether you put nothing or

User-agent: *
Disallow: 

Code (markup):

in your robots.txt file, it will do the exact same thing and is completely correct according to the standard. Why not choose the simplest solution? That way you save having to look up the syntax and you don't risk mistyping something and it even saves a couple of bytes of bandwidth.

Also you haven't responded to

 

Hi Guys,

I have recently installed a PHPbb forum.
I used some keyword analyser and realized that words like, class, post, moderator, user names come as the most used words on the site.
Can i write certain words in the robots.txt file which would be ignored from the crawlers?

Cheers ,
LittleG

 

No. You can only exclude files and directories

 

is there any solution to this problem?

 

You can exclude certain phpBB directories from indexing -- add this to robots.txt and make sure the robots.txt file is in the ROOT directory of your site:

User-agent: *
Disallow: /phpbb/admin/ 
Disallow: /phpbb/db/ 
Disallow: /phpbb/images/ 
Disallow: /phpbb/includes/ 
Disallow: /phpbb/language/ 
Disallow: /phpbb/profile.php 
Disallow: /phpbb/groupcp.php 
Disallow: /phpbb/memberlist.php 
Disallow: /phpbb/login.php 
Disallow: /phpbb/modcp.php 
Disallow: /phpbb/posting.php 
Disallow: /phpbb/privmsg.php 
Disallow: /phpbb/search.php 

Code (markup):

If your forum is in the root directory, admit the /phpbb part above. If the forum is in a directory with a different name, make the appropriate corrections.

Then download and apply Parts 1 and 2 of the following phpBB MOD:

## MOD Title: Able2Know.com Search Engine Optimization 2.0.0
## MOD Author: Craven de Kere (N/A) http://www.Able2Know.com
## MOD Description: The definitive guide to phpBB SEO

 

I downloaded the Able2Know.com Search Engine Optimization 2.0.0.txt 2 days ago.
It makes too many changes which i don't really want to do.
I am actually interested in the 4th.
4) Remove listing of moderators on index.php (AbelaJohnB)
but i can't really find this mod.
I will be really happy if You can help!
Cheers ,
LittleG

 

LittleG - doing that isn't going to help your forum SEO much. You NEED part 1 - you do NOT need any of the mod_rewrite stuff.

See attached file -- in there you will see this, which should remove the moderators listing for index.php:

# 
#
#-----[ OPEN ]------------------------------------------ 
# 

index.php 

# 
#-----[ FIND ]------------------------------------------ 
# 

	//
	// Obtain list of moderators of each forum
	// First users, then groups ... broken into two queries
	//

# 
#-----[ BEFORE, ADD ]------------------------------------ 
# 

/*/ 

# 
#-----[ FIND ]------------------------------------------ 
# 

   // 
   // Find which forums are visible for this user 
   // 

# 
#-----[ BEFORE, ADD ]------------------------------------ 
# 

/*/ 

# 
#-----[ FIND ]------------------------------------------ 
# 

'L_MODERATOR' => $lang['Moderators'], 

# 
#-----[ REPLACE WITH ]------------------------------------ 
# 

// 'L_MODERATOR' => $lang['Moderators'], 

# 
#-----[ FIND ]------------------------------------------ 
# 

if ( count($forum_moderators[$forum_id]) > 0 ) 
{ 
   $l_moderators = ( count($forum_moderators[$forum_id]) == 1 ) ? $lang['Moderator'] : $lang['Moderators']; 
   $moderator_list = implode(', ', $forum_moderators[$forum_id]); 
} 
else 
{ 
   $l_moderators = ' '; 
   $moderator_list = ' '; 
} 

# 
#-----[ REPLACE WITH ]------------------------------------ 
# 

/* 
if ( count($forum_moderators[$forum_id]) > 0 ) 
{ 
   $l_moderators = ( count($forum_moderators[$forum_id]) == 1 ) ? $lang['Moderator'] : $lang['Moderators']; 
   $moderator_list = implode(', ', $forum_moderators[$forum_id]); 
} 
else 
{ 
   $l_moderators = ' '; 
   $moderator_list = ' '; 
} 
*/ 

# 
#-----[ FIND ]------------------------------------------ 
# 

'MODERATORS' => $moderator_list, 

# 
#-----[ REPLACE WITH ]------------------------------------ 
# 

// 'MODERATORS' => $moderator_list, 

# 
#-----[ FIND ]------------------------------------------ 
# 

'L_MODERATOR' => $l_moderators, 

# 
#-----[ REPLACE WITH ]------------------------------------ 
# 

// 'L_MODERATOR' => $l_moderators, 

Code (markup):

This basically comments out the part that lists moderators.

 

Hi.
I've had a site online for years and I don't have a robot.txt file.

I was just given a free site review and was told that:

No robot.txt file in root / not good


Do I really need to add one? Will it change anything by doing it now?
I don't want to risk messing anything up.
My site ranks very well in MSN already. I'm listed in all the SE.

I do get those errors in my logs but I ignore them.


Thank you.

 

Go ahead and create a blank one. It won't hurt anything and you'll get fewer 404's in the logs.

 

1. you don't need one unless you wish to prevent certain files or directories from being indexed, or if you want to TRY to block certain spiders

2. you CAN upload a blank one but it's vey easy to create the default one and, IMO, makes more sense that way. Just create a normal ASCII text file as follows (you can even cut and paste this one):

User-agent: *
Disallow: 

Code (markup):

Save this file as robots.txt and upload it to the root of your website.

 

Ok, ASCII text file?

I know I have heard of ASCII, but how do I create one.

I would have just opened a new file in Notepad and saved it as robot.txt

What do I have to do to save it as ASCII ?

I think there is an option on my FTP software that lets me upload files as ASCII or Binary. It's default is on Binary. Do I just upload the file I create in Notepad with the ASCII option checked ?

Sorry, I'm such a newbie with many of these things...

 

It just means plain text, aka ANSI text.

If you use Notepad, I believe recent versions at least will default to saving the file as Unicode. Sometimes that can cause problems so use the drop-down box under Encoding and select ANSI.

If you use a different text editor, just save as any one opf ASCII, ANSI, Plain Text...

 

Thank you, Minstrel.

My Notepad default is ANSI.

So, it should be ok.


Thanks again.