Robots.txt

New to SEO however understanding keyword analysis, sand bagging, page names etc., in looking at my raw log files can see where spiders are constantly looking for /robots.txt of which I do not have.

I guess should I..would it help with ranking and placement?
I stumbled onto this site that offers a free robot txt generator and wondeirned what the group thinks if its good to use?
Thanks.

 

If you there's nothing on your site that you don't want spidered, you could just create an empty or very simple robots.txt if it makes you feel any better not getting 404's on it.

Personally I think having one means nothing with ranking/placement.

The site looks ok, the robots.txt specification is pretty simple... can't go too wrong if you read a little on it.

 

thanks..will research and read up on it further.

 

And don't forget about http://www.robotstxt.org/wc/robots.html

 

I have a follow up questions about this.

On this website

http://www.webseo.com.au/robot.html

it says put robots in your meta and on this

http://www.sabrewebdesign.com/support/tools/robots.txt.php

put a seperate file in index folder?

Is it the same thing or not?

AND

Should i change my meta to

Now i have

<META NAME="Robots" CONTENT="All">

should i change to

<META NAME="Robots" CONTENT="ALL=INDEX,FOLLOW">

 

check this link from google.com

google.com/support/webmasters/bin/topic.py?topic=8475

This will help you out.

 

it helped me out but i still want to know about the meta tags nothing about it there