Robots.txt

I currently do not have a robots.txt, but feel I should have one. Now, I would like to accept all search engines - perhaps not google image, but other than that...? As well I have one folder I do not want searched "adminpanel". How would I write that? Anything else I should keep in mind?

Thanks very much

CWN

 

Disalow:/the folder/

it's that your prob?

 

Copy and past the code below into your robots.txt file

User-agent: *
Disallow: /adminpanel/

 

Hi

It is easy. It must be a .txt file and saved in the root directory with the exact name of "robots.txt"

A good example is go to Google's: http:// www. google. com/robots. txt

or look at mine. I've added some examples to it for you.

------------------------------------

# Robots.txt file created by 6/21/06
# For domain: http:// www. catanich. com

# All other robots will spider the domain
User-agent: * # index everything, all bots

Disallow: /_* # do not index any file starting with "_"

Disallow: /catanich.html # do not index the file catanich.html

Disallow: /catanich/ # do not index the directory catanich

-------------------------------
Note: the # is the start of a robots.txt comment

But a quick Google Search on robots.txt will help you as well.

Try http://www. javascriptkit. com/ howto/ robots. shtml for a quick overview. It was the first one I came to that made sence.

Good luck

Jim Catanich
www. catanich. com

 

If anything, you should at the very least have a blank robots.txt file

 

Thanks everyone, very helpful!

 

Whether it's an open-door robots.txt or a restrictive one, it's still smart to have one ... without it, many bots will 'unfairly' register 404s looking for it

 

open-door robots.txt? does that mean, allow everything? anyone have a robots.txt that allows everything? this is what i am looking for

 

Yes, that's an open-door as I call it.

However, it's very easy to restrict some folders, which you might want to do, for example:

User-agent: *
Disallow: /test/
Disallow: /cgi-bin/
Disallow: /cp/

 

You guys ROCK! Thanks very much!

 

This is my robot.txt ...

User-agent: *
Disallow: /adminpanel/
Disallow: /cgi-bin/

It seems simple - would this be considered "efficient"? I think it suits my requirements

 

Good move on keeping bots out of your admin and cgi. Yes, seems efficient and it will mitigate unnecessary 404s when bots can't find the robots.txt file.

The next thing you need is a favico