First time setting up robots.txt

[Jinx3000]

Hi Guys and Girls!


I've been making websites for a little over ten years -- and I currently enjoy it more than I ever have. I've always wanted to learn, but for some reason have never implemented a robots.txt file on ANY of my domain names...

Now that one of my sites is pulling in more and more traffic, I want to make sure I'm not missing out on something vital here, hence this post.

I've just made my first attempt -- located here: the-lingerie-post.com/robots.txt

Is this a good start? What would you guys change?


Second, my forums would also benefit from a good robots file -- but it's on a subdomain. Does that mean it should have its own robots.txt file? What pages would you exclude? (I'm thinking member profiles, memberlist, search, faq, ...). Any practical hints? The forums are located at forum.the-lingerie-post.com.


Thanks in advance!

[SwapsRulez]

First of all your first robots.txt is completely OK.
Second you can always set the robots.txt for the subdomain
& third one your forum link is wrong as its not opening ;-)

[Jinx3000]

1. Thanks!
2. so i make a robots.txt file at forum.the-lingerie-post.com/robots.txt ? google will automaticallty find this?

3. sorry -- fixed! ;) forum.the-lingerie-post.com it is! :)

[SwapsRulez]

Yes... every subdomain is also searched for the robots.txt, therefore you can include the robots.txt in every subdomain you have ;)
& Yes, you'll have to make this file forum.the-lingerie-post.com/robots.txt
Peace!

[Jinx3000]

thanks, going at it right away!

[Jinx3000]

This good for my forums robots.txt file?

forum.the-lingerie-post.com/robots.txt


Thanks for your help, SwapsRulez!

[SwapsRulez]

No problem mate... its my job to help peoples... ;)

[catanich]

You only need one robots.txt file for http://www.the-lingerie-post.com/ and what you have in the robots is correct:

User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /comments/
Disallow: /cgi-bin/

If this is what you need if you DON'T want the SEs to index.