Just curious whether anybody could suggest some good basic robots.txt file for phpBB3 to exclude useless pages from index. The forum runs on Lighttpd and URLs are not rewritten. Spore Forums Thanks in advance
From the PHPBB Knowledgebase Thwarting The Googlebot (and other helpful tips) User-agent: Googlebot Disallow: /posting.php Disallow: /admin Disallow: /privmsg.php Disallow: /search.php Disallow: /login.php Disallow: /memberlist.php Disallow: /images Disallow: /includes Disallow: /profile.php There are other tips there as well, so be sure to read it. (I'm unable to post links, being a newb, so copy/paste the link below) http://www.phpbb.com/kb/article/thwarting-the-googlebot-and-other-helpful-tips/ Good luck!