Wow is ROBOTS.txt important?

Google just index limited pages from each site everyday depending of the strongth of it.Robots.txt will be useful for the newly site to index the important pages firest
You can also use this If you don`t want spider prevent any thing of your pages

# Allow all
User-agent:  *
Disallow:

Code (markup):

 

you are welcome to the webmaster club robots.txt is one of the key factor web site designing

 

you r right skionxb

if u really want to disallow anything then it matters otherwise i dont think its really important..to include robots.txt

 

Would a phpBB2 forum need a robots.txt file or only html based sites? What about flash sites?

 

Yes, you can use a robots.txt for .php pages if you want to - to keep bots out of places that you don't want to appear in SE results. If your forum is in a folder called forum, you would use a robots.txt to exclude bots from pages like avatar.php and faq.php as follows.

User-agent: *
Disallow: /forum/avatar.php
Disallow: /forum/faq.php

 

all the files with no content should be coded out

 

I don't know if anyone has posted it yet, but there is some nice information here: http://www.thesitewizard.com/archive/robotstxt.shtml