I dont have a robots.txt

Hey guyz, I dont have a robots.txt file, all the time i was thinking its automatically in our sites, but today when i went on google it said i dont have it
can someone tell me how we add a robots.txt file ? and what do we have to type in it ?
(Sorry, but i am new, to robots.txt thingy )

 

This will help get you started.

http://www.robotstxt.org/robotstxt.html

 

i don't have one either. I don't think they are really necessary.

 

They're actually a pretty darn good idea to have. Not only will they keep your server error logs free of "false positives" but they'll also help you channel your PageRank flow to and from the pages you want to be indexed (not to mention keep the legitimate bots out of places where they don't belong).

 

ok, i added a robots.txt generated from google, and then i just typed these where i dont want bots to go like
disallow: /forum/calender.php
is that fine ?

 

That should be fine.

 

IMO is safer if you hide your .php files from malicious bots
You can place your .php files in a directory, example /myphps and add:
disallow: /myphps

Also you may need to add this code on your .htaccess file on /myphps directory to avoid directory listing:

Options -Indexes -Includes
Order allow,deny
Allow from all

Code (markup):

 

hmm, alright, thanks.

 

Honestly not having a robots.txt is not a big thing. But still its good to hide your folders from getting indexed. DP has its own Robots.txt section which should help you in getting a start.

 

You don't need a robots.txt file if you don't have anything to hide from search engines (like a folder or a file on your website)

 

You really don' need a robots.txt.

 

Can you explain why ?.
I agree with Dan Schulz: helps to keep your logfiles free of HTTP 404 errors and keep the legitimate bots out of places where they don't belong.