Newbie Question - What is Robots.txt and what does it do?

Question is in the title.

What does it do?

I ask this as most sites have them and when I checked mine it doesn't.

Thanks

 

It helps you block certain pages from getting indexed on google.

If you do not want your images folder to get crawled, then you would add it to your robots.txt file

If you are only blocking images folder, your robots.txt file would look like this:

 

Thank you very much. Appreciate that alot.

 

Nicely Explained I Do Agree with. Thanks

 

neatly explained..
Great

 

thanks... I was going to ask this question but got the answer here