Giv me an intro to robots.txt

[birdsq]

I am new to this topic
can you please give a short description
on the use of robots.txt

[ajsa52]

Read this page: http://en.wikipedia.org/wiki/Robots.txt

[birdsq]

How do we create this
simply make a file robots.txt?

[mistnee]

How do we create this
simply make a file robots.txt?
yes use notepad to create one and upload it to your root folder of your website

[birdsq]

I need google to crawl all content in my forum
And i dont have a robots.txt

[ajsa52]

I need google to crawl all content in my forum
And i dont have a robots.txt

For that, you can make a sitemap and submit to Google.
Read this page: http://en.wikipedia.org/wiki/Sitemaps
Also you can use a tool to create your sitemap automatically.

[birdsq]

which tool?

[ajsa52]

Google provides "The Google Sitemap Generator (https://www.google.com/webmasters/tools/docs/en/sitemap-generator.html)" is a Python script (for advanced users).

Also you can search the web for other online tools like this: http://www.xml-sitemaps.com/

[birdsq]

Since i am new to website programming
For me it was a great help .....
thanks ajsa52

[just wondering]

great info

[seoworld]

I suggest that you read this http://www.robotstxt.org/wc/robots.html