Robot.txt

How can i create robot txt?
what are the circumstances to put portion for not indexing?
what should i write in robot txt?

 

Here's an intoductory article about robots.txt
http://www.javascriptkit.com/howto/robots.shtml

 

Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

Friend, you can try online free tool for generating robots.txt file. ex:- http://www.seochat.com/seo-tools/robots-generator/

 

Login into webmaster tools and go to robots section.. their you can create sitemap quickly/..

 

If your using WP, there are plugins to do it.

 

+1 I use PC Robots or something very closely named to that. Creating a robots.txt file isn't hard by any means, but WordPress really takes the effort out of it.

 

After making a robots.txt file, the forum section on my site seems to have picked up the robots really quickly.

 

Mind you name the file as robots.txt and not robot.txt or else bots will not recognize it.

 

Robots.txt is simply a text file. Search Engines send a spider or agent to visit your site and index your content but even now and then there are a case when indexing parts of your online content is not what you want. Hence, you just want to hide that details. The location of this file is important i.e it should be placed in main directory.