Hi, I just now read robot.txt should be in seo. I want to know how to do robot.txt in my site. is there any problem if not in my site. please reply me in short and clear......
robot.txt is use when you want a certain page in your site not to crawl by search engine spider and you can create robot.txt in google webmaster tools.
Yes. You can do that. It might be confusing when you try for the first time. If you are using WordPress, it creates a virtual robots.txt file by default
Creating the robots.txt file It can be created using notepad or whatever is your favorite text editor. Each entry has just two lines: User-Agent: [Spider or Bot name] Disallow: [Directory or File Name]
You could simply create a text document & save the new document as robot.txt. but don't use a html editor, could use the notepad for this purpose.
Most websites don't even need a robots.txt Be sure to double check before you upload one on your site. Half of the people on the forum complaining that their site is not getting indexed have a bad robots.txt file ;-)
If we can define Robot.txt file on webmaster tool it will be easy to index your WebPages .In robot text file we need to define which file of folder need not to be indexed.
Most of sites are not used robots.txt files. Robots.txt file simply a notepad file. Help of this file, we mention some guideline for search engine bots which part of site is access by bots or which is not access. So it is not harmful for our site. enjoy posting.