Hey All, I dont want google to index my wordpress admin area. Is my robot.txt file ok? I havent uploaded it. Can you people tell if its ok. User-agent: * Disallow: /wp- Disallow: /feed/ Disallow: /trackback/ Disallow: /rss/ Disallow: /comments/feed/ Disallow: /page/ Disallow: /date/ Disallow: /comments/ Disallow: /cgi-bin/ Disallow: /2008/ Disallow: /2009/ Disallow: /*?* Disallow: /iframes/ Disallow: /recommends/ Sitemap: http://mysite.com/sitemap.xml Code (markup): This is the first time I am going to upload a robot.txt file Please let me know if there is any mistake in it.
I dont think thats correct. Many people are using robot.txt file in the their websites and they are not getting banned. Robot.txt file is basically used to prevent indexing some of the pages in search engines.
What does Disallow: /*?* mean? Disallow to index all folders with 3 and more letters? Are you sure you want that?
It means you do not know what you are doing. I'm sure the idea was to prevent bots from spidering URL's that contain a "?". User-agent: * Disallow: /wp- Disallow: /feed/ Disallow: /trackback/ Disallow: /rss/ Disallow: /comments/feed/ Disallow: /page/ Disallow: /date/ Disallow: /comments/ Disallow: /cgi-bin/ Disallow: /2008/ Disallow: /2009/ Disallow: /iframes/ Disallow: /recommends/ Code (markup): This should be valid. All User-agents are prohibited to crawl the folders. Keep away the slash at the end to make sure, even subfolders are not spidered. For example: Disallow: /comments disallows the folder comments and all subfolders. Disallow: /comments/feed/ disallows only the folder feed in comments.
Thanks for the heads up man. It was a helpful explanation. Rep added. I was not going to upload it till I was sure.