I need a good robot.txt which must be a google friendly and seo friendly! My website: http://www.sportsstatsblog.com/ Platform: joomla site map: http://www.sportsstatsblog.com/sitemap.xml Site map generated with help of JCrawler here:http://extensions.joomla.org/extensions/structure-a-navigation/site-map/5386 current robot.txt according to webmaster tool User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /media/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates/ Disallow: /tmp/ Disallow: /xmlrpc/ Code (markup): I know there is something wrong with current robot.txt but don't know anything about it! so need your help! So robot.txt guru help me!
what makes you think there is something wrong with it? If they are the directories you don't want bots to index then it's perfect.
Quick lesson: User-agent: * = All users Disallow: = don't allow spiders (Google) to see what is after this Disallow: /libraries/ = don't allow spiders to see the folder libraries
I am not sure what suspicious you find in this robots.txt file. But, I do not find anything wrong there.
means it's a good robot.txt? I was in doubt because never deal with that number of "Disallow". I still don't know whether these help my site or not! Guys just tell me will it work good for my sports blog: (http://www.sportsstatsblog.com/) or not!
you can use "Robots.txt syntax checkers" some of them i m mentioning below: 1:Google Robots.txt Analyzer ( now it is 404 use below checkers) 2:UKOLN WebWatch /robots.txt checker 3:SearchEnginePromotion's robots.txt Checker 4:Simon Wilkinson's Robots.txt syntax checker 5:Motoricerca Robots.txt Syntax Checker plz keep your eye in my reply if i will get any thing else i will update my contents. thanks alot
what you should add to the bottom of that list is Allow: / Code (markup): Just to keep it all in order , and show the bots that there is something to be crawled Make sure it's UNDER the Disallow though...