Here is what my robot txt. file looks like. User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /help/ Disallow: /images/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ Is it configured properly or should it be changed?
looks okay. you also use the below link to check http://www.searchenginepromotionhelp.com/m/robots-text-tester/robots-checker.php
thanks for this information..I was new and only not getting the right answer to what to put in my robot.txt file..that is why I never made one ..Now I think I can and can test it too..I hope this is all to put in this file nothing else to put in this file..
noob question. What is the robot txt thing about anyway? what is it? laugh if you want. Im a noob but Im here too learn.
No laughing--its better to ask then guess. I recommend you check out my blog, it answers a lot of noob questions among other things. In short, a robots.txt controls what files a "robot" (or spider) from a search engine will index. So for example, if you have a website, and you place all your pictures in an "images" folder, but you don't want Google to index these Photos in Google Images, you write in the Robots text to not allow Google to Index this folder. read more here http://www.mark8t.com/2008/09/03/why-you-need-a-robotstxt/