hows my robots.txt

User-Agent: *
Disallow: /lang
Disallow: /img
Disallow: /style
Disallow: /cache
Allow: /
Disallow: /admin

Code (markup):

 

If You are using Google Adsense in your website then all the page should be allowed to the Adsense Crawler so add this lines too:

User-agent: Mediapartners-Google
Disallow:

Thanks

 

Everyone should drop this subject for robots.txt. Firstly, robots.txt is extremely stupid. Google may not snoop around the areas you choose to block, but anyone with a browser can enter /robots.txt at the end of your domain and clearly see all of your directories. Do yourself a huge favor and simply remove /robots.txt. The countermeasure would simply be to lock or restrict the folders with a password or some kind of authorization/authentication scheme. Lastly, add an index.html file to each folder, and call it quits. That's your best bet for the state of security. As far as SEO and other optimization methods, it might help, but security matters more than functionality

 

I would drop the "Allow: /", it is not needed.

User-Agent: *
Disallow: /lang
Disallow: /img
Disallow: /style
Disallow: /cache
Disallow: /admin

 

I couldnt agree more! I have never addded robot.txt to any of my sites just for that reason.

 

excellent excellent. i second that.

 

what is the txt for can you get a text that displays sertain ads?

 

Plz suggest me google adsence pin no

 

Great security advice.

 

Question about not using the robots.txt, when I don't use it on my site, in my (Cpanel - Awstats) it shows up in my "HTTP error codes" as a 404 - Document not found.

I checked it earlier today and it shows 56 hits for the robots.txt, so wouldn't this be a problem?

Let me know what you think
Mike

 

lol! 56 hits for the robots.txt. Maybe I should make the includes folder called "h2kld9wj" and the other useful stuff in one big folder : "ao47hgaiwuhga9w83hawyt9awyg30awu30t"

 

@fireworking: that's just security by obscurity. Don't do it.

- Your include files should always be named .php and start with <?php. No .inc, .txt, .htm etc.
- Set correct file permissions & use .htaccess to deny access to directories & files.

Also, not having a robots.txt is a bad thing SEOwise.

 

It is a common misconception to believe that robots.txt can improve security. But still robots.txt is very useful to pass crawling instructions to polite robots.

Jean-LUc