My site is http://www.foreverfreestuff.com sitemap: http://www.foreverfreestuff.com/sitemap.xml I have paste this code at my wordpress header.php to avoid duplicate: <?php if(is_single() || is_page() || is_home()) { ?> <meta name="googlebot" content="index,noarchive,follow,noodp" /> <meta name="robots" content="all,index,follow" /> <meta name="msnbot" content="all,index,follow" /> <?php } else { ?> <meta name="googlebot" content="noindex,noarchive,follow,noodp" /> <meta name="robots" content="noindex,follow" /> <meta name="msnbot" content="noindex,follow" /> <?php }?> and i have installed Plug-in of All-In-one-SEO also. and i juz confused my robots.txt, who can help me to correct my robots.txt? because my robots.txt cannot pass at Google robot.txt analysis. my Robots.txt is: User-agent: * # disallow all files in these directories Disallow: /cgi-bin/ Disallow: /z/j/ Disallow: /z/c/ Disallow: /stats/ Disallow: /dh_ Disallow: /wp-admin Disallow: /wp-includes/ Disallow: /contact/ Disallow: /tag/ Disallow: /wp-content/ Disallow: /wp-content/themes/ Disallow: /trackback/ Disallow: /*?* Disallow: */trackback/ User-agent: Googlebot # disallow all files ending with these extensions Disallow: /*.php$ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.cgi$ Disallow: /*.wmv$ Disallow: /*.png$ Disallow: /*.gif$ Disallow: /*.jpg$ Disallow: /*.xhtml$ Disallow: /*.php* Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ Disallow: /*?* Disallow: /z/ Disallow: /wp-* Allow: /wp-content/uploads # allow google image bot to search all images User-agent: Googlebot-Image Allow: /* # allow adsense bot on entire site User-agent: Mediapartners-Google* Disallow: /*?* Allow: /z/ Allow: /about/ Allow: /contact/ Allow: /wp-content/ Allow: /tag/ Allow: /manual/* Allow: /docs/* Allow: /*.php? Allow: /*.js$ Allow: /*.inc$ Allow: /*.css$ Allow: /*.gz$ Allow: /*.cgi$ Allow: /*.wmv$ Allow: /*.cgi$ Allow: /*.xhtml? Allow: /*.php* Allow: /*.gif$ Allow: /*.jpg$ Allow: /*.png$ # disallow archiving site User-agent: ia_archiver Disallow: / # disable duggmirror User-agent: duggmirror Disallow: /
I simply have the following: Do you have a problem with Google indexing your WordPress files or are trying to protect your WordPress? If you are trying to protect, you should be doing it with .htaccess, not robots.txt.
I have a problem with Google indexing my wordpress files, is this robots.txt got problem? how to change it better? please teach me...my friend...
Yes, the issue is with your robots.txt, your blocking all file extensions that end with the wild card *.php, remove this so Google can index your web pages.
But my website is HTML, so if i cancel disallow /php*, google will index all my php files? include wp-admin.php
I have use the robots.txt on my blog like this and it can avoid from the SR Sitemap: http://www.mysite.com/sitemap.xml # This rule means it applies to all user-agents User-agent: * # Disallow all directories and files within Disallow: /cgi-bin/ Disallow: /stats/ #Disallow: /tag/ Disallow: /wp-admin/ Disallow: /wp-includes/ # The Googlebot is the main search bot for google User-agent: Googlebot # Disallow all files ending with these extensions Disallow: /*.php$ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*.tar$ Disallow: /*.tgz$ Disallow: /*.cgi$ Disallow: /*.xhtml$ # Disallow Google from parsing indididual post feeds and trackbacks.. Disallow: */feed/ Disallow: */trackback/ Allow: /feed/ # Disallow all files with ? in url Disallow: /*?* Disallow: /*? # The Googlebot-Image is the image bot for google User-agent: Googlebot-Image # Allow Everything Allow: /* # This is the ad bot for google User-agent: Mediapartners-Google* # Allow Everything Allow: /*
So can i use your robots in my website? or anything i need to change at my robots file? can you teach me? thank all who reply and help me.. thank you very much and much!!