How often do search engines hit up this file? I know .htaccess is hit for every single website request, but with robots.txt, it seems like google/etc only do it every once in a while. Any idea of how often they do it, and/or how to force them to realize it's been updated?
Hi 1) you need to create sitemap.xml and sitemap.txt 2) add/edit yoursite.com/robots.txt url in those sitemaps (both) 3) have metatag commands like <meta name=robots content="index, follow, all"> 4) join google webmaster tools and submit your robots.txt, sitemap.xml, verify your site by add google give codes in your site. (if you don't verify it is not worth it so you must have to verify, now if you don't know we can help you so please pm) 5) submit your sitemap URL to yahoo as you submiting your site URL normaly to yahoo. please PM me know if you not sure of anything
By the way does anyone have a sitemap generator I can use? I really need one. Is there a way to create sitemap on blogspot?
Hi, Here is the quick & dirty way to get google to load your latest robots.txt file: Login to webmaster tools, select Url Tools at the bottom of the list, next select the url removal tool and select a page on your site for removal. It will now say pending, now watch your httpd logs, in less than 1 hour(most times about 10 - 15 mins.) googlebot will grab your robots.txt file. Now go back to Webmaster tools url removal tool and select cancel for the page you selected earlier. Now check the robots.txt file in Webmaster Tools and you will see it is only a few minutes old. That is how you do it (After google reads this, it might change)...... lhughes33309