Hello, I found that Google is downloading robots.txt file daily once. But, it is reflecting the change even after 1 week. We restricted the entry of googlebot for some pages, even it is showing in the serp from few days but not storing the cache. Any replies? Thanks, PKR.
I have always learned and exp that google never catch the robot.txt pages...nothing to say about it..
Google crawls robot.txt file but it is not sure when it is crawled and the result is shown as you will mention in the robot.txt.
The best option is check your webmaster tool. There is robot.txt section There you will find your actual robot.txt file which is recently download by google. You will easily come to know whether google reflected your changes or not.
Tom, i already checked there it showed the functioning very correct as per the instructions in the robots. And the user generation2020, vl09 pls try to understand the question. Pls don't mis understood. Google daily downloads the robots.txt file. But, not stores the cache file. I asked about the cache of the restricted url's, mentioned in the robots.txt file. Ok. And, i think Googlebot's are getting old. In fact, taking much time to reflect important updates in the robots.tx file. And, i am sure and strong in my case. Google
i have strange situation with my robots.txt file. I made robots file about 3 weeks ago with list of php files robots should't crawl. By the time, I saw the number of detected errors are reducing, but today when I checked the situation i saw there are 2 new "errors" with php files I put in robots file. Anyone know how is it possible? My robots file is ok. How is it possible to robots detect new "errors"?
Mares, you can check the functioning of robots.txt from your webmaster tools -> Tools section. It will help ....
i've just checked as you said and still dont understand why bot detects problems. Two days ago I got one new url restricted by robot (and this url is too included in robots.txt). I checked tools -> anaylze robots.txt, status is: success. When I type in field "Test URLs against this robots.txt file" the restricted url (which is in robots.txt file), and press "check" buttion, I got result "blocked by line ..." which means it works properly, right? But why bots detects same php file as restricted? My robots.txt file is made properly, code is: User-Agent: * Disallow: /example1.php Disallow: /example2.php Disallow: /example3.php etc...
delete the robots.txt from server, delete the reference from google webmaster tools, and it will find it and add it again but this time it will reflect the changes made
Just because it's not showing as updated in Webmaster Tools doesn't mean it isn't. Google doesn't always export data there in real time, just like the toolbar PageRank scores... those are usually months old.
It is one thing to add a "disallow" command, but if it has already been indexed before, it will still be there. You MUST use the URL removal tool in WMT to remove the pages/directory. And I have had no problem with any of our robots.txt files and getting the major SEs to follow it.