I just saw in the server logs that now google don't (always) downland a page if no change were found. If no change from the last visit, the robot will get a 304 status code... * /main.txt Http Code: 304 Date: Feb 03 10:50:34 Http Version: HTTP/1.1 Size in Bytes: - Referer: - Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) .... I also notice that for a URL like http://server.com/directories it will do a redirect to http://server.com/directories/ so if you will link direct to http://server.com/directories/ (with the last /) it will index much more pages and much more fast