My site's URL is: www.twomap.com Google[BOT] used to come into my site minimum twice a day and start to crawling, but crawling is not done. In Google Search Results my old database pages are still there. I lost that database. So I created a new database, new pages in new database are not showing in search results(Google bot visited now GMT +5:30 10:22AM). Still in search results it showing the pages that on 10th march. And how to get indexed fastly on google.
there is one user called Google[BOT] for google(with their ip). it have all abilities of guest. so no need of robot.txt, i think
ya i know , but it can be specifically used to define when to revisit your site and what matter to crawl, it like this google can do it own its own , but robots.txt only reminds google to do that specifically
It seems your site is still a new one. Google crawling rate depends on two main factors 1. Site update history. Google bots take a month to collect how frequently your site content being updated or added new one. If you update everyday then Google bots would come and cached a new page everyday. 2. Number of inbound links. The more links, the more crawling ratio.