I put up a new site on the 2nd of November and established five links to the site from well ranked/indexed websites. From the logs, I know that googlebot visits the site every day between 2am and 7am. It grabs the robots.txt file and the home page. That's it. It does not go any further. The home page has about 300 words of text and about 100 links to subfolders and pages in the site. Now before you ask, I'm not listing the domain because this is just an experiment and I don't want a lot of extraneous traffic to the site making it harder to find googlebot in the logs. The only thing I have not tried is deep linking and that's my next strategy. This is my first real venture in setting up a site from scratch and working to get it indexed. All of my previous work has been with sites that have existed for quite some time and the SEO has been easy in this regard. I've read all the theories (sandbox, etc.) and I'd like to hear your thoughts as to why googlebot visits daily but only the homepage.
I thought it might, but I wasn't sure. I just needed a little reassurance that this was the normal course of getting a new site in the index. It just gets a little frustrating seeing that googlebot gremlin not going past the home page. I wonder what the purpose is? See if the site exists for so long before including it?
Correct - I set up Food-Network-OK.com [one page] on 17/10 and then added another "directory" [actually Mambo in a directory] on 30/10 - it took 1 day for the first page and then after the 30/10 it took 4-5 days to show the next level..I expect the next levels/pages to show soon...as each level shows I do some more to see the effect. So far for "food network" I am no 2 +3 for all in url, 54 for allinanchor and 40 for allintext both on the site page. Theoretically I should have a number of pages [because of the nature of Mambo] and links showing in google under the site: search but at the moment I have only 2! and link: shows none [which is interesting beacuse I am 54 in allinanchor] but @ shows 3! Another interesting point is that this second page/directory is now at 717Â http://www.food-network-ok.com/food-network/ for "natural food network" which is only on text!
It usually takes a couple of weeks before Gbot goes further, but a funny thing happened this week. We put a new site up and the index page showed up about 3 days after a link was pointed to the / page. Then the next day we had 30 pages in G....and had 6 search visits from some pretty random search queries. I had never seen 30 pages show up that quickly....but, even more strange was the search traffic on the same day. Anyone had that happen?
It seems to be that the higher the site PR, the more likely the site will be crawled more deeply - and more quickly, too.
Indeed, I also believe that to be true. Then, of course, a new site won't have any PR. It's just frustrating to see gbot come and go w/o doing anything more. Curse that gbot!
I see this type of bot behavior regularly, even though my site is 6 months old and even though all pages are indexed. Googlebot will visit my default page 6 to 15 times in a day, but only the default page. I may then see no bot activity for 10 days. I might then see a deep crawl of 75% of my pages in a day with the remaining 25% getting picked up the next day. My theory is that the single hits on the default page are actually fresh crawls from those sites that link to you. Gbot is crawling that page, finds your link, follows it, checks your robots.txt for permission, then requests the page. If it loads successfully, the link is verified. (again my speculation). I also believe that when the link is verified, Gbot phones home and requests that the visited page be scheduled for a crawl of its own. Then, Gbot returns to the page it came from for further spidering of that page. At some point, your site’s default page’s crawl time will arrive and a Gbot will begin logging your page attributes and crawling links to your other pages. Again, if your other pages load, each link is verified and a crawl of that page scheduled. With 100 links from your default page, I suspect that it will take several weeks for all pages to be link verified, crawls scheduled, and actual page crawls completed. I think most people agree that normally (if there is such a thing as normal) new sites are given less crawl time than older high PR sites. It also seems that sites that are updated routinely get more crawls than those that are not. /*tom*/
What I observed is that PR5 pages get the pages linked on them crawled daily. Like if home is PR5, then 1st level pages linked on it will get crawled daily. If home is PR6 and 1st level pages are PR5 then 2nd level pages are crawled daily also. As long has you do not have 200 links on the page, of course.
hi Mopacfan, what you describe happened to J1UK also...and lasted 4 months.GBot did actually take a lot of pages one time...but they slowly disappeared. Launched a recipe site this week with a buddy...and we were looking for some recipes to add, on google....so we put in a search ...and clicked on a random result....and it was OUR SITE !!! you've never seen 2 grown men cry?..it was so funny! (obviously you would have had to have been there) Now, i dont think it even shows up for the same search, but we are in no rush. So don't go getting all worried about it....just work on links...you know enuff about SEO not to worry, i'm sure GEM
Can any explain the following: So far for "food network" I am no 2 +3 for all in url, 54 for allinanchor and 40 for allintext both on the site page. Thanks Ian