hello all, Googlebot came to pay a visit to my new site, http://www.salesjobsdaily.com, looked to see if there was a robots.txt (there was none at the time), then looked at the main page only and left. Is this normal? How many pages does Googlebot usually visit for a brand new site with hindreds of pages all linked up via straight html and no meta robots tags? Or is this number unpredictable? thanks, james
Backlinks seem to grab the attention of the Spiders. Keep building them and, as time goes on, you'll be indexed. I've had sites I've built months ago that have only just recently been completely indexed. I wouldn't worry about it too much unless if you notice the engines are specifically ignoring you. I also highly suggest you try Google Sitemaps.
Google will sometimes hit the page many times daily, just taking one page, then all of a sudden BANG it will start taking more. As posted above, it depends on the weight of the links it has found. I had one site that I put up, I submitted to the ODP, it was up and accepted in 3 days, Google picked up the backlink from the odp. and within 6 weks had taken around 4000 pages. Other times I have had to wait 6 weeks for the spider to go beyond the front page. It is just one of those things. That said, I would suggest you make sure there is nothing stopping the spider getting past the first page such as unspiderable links etc.
I think it is quite common to get a few home page only visits for a new site. Be patient, give Googlebot a text link to a sitemap from the home page (I mean just a regular sitemap with text links to all your main pages), and in due course you'll get the rest of the site spidered. It's almost like Googlebot is following a link to your home page from another site, makes a "note to self" that the site is there, and then when it gets a moment after it finishes cleaning the bathrooms, it stops by for a proper lok at the rest of the site. Follow any woman around the mall and you'll get an idea what it's all about. Well, maybe not just ANY woman... it's probably safer if you know her and she knows you're there so she doesn't call security and report you as a stalker...
I should probably create a sitemap, I have been getting the same problem. I do have robots.txt though
Yes. Any site with more pages than can comfortably fit in a simple nav bar should have a sitemap, linked from the home page.
yeah, despite the fact that all of my pages should be easily spiderable because of the way they are interlinked, i will create a simple sitemap as well. btw, i just noticed that the main page already appeared in google index and is returned in search results if the exact site title is queried