GoogleBot Stopping at "Home"

[GuyFromChicago]

What’s the best way to guide the GoogleBot through your complete site?

I’ve noticed that sometime when I add pages the GoogleBot comes around and visits my site, but doesn’t follow all the links and index all pages. They always make it in eventually, just not as quick as I would like sometimes.

Any suggestions on keeping sites GoogleBot friendly?

[digitalpoint]

The only thing you can do to make Google visit more often, and spider faster is to get more links. Google judges the importance of a site by the number of inbound links they have and how important the sites linking are (basically PageRank).

For new sites, it just takes time... there is no way to force Google to visit more often. Just need to get more links so Google perceives the site as being more important.

- Shawn

[GuyFromChicago]

Thanks for the quick reply.

Links & content, the quest continues!

[hans]

yes i agtree that back links etc may be important
but it appears another factor also may be the activities on your site - the number of NEW pages and changes ..
i.e. the number of 200 responses the google bot gets s to 304
may be ..
and may be another factor
the word "forum" appears to be magic to bots and google-bot as well - hence having a forum and linking from that forum to NEW sections or pages - or having a NEWS page/section .. with announcements of new pages ..

seems to attract google daily basis AT LEAST !!
i am happy to have google alomost hourly ..
and like you
many bots at same time
only 1-10 pages then off again

but

a recent page i made just for fun ..

24 hrs later was in the google base
how the get a full picture form the small salad of visits may be a miracle ..
somehow pages are in anyway about female viagra or love (http://www.kriyayoga.com/viagra/female_viagra.html)

also having a site-map ( like the one created with webcheck (http://mired.org:8080/webcheck/) ) with links to all pages appear to make it somehow easier for many bots to get more pages out of a domain ..

[hans]

there is one more thing you may be able to do ..
or 2

on your domain start page it appears you have NO <title>title</title> tag ..
some bot do NOT spider / index pages that are very incomplete
OR
at least you would LOOSE lots of ranks by having a page without title tag in meta ..

then other point

example:
Upgrades and Repairs (http://www.jjscomputing.com/Upgrades%20and%20Repairs.htm)
having the URL:
http://www.jjscomputing.com/Upgrades and Repairs.htm

i am NOT sure if at all bots do like windows type URLs with spaces instead of regular _ or - between words ..
even when displayed or written as
http://www.jjscomputing.com/Upgrades%20and%20Repairs.htm
as it may appear in some browsers when looking at page source

it may be viewed differently by different bots.

to assure bots DO follow links - best is always to have clean straight URL
just keep in mind that ONE of many methods of INVITING google bot is to be in many other directories ... and they use different bots behaving differently :-)

of course your intention was to have clean words in URL - that makes Yahoo-bot slurp happy as Yahoo uses apparently search strings as opposed to Google using whole words only .. Yahoo uses / highlights search strings found in URLs Google seems NOT.

an easy method to see at a glance if you have title tags in meta section is once again a full sitemap report such as created by webcheck - it shows you in ONE of the several report pages pages with missing titles and it also shows you WHAT tiltes you have at a glance - just in case you forgot to change tile when using copy and paste to create another page form an existing one ..

enjoy

[GuyFromChicago]

Thanks for the look over of my site. I had just updated my meta tags on the index page a couple days ago, and most have accidently deleted the Title tag. Thanks again.

[hans]

another point about inviting googlebot and OTHER bots as well to go BEYOND home .. in addition to CREATE fresh updated CONTENT:

i have had several different domain / pages during the past almost 7 years
and the only BEST option is

- to have a direct LINK from your domain - HOME to all direct pages on your site ! DEEP into any folder !!

anything else has caused some spider to break down or loop underway and missing pages were the result.

for my site now almost impossible
but i still have to most pages a direct link and all other pagers have a door-page linked FROM the domain - home.

out there are many NICE ( GOOD looking !!! ) pages ... but
of course a domain / is nicer with a pretty picture or even flash and only a few nice titles or keywords and very few links
but the experince still has shown that creating a / page like a sitemap is more bot efficient than any other option i have tried.

google bot and a FEW others MAY climb into all subfolders .. many get lost and need help.
another way to SUPPORT bots would be to have HIGH up on the page among the first few links a link to a site-map created with a tool that gets ALL pages linked.

2 options i have to do that myself

a.
my own HomePageSearchEngine can create a LIST of ALL files - that list is similar to Unix / Linux command

ls -R

and REALLY shows ALL pages whether they are linked or forgotten to link - they will be listed with a full URL

b.
with webcheck - link-check tool a full crawl of all links comparable to a real bots can be made
the output - mani pages - one of it called sitemap.html can then easily be uploaded on the site and linked to from he home-page UP near beginning ..
to assure all pages are found

IF webcheck can NOT find all links / pages - then most likely at least SOME of the major SE - bots also may FAIL to go all the way

[Foxy]

Blogs are good

They are real fresh content and google aprrently loves them

[Foxy]

that was

http://forums.digitalpoint.com/showthread.php?p=1000#post1000

post 1000

:D