I have noticed something that I don't understand, and wonder if someone can make sense of it. I run a site about "speculative fiction", which is an umbrella term for two genres usually called "fantasy" and "science fiction"; I consider those two genre names to be my keywords. The very name of the site uses them both: "Great Science-Fiction and Fantasy Works", and unless context utterly forbids, when I mention one I mention the other. My site has, effectively, over 11,000 pages, of which Google seems so far to have become aware (count courtesy of Shawn's keywords tool) of 5,080; on each of those pages, both terms appear--as a pair--several times: in the page title, in the meta description, and in the meta keywords; and on most they appear in body text as well. Last night I did a little checking of that body (and header) text, and over all those thousands of pages, with their corresponding thousands of uses of each term, I believe that the keyword "fantasy" is used more often than the keyword "science fiction" perhaps 20 or 30 times total. Now: on a search for <fantasy science fiction> the site turns up #27; on a search for <science fiction fantasy> it turns up #88. Hmmm?
If I understand your question properly you think that because 'fanasty' as a word occurs more often than 'science fiction' that when it is placed at the beginning of a keyword phrase it will get a higher rating than when at the end. Is that correct? I doubt that the number of time these words occur in the on-page content is the determining factor. How many backlinks have you got? And what is the most common anchor text? People continue to believe that on-page content is the major determinent of index ranking. And it may be for very niche terms. But for popular search terms the anchor text of links has much more to do with SERP placement than the on-page content.
If I understand your question properly you think that because 'fanasty' as a word occurs more often than 'science fiction' that when it is placed at the beginning of a keyword phrase it will get a higher rating than when at the end. Is that correct? Not exactly: I know that it gets a higher rating: the two search-pattern forms, A B vs. B A, return drastically different placement results--despite the fact that the difference in their rate of use on the site is small, one might say negligible. I doubt that the number of time these words occur in the on-page content is the determining factor. How many backlinks have you got? And what is the most common anchor text? Most backlinks will be to the name of the site, which contains both terms (moreover, the term getting the way-lower rank is the first in order); the remaining backlinks will probably be to the site URL. People continue to believe that on-page content is the major determinent of index ranking. And it may be for very niche terms. But for popular search terms the anchor text of links has much more to do with SERP placement than the on-page content. So no backlink will contain one term but not the other: a backlink will contain both or neither. Those things are why the wide discrepancy puzzles me.
Isn't <science fiction fantasy> including surrounding text as well as fill words the more likely used phrase thus faces more competition. Fanatasy being a genre of sf and thus a qualifier. <fantasy sience fiction> is the less likely used phrase and harder to qualify even using semantics thus faces less competion. Thus backlinks or internal links as well as on page usage of - author of fantasy, writing great sience fiction -may have more impact than - author of sience fiction and fantasy books - M
Isn't <science fiction fantasy> including surrounding text as well as fill words the more likely used phrase thus faces more competition. Fanatasy being a genre of sf and thus a qualifier. On sites that discuss such things, "fantasy" would not be considered a subdivision of "science fiction" (if anything, the other way round: once it was all called "fantastic fiction"). <fantasy sience fiction> is the less likely used phrase and harder to qualify even using semantics thus faces less competion. Thus backlinks or internal links as well as on page usage of - author of fantasy, writing great sience fiction -may have more impact than - author of sience fiction and fantasy books. If I understand Google, neither <fantasy science fiction> nor <science fiction fantasy> is a "phrase": they are just individual words. But I checked some more, and it is all even weirder than it looked at first: fantasy science fiction : #29 fantasy "science fiction" : #51 science fiction fantasy : #89 "science fiction" fantasy : #90 Holy counter-intuitive, Batman! Never--I think literally nowhere at all--will the word "science" appear on the site unless as part of the phrase "science fiction" (or "science-fiction"); yet using the phrase as a search term materially lowers the site rank compared to using the words individually. My expectations for relative rank are based on this: the word "fantasy" and the phrase "science fiction" are each used on the site thousands of times, always several times a page; over the whole site of--so far as Google has yet counted--over 5,000 pages, the frequency of "fantasy" exceeds that of "science fiction" by perhaps two or three dozen occurrences total; and links to any page will either contain both of the terms (the commonest case) or neither. (Oh, and the site index page assuredly contains both terms an exactly equal number of times.) Those expectations are that whatever the actual rank assigned the site under a search for <fantasy "science fiction">, it should be higher, but only by a quite small degree, than a search for <"science fiction" fantasy>. Moreover, a search that uses "science" and "fiction" as separate words rather than as a phrase ought to yield similar but slightly higher rank for either order. I find the argument of "less competition" for f/sf compared to sf/f plausible but not convincing: I see pretty much the same sites on both searches, only in somewhat different order. I am moderately familiar with many of those sites, and my anecdotal experience--obviously, I have not combed every page of 90 sites for term use--is that "fantasy" gets a pretty closely comparable use frequency to "science fiction", though that could be a small-sample or personal-memory error. It all remains most puzzling . . . .
you have NO real title H1 !! the picture used as H1 never counts in full with same priority/weight value for H1 - you lose LOTS of positions for that omission !!! you have too many words in your main tile IF "science fiction and fantasy" should be your core main words .. your home page has too much content - too difficult to get in top 10 with that many pages on your site and in Google you have no hurry to get all info on first page !!! only a tiny small % of my visitors come through the "front door" with Google's excellent deep crawl most visitors come directly into the most relevant pages according to their individual precise query. so .. first a few entry facts from "Keyword Suggestion Tool" both sources ------------------------------------- snip fantasy 4,658.0 /day fantasy 11,860.9 /day science fiction fantasy 61.0 /day science fiction 670.0 /day science fiction 4,697.7 /day -------------------------------------- snap If you look at Google and see the search results for science fiction science fiction and fantasy then you see clearly that IF you want - you have a good chance of getting into easy top 10 ! but never have that many words in your titles .. never make a mixed salade having ALL in one and ALL in ALL pages ! focus on 2-3 keywords in main title and main description ... my recommendation JUST for fun and educational purpose make 3 NEW pages -. just for fun - but REAL pages with REAL content - for your SEO leanring process. 1. Science fiction 2. Science fiction and fantasy 3. Fantasy and science fiction ONLY THESE precise words in title - nothing added ... and according to these short titles adapted description meta tag and page titles H1 and sub titles H2 and link text all always with small variations - like an extra added word or 2 that make sense and relate to the linked page and are accurate to the real and true context of that page or chapter or paragraph. full example page 1: <h1>1. page</h1> title: science fiction and that "science fiction" can occurs easily in CORRECT context at least 2 or 3 times in description meta tag YOU are the expert - else you would never that that number of pages . ... right but for example from the little i know about science fiction from early childhood ( and early teenage years ) title science fiction meta: description: "Science fiction - science or fiction or science disguised as fiction and fantasy - works and comments about popular science fiction authors and published science fiction works." meta: keywords according to real keywords on that page .. including UNIQUE special words or names form that page ( is only used by few SE nowadays .. - as far as i know neither by Y nor by G ) page title H1: <h1>science fiction</h1> page sub-titles H2: <h2>science fiction and fantasy </h2> <h1>Definition of</h1> <h2>Science</h2> <h2>Fiction</h2> <h2>Fantasy</h2> <h1>"Science fiction and fantasy" works</h2> <h1>"Science fiction and fantasy" authors</h2> <h2>"Science fiction" <i>science</i> or <i>fiction</i> and only <i>fantasy</i> ?</h2> <h2>Examples of science fiction where fantasy has become true</h2> anchor texts and alt text accordingly have links to "science fiction and fantasy" page and to "fantasy and science fiction" page and to "fantasy" page and to "science" page and to "fiction" page and to "definition of science fiction" page and to "definition of fantasy" page CREATE above pages if missing. with exact procedure as for page 1 - there will be facts and content you have ME - i am only an amateur - many decades ago a little "Jules Verne" wasn't there something that became true form what he wrote ? that can combine science - fantasy and fiction ? you may also include a link to collection of Jules Verne - popular "author of science fiction" "quoted" text being anchor text of link ... remember - its just a recommendation - a learnign game to aim at a top 10 page you have nothing to lose - but all to win ! just for fun - it takes but a day or so . make sure Google finds the page within a day or 2 by having a link form the home page and 1 or 2 other frequently visited ( news) pages on your site.) and if you can NOT get #1 page - then only for ONE single factor - your domain name has neither science NOR fiction in it ! i think you can get a # page in G - because the others are far form SEO top optimized smaller pages are easier to get in top 10 or all the way on top .. ( my experience ) and then link to OTHER relevant pages on your site with correct topic relevant anchor texts good luck
Hi, sorry I'm not an expert in SF. With competion of phrase / words I ment word or phrase competition as such not sites. E G 10 sites using SF once each would give you 10 competing all things equal this would force a sequence by PR age etc. If the same sites would use it twice each you have 20 competing. (Which gave us keyword stuffing). Now by using dictonaries and thesaurus it is easy to come up with synonyms describing this without using the actual phrase and by the same token allowing to ringfence themed or authoriative sites. Word combinations like (they are a bit over the top) ...futuristic novels about worlds and societies .... ...theoretical exploration of the parallel universe a novel ... But they are associative to sience fiction fantasy and when G (respectively Applied Semantics) strips out the actual search phrase these would give a pretty good indication what a page or site is about. That is one reason why there is such a vast difference between exact match "sience fiction & fantasy" and sience fiction fantasy just my .02$ M
Hans advice is excellent, but all you guys continue to concentrate on on-page issues. The real answer is to take Han's advice about the number of words in the title and apply it to the words in you anchor text. To quote Hans you anchor text is also "a mixed salade". One more time! It is links that get you ranked for popular search terms -- not on-page content. According to Hans keyword research your prime search terms are popular. Get more links and use only one or the other search terms as anchor text. No more "mixed salade".
If it only would be so easy - all is relative adoption agency london brings up I-A-S.org.uk p1 no H1 no standard SEO (no black hat either) OK just 373,000 competing adoption agency UK P3 - 850.000 design is debatable but the text is not. Links, yes some very careful selected and certainly none of the reciprocal nonsense. M
a quick look at what Google knows about your site's pages shows your page titles are: Great Science Fiction and Fantasy, Buy Links: "Wildeblood's Empire ... Great Science Fiction and Fantasy, Buy Links: "Modern English ... Great Science Fiction and Fantasy, Buy Links: "What Entropy Means ... Great Science Fiction and Fantasy, Buy Links: "Deliverers of Their ... i have looked at some 253 of your page titles ( thats what Google found from your site when i checked .. i did NOT have a deep look into code and LINK details - that takes days of work BUT W3C validator says http://validator.w3.org/check?uri=http://greatsfandf.com/editions.shtml This page is not Valid XHTML 1.0 Transitional! to me it looks very much like HTML4.01 - but you may have a look yourself why that matters ? bots crawl CLEAN stuff much easier - if html tags are screw up - then bot is messed up OR confused and gives up before crawling/indexing a page completely and rank goes down or pages are dumped ... and all your pages .. they all start like above withthe same title part BEFORE the actual individual page title starts ... i think to remember that i wrote about some "mathematics" of number of words in titles related to actual used key words .. you lose at least some 50% of all relevancy in all searches by any searcher ! just a rough guess - may be more %. why ? approx 50% of title volume of ALL pages have identical part of title ... people search OTHER than that full query the may search by author OR by title such as to repeat one and same string in hundreds of pages does NO GOOD to your domains reputation at all IF serving large number of different persons, if answering or meeting request of large number of different queries is your true goal then you get MUCH better visitor numbers if you just remove on ALL pages the "Great Science Fiction and Fantasy, Buy Links: " - part of each page title and keep the rest of title if you ad the Buy book or Buy online - in the first sentence of your description tag - then you have a highly competitive title and rank probably easy in top ten @ Yahoo and Google for each or most of your titles. one by one. and after full make up of each page ! 11'000 pages have fun and you may get MULTIPLE of visitors .. for your site .. example page "Peter Pan in Kensington Gardens" i have a version how it could look like http://www.kriyayoga.com/science_fiction_fantasy/Peter_Pan_in_Kensington_Gardens.html its a demo page and will be removed after a few days. of course style, colors, etc are my style and need to be adapted to your site - but keywords etc are most likely to give far better Google and Yahoo ranks this way. i am a layman in your business and have zero knowledge about your job. it was just a little game half night long ( its 3:45 AM now ). whatever you really want to achieve you can do - and every year you can do better because SE are better and skills and expertise and experience grow proportionally with work. if you want to improve ranking in G and Y then you can easily - week after week. page by page. if that is important to you for your family, business or success .. you only have to compete with some 1.5 or so Miollion other web sites - many of them peresonal pages others large companies - my experience, the larger the company the less optimized their site because they love to have ALL their company title in every page or phrase ..
Hans and others have raised several points, and I thank everybody. Let me remark on a few of those points. First, <h1> content:: I had assumed--foolishly, I suppose--that Google would either use the alt= text for an image used as an h1 header, or else default to using the h2 text as the "topmost"; but I have now changed things so that every page has actual h1 text. At present, all the pages have the same h1 text--is that likely to be a problem? (It is a generic header containing my keywords in a valid and useful heading.) Second: front-page length. With a certain amount of trepidation, I have split off much of the original content of the front page to a second page, to which the front page points in large text at its bottom. Sentences and headers with my keywords will now be a somewhat higher percentage of the page content, and the page is materially shorter. I am unsure why a longer front page is likely to Lower Google's opinion of the site, but I have followed the recommendation. Third: title length. I do not understand why this is a problem. Each site page has a title of the form "Name of the Site: Description of the Page". As it happens ("happens" because I was ignorant of SEO when I titled the site), the site name is essentially my keywords: "Great Science-Fiction & Fantasy Works", where I consider "science fiction" and "fantasy" to be those keywords. But I like to see that information displayed to visitors in their browser title bar, and none of it is so long that it doesn't fit on my browser bar (full screen at 800 x 600). What is the downside of page titles of that form? As to general page structure and titling: while I would love, like anyone who runs a site, to rank high in SERP, I am not a commercial site. I am very wary of letting the tail, SEO, wag the dog--actual site content and layout. One wants to do the simple things to maximize SERP, but I am very reluctant to construct pages whose prime function is to be search-engine chum rather than useful content for my visitors. Some things can be done for SEO that do not conflict with content optimization, but some SEO considerations would--in my opinion--too drastically modify what I want people to see and read. As to what Google knows of my pages: I have changed the titling of the many book pages so that "Buy Links" is now "Edition Data", which is probably more accurate anyway. The site organization is somewhat like a solar system: the "sun" is a master list of recommended authors and certain books by each of those authors; the "planets" are individual pages about each author, which do--or eventually will--not only repeat the entries from the master list but also include an essay on that author; the "moons" that circle each "planet" are pages about the individual books recommended on each author "planet" page; and there are also "moonlets" that circle each "moon", being pages created on the fly by php (but aliased by mod_rewrite to fixed URLs), one per particular edition of each book-page "moon". (There is also a sizeable "asteroid belt" of pages of general information on the subjects of the site.) The point of that explanation is numbers: there is 1 "sun" page, 208 planet" pages", 1,279 "moon" pages", and--an estimate--9,689 "moonlet" pages, plus 71 "asteroids". Since I switched to aliasing php not very long ago, I'd reckon that Google is far from having found all my pages yet (Shawn's Keyword tool tells me of 5,080 pages). So the bulk of pages found will be moons and moonlets; but the heart of the site is the sun and planets, and it is those to which outside links point. The bad-xhtml page particularly burns me, because I take a lot of trouble to be sure all my pages are 100% compliant--indeed, each page has a "test me" w3c click-on image link to the w3c validator. The thing I can never seem to remember is that there is NO such thing as a "trivial" page change, and so i forget to re-check after such: the "88" errors the validator reported were two omitted anchor closing tags, missed because the links were so long that their ends ran off my screen, so that I forgot that they weren't closed. Needless to say, that's fixed. But so far as I know, and I have check all the non-moon/non-moonlet pages individually, and the rest by sampling, every one of those 11,248 pages ought to be 100% xhtml 1.0 Transitional compliant. I have downloaded the sample page Hans kindly made, and will spend some time looking it over in the light of his comments. As those 1,279 pages are re-made nightly, it would be easy enough to change them all in a standard way. But it should be remembered that I would be much surprised if anyone came into the site through a search that landed them on a "moon" (book) or "moonlet" (edition) page. My feeling is that people searching for what I offer will either come in on the chief keywords--"scienc fiction" and/or "fantasy"--or on the name of a particular "planet"-page author. An aside on links: I would love to have more ranking inbound links. The problem is not, I think I can say without vanity, the quality of the site--I have gotten good reviews from some people who count in the field, including some notable authors--it is that the vast majority of sites in my niche are amateur, not professional, and emails to webmasters might as well be sent to the bit bucket. There are many that invite link requests to which I have submitted, and re-submitted after some months, with no reply whatever; several others have eagerly promised links and done nothing weeks or months later. The site that invariably comes up #1 on my keywords is, according to Alexa--a blunt but by no means useless tool--#2,500,000 on the web, whereas the sites right behind it are in the #65,000 range; but that #1 site has a vast number of inlinks owing to, I would say (without naming the site) "political correctness", and hasn't updated its links list in a year or two (despite a whining plea for linkers to please come forward). Sigh . . . . Anyway, thanks again to everyone, and especially Hans, for the helpful advice. Oh, another thing that might help: I finally managed to get mod_rewrite to send 301s to all requests for any site page not in canonical form: I recently had a "webmaster" whom I asked to fix their link to eliminate the leading "www." email-lecture me about how not including such a leading "www." was a breach of some unnamed protocol which some browsers "repair"; when I responded, his "big brother" apologized for that nonsense, but many webmasters, amateur and, it seems, professional, don't seem to get the SE importance of linking to the URL as given by the linked site, or of updating it when asked. Come to think of it, does Google treat a link to a 301-redirect as a link to the new target? If Site X links to http://www.greatsfandf.com and gets a 301 to http://greatsfandf.com, will it "count" that link toward the correct page? (I see Google carrying two links to a single page of mine, one with and one without the leading www.)
you have an excellent site in a high competition business and you are also competing against amazon i guess in the ranks now about the all pages same H1 yes it is a real problem because even if all page-content is different in OTHER content the title itself of <title> and possibly H1 has such a high weight that the individuality of EACH page is lost. let me explain in other words if 100 surfers searching each 1 different title of book that may be found - your site however delivers mostly VERY similar data base entries and the difference between the different books is mostly lost. THE more your pages vary - the GREATER the potential number of surfers you attract because your Google and other database entries vary a lot or in full among all the many 1000 pages you have ... and make you more competitive .. surfers seldom come trough the domain root page - only if they come form directories such as DMOZ or similar that refer to domain root only - but most of the surfers jump directly into the MOST relevant page to their query. the more such individual pages are adapted to the real content the higher within all your competition that one page will rank. one other help to improve the rank on each page among similar competitors pages is to add pics, gifs, jpgs FROM or about the story on that page .. all these pics named correctly accordingly to the title and story of each book with correct alt texts .. its just one additional plus point among many in a high competition field where every point counts re alt text and G i saw that on pages with PLENTY of pictures - almost ONLY pics like my LoveCards - the alt tag is EXTREMELY valuable @ G while on pages with lots of text and content it appears almost no value and far more value is contributed to H1 H1 description meta and <title> plus actual text. i have modified on pure content pages alt text with no visible positive result on better rank while my ecards pages having only a picture gallery lives from alt text and H1 only ! and here we have already another way of getting more customers / visitors out of SE - my ecards are NO pick up but actually delivered. and highly configurable ( and FREE for non-commercial use ) and links can be added to various parts of your site - and pictures, artwork, gif, animated gif, etc FROM science fiction, fairytales, fantasy .. whatever is FREE of copyright could be used as eCard motives ... attracting hundreds of additional visitors each months .. re omitted anchor closing tags i do a code check almost each time i modify even smallest part of pate. ( i use quanta plus as editor ) and hence such things are found. another check i do usually daily before uploading is a full link check on full site - offline version - using the webcheck in file:/// mode. it takes but a few seconds to run trough all the 6'000-10'000 or so of my links - but every once in a while i find errors - specially after renaming a folder or file name. missing anchor tags or missing titles etc are then reported here again it may pay off to have one theme per page .. like peter pan, Jules Verne, etc .. i have seen one other site using same script than i have - a PHARMACY in germany - offering toys for sale online - and adding ecards with the picture of the purchase toy on ecards ... like announcing the gift on the way . and promoting the business at the same time another point for your 11'000 pages if you would add on EACH of the book-title pages a Google ad - then people may find amazing value added ads on your page as well. one example i saw ( i tested it of course was about peter pans title attracts among others a site offering peter pan T-shirts .. - peter pan musical, or i think there also was another one peter pan on video, .. i could imagine that most titles in science fiction when from popular authors - have some kind of fan comunity collecting such items like t-shirts, ... hence you can but win by adding a 2-4 ads G ads on each page .. and you have nicely place there you may give it a try on 100 most frequented individual book-title pages and see .. re links another approach getting high ranking links - specially for you since you have a GLOBAL product with fans all around the world ( check once your access_log for countries ) when i checked my access i found white areas of no surfers accessing in china, arab countries and other geographical locations, when you know these cultures then you understand WHY they make access to www difficult and controlled. intense search for their local/national SEs and entries therein resulted in more than 40 countries being added to my visitors if you have blank areas on the world map .. you may try as well
Owlcroft, Run your information through this tool Keyword Analysis Tool and see what an important role Anchor Text plays in the results. Forget tweeking on-page factors for now and focus on getting more links to your site with the exact keyword phrase you are targeting in the anchor text. Caryl
I just went and did, on Google, adoption agency london took the number one http://www.i-a-s.org.uk/ put it into the mcdar test machine using the above and look what it gave ONLY http://www.i-a-s.org.uk/ had an anchor link back which supports compar
A few quick thoughts. 1) Allinanchor seems, from browsing recent net information, to be of rapidly fading significance (GoogleGuy, for one, suggests as much), which many seem to feel is A Good Thing, inasmuch as it is hard to control what the other guy will use as link text, especially if the other guy is not a professional webmaster. 2) I do already have an automated overnight check of every link on every page; I also have an on-demand tool for checking the continued presence of backlinks. What I don't have yet, but will add very soon now, is an automated overnight w3c Validator check of every page, so that sleepy-headed dumb mistakes made when I make "trivial" page changes that I forget to re-validate will get caught for me. I know I am being obtuse, but I still don't get the matter of title text. I did a run on the book title Hans used for the sample page he kindly made up for me, "Peter Pan in Kensington Gardens", and I must say I was surprised almost to shock that my page came up in the top 20 (#18, in fact). There are two things about that: one is that even for so popular a book (though not as much so as "Peter Pan" itself), my page, with its full title <Great Science Fiction and Fantasy, Buy Links: "Peter Pan in Kensington Gardens"> (which, at the next cache update, will become <Great Science Fiction and Fantasy, Edition Data: "Peter Pan in Kensington Gardens">) still placed pretty high even though not SE-optimized. That last relates to the second point, which is that I have never thought of those book pages as ways anyone would find or come into my site--only as pages one would go to once already in the site. Perhaps I need to re-think that, and optimize those pages. My emphasis has been on front-door entry for visitors having heard of the site somewhere, and on entry to the individual by-author pages; for some of the less-famous authors I list, my pages--to my constant surprise--come up in the top few hits, even with few or no backlinks.
I was about to start a new thread on the freshbot and the significance of the title but I will put it here in a brief form as I think it may answer some thoughts. About 1.5 months ago I placed a new site on the Thursday and it came on line DNS wise on the Friday and I just put down the first page thinking I will place the others soon. By coincidence on the following Monday I saw a Keyword phrase, in the stats on the site, that I was just initiating, that made me sit up, go and check on Google, and scramble to put some pages down quickly as Google had been there and taken stock and Serp'd the page as Number 1 for that title/H1/text in just 4 days max. Now I was simply amazed at the speed and there is a salutory lesson about placing a whole "site" down when you decide to go - but that is not what this is about in total, which is, that what was first placed there has determined the overall context of what the site was/is about as by the next day I had placed down 12 pages all interlinked with keywords and crosslinked from other similar context sites with a broad range of keywords that were relevant and yet only now am I seeing some new keyword positions and those are tightly related to the first Title/H1/text and of course the original anchor link text that google used to find the site. So far I have not seen the other pages in any way as they all have different keyword uses as different pages of entry - which brings me to your scenario Owlcroft of not using other pages as entry pages. I have always regarded EVERY page as an entry page [even frames! with some amazing successes!] as some pages with relevant content can act as "open door" pages to that which you are trying to sell. For example, recently, Austin not Florida, I got hammered by Google, but the pages that survived were the pages about towns where we had holiday rental properties and it was from there that we continued to do business until the others recovered which I assume was because of the links from these pages, but also other pages with little content can sometimes give amazing results and can help you link navigationally around the site continuing the theme. So to sum up, and I know that this may not be rocket science, and it is something that we all "know", but it is the first time I have actually 'proved' the workings: Freshbot determines from the index page the overall "category" of the site which is "stable" and the other bots place all the other pages in their different sub categories which is how we get these other points of entry.
Again may be slightly of target but not only do I agree with FW. We are using the page concept to it's max on a travel related site. This site has an index and site map but we do not expect any visitors to go there as it's much to unfocussed. We wish our visitors to find a page that precisely matches their query - if the query is precise enough or a sub hub page if it's not. We provide a large amount of precise informative pages (like a map to the actual airport not a map to the hotel on the airport one sells) here we take the chance of "advertising" and introducing the rest of the site. We also provide (and shawns site is a brilliant example) some tools for travellers, thus we attract natural linking and bookmarking which also allows us to gain entrance into highly informative directories or sites that normally frown on just commercially driven content sites. Specifically in congested areas a lot of ASP's (alternative supplier pages - i don't do competition) are focussed on their commerce aims and thus leave a large void to attract visitors by providing unbiased answers. Again an example I just checked is "Bristol airport by train" (G.com) just one page / site that answers the question (Y is much better). That page (sadly not mine) will get a lot of traffic and advertising opportunity. Thus I hope G is not or will stop classify a site by its index page rather by clusters derived from site maps, interlinkage and overall content. Hope this helps in any way M