AdSense states that it can only serve PSAs for pages it has never crawled. It also states that running AdSense on a page will not in and of itself get that page crawled. Consider a fairly new site with a huge dynamic-page count (all mod-rewritten to static). Its front page has links from a PR6, three PR5s, and a couple of PR4s (all with only a few outbound links). That front page links to about six dozen pages, each of which links to a hundred sub-pages, each of which links to a hundred content pages (yes, over half a million pages). (All pages link only to their subpages and back to the index page.) Google suggests that uncrawled pages be manually submitted. Right: all half million. I don't theenk so. How best to proceed to expedite crawling of the topic pages?
I guess you could find some sort of bot to go to every link on your site, but they'd also need to trigger the javascript. if you find one that does this, let me know.
That is the way that AdSense used to work. That is not the way that AdSense currently works. When I add a new page now, AdSense immediately displays an ad which is relevant to my web site. I added over 1,100 pages yesterday. All of them that I tested popped up site-appropriate AdSense ads automatically, first time. And yes, this happened before the Mediapartners bot visited. How does it really work internally? I have no idea... This won't help, of course, if each of your pages is unrelated to each of your other pages.
Do you mean manually submit to the Google search engine? All you need to do is submit the root URL e.g. site.com, the Google bot will follow the links on the pages from there, the speed of spidering is normally determined by PR.
.Just so. As the page count suggests, the site is essentially an encyclopedia, so indeed each page will have a very different theme/topic. Again, just so. Google already has the index page, and the PR is about all I can personally give it. But I suspect that to be enticed out onto a half million pages will take more PR firepower than I can immediately bring to bear. I had thought to perhaps engage a listing in, for example BluePoint, but am unsure if a PR 7 link is worth $50, having never before even considered a link purchase. If I just leave it as is, I have this dreadful fear that the spidering will be completed sometime in A.D. 2136.
So it seems. I was clicking a few random pages of the massive whole, and seeing mainly PSAs, but, by gum, here and there I do see a few relevant ads. Most pages still produce, even on multiple loadings, only PSAs--I suppose those are the ones where it is harder to immediately determine relevance (though I wouldn't have thought so from their content). By the way, for those who were wondering on another thread: yes, the ads tend strongly to be themed to the immediately preceding text, rather than the page as a whole: a page on a baseball site that is of some length and quite general about baseball (which one would thus expect to be the key) contained a one-shot mention of the Red Sox in the last short paragraph above the ads, and sure enough two of the four were Red-Sox-specific ads. But still, if anyone has any suggestions beyond "get PR for the index page", I would be grateful.
How often is the site getting crawled? Would a rolling program of linking direct from the front page to each batch of sub-pages (the ones I highlighted in Bold above) help the situation. The Bot might then come back directly to the sub-pages and recommence from there. If the front page got indexed every day, then by swapping the list of sub-pages on the front page daily, you might have all the sub-pages indexes with 72 days. And, if the front page was changing daily, I imagine the Bot would be back daily.
The front page is rebuilt daily. It's actual content may or may not change, but there is a "last updated" line that will, of course, be changed text from day to day. The front page does seem to be getting a daily crawl. The site is very new, but so it seems so far. But it has not yet cached even the first of the 56 subpages . Well, we will see what time brings. (I'll keep that swap rotation in mind, but the hierarchical structure suggests that if the 56 pages actually linked direct from the front page aren't yet crawled, there's no rush to move on to the next layer.)
I have noticed that when I add new pages with AdSense, that mediabot is there within a minute or two crawling the page. This could be why you are seeing relevant info.
Hmmm... this could be extremely useful information. Check out this site where AdSense picks up the anchor text from a link directly after the AdSense box: Dictator For Hire How is that for relevancy?
Yes, I supose that is the more general form of the statement--AdSense looks immediately *around* the ad-placement spot to determine relevance. Well, well, well. Must remember to beware those parenthetical remarks about Zippy the Pinhead . . . .
Do you mean manually submit to the Google search engine? All you need to do is submit the root URL e.g. site.com, the Google bot will follow the links on the pages from there, the speed of spidering is normally determined by PR.