Matt Cutts Q&A on Big Daddy and other issues

[minstrel]

Excerpts form Matt Cutts' Q&A on Big Daddy and other issues
March 27, 2006

Q: “Is Bigdaddy fully deployed?”
A: Yes, I believe every data center now has the Bigdaddy upgrade in software infrastructure, as of this weekend.

Q: “What’s the story on the Mozilla Googlebot? Is that what Bigdaddy sends out?”
A: Yes, I believe so. You will probably see less crawling by the older Googlebot, which has a User-Agent of “Googlebot/2.1 (+http://www.google.com/bot.html)”. I believe crawling from the Bigdaddy infrastructure has a new User-Agent, which is “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

Q: “Any new word on sites that were showing more supplemental results?”
A: An additional crawling change to show more sites from those sites was checked in late last week, but it may still take a little bit of time (another few days) for that to show up in the index. I’ll keep an eye on sites that people have given as examples to see how those sites are showing up.

Q: “Is the RK parameter turned off, or should we expect to see it again?”
A: I wouldn’t expect to see the RK parameter have a non-zero value again.

Q: “What’s an RK parameter?”
A: It’s a parameter that you could see in a Google toolbar query. Some people outside of Google had speculated that it was live PageRank, that PageRank differed between Bigdaddy and the older infrastructure, etc.

Q: “Now that Bigdaddy is out, will there be a new export of PageRank anytime soon?” and “Will the deployment of BigDaddy stabilise the rolling PR issues we are experiencing at present?”
A: I’ll ask around about that. If there aren’t any logistical obstacles, I’ll ask if we could make a new set of PageRanks visible within the next couple weeks. I’d expect that as Bigdaddy stabilizes everywhere, the variation in toolbar PR for individual urls is more like to settle down too.

Q: “This datacentre http://64.233.185.104/ works differently to all of the others. Noticed just a few hours ago. . . . . Where does that DC fit into the scheme of things? Is it mainly made from newly spidered data?”
A: Sharp eyes, g1smd. That wouldn’t surprise me. As Bigdaddy cools down, that frees us up to do new/other things.

Q: “My sitemap has about 1350 urls in it. . . . . its been around for 2+ years, but I cannot seem to get all the pages indexed. Am I missing something here?”
A: One of the classic crawling strategies that Google has used is the amount of PageRank on your pages. So just because your site has been around for a couple years (or that you submit a sitemap), that doesn’t mean that we’ll automatically crawl every page on your site. In general, getting good quality links would probably help us know to crawl your site more deeply. You might also want to look at the remaining unindexed urls; do they have a ton of parameters (we typically prefer urls with 1-2 parameters)? Is there a robots.txt? Is it possible to reach the unindexed urls easily by following static text links (no Flash, JavaScript, AJAX, cookies, frames, etc. in the way)? That’s what I would recommend looking at.

Q: “When I change a robots.txt to exclude more existing files from being crawled, how long does it take for them to be removed from the index? Perhaps the answer is a function of how often the site is crawled and it’s PR?”
A: It is a function of how often the site is crawled. I believe in the past that every several hundred page fetches or several days, the bot would re-check the robots.txt. Note that for supplemental results, you need recrawling to happen by the supplemental Googlebot in order for the robots.txt file to take affect on those pages. If you’re really sure you never want those pages to be seen, you can use our url removal tool to remove urls for six months at a time. But I’d be very careful with the url removal tool unless you’re an expert. If you make a mistake and (for example) remove your entire site, that’s your responsibility. Google can sometimes clear out self-removals, but we don’t guarantee it.

Q: “If one were to offer to sell space on their site (or consider purchasing it on another), would it be a good idea to offer to add a NOFOLLOW tag so to generate the traffic from the advertisement, but not have the appearence of artificial PR manipulation through purchasing of links?”
A: Yes, if you sell links, you should mark them with the nofollow tag. Not doing so can affect your reputation in Google.

Q: “On sites directed to international audiences with the same (high quality) content in several languages is it better to do several TLDs like mydomain.com, mydomain.de, mydomain.fr, mydomain.eu and so on or do subdomains like en.mydomain.eu, de.mydomain.eu, fr.mydomain.eu or something else like mydomain.com/en, mydomain.com/de, mydomain.com/fr?”
A: Good question. If you’ve only got a small number of pages, I might start out with subdomains, e.g. de.mydomain.eu or de.mydomain.com. Once you develop a substantial presence or number of pages in each language, that’s where it often makes sense to start developing separate domains.

Q: “What about the problem of directories and shopping comparison spam overriding real pages?”
A: Fair feedback. I heard that recently from a Googler, too. Sometimes we think of spam as strictly things like hidden text, cloaking, etc. But users think of spam as noise: things that they don’t want. If they’re trying to get information, fix a problem, read reviews, etc., then sites that like aren’t as helpful.

Q: “The one thing that seems to be getting to people generally, is what are the post Big Daddy intentions? Fixes, spam issues, regeneration of ‘pure’ indices, supp. issues, PR and BL update, etc.”
A: I can’t give a timeline (e.g. “scaling up communication in April, more work on canonicalization in May”) because priorities can change, esp. depending on machine issues, deployments of new binaries, webspam developments, etc. Short-term, I wouldn’t be surprised to see some refreshing in supplemental results relatively soon, and potentially different PageRanks visible in the next couple weeks.

Q: “Even Matt is afraid to use a redirect from www.mattcutts.com/ to www.mattcutts.com/blog/ because Google might penalize his website and put it into supplemental hell.”
A: Heh. No, that’s not it. I’m deliberately leaving them separate as a test case to see how we do now and down the road.

[Nintendo]

Nice FAQ!!!

Originally Posted by minstrel

Q: “Now that Bigdaddy is out, will there be a new export of PageRank anytime soon?” and “Will the deployment of BigDaddy stabilise the rolling PR issues we are experiencing at present?”
A: I’ll ask around about that. If there aren’t any logistical obstacles, I’ll ask if we could make a new set of PageRanks visible within the next couple weeks. I’d expect that as Bigdaddy stabilizes everywhere, the variation in toolbar PR for individual urls is more like to settle down too.

I translate that to meaning a possible completly new PR update might come with in a few weeks!!!

[fsmedia]

Originally Posted by Nintendo

Nice FAQ!!!



I translate that to meaning a possible completly new PR update might come with in a few weeks!!!

amen, I agree to that. Hopefully the updates will come more often so we can get a more accurate image of PR and the real rankings (Since they removed LivePagerank from the datacenters).

[mobilebay]

Good to see it all explained

[Jim4767]

In his March 27 blog comment (http://www.mattcutts.com/blog/) Matt Cutts said:

Quote: "Q: “My sitemap has about 1350 urls in it. . . . . its been around for 2+ years, but I cannot seem to get all the pages indexed. Am I missing something here?”
A: One of the classic crawling strategies that Google has used is the amount of PageRank on your pages. So just because your site has been around for a couple years (or that you submit a sitemap), that doesn’t mean that we’ll automatically crawl every page on your site. In general, getting good quality links would probably help us know to crawl your site more deeply..."

Contrary to the many who say that "Pagerank doesn't matter", Matt pretty strongly suggests that better crawling by Google is at least in part a factor of your pagerank.

[cyanide]

ahhh, so that's why pagerank is important

good post !

[T0PS3O]

But still, it's the chicken and the egg story.

What's important for crawling is having as many good links as you can. Side effect: Get higher PR. So yes, one could argue that PR is important, but I woud say links are important. PR for the sake of PR isn't.

[jrd1mra]

This has been a fact since the inception of PR. Higher page rank pages get crawled much more often.

[Jim4767]

Originally Posted by T0PS3O

But still, it's the chicken and the egg story.

What's important for crawling is having as many good links as you can. Side effect: Get higher PR. So yes, one could argue that PR is important, but I woud say links are important. PR for the sake of PR isn't.

I agree. And I think that's the point of the quote from Matt: "Getting good quality links" is important, because that will raise pagerank, which will induce Google to crawl your site better.

So whether it's cause or effect, chicken or egg, focusing on raising one's pagerank pays off in Google with better crawling.

[mdvaldosta]

Yep, not only do you get crawled deeper - but also more frequently.

[maldives]

Links are more important. Having more links will not only give you better ranking in SERPs but also better chance of high PR.

I feel the most important thing is securing good quality links (the more the better) that are highly related to the site.

[minstrel]

This has been a fact since the inception of PR. Higher page rank pages get crawled much more often.

That's correct.

[T0PS3O]

I'd like to know from Google then whether many lower PR (adding up to the same in total) links get you crawled more frequently/faster/deeper than a few high PR links.

I personally think the bots will drop by more often with more links. And because with more links you can spread them over more internal pages, you will probably get crawled deeper with more lower PR links as well. Just my hunch of course.

And yeah, maldives has a point too, with more links you have a higher chance the total PR will increase more as well.

[cyanide]

This thread serves as a good reminder

Maybe it should be stickied, as alot of people are still confused as to what pr is all about

[Kagemedia]

There has been no mention of external links to deeper directories as a means of getting clawled deeper. Is there any benifit to linking to sub pages from other sites? IE, I have a small tutorials section one level off the main page, does linking directly to the tutorials section from help to get the articles within crawled better? I thought it best practice to have all inward links point to the same page on any particular domain, is that correct?


Newbie question I'm sure but I'm curious none the same.

[Jim4767]

Originally Posted by T0PS3O

I'd like to know from Google then whether many lower PR (adding up to the same in total) links get you crawled more frequently/faster/deeper than a few high PR links.

I personally think the bots will drop by more often with more links. And because with more links you can spread them over more internal pages, you will probably get crawled deeper with more lower PR links as well. Just my hunch of course...

Interesting point. Matt's exact quote was: "One of the classic crawling strategies that Google has used is the amount of PageRank on your pages." I wonder precisely what he means by "the amount ... on your pages" [plural].

[Phynder]

Okay, so my question is... Do you concentrate on getting links to the home page or focus on spreading them around on internal pages?

I know it isn't either or - just curious to hear ideas.

[T0PS3O]

You get links to the pages you want to rank well and where you want the traffic, where the money is for you.

For some sites that means the homepage only, for most of my sites (shops) that means almost none to the homepage but all to individual products or category sections if they're closely themed.

[SEO_AM]

Just a thought..

I think what Matt is speaking of is what goes into the algo that makes a higher page rank not necessarily the PR seen on the toolbar itself. Public PR rankings unto itself does not guarantee higher SERPs for any given keyword, but some part of what goes into the PR algo is likely relevance to the subject matter or theme in the broader sense. Bottomline, IMHO, it is the finite elements of the algo taken into consideration to establish PR that drives most actions by Google; to include frequency of crawls by Gbot (Matt said BLs is one of the main elements for crawls).

[noppid]

I think everyone of you misquoted and misunderstood Matt's point.

He did not say go out and get links on pages with good pr.

What he appears to be saying is that if google is not finding links at other places to your site occuring not only in the past but the current as well, that your site is probably not interesting nor active. Remember the term natural organic links?

In fact, I'm pretty sure it says somewhere buying high PR links diliberatly could backfire?