semi-disclaimer: I was totally never quite sure what the boundries were between threads that go in here vs seo site reviews. if this isn't appropriate, I apologize and feel free to move it. my logic was that I'm mostly interested in the theory behind it and I'm not even showing an actual site's url. -- condensed version: I go into detail below about a specific example. but ultimately the question is this: the fewer the links on a page, the more value they pass along. if some links are disallowed via robots.txt, how does that affect the value of the links? if you have 25 links and 20 are blocked by robots.txt, is it treated the same as having a page with 5 links? -- I hesitate to bring up the concept of PR bleeding here because I have a feeling a lot of people will take it the wrong way. shrug. read this before you flip out and start removing links. for those of you unfamiliar: the concept is essentially that the more links on a page, the less PR / link popularity / strength is passed on to each link. I'm just concerned with internal links. structure is like this mainpage.php -> mainpage.php/x/ -> mainpage.php/x/y/ mainpage.php links to ~10,000 mainpage.php/x/ pages. not all on one page, there's mainpage.php?page=2, obviously. each mainpage.php/x/ page links to 20 mainpage.php/x/y/ pages on average. sometimes 5, sometimes 200, but ~20 is by far the norm. I want mainpage.php/x/ to rank, but I don't realy care about mainpage.php/x/y/ pages at this point-- because NOTHING is ranking despite an absurd number of incoming links and the site being on an old, trusted domain. it won't even rank for its obscure site name. 10,000 x 20 = 200,000 pages I don't really need in google, which I'm assuming may be diluting my rankings and hurting me. if I block all of the /y/ pages via robots.txt, would that be enough to stop the bleeding of link strength? do I need to take more drastic measures? I have no intentions of using nofollow, so don't even mention it it's not what it was meant for. can you think of anything else that may be hurting this site as much as it's being hurt? I have a few other ideas, but curious what other people think.
If all those thousands of internal pages are linking back to mainpage.php then, in theory, all you PR bleeding eventually flows right back to the source. Thus a null effect. Maybe??
not quite that simple. the main thing is, by the time it gets down to the mainpage.php/x/y/ pages, it's likely diluted enough that no links are followed from that page. thus I'm passing a miniscule amount to these pages, but because of the volume of them (since there are 200,000+ pages) all of that miniscule link strength that's passed to them really adds up and doesn't do me any good. the pages don't have enough strength to rank, nor do they have enough strength to pass internal link strength back at the main site / page.
i think either blocking the /x/y/ directory via robots.txt, or adding the robots meta tag (with noindex) would stop it from bleeding. i dont think you have many other choices, apart from nofollow.
If those deeper pages don't offer unique enough content (compared to all the other pages) then they may get placed in supplemental results. A dynamic site that generates that many pages likely does no make them with enough unique text content. If thats the case then go ahead and disallow the bots. You'll have to trade off the potential that all those pages could possibly be pulling in more variations of search queries (if the content is unique) vs. the potential loss of weight to your main pages. Long tail vs. short tail
I'm not of the opinion nofollow works that way, from my experience. added it to robots.txt already, but crux of what I'm questioning is this: if a page links to 25 pages, but 20 of them are blocked by robots.txt (or no index), is that the same as having a page with 5 links? does anyone have any evidence-- concrete or otherwise-- that may indicate how it'd be treated? some are in the supplementals, most aren't. despite that though, they never rank for anything. at all. not the main site, nor main.php, nor main.php/x/, nor main.php/x/y/. the traffic coming into /x/y/ pages is the rarest by far though. I have no qualms with blocking /x/y/, but I'd like to know if that'd actually stop the bleeding.
nofollow tags may be the only thing that stops the bleeding. robots.txt just tells the robots to stay out. Links to those pages may still pass PR but passes them into the abyss of a page a robot is not allowed to crawl. Simply using a rel="nofollow" in the a href tag stops the PR bleed. But pages are still allowed to be indexed in case somebody else decides to link to one of your pages. robots.txt may prevent any benefits of external links to those pages.
what evidence do you have that nofollow stops the bleed? google'd have no reason to do this-- it'd encourage webmasters to use nofollow on every external link regardless of the reasoning behind it because it'd keep their "link strength" flowing only within their own site.
evidence?? That is the purpose of a nofollow tag. Tells a bot to "stop right there, do not follow this link. There is nothing for you on the other side". I use them to block the bots from entering pages like "privacy policy", "login" etc. Anything i don't need indexed. If you have "SEO for Firefox" extension, turn it on and view this forum. It highlights nofollow links in red. You will see that Shawn (or the VB programmers) has selectively chosen which links have a nofollow and which do not. As for webmasters doing it to not allow PR to pass to external sites, well it happens. You just don't purchase links, or trade links with those that do. I always check for this, and redirects, when buying links, or submitting to directories, etc. Google even recommends using nofollow on links you sell so as to discourage the business of buying PR. But who is going to listen to that? It like asking oil companies to voluntarily adhere to environmental regulations.
that is not the purpose. it was designed for a very specific reason-- to flag user submitted content. it also does not say "do not follow this link." it says "do not use this link to pass link popularity credit." if a page is linked to with nofollow, and no other page in the world links to it, yahoo, msn, AND google will index the page. msn and yahoo even show them in the backlinks count. Shawn didn't enable most of the nofollows on VB; they're built in and VB has very poor SEO, honestly. I went into a rant about it on their forums some people agreed, others didn't. but that was mostly out of ignorance. people seem to think nofollow means, like you suggested, "don't follow this link." that's not what it means, which is why it's used very poorly in vbulletin. ironically, vbulletin doesn't even use nofollow in the way it was created for. not all link development involves trades or purchases. the far far majority of linking is natural. the only comment ever made by by "google" that I'm aware of is matt cutts saying he would personally use nofollow when buying text links to mark it as not being an "editorial vote." extend that further. some nofollows make sense. but if a site goes overboard, why would google want to reward that? not voting isn't a good thing. the people that praise excessive nofollow use are generally the same people who once claimed it was best, in terms of seo, to never link to anything externally, ever. that's obviously bunk. blogs link more than just about anything in the world. look at how blogs do in google. I don't feel they're punishing nofollow but I also don't believe you'll benefit from it. and I do certainly believe that excessive, inappropriate use will eventually "flag" you in google's algos. they'll know you're trying to manipulate things.
well there is your answer to how to stop your PR bleed. Like i said, you can still allow your pages to get indexed if another site links to a page, but your own PR will not bleed away to those pages. robots.txt can be used to keep the pages from being indexed, if thats what you want This from http://blog.searchenginewatch.com/blog/050118-204728 Sure, it was originally devised for blog comment spam, but it has other uses.
nofollow links not passing PR / link popularity does not neccessarily mean it doesn't count as a link when it comes to the value being passed; again, for the reasons stated above, there's no reason for google to reward this behavior. which is essentially what that would be doing. just because someone said it on SEW doesn't mean it's gospel. I'm hardly the only one that has qualms with that SEW post, by the way: http://dallas-seo.blogspot.com/2006/11/nofollow-does-it-really-work-like.html
hmmm. I've officially change my tune on nofollow. I think. Need to rework how i mask my affiliate links.
Can anyone tell me how robots.txt can tell bots not to follow/index a link? Anyone have a example of this?
put it in something they can't access (javascript, etc), the meta tag for it, or robots.txt if it's on your domain.