Dropped from dmoz after a year and a half.

Thanks Minstrel - some good insight.

I saw the demoting of the Google Directory to the "More..." pages. I don't think there is any doubt in my mind that most internet users don't use directories, unless they are interested in some specific niche, so that doesn't surprise me - why wade through a load of categories when you can just type in a semi-decent search term and get links to click on?

No, I mean more the stuff you guys are interested in - I don't think you care whether Google has a directory or not - you guys talk about PR and where something comes in the search results, and my understanding is that the data from the ODP is factored in in some semi-arcane way (and again, I don't need another debate about how Google do it - the empirical data seems to indicate a correlation between a listing in the ODP doing at least somewhat better in the Google search results, right?)

If the ODP is really this flakey/corrupt, etc, why aren't Google downgrading this weighting? (Sorry about not knowing the specific terminology you use)

 

That's an excellent question... one that I've asked before as well. The reason DMOZ currently "enjoys" all this attention is because a link from DMOZ isn't just a link from DMOZ... it's also mirrored to numerous other sites using a DMOZ feed. So potentially one link in DMOZ can give you several to dozens to hundreds of backlinks to your site, thereby increasing PageRank for the main page of your site, thereby improving your ranking in Google, at least for the search terms in the DMOZ description.

What puzzles me, though, is that this is basically duplicate content. Why doesn't Google discount it? Or perhaps they do but just haven't publicized it?

Or... maybe it's all folklore and Google is already ignoring DMOZ links...

 

And it MUST be duplicate content, right? Same URL, same link text? Even the same text around it.

You guys aren't getting this concerned about something that is folklore, I'll wager. There has to be something behind it.

 

The thing is, no one but Google knows what the threshold is to trigger the duplicate content filter. Take two pages: if they are 100% identical, simple copies, Google certainly recognizes that. What if you change 1 word? 2 words? 10 words? At what point does Google no longer see the pages as duplicate content?

With respect to the DMOZ mirror/dump sites, if it's a DMOZ feed, then the items are drawn directly from DMOZ so the individual items would be identical. I've never tried to download a DMOZ feed but I assume once you have it you can search it SQL style as a database, so you could end up with a page that doesn't actually appear (isn't an exact copy) anywhere at dmoz.org but does consists solely of items from DMOZ.

There must be some fairly simple way to determine that such pages are merely reconstructed DMOZ pages but whether Google cares enough to work that into a filter isn't known. However, I would think it WOULD be important given that backlinks are the foundation of the Google algorithm. Consequently, I do think at some point Google will want to discount all those duplicate backlinks -- and for all I know maybe they are already doing so, or at least to some extent...

 

If they care enough about the accuracy of their rankings and they really think that the ODP data is really bad, I'm sure they would have done something about it by now, don't you think? They seem to be pretty on the ball, in general.

Is it at all possible that they think that for some reason ODP data is actually useful and that this isn't some oversight on their part?

 

The issue addressed in my last post doesn't have anything to do with whether the DMOZ data is good or bad... my post before the last one addressed that question.

The current issue has to do with the DMOZ directory spawning multiple backlinks from a single directory listing. As I've said a couple of times, i don't know what if anything Google does currently about this problem but it IS a problem because it artifically inflates the "vote" for the webpage all those links point to... and anything that does that gets at the heart and integrity of Google's algorithm.

Even if you believe that DMOZ is the single most important and most valuable site/directory on the planet, it's still a problem for Google.

So this doesn't mean that Google thinks DMOZ is valuable... not at all. I might imply just the opposite in fact, conceivably -- maybe Google thinks so little of the DMOZ entries that they haven't yet cared enough to discount them???

 

But if they so badly (and innaccurately) distort their ranking system, and they seem to keen on tweaking it to "get it right", why wouldn't they address something that seems, to me at least, easy to fix? That's what I don't get. Something doesn't add up for me.

 

That's the $64K question, isn't it.

But do we know that Google isn't already addressing it?

Most web developers and SEO types point to the multitude of DMOZ feeds as the reason for coveting a DMOZ listing. If those DMOZ feed sites weren't "counting" to increase their PR, would they care as much? Definitely not.

If all those DMOZ feed site links DO add to PR, then it's a major distortion of Google SERPs, in my opinion. And I think either it's already been addressed by Google (quietly), or it is (quietly) being addressed currently, or it will need to be addressed in the near future.

But what if this is a myth? What if Google already has applied a filter and a DMOZ listing is only worth a single backlink no matter how often it is mirrored?

That would actually be quite funny, in view of how much effort and anguish and arguing has gone on in places like RZ...

It's late... I'm tired... I apologize if I'm making less and less sense as I go on here tonight... I hope at least the gist is clear...

 

Alucard,

How is it easy to fix? I think it would be at least slightly difficult for google to tell when dmoz data is being mirrored. How do you separate a link pulled from dmoz from a genuine link? If it's exact links in all the same order, that still doesn't tell you it's a dmoz feed, and not just someone choosing to copy dmoz once and not mirror it anymore, or just have the same sites listed in a sequence.

Even if google dropped dmoz, that backlink problem would still exist, at least for a while. If DMOZ was truly interested in being a human edited resource that was meant to find sites in specific categories, why don't they get on the ball and use the nofollow tag instead of wondering why google hasn't done their job for them? (and by do their job, got rid of this backlink/PR nonsense and followed what they CLAIM to be their "mission," we hear so much about it, but it really seems to be a joke.)

I mean, that is the EXACT purpose of the nofollow tag :

To prevent spam. If DMOZ was so concerned with spam as Hutcheson claims, why don't they back it up and use the nofollow tag. You still get the largest, (perhaps best), human edited directory without all the complaints about webmasters spamming for PR and backlinks. Why does DMOZ need to let crawlers follow their outbound links anyway?

Is DMOZ there for spiders or for human users?

 

nddb,

My understanding is the same as the quote - the nofollow flag is intended for blogs and forums. The ODP is neither.

You are saying that the ODP should help Google by somehow blocking the significance of the mirrors? Since the ODP really doesn't concern itself with Page Rank, (That is Google's domain, as I understand it) then if you have a problem with how the Page Rank is calculated, shouldn't you folks be taking it up with Google?

The ODP is a essentially a bunch of volunteers doing the best job they can (and yes, I can see the cynical look on most of your faces) to produce a directory. From the other posts in this forum, there are quite a few other people doing something similar. Some are better than others. If it's really not good enough for Google then they shouldn't use it. Because as we have discussed here, the problem you guys have isn't with the directory, it's that Google puts so much store in what is in it for their PR calculations (directly or indirectly). If Google didn't use it, you wouldn't care about the ODP.

Sorry if I am restating things, here - I just want to try to understand what is going on, once you cut away the cyncism, bitterness and anger (from all sides).

And that brings up a thought I had as a consequence of this discussion here... presumably there is some forum that Google uses to communicate with its users, yes? Somewhere where you can voice your concerns and get them addressed? Or do employees of Google post in forums like this?

 

It seems simple to me. And nofollow is not JUST for blogs and forums. It's a way to keep from passing pagerank to any link. No where does google say it's JUST for blogs in forums. It's useful for them, yes, as it would be useful for ODP. ODP is basically a giant communal blog anyway, "me and my friends like these links." As I understand it, wikipedia has been using it in some places.

The point is a webmaster isn't going to waste time submitting 100s of links to dmoz if they have the nofollow tag. People with feeds could strip out the nofollow tag, but many won't and dmoz won't be passing any PR.

Like I said... why is DMOZ there Alucard?

1) For the webmasters to get PR and for spiders to follow links and give it to them?
-OR-
2) For the human users out there who actually come to DMOZ to find a site? Who don't need to know anything about PR to click a link.

 

That's pretty much it in a nutshell.

There are, of course, problems with DMOZ as well, but without Google no one would really care - they'd just ignore DMOZ and leave it for the editors to with as they wish.

 

An interesting suggestion. I think it would in time correct the Google problem. Of course, it would also be the death knell for DMOZ as anything but a special interest club...

I also see Alucard's point, though. Why should DMOZ do this? It would take some work to alter the existing links (although not as much as some might imagine) and there is really no benefit to them other than that people will stop submitting links, people will stop complaining publicly about DMOZ in forums like this one, and people will pretty much stop applying to be editors. (Wait... maybe there IS a big advantage to DMOZ!).

It IS Google's problem, after all... so Alucard's point that it should be up to them to fix it has some validity.

 

But it's not a question is stripping tags out. The data dump that most downstream users use contains a structured syntax of the content, not the format. If nofollow tags were put on all the ODP pages, they wouldn't get mirrored to any of the downstream data users. My understanding is that the nofollow tag is at the PAGE level and not at the LINK level, yes?

So that might stop a little PR from the ODP site itself, but if, as the rumours seem to indicate, most of the PR is coming from the sites that mirror the ODP data, then it will not serve the purpose you are trying to achieve, I think.

OK, forgive me if I am going to disappoint by quoting some ODP by-line, but you have asked a fundamental question. And I'm assuming that your question isn't rhetorical.

The ODP is there to build a directory that catalogues what the volunteer editors regard as useful sites on the web. Yes, it's their opinion - that can be a blessing or a curse, depending on how you look at it (and it is patently obvious to me how most on here look at that), but that is a part of the fundamental design of the project. That is about as far as the "purpose" of the ODP goes - someone wants to use the directory through the web pages? Fine, go ahead. Someone wants to use the data dumps of the directory? Fine, as long as you adhere to the ODP usage terms for the data.

What is the responsibility of the editors, given the charter? To build what in their opinion is the best catalogue of sites, sorted by topic. Period. No more than that. What others do with that data isn't for us to wonder.

See, that's where I am going with this discussion - if the data generated is garbage, then it's up to the people that use the data to decide. If Google decide that the ODP data is garbage they will drop it. As many of you have said here you don't actually care about the ODP (and rightly so) - you only care because Google have decided to use it to affect the results of searches which directly affects your livelihoods.

I've briefly looked through the Google forum on here - and I couldn't find one word asking about, or complaining about why they use the ODP data. That's why I asked the question - do Google run a forum where you can ask questions? Do Google employees post on any forum that we can get answers to these questions?

[Edit to add]
OK, I looked up the nofollow thing, for my own education - page level and link level. Neither are reflected the data dump from the ODP that all the downstream users get. It doesn't contain HTML, but the URL, Title, Category and Site description. So putting that in the ODP wouldn't solve your issue in the way you hope, I think.

If you'd like to verify my interpretation, check out "rdf dot dmoz dot org" - I think it's pretty well documented there.

 

No.

Google employees read somewhere between a couple and several forums, including this one apparently. An individual reputed to be a Google employee and going by the name of GoogleGuy posts (when he feels like it) and answers questions (the ones he feels like answering) at WebMasterWorld.

 

Interesting. Thanks, Minstrel. So why do I not see comments being made about how closed Google are being, and not listening to your needs? Aren't they a worse ivory tower than the ODP? I mean, they are the ones that can make or break you, right?

 

True.

I actually see this quite a bit, but in fairness Google tend to be informative within the forum circuit.

Sure, I'd love them to tell me how to get a #1 for anything I want, but they just aren't going to do that.

 

I think the major difference is that people (well, I do at least) perceive Google to be an organization than is run in a professional manner with some reason and logic behind their decision-making, rather than being an organization of non-professional ego-feeders engaged in whimsical and arbitrary descision-making that is often self-serving. Whether or not I like everything Google does, I at least have confidence that the goal is to deliver relevant and complete search results and to add new sites to the database as quickly as possible. Additonally, when I have a question, I email Google and get a reply, albeit sometimes a form letter telling me where to look for more information, which doesn't insult me.

Summary of common public perceptions:

• Google conducts itself in a professional manner
• Google is efficient and timely in its actions
• Google doesn't insult people who ask questions
• Google has a goal (even if it isn't always realized) of fairness and objectivity

A list of common public perceptions of DMOZ would not look anything like that one.

 

Boring.

Take it up with Google. Alucard has finally brought reason to this ridiculous "discussion of the ages." Oh, take it up with Google.

- Site not ranking well in Google? Take it up with Google.

-Don't think the ODP should be the basis of the Google Directory? Take it up with Google.

- Think that the ODP is the biggest joke of a Directory? Build a better one.

Move on people. Life is too short.

NOTE: Iconrate and Fryman - Sending a message via the editor links in the categories above the one you were in helps. I have had a site or two removed by competitors over the years and messaging senior editors in categories above where my listing was got the listing back shortly. And, the boring comment was not for you starting this thread.

 

Then what is their response as to why they aren't downgrading the effect of the ODP listings in their rankings? Why are they using this directory for their directory?