We all have to be little honest and open up our mind here. History of web directories: As most of us know web directories were initially started over 15 years to organize websites for end users. Those were the days when internet was a baby. It would take $50 to $100 to register a domain and insane cost to develop and maintain a website. The growth was very small and the total number of websites in the planet was handful. I think this was the time when web directories were started to organize world wide web on a category basis just like yellow page. User were easily able to visit these directories and spot the best website in each category. Today's web: But this isn't the way we live today. The web has exploded since then. Today one can register a domain for $2 and host a site in a minute. Total number of web pages indexed by search engines are close to 11.44 Billion pages (Source: http://www.worldwidewebsize.com). Obviously you know that search engines miss a considerable number of pages that are present in the world wide web. Total number of websites in 2008 was around 156 Million (Source: http://www.labnol.org/internet/blogging/the-total-number-of-websites-on-earth/2257/) and with the YOY growth this might have crossed 200 Million by now. Web Directories in today's life: Any one please clarify whether today's web directories are useful for end users or not? Seriously I can challenge you on that. What is the role of web directories in today's user's life? Now I am not talking about SEO, Link Building and stuffs like that. Let's forget that we are webmasters and wear an end user's cap and think. Are the so called "best web directories" useful for end users? Are they helping them to find relevant information they need? Quick Analysis I did a quick analysis. Let's say that I am a student in NC who want to get a loan. The user is badly in need to loan lenders, bankers who offer student loan in North Carolina. Here is what I got from so called "best web directories" for the keyword 'get student loan in NC'. DMOZ Result: None of the results were relevant. Local results were missing and as a user I was not able to get the relevant information. Best of the Web Result: Nothing impressive and relevant to solve the purpose of the user. The Great Yahoo! Directory Result :No comments. Business.com directory was clever enough to serve relevant Adsense ads. The actual problem: Actually problem of achieving relevant information for end users is the size of the internet. Every internet category has millions of websites that it is humanly impossible to review them, analyze them and categories them. Now let us see some useful end user features in web directories. BOTW Online Degree: I love this. It can really help users narrow down and search the universities that they want. But even here I am not fully convinced whether BOTW will be able to find every university and provide relevant information. Even though they might be able to list in this case - Probably there might be 10,000 top tier universities in the world, how can they offer them in a indexed manner based on their reputation? Who will determine which university is best in a particular locality? What algorithm that they can follow? There is no way that the indexed results are based on quality right? Search does this pretty well I guess. Through link quality, online reputation etc., which are a part of its algorithm. Even this has a flip side and one can manipulate it easily. Google and Web Directory: I think most of us would know the relationship between Google and DMOZ. Google directory was pulling out information from DMOZ for a long period of time. Now they have shifted their gear and said they no longer offer something called Directory and they feel "web search" is the best way to find relevant information. That's true. Niche Classifieds I am not sure if this will fall under directory but niche classified websites are the best to find out relevant information that Google fails to offer. Sites that users love and use over and over again are something like this: Jobs - monster.com and other job portals. House rentals - Rent.com, apartments.com etc., Real Estate - realtor.com etc., These sites do a good job. Primarily they are not directories IMO. They are niche classified websites where they run a back office business to collect these information and have a wonderful business and revenue model. Can a general web directory offer such level of deep info from every industry? That is not possible I guess. Social Networking: Now let's see the other facet of directory "type" websites. Social networking or aggregators like reddit.com and Digg.com. They do a good job in highlighting the best articles that are being shared / talked in the internet. They are essentially folks who highlight hot topics in the internet. Can a directory do this as a part of their service? I don't think this is possible too... The working, backend and utilization of these sites are entirely different. What does "Quality web directory" mean for end users? So, after all this I still do not know what is a "quality web directory" mean for an end user? I think users have never (after web 2.0 era) and will never find relevant piece of information from a web directory. I may be wrong. If you feel so, please give accurate information like web directory url and how it can help a user to find information that are relevant and credible. Web Directory and SEO Now if we shift our gear and think from a marketing / SEO perspective, directories can be definitely useful. Submitting to quality (in terms of search engine credibility) web directories can improve search rankings. But this is valid till the day Google say 'No' to this. Who knows what might happen in the future? There is lot of spam happening in the name of web directories. There might come a day where Matt Cutts might host a video and say that submitting to web directories is just like paid links and considered to be black hat. Boom, that's it and this will end the chapter of avalanche of mushroom web directories that we see each day. I think Google is not still making this announcement just because they have endorsed web directories in the past and shifting this gear may cause grudge and howl in the internet. Just like how Page Rank is sticking with Google, the chapter of web directory is sticking with them. That's what I think. Conclusion: As a final note, all I want to conclude is at present web directories are not useful for end users and are only useful for web masters and SEO folks to add a link and boost their rankings. If you think this is wrong, please prove me otherwise by pointing out relevant example. DMOZ's Alexa rank is 656. How many of us really go to find relevant information to solve our day to day problem? I think the traffic is fully generated by web masters and SEO folks who want to get their site listed there. That's it folks. Now open to talk !
Well, I believe that the function of directories have changed from being a cataloger of the WWW to a service offering webmasters and site owners the opportunity to list their websites according to categories. Even that function is not just being offered by a directory, for example there are directories which post articles for their readers (like a webzine), such as how this PR6 directory FOI.net does, or even like how DirJournal does with their multiple niche blogs. Directories, in my opinion, will not go away if the industry reinvent itself as a resource of not only useful web links, but also helpful articles to pull in loyal readers and have a following.
What you have explained is correct but people care to submit to directories to get in search results. Directory submission is one of the many ways of doing this and there is no harm at all. Yes, there importance decreases over the time but stills has a time that business can get benefit from these.
But if directories were to do that, they're not really classified as web directories anymore, are they?
I haven't done the math lately but one of my directories has always ran about 95% visitors and only 5% submitters. You made a big assumption that Google indexes all topics equally. In both of my niches it's often heavily SEO'ed MFA sites that are outranking quality sites. The quality sites are often done with a passion for the topic and their owners know very little about webmastering or SEO. There's so much money in the two topics you mentioned that the top sites probably have people on staff who's sole responsibility is SEO and link building.
To put this in scope: You went through the major general directories looking for something very niche... You could have chosen directories that are focused on that niche (there wouldn't be many), and the outcome would *probably* be different. Everyone knows that people use "search engines" to find things. Why create a blog then? Why social bookmark? Why even have a website at all? No one goes to directories first, the same as no one goes to any of those social bookmark places first, to find anything. I can't ever recall going to Digg, or Stumbleupon, or delicious - or whatever - to find anything. I go to google, if it pops up an interesting article on Digg as a result, then I read it on Digg. Same can be said of directories. They catalogue websites. If I do a search and what is returned is a page on a directory then I go to that directory (happens a lot). People need to stop thinking of directories as a spot for end users to start their day. It doesn't happen. The same as, your blog, your website or your facebook/twitter profile isn't a spot that anyone starts their day. Do we stop creating all those things though because people aren't visiting those places directly? If facebook didn't have a feed, that showed what people were up to, would you even bother visiting half your friends pages ever? But when something interesting pops up that catches your eye... Boom you are there. Google works in the same way for directories and their content... Which is why I'm still prattling on 10 years later about directory owners ensuring their content is quality. It's the only way to get it indexed and ranking. If it aint doing that, then how useful is it?
Michelle, really? That is awesome. Can you please post that url here? If you want anonymity please PM me. I would love to see what made your directory really useful for users. Dan, I agree with you. Love the way you slotted it down. Ok, now let's put aside the topic of directories being useful for end users in such a way that they are gonna start their search there. It's not gonna happen and every one knows that. So we will not rate a directory based on this outdated fact. Coming back to the direction your thought process leads to - Content is very important. I agree with you 100%. When you say content, are you talking about site description? As you know, a site / directory to rank pretty well, it has to have substantial level of content to enable search engines to crawl, index and show them. A site description of 50 words often prove to be less for search engines to think that the content is of any value to end users. Even though they rank well, 50 words often isn't enough for users to get a grip of a site. I am creating some use cases based on your reply. Case 1: Directories with category pages being listed in search Let's take a keyword. Say: Real estate directory In this case, the user who uses this keyword want to know the best real estate company that will help him get the property he wants. Or he might want some great DIY informative website that can help him get the property that he wants to buy or rent. Now to get in search result for such keyword a directory has to satisfy what the user is looking for. You would agree that listing some 10 real estate websites and provide 30 words description isn't gonna help the user. They want more than that, right? Here is what I think will benefit the user: The user would want real estate companies classified based on location. Obviously he would want to get in touch with local company to find the property he wants to. Then the user would want them to be ranked based on value of their service right? I would be happy to get top 10 real estate agencies in a local area - eg: TX. So there must be some sort of rating system. Right now the ratings system that some scripts offer is a BS. User based rating is not going to help and it is often biased. Then there must be enough information about the company. Their values, their services and reviews that are not biased and that are not provided by the site owner. Probably all the above will give some value to the end user. Personally giving 10 links and having some description is by no mean called "quality". At least that is what I think. Please correct me if I am wrong. Case 2: Directories whose inner link details page getting ranked: Say for example the user is searching for a company. Let's say that keyword is "1800flowers" To rank for this keyword the page has to have substantial content right? A listing in a directory details page must give end to end details about this company so that it is truly useful to end user. Mostly in these types of keywords, Wikipedia articles get ranked. That is because they have good amount of detail about that company. Hence a directory listing details page must have enough information (content) in terms of about the company, reviews, what other people talk about this company, how is their product pricing, how is their customer support etc.,etc., I am also thinking some other business cases. In all this case today's directories fall short because they have just 50 word description and it is not enough for the user to benefit or for search engines to rank. I think you will agree with me on this. So to benefit for users and for search engines, directories must have a clear cut content strategy. Today's content is not enough in any form. Now, please tell me if there are any directory out there that is unique in terms of content that is really "beneficial" to end user.
Great post Dan! Sam, in reply to your latest post... Under your Case 1, you are still thinking like a directory owner and not an end user. Other than people looking to submit to a 'real estate directory' chances are good no one looking for a realtor in TX is going to search for that phrase. They might, however, search for 'San Antonio Realtors' or 'list of Realtors in San Antonio.' For a directory to be found by users, you have to research keywords like you would for a content site. I have never understood why directory owners don't get this. Under Case 2...perhaps it's because of the niche I deal with but I've had a number of details pages rank well (even higher than the original site). I don't generally put extra content on the details pages. Some do have additional content but the ones ranking well generally don't. Then you have to consider realistic expectations. For example, let's say your general directory has a section for Mustang cars. The category is 5-6 pages deep and the directory owner is trying to rank for Mustang Automobiles. Do you really think a directory with 200+ categories is going to rank for something so competitive on a page that might have listings for Mustang Collector Clubs, Ford Dealerships, Mustang Repair Specialists, Mustang Replacement Parts Manufacturers, etc.? Now, let's take that same 50-60 listings and group them into those same categories. We'll make a page called Mustang Collector Clubs where every single listing on the page will have a description that talks about the topic. Now you have a page that will attract visitors. Now you have a page that has a possibility of ranking well. Now you are building a directory that people will view as a resource. I've been running a directory and visiting these forums for over 5 years now. It continues to amaze me that directory owners think they are going to get anything other than submission traffic when they keep optimizing their sites for submission-related keywords.
Michelle, valuable contribution, thanks. Pardon my ignorance - I am just 2 months old in the industry and I am figuring out what is there in this industry and how one can do something different. Getting back to your reply. I agree - Perhaps I was driven too much as a directory owner. So, to be useful for an end user a page has to have list of 'companies' in a locality properly categorized, formatted, rated and given with comprehensive information. Learnings for me, thanks Awesome. So, point 2 is that a directory has to be like a banyan tree. With multiple level of categorization and arranging it very carefully so that it is useful for the end user. Generalizing categories sucks and will not attract users or search engines. Great point! I agree. It's perhaps your experience that has given such deep level of understanding. I think people (including me) should focus on end user related keywords, build categories and pages accordingly and start optimizing. In this way the website that is being listed in a directory not only gets link benefits but also gets traffic from targeted and potential buyers. Wow, this thread turned out to be useful for me! Thanks Dan / Michelle.
The question that's bouncing about here is "What is the function of a directory?" Before the emergence of worthwhile Search Engines it was clearcut Directories were a hands-on tool that end users went to in order to drill down and find websites on specific topics. It was a huge list of bookmarks categorized in a hierarchy for ease of use. Being in as many directories as possible was the way web masters insured people could find their website. Not only did users actually USE directories directly, some SEs early on did not have a proprietary database and didnt have the current spider technology. Back then, many SEs were nothing more than a search of the dmoz database. That's why dmoz inclusion THEN could literally guarantee boatloads of real traffic, both direct or SE driven. Given that dmoz proved not to scale well to the growth of the web and its own size, rendering the database less reliable as it aged, it was fortunate that SEs developed their own systems of creating search DBs. That opened phase two for directories, where the function remained the same but the method altered. Directories in the age of the Search engine The direct traffic from ANY directory is a tiny fraction of the traffic they once provided, but SEs still factor the presence of a site in quality human edited directories when deciding where a site comes up in the SERPS. An algorithm can do so much, but a pair of human eyes still beats an algorithm when it comes to separating wheat from chaff, so the guys that write the algorithms use inclusion (in those directories known to refuse spam) as a ranking consideration. As a result, directories perform the same function they did at the start... Helping get sites they list in front of people... but indirectly in the majority of cases. Local directories If you look closely, when searching for a vendor on a search engine, the highest returns tend to be based on the databases of Local directories. It'd be tough to argue that directories are useless if being found by a search is connected in any way to being listed in some directories. University directories This is an example of a directory that, like a local directory, still tends to have dramatically more direct use benefit than either a search engine or a general directory. Why? Well, despite the info in the opening post, it has nothing to do with rankings. Mostly it has to do with providing data on multiple institutions in a formatted template. This enables a student making decisions on where to go to easily perform side-by-side comparisons of factors important to them. The ones that pass their threshold comparisons will then be considered by following the link to the schools personal websites... not by looking at some ranking a webmaster gave it. Is deciding the worth of a vendor the real function of a directory? As someone familiar w/ RE websites I don't think users look to directories to tell them who is the best. They prefer to evaluate that themselves. Most are not looking for an agent at all, they find an agent while searching for a property. I've had a brokers license since the 80s and it was my primary income for 25 years. The typical web surfer hunting real estate would prefer to look at houses, not self-important crap agents write about themselves in third person. The agent is considered a necessary evil at most, at least at the start. I've seen idiot web masters that optimized terms like "realtor in {town name}" when they'd be concentrating on terms that involved properties in {town name} if they understood the buyer at all. So if buyers are not hunting for an agent to begin with, I see little value in a site that claims to rank them. That brings us to the question of whether ranking is really a necessary function of a directory to begin with. Rating vendors or sites is just too subjective (or too rigged) It is not realistic for directory editors to be the arbiter of ranking vendors or websites they add. Maybe it's a function small niche directories could do, but generally Directories are better served helping weed out the junk and get information in front of people to make their own determinations on rank. In general, ranking is subject to being gamed, even if the ranking decision is supposedly left to users. Check the hordes of "host review" sites that are clearly biased in favor of vendors with which the owners are affiliated as an example of where that function can be problematic. Conclusion Getting info in front of the surfer for him to make his own decisions is the function of a directory, and whether directly or indirectly, a function many directories have performed for years and continue to do.
Ahh Rob, the days before Google ever existed. What fun times were those. Almost everyone had a Geocities site and links were passed around like golden nuggets unburied from a stream bed. If only the two of us saw what was coming and had figured out how to capitalize on it, we wouldn't be here having these conversations and would instead be living a life of leisure sipping margaritas somewhere. sigh. Great informative post. p.s. As someone who has worked with Realtors and their sites, it always amazes me how little they continue to understand how to appropriately use the Internet to build their business. Glad to see there's at least one out there that truly gets it.
Lol. The early days of the SEs. In the early days of Google I had the #1 ranked website for the term "conventional loan". Entirely by accident. It was my agent website and I wasn't in the mortgage industry. If I'd known a darned thing then I'd have sold the URL to some corporation with deep pockets and today I'd be on a beach clipping coupons instead of chatting with you guys. That reminds me, still need to repurpose that site... Think it still has stuff from about 5 hrs ago on there. Geeze, the to-do list just keeps getting bigger.
Rob, thanks for the detailed post. Adds clarity to the conversation. Appreciate it. Ah... Those were days when I was in school back home in India. Computers were still new to us and I remember reading about domain registration in newspaper. We were middle class family and didn't have computer in home. Those were days when only rich folks had computers in home! I had that section cut and went to a browsing center to see what was it all about. Unfortunately the cost of domain registration was so high for me as a kid that my pocket money simply couldn't afford to it. I think it was around $35 in register.com and networksolution.com. Today I have hundreds of domains and flip at least handful of them each month. Fate isn't it??!! Had I been in the US or had I been a little aged guy, I may have realized the potential. But who knows... Apparently nothing can be done thinking about past and missed opportunities. Hope we foresee things in the future.