History of Search Engines and SEM

Discussion in 'Search Engine Optimization' started by arale, Aug 14, 2007.

  1. #1
    The Spam Era

    1997 – 1999 were the early years of search engine era. SEO mainly consisted of submitting your sites to the search engines. These "voices from the past" are still heard today when this or that SEO company and / or automatic submitting software will claim to do search engine promotion for you by submitting your site to hundreds and thousands engines and directories. The search engines' indexing programs, called "robots" or "spiders", looked through all of the HTML code of a page and used some page ranking algorithms that they kept in secret.

    Those days were spammer's Heaven: it was rather easy to get your site ranked high. You could just use your keywords lots of times on the page, in the META tags, HTML comments etc. and hide it from the human visitors by making the text tiny or completely invisible with the help of HTML tricks. The search engines didn't have any sophisticated technique to recognize this kind of spam, and such sites usually got high rankings very easily. Today, you still can find some samples of this primitive optimization (however you will have to give it a hard try, because nowadays most of such Web projects have been banned by the search engines for excessive keyword usage).

    The only exception was Yahoo which has always been indexed by humans who could, in most cases identify and ban spamming pages.

    Gradually, search engines started recognizing spam and applying corresponding penalties to Web pages using spam methods. However, search engine optimizers were always one step ahead of the search engines in finding new ways of cheating the indexing algorithms. Since each search engine is committed to delivering only relevant results to its visitors, the engines needed to take control away from the spammers and auto-submitters. Many began to try different ways of indexing.

    Off-the-page Factors as a Solution

    By mid-1999, search portals began using the logic of an ordinary Web surfer to improve the quality of search results. Before this, search engines ranked pages primarily by keyword distribution, which was easily manipulated by spammers. Another way was needed for search engines to estimate a site's importance. Thus, algorithms were developed that utilized click tracking. Sites receiving more clicks from particular keyword search results were considered more important, and ranked higher for subsequent searches based on those keywords. DirectHit (since acquired by AskJeeves / Teoma) introduced a technology that watched which sites were chosen by Web surfers. If the surfers often choose a certain Web site for a given keyword, this site will rise in position for that keyword.

    Another way to make indexing algorithms more bulletproof is to rank pages based on how many other pages are linked to it. This "reference" principle has come from regular libraries and archives and now plays the leading role in governing the traffic flow in the Internet. It has been named "link popularity" and remains a huge factor in ranking.

    Both of the above parameters are known as "off-the-page" factors because they are factors that are not directly based on the content on your Web page. The claim of "off-the-page" factors is being liberated from the spammers' influence.

    The Search Engine Optimization industry has found a temporarily appropriate response by creating so-called "link farms". The idea is the following: if the search engines consider how many inbound links a Web site has got, let's create a special link page outside our site which is of low interest to the visitors but which can be fed to the spidering robot. We can place many links to our site on this page and on lots of similar pages, and these links will have their say when the robot comes to index.

    The link to a link farmer's Web site was placed on a great number of similar link pages of other participants, and in exchange this farmer had to place their links on his own link page . The official aim of the link farms was to create a community united by total cross-linking, however the hidden goal was achieving high search engine rankings.

    While link farms were enjoying great popularity even until recently, and loads of special software for link exchange between the link farmers has been written, the search engine spiders became aware of this long ago. The contemporary search engines don't care a straw about these kinds of link pages when ranking a site.

    Community-Edited Directories

    Yahoo was the first and is still one of the most popular search engines. Primarily a human-edited directory, Yahoo faced the problem of finding human resources to deal with an exponentially increasing number of pages queued for indexation. While human-edited directories provide quality results and are almost spam-insensitive, the limited number of editors makes it troublesome to achieve the same perfection in quantity.

    Community-edited directories first appeared in 1999. The concept allows for thousands of editors, organized in a system of self-governance, to constantly improve and expand the directory with the new resources. The first to come was the Netscape Open Directory; the Go.com directory (now not functioning) was another early leader. Zeal.com, which feeds results into LookSmart.com and MSN search, is a newer addition to this category.

    The Netscape Open Directory, besides being community-edited, also was a kind of open source directory. Any developer wanting to create or improve their own search portal had access to its content. This resulted in the fact that by the year 2000, listings from Netscape Open Directory started showing up on almost every major search engine.

    The community-edited directories have developed a successful junction of quantity and quality, becoming a weighty part of the search world. Human-edited directories in general tend to play the role of "searcher advocate", since they produce very relevant results for any given search. With the rise in importance of directory listings in 2000, search engine marketers began to concentrate on optimizing their sites for focused, targeted, and quality content.

    Paid Listings

    In 1999, AltaVista has made an attempt to introduce paid listings, i.e. to take fees for inclusion into the search index. This attempt was totally uncompetitive and was universaly denounced. Alta Vista soon quit that project. Nevertheless, by the end of 2000 all major search engines offered some kind of paid listing option. One of the brightest current examples is Overture.com (previously Goto.com), which began as an independent search player and is now owned by Yahoo. It is still one of the major paid listing / advertising resources, offering ranking for keywords based on an auction system: the higher the bid for a certain keyword, the higher your rank for this keyword. The bid amount is charged every time a user clicks on your site listing, a payment model known as Pay-Per-Click (PPC).

    In 2001 most directories and search engines introduced various payment models for paid listings: fees for indexing by the search engine spiders, fees for listings in human-edited directories, ad placement opportunities etc. Yahoo established a one-time submission fee, which was later changed to a yearly submission fee. LookSmart switched from a one-time submission fee to a PPC model. Spider-based engines such as FAST, AltaVista, and Inktomi introduced paid inclusion, i.e. a recurring fee that ensures a site will be listed and regularly re-spidered.

    tobe cotinue, read more at [B]www.webuniver.com[/B]
     
    arale, Aug 14, 2007 IP
  2. sweetfunny

    sweetfunny Banned

    Messages:
    5,743
    Likes Received:
    467
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Just curious where you got your info from?

    Also i'm surprised AltaVista didn't get more of a mention, as it was a pretty big player at one stage. I've actually got a couple of articles and Infopop/UBB forum posts from 1999 in Archive.org on Optimization for AltaVista.
     
    sweetfunny, Aug 14, 2007 IP