Search engine theory - post yours here!

So a lot of webmaster forums have plenty of threads saying why Google is broken, or not as good as it could be. Here's your chance to say how you think a search engine should work! Serious or silly ideas both welcome. I'm personally semi-serious about this, from a hobby/programming perspective. My own theory is to take the original Google idea to the extreme, and base almost everything off the "links are votes" concept - give surrounding text of the backlink, anchor text (and rep of page) most of the weight.

If I started my own engine, I'd probably start by crawling dmoz and yahoo, giving pages on those directories an arbitrary high weight as a placeholder until I could write a decent ranking algo.

Anyone else have mad whacky theories? I remember some posts by Owlcroft a few weeks back that held the complete opposite view to mine -- that a really decent SE should be able to determine the relevancy of a site by its content. I love discussing this stuff!

Edit - also, any constructive points about how other search engines (Google) fall down would also be great. i.e. the 302 redirect problem.

 

A search engine actually isn't that hard to program; I made one for a class project in college. However, the key is the algorithms used to sort and search through all the data that you have spidered. You'll need a background in computer science or math because the algorithms used are very theoretical. All search engine theory relies on the same type of algorithms and most of the information is public because it's presented in conferences. So, over time all search engines have to evolve or clever people will take the public information and start using it to manipulate the results.

My whole point is that search engine theory is a young, evolving field based on one of the oldest hard sciences - math. If you have a strong math background, then you can create and manipulate search engines. Troof.

 

Google is not broken... It is a smart move on google part to keep you in the sandbox (many of my sites are in it now, so it exists) so you that you end up paying them through Adwords for traffic. However it does look like google is not the only game in town and I may consider not optimizing for them altogether.

 

BSc Hons. Computer Science, check

In agreement there. Even though Google has evolved and updated its algorithm over time, the basic theory underlying it still holds and link buying, 302's etc. are examples of people using the knowledge of how the algos work to manipulate the results. Maybe there's some way that's not the same type of algorithm though, kinda my reason for starting this thread to see if anyone has any original ideas...

Yeah, another reason I'm interested in it, precisely because it's a young field. Although I don't really have any truly new & original ideas I firmly believe there's still some waiting to be discovered.

 

Good post and I agree with you 100%, a good background in Math somehow plays a role in understanding the algo....

 

I think Google was good when there were thousands of websites, but now when the count is in millions, their results are less than good, except when you are looking for a very specific term. With many searches I end up getting back tons of forum posts dated back to 2002 and before, link-exchage pages, some blogs and it just is getting longer and longer to sieve through these.

As for page ranking, it simply can't be just links. Otherwise cross-linkers would be the only ones in the search results. I can guarantee you that they are sampling actual clicks and browsing patterns.

J.D.

 

How do they sample browsing patterns, by using the G. cookie?

 

They stick JS into their links from time to time and use redirection.

J.D.