Superunits and Overture

There seems to be a popular conception that the Yahoo search Also try: list of 100 links is derived from Overture data, see for example this recent thread on WMW.

I have been researching the possibility that the search server includes data from the concept server in the Yahoo algorithm. I think that this is at least a possibility after studying the superunits patent application and in particular the block diagram FIG.11.

As part of this research I have accumulated data for analysis and although I have yet to reach a conclusion on superunits and the algorithm I have concluded that the Also try: list is not derived from Overture but is a list of units or a superunit provided by the concept server.

I have looked at the number of times a keyphrase in the Also try: list for a given search also appears in the Overture keyword list for the same search term. If one was derived from the other you would expect the percentage of occurrences to be very high. They are not and here are some representative examples:

First the search term used to provide the Also try: list followed by the percentage of the list in Overture.

real estate 66%
mp3 21%
new york 33%

Section [0047] of the patent says in part "....a concept discovery process based on some number (preferably a large number, e.g., at least several hundred thousand) of previous queries. Concept discovery....involves analysis of the queries to generate a concept network....". This would indicate that the original data from which the superunits are derived is based in part on the analysis of data accumulated from search queries. So of course is the Overture data which is probably why the Also try: list is partially the same as the Overture data.

Of course having established that the results from the concept server are passed through the search server to the client in the form of the Also Try: list the big question is - are these results also used on the way through in constructing the SERPs? I think they may be but I need to do a lot more analysis before I am sure.

- Michael