How would you build a small, specialized search engine?

The engine would have a basic algorithim, nothing too fancy. No link popularity would be necessary. For example, say I have 30 fairly large sites in my network. I want to be able to have a search box on my "corporate" site that searches all the sites in the network. They're all hosted separately.

So what language would you use to code this kind of engine? Any insight on how a spider fitting these needs would be built would be cool too.

Thanks in advance.

 

This thread has some good points.
http://forums.digitalpoint.com/showthread.php?p=424757

 

Some good stuff there Dave, thanks.

I did some more searching, came across a thread that mentioned a SE script called Fluid Dynamics Search Engine. It seems pretty cool. But depending on the power of your server and some other factors, the most pages/documents you can effectively search is 10,000 to 100,000, at the very most.

http://www.xav.com/scripts/search/help/

 

What about this guy? http://www.freefind.com/plans.html
But they have a 5000 page limit... contact them for more pages.

 

check dtSearch. they have some kind of ActiveX which can index text for webpages.

 

Atomz Search / free for 500 pages -> http://www.searchtools.com/tools/atomz.html

 

Thanks for the link, I need around 100k pg capability though. I guess I'll have to face up and pay some serious $ if I want something like that. Atomz Search is $15k + per year, out of my budget.