I was wondering if there is any search engine whose database you can use to build your own custom engine on. So to be clear, I want a search engine that has crawled the web and I then want to categorize and present this data in a site of my own. Since its against googles policy to use their results perhaps you guys could give me a point in the right direction.
Sphider has a fairly good open source script www.sphider.eu that I have been working with for some time now.
Hi, thanks for your comments. I was looking for the database that crawlers gather instead of the crawler itself though... I want to make a frontend myself and use the database of another source because of bandwidth considerations..