I think the previous comment about using existing search engines and displaying it using your own custom branded UI is an excellent idea. What teh fcuk?
Exactly! I've made a web crawler in C++ before. It caches websites and uses an extremely basic page rank system.
If you were to start a search engine, you would start simple and work through a stack of challenges around spidering data, storing data, search display, sort order, and if successful then load balancing. It is not rocket science, and you can grow it with traffic and content over time if your successful. IMhO it would be very hard to knock off google, yahoo and Microsoft. That being said, none of the engines are perfect so I think there is room for another good engine.
If you wanted to make a custom search engine to compete in the market you would need a team of good programmers. Even though some would recommend to use an open source search engine for ease. On top of that you need some heavy duty hardware. Plus you need traffic so its not easy task, especially with the big three (google, yahoo, bing) dominating.
Can you share with me how you made the crawler? You can PM me the details if you don't want to share it openly. I can code a little in PHP and I want to learn C++, because of my interest in developing a unique search engine. Any help will be appreciated. I'm not interested in beating Google, Bing and Yahoo, but in taking my chunk (share) of the market. Three isn't enough, but I believe it's a wrong approach to go into the business professing to be a Google, Bing or Yahoo killer. However, I respect the views of those who feel that way.
Unfortunately this was a job I did for someone, based on code I had already written. I recommend looking at Chilkat or this
i do have a plan with Custom SE, but after few hundred spend (con) n month,by those I-programmer, i give up now .....
I've created one not intending to go into competition with Google but to spider my link partners sites and send out traffic to them which is reciprocated back.
writing a web crawler is not rocket science. ranking alghoritm/system it is rocket sience but crawler is simple, you don't need to use components, go to planetsourcecode and you will find a lot of useful code, mostly i vc++. I was doing some with crawlers and you just need powerful hardware and good ,structured database for data storage. But when you storage data you need alghoritm to do 'something' with that data it is not just page ranking. that is place where I stoped, and decided to start playing with custom meta search engines, it is much easier and easier to gain some traffic.just my 2 cents.
In danger of reinventing the wheel? I'd leave it to the ones who are currently dominating the market. Fair enough if it is an ambition as such, but don't rely on it for anything. My 2 cents!
If you feel like you need to learn more about them they Toolz has the answer. Just start with a little basic design for your program and start filling in the blanks. You will learn so much about the various components that you will gain a good understanding of the total product of a search engine. Don't expect to be done before xmass though.. xmass 2010 that is The tricky parts of a SE if you ask me are the ranking algoritm, queueing and database optimization. 3 tech components which may turn your hair grey.. if you did not pull it out yet And concerning the language to use you can get started in many just remember that the backend indexer and retrieval components run in the language which runs best on your platform. PHP still rocks but perl ereg features run alot faster.
I think I couldn't. I rather make my own webpage. Thats' so complicated and a lot thinks to be done. ________________ How To Make A Website FirmMonster.Com
that's quite not just quite but rather very complicated to make..;p you need to team with the best developers out there in order to create a competing one, not just a mediocre type..;p
Handy, just ran into a thread with how to start crawling. So for those interested read on forums.digitalpoint.com/showthread.php?t=1536208