How can I create my own Search Engine?

[misohoni]

I bet you are thinking, ah it's been done...but I want to create something for my country only and base it on META tags.

I've seen sites like Hoppa and searchpole and want to create something like that.

Could anyone suggest any software/advice on what I need to do?

Thanks guys

[Old Welsh Guy]

There are good scripts you can use out there, but you say search engine, do you mean a full blown search engine with its own spider that runs around adding sites, or are you talking about a site with spidered sites included, but manually apporved, that has a search engine algorithm?

I would recommend the fluid dynamics search engine, it is a good thing, written in perl, so is mlimited in the volume of pages it will run with before getting slow. It has its own spider, and you can add weight to the algo on things like , newness of page, words in metas, copy repetition etc.

http://www.xav.com/scripts/search/

[misohoni]

thanks, it's pretty good. Yep I'm looking to manually approve and charge people if they want to submit it quickly.

[nriweb]

Curious ... which country are u building the search engine for .... i would suggest you to download the dmoz.org dir and use the category info to just filter out sites not related to ur country...

[misohoni]

hi, how can I download the Dmoz directory?

Do you mean I should copy the structure?

[Sholva]

http://rdf.dmoz.org/

It's big...so hope you have a fast connection, I'd recommending downloading directly to your web server than to your desktop first ;) (if you can do that)

[misohoni]

thanks for this, I'll give it a go! Any other tips on creating a Search Engine/Promoting it etc., I've developed the rough code for it - so at least it's a start.

[disgust]

sure, I'd try this:

start with dmoz's data. crawl each site there. only keep the "main" domain pages- ie store www.google.com, but no internal pages. have your bot follow everything in the dmoz mirror you set up. after that, have it continue to crawl. maybe set up some sort of cap on how much space you want it to take before it'll give up. once you have that done, I'm sure you'd have enough to play around with.

if you allocated 2 gigs, and each page was 50KB, you could get something like 40,000 sites' main page indexed.

[donteatchicken]

Can you legally run a full blown spider on your server?

What scripts are out there to do this?

Thanks in advance.

[eklim8]

If my memory still serve me right, i found a search engine script coded using asp.net/c++. It's great nevertheless. The programmer used 1 day to code it. He's an expert btw. :)

[Mystique]

This is the best search engine software I've ever tried:

www.webwizguide.com/asp/sample_scripts/site_search_script.asp

users submit a site, you approve them manually and surfers can rate the sites as well.

Only con: you need a Windows based server because it is coded in ASP using MS access database.

[misohoni]

Thanks, just wonder what the quality will be after a day's work...I can actually answer my own question now, I made my own search engine...

[finaldestination]

misohoni please explain what u di?

also if u cud share the code with us

[frankcow]

I created and posted the ASP code to create a spider here: http://www.justin-cook.com/wp/2006/04/14/how-to-write-a-spiderbot-with-asp/

[Psychotomus]

if you allocated 2 gigs, and each page was 50KB, you could get something like 40,000 sites' main page indexed.

about 27,000 html pages that are 40kb is 1GB. It also took about 10 hours to download them all at about 200kb/sec.


your need a lot of computers and bandwidth to create a good spider. Id say 10 computers would be sufficent enough.

[Nick_Mayhem]

There are many options like:

mngosearch, Datapark, ASPSeek, Sphider, phpDig, SMEMETA, Curryguide script, Jomo PPC Search.

I have used all of the above and have experience in it. If you are thinking anything commercial then let me know. I can help you out.

[paulmcdonald]

Hi everyone. it looks that there are some pretty switched on cookies here.
im looking to develop a search engine based on post below

users submit a site, you approve them manually and surfers can rate the sites as well.
required to run on a Windows based server using MYSQL
and have additional modules added based on fuctionalitys alexa has for example a user can request a reindex of there individulal website

Would anyone be interested in building such a search engine.
Final end development would everyones to use who was part of this development
It also must be able to index selected domain extentions

Any information on pre built search engines or anyone interested in helping me build this are more than welcome to contact me


:)

[Phynder]

required to run on a Windows based server


I was interested till I saw this part of the post. Golly, I would be willing to setup a nice little linux box if there were enough people interested.

But, the reality is... What will YASE (yet another search engine) really do? What is the business model? Or is this a "fun" project? If it is a fun project, then I am all for it.

[Nick_Mayhem]

If you are thinking about users to submit a site then it is not called search engine. It is called Directory and then in it the users surf and rate the things. If you want it to run on windows and want to go with ASP then I can suggest UUDir which is also developed by a DPer here. Or if you want to go with PHP then you can get esyndicat which is free one.

It has that alexa and Page rank type of thing in it.

But If you want registered members to rate and review then there will be some custom coding involved in it. As for approving and other features. that can be done easily.

are you interested commercially? If yes then you can PM me I can develop this kind of things.

[Phynder]

I think everyone at DP already has their own directory. I am not sure just because people submit web sites, that makes it a directory - help me to understand that distinction.

[MrSupplier]

search @ sf.net for opensource spider projects

[ajarchibald3]

I'm currently using nutch for developing a vertical search engine. Anyone else out there using nutch.

[masterc19]

yes i have a domain name it is www. ineedtoknowitall.com perfect name for a search engine
and i would love to create a full blown search engine for it like google.com but i do not know how to build a search engine could anyone tell me how to do this thanks :confused:

[Vilims]

Thanks for valuable information on search engine programming and i will try also for it.
thanks again

[daringtakers]

Creating a full blown search engine isn't an easy task, it involves lots lots lots of complexities like creating a spider, parsers, indexing/searching algorithms, and lot's of other stuff.

Forget search engine... Developing an efficient spider only, is a big project.

I will suggest don't try to do,specially when your are doing it alone, instead use google custom search and save your self.