i wish to start my own search engine i have a found a basic search engine that will crawl the page i have specified but it will not crawl the entire internet, is there away i can generate a sitemap of the entire internet so that i can crawl it and have my very own search engine similiar to google thanks alot
are you really sure about what you are starting? anyway... feed your searchengine with common startpages like http://www.google.com/search?q=site:com http://www.google.com/search?q=site:net http://www.google.com/search?q=site:org http://www.google.com/search?q=home whatever... be creative
I dont understand. u r planning a site engine for ur site or of the whole internet? If later and if u r serious, well, i will suggest you to give some time in planning before just jumping to work. cause, its hard to build google (as u said) from ur home. You may think of make a team of some experts to do it. best of luck!
I don't think there is any tool that can crawl all pages and create a sitemap for it. I would be more than happy to be informed otherwise.