Hello, Is there any working model sites where anyone can make a web crawler go according to their need , where we can use the internet as a database and build necessary applications.
Sphider is the only open source spider ive used... not sure if thats what you are after..? You can have it follow links on any site....