I'm building a clustered web-crawler that is able to crawl from many servers and easily scalable. Right now I'm in the test phase, everything basically just works, few things need a little adjustment. I'd really like to focus on this field full time, so I'm looking for opportunities to provide a paid service. I'd like to know your opinion if such service would be on demand on the market and what would be an appropriate pricing for it. What I can offer now is solving individual crawling tasks. A customer explains what he needs to crawl and the speed he needs to achieve, amount of data he wants to collect, etc and I'm solving this task. It can be a one time job or providing the fresh data on a regular basis. Basically it's good if you need to collect some content for your projects or to analyse some data, to get any statistics. Just a few examples of what can be done: Collecting a database with the products from different sites Collecting a database of product reviews Collecting a database of jobs from different job sites Collecting a database of real estate and property for rent Crawling profiles from social networks and dating websites Crawling the internet for some particular data Different data mining tasks To solve such task now, people usually hire some free-lancer to write a simple PHP script that will crawl for ages. For now I have a crawling speed about 15mln pages per day from a single server, but it's a scalable number almost without limits (depends on amount of servers I'll add to the cloud) What do you guys think? I'm also open for any ideas and business opportunities related to field of web-crawling and processing large amounts of data.