Web Crawler to pull in news

Discussion in 'Products & Tools' started by kevvy, Nov 22, 2009.

  1. #1
    I am looking for a web crawler that will search news sites for specific keywords. Basically I want to be able to type in specific key words and then have the web crawler go out and pull in all pages with those keyword on specific sites. The crawler program should be able to display the url, page title and page description. The crawler also needs to have an export function, perhaps to a CSV file.

    We cannot use any type of RSS feed crawler since most of the terms we will be searching for don't have RSS feeds created for them.

    We have been trying to use Visual Web Spider from Newprosoft which works very well. The problem is that this program only allows you to do one search at a time. So we can search for a specfic keyword (or multiple keywords) for the site we enter. But we have 20 different keywords for 20 different sites that we need to do daily. So we have to launch each search separately, one after another, rather than just being able to launch all searches at once.

    Anyone have any suggestions?
     
    kevvy, Nov 22, 2009 IP