Hi There, I'm looking for a script that can automatically search a number of (pre-defined) websites for a given term (say a name pulled from a Database) Are there any such scripts avaliable on the interweb? (it doesn't even have to do everything, as it could be used only as a starting point) Or failing one being already in existence, would there be any advice you could give me in making one of my own? Thanks Doctor
would i be able to set up a script to use google to scan dozens of sites for 100's of terms at regular times? (obviously, the timing could be executed with a cron job) regardless, thanks for your speedy reply Doctor
Well, the advantage of going via a search engine is you don't have to crawl all the sites yourself which... a) Is going to make the process much easier b) Isn't going to leave a trail in the site's server logs that you're crawling their site But, on the downside: a) I suspect Google will block you unless you limit the frequency of your requests b) You'll only know of the term if it's on a page Google (or which ever engine you use) indexes So, it's a little swings and roundabouts. But yes, in theory, you can do that. You can drive it from a CRON job, and then parse the results google (or whatever) returns to double check your required term does actually appear.
well, the plan i had was to search the "latest news" pages for the latest news regarding football teams, businesses and the like and also the frequency they appear, but as you said, i'd be limited to pages actually within the search engine. thanks for your help
Again, I would suggest that sounds like something that you'd get without much coding by using.... Google Alerts The Yahoo Pipes thingy An RSS aggregator I'm not trying to be deliberately negative... just sounds like re-inventing the wheel to me?