After just a few queries Google starts asking for CAPTCHA, and if search continues, G blocks IP. Perfectly normal, since it's against G TOS to query it programatically. Google API script means nothing, as API is abandoned long time ago, so it makes no difference whether I use ex-API, or for instance, my own, custom cURL/regex search. And I need results, cause I can't finish blog searching script without help of major Search Engine. So, having in mind that G has results with top quality, is there a way to get them without getting banned AKA by respecting G's TOS, or NO? [sure I could code proxy-based script, but that means making application which is dependant on other machines (actual proxies), equals lack of stability which is something I don't prefer at all, no matter that its STILL against G TOS] If not, what's the alternative you suggest? Y! SDK/API, with 5k queries per day, with up to 100 results per query; something else? Any idea/suggestion is highly appreciated, I already broke all deadlines and I really care for customer I'm building the solution for, as he's extremely polite person; so I'd like to deliver best possible thing, and I just need to find a way to get [quality] search results for specific keyword/phrase [no code needed, thats what I do ;-)] Thanks to all good people!
I would use Yahoo API as you said, but the other possibility would be to buy from someone Google SOAP API key, this is very similar to Yahoo API however Google stoped giving keys to this API few years ago.
You mean the same key we could optionally put in DP's coop ad network? -I have that key. Script I mentioned above requires it. And G still forbids search
I think that if you want to use google, your only real choice will be proxies. proxy.org scraping *cough*
1) Use all available server IP's. Rotate and request. From experience, 5 did the trick for me (but see point 2) 2) Add a timed delay, don't execute request after request. I hope this helps you,