How do the more successful Craigslist Clone sites (or scripts that allow you to pull CL classifieds from RSS feeds) get around the fact that sooner or later... CL will ban the IP address of the server who is making the repeated requests? I'd like to build a site of my own, to be used by a small group of people, but I don't want to have my server's IP banned, temporarily banned or blacklisted. Can anyone shed any light as to how I can make this *always* work? Are there any workarounds I can code? TIA... cfdev
First thought that comes to mind is to drop the craiglist feeds into a Yahoo Pipes account and pull from there. They're not going to ban Yahoo, are they? Damn, I spend to much time around here. I'm beginning to think like you guys. Tarnish my whitehat image.
I tried the Yahoo Pipes idea. Unfortunately, CL has banned all requests coming from pipes.yahoo.com earlier this month. That said, does anyone know how a site like http://craiglook.com/ or http://searchtempest.com/ pulls results w/o getting banned? Are they parsing search results from Google? Is that even possible? Any ideas as to how this is done would be appreciated. TIA... cfdev