Error loading HTML in variable from remote site (scraping)

I am trying to scrape some content from a website, and it keeps returning errors, but if I change the URL it works OK on other sites.

I have tried pretty much every method listed here:
http://blog.oscarliang.net/six-ways-retrieving-webpage-content-php/

The curl method is the only one that gets any result, and this is what you get:
Object moved to here.

Is there some method they may be using to prevent accessing the source via this method?

I can send URL in PM if you have any suggestions.

 

Send the url AND the current, almost working code, and I can have a look.

 

Without knowing the URL you are trying to parse it's hard to say, but it could simply be the site you are trying to scrape is such a poorly written steaming pile of crap, it can't be processed well if at all by a normal processor.

You've pointed at methods for LOADING the page content to a text file, but how are you actually trying to PROCESS it? DOMDocument? regex?

 

Alright, he sent me the URI via PM, and this is the same issue someone else ran into last week.

preg_replace: How to remove it

the problem is the URL is a redirect, and most of those techniques he linked to can't handle that... cURL can, but you have two ways of doing it and you have to code "around" that issue. The code presented in that thread SHOULD get you most if not all of the way to a solution. You have to tell cURL to follow redirects, and if that fails you have to trap the error code responses.

Note the method presented in that thread ONLY works if you have PHP 5.4 or newer. Older versions do not provide one of the cURL response variables -- you could code around that with a regex, but to be frank that's NOT the correct answer. The correct answer is to update to a version of PHP from THIS century.

That some people are tool lazy or cheap to update their codebases disgusts me no end... particularly since it seems 99.999% of my code of the past decade runs unmodified in PHP 7 because I *SHOCK* paid attention to the things they told us to stop doing. That web hosts are simply molly-coddling people by keeping decade out of date PHP versions in circulation only exacerbates the situation.

 

Tested with the cURL redirect-follow, it doesn't work (well, at least it didn't when I tried, but I'm not very well versed in cURL, so it might be that there are ways to make it work). The redirect is done in code, not .htaccess or similar, as far as I can see (or was the code ASP... don't remember). Anyway, the plain follow redirect cURL setoption doesn't work.

 

when i use curl, i get a redirect to an ASP error message... how do you get the correct redirect URL