Drupal Questions - Web Scraping using Drupal Custom module

Discussion in 'Programming' started by i_am_dhaval, Sep 11, 2014.

  1. #1
    I am trying to get code from our old site using Web Scraping using drupal custom module.

    The code I am using is
    /*$mainUrl is the link of the page from where we want to fetch the HTML*/
    $response_data = drupal_http_request($mainUrl);
    $dom = new DOMDocument();
    @$dom->loadHTML($response_data->data);

    /*We want to get the titles in from a Div*/
    $titles = $dom->getElementsByTagName('div');
    foreach($titles as $title) {
    if ($title->getAttribute('class') == "movie_results") {
    $result = $dom->saveHTML($title); // This result works fine in my Local System but not showing result in Live Site.
    }
    }

    The $result variable is giving result in my local system but when I am using the same code for live site it is not showing the result.Basically "$dom->saveHTML($title)" is not working. I have tried "$dom->nodeValue($title)" which works both in local as well as in live site but problem is it won't return HTML structure.

    Any suggestions how to get whole result of div using saveHTML or using any other method?

    Or let me know if I am doing anything wrong? Or anyone is having any suggestions please do let me know.
    Thanks in Advance.
     
    i_am_dhaval, Sep 11, 2014 IP