cURL, JavaScript and Php

Discussion in 'PHP' started by afimafis, Jan 30, 2012.

0
  1. #1
    I’ve developed a scraper using cURL and PHP what can scrape from different websites very well. But, if a website uses JavaScript POST operation, cURL code doesn’t work. Do you have suggestion to solve this problem?
    Thank you in advance.
     
    afimafis, Jan 30, 2012 IP
  2. hip_hop_x

    hip_hop_x Active Member

    Messages:
    522
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    90
    #2
    javascript can't be processed from php directly, so you have to interpret javascript code differntly, there's the v8 engine extension for php, but i never used it.
     
    hip_hop_x, Jan 31, 2012 IP
  3. maureeeteeeee

    maureeeteeeee Member

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    2
    Trophy Points:
    41
    #3
    Exactly as hip_hop_x said.

    Javascript is client-side working, and PHP server-side.
    You should try making a Javascript code that makes an AJAX call to your script, and then parse the response.

    Some tips, when getting bigger response or many elements, try using json_Encode() in your PHP so it's easier for the Javascirpt to work with.

    Other tip, take a look at jQuery's .AJAX(). It's really good to work with! :)
     
    maureeeteeeee, Feb 1, 2012 IP
  4. afimafis

    afimafis Greenhorn

    Messages:
    90
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #4
    thanks maureeteeee :)

    but how to trig this "Javascript code" from php curl or what else ?.

    another point is if all ways go to URL and POST Fields,why we need to trig javascript code ?

    thanks a lot



     
    afimafis, Feb 1, 2012 IP
  5. maureeeteeeee

    maureeeteeeee Member

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    2
    Trophy Points:
    41
    #5
    This Javascript code makes your site usage seamlessly, without pages reloading. That's why it's good!

    Check out jQuery $.ajax() reference and you'll get it out within minutes!
    It's really easy to use an implement!

    If you need any further help, let me know.
     
    maureeeteeeee, Feb 1, 2012 IP
  6. stephan2307

    stephan2307 Well-Known Member

    Messages:
    1,277
    Likes Received:
    33
    Best Answers:
    7
    Trophy Points:
    150
    #6
    There is a very easy way.

    What you need to do is to use Google Chrome. Google makes it very easy to figure out what is happening on a website including all the ajax calls.

    What you need to do it load the page in Chrome and then right click anywhere and select the last option called INSPECT ELEMENT. The current window will be split and at the bottom you will see a series of tabs.

    Not the tab of interest is the NETWORK tab. When you look at it it will be empty. So reload the page and see the magic.

    In the Name column you will see all requests that have been made and continue to be made. This includes images, css and javascript files.

    Now what you need to do is look through the list and find the files of interest. If you click on the file name a new window appears with tabs at the top.

    There you will see exactly what the request looked like and if you go to the Response tab you can see what the server returned.

    You just need to go through the list and find the files of interest. You can filter the files by type at the bottom of the page and you can also clear or freeze the list of files.

    Have a play and learn to love it.
     
    stephan2307, Feb 3, 2012 IP