regarding Crawl Function

Discussion in 'PHP' started by bharan_n, Nov 29, 2008.

  1. #1
    I am writing a crawl function for Vbforum.

    I was successful in logging into the website forum.

    after that it is redirecting to the index page.

    At that point it is asking for username and password again.

    I think i need to pass security token. I tried that as well. But i was not successful in crawling the website forum

    Could anyone give me any idea or modify the code.

    The Below Code is to login into VB Forum
    -------------------------------------------
    <?php
    $username = 'johny';
    $password = 'lak8shmi';
    $post_fields = array(
    'vb_login_username' => $username,
    'vb_login_password' => $password,
    'do' => "login",
    's' => '',
    );

    $url='http://www.indiancinemafans.com/board/upload/login.php';
    $cookie_file='cookie.txt';
    $ch = curl_init();
    curl_setopt ($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_POST, 4);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file); // Write cookie file
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file); // Read cookie file
    $data = curl_exec ($ch);
    curl_close($ch);
    preg_match('%name="securitytoken" value="(.*)"\ /\>%',$data,$security123);
    echo $data;
    ?>
    -------------------------------------------


    example website where i tried the code :

    www.indiancinemafans.com

    Forum : Telugu DVD rips.(forum : 11). Can be seen only when logged into forum.

    What i want here is. Once i am able to log in. I should be successful in crawling the website. After this code how i should write the code to crawl the website.


    Please give some sort of idea or post me any example file.

    The rest i will try to manage myself.

    regards,
    bharan.
     
    bharan_n, Nov 29, 2008 IP
  2. harrisunderwork

    harrisunderwork Well-Known Member

    Messages:
    1,005
    Likes Received:
    21
    Best Answers:
    0
    Trophy Points:
    135
    #2
    What u exactly want to do after login in ?
     
    harrisunderwork, Nov 29, 2008 IP
  3. bharan_n

    bharan_n Well-Known Member

    Messages:
    419
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    110
    #3
    i want to crawl the site content and get thread links as well.
     
    bharan_n, Dec 8, 2008 IP