Using the Google API to Scrape URLs?

Discussion in 'Programming' started by travelhead, Nov 28, 2006.

0
  1. #1
    I'm not a programmer :/ I'm looking for a script to scrape the URLs from the Google Serps using their Google Search API.

    I was thinking about modifying this script:
    http://iconocla.st/hacks/rtf-catalog.pl

    Thanks for any help.
     
    travelhead, Nov 28, 2006 IP
  2. T0PS3O

    T0PS3O Feel Good PLC

    Messages:
    13,219
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Using the API means you're not scraping. So what do you want to do, scrape or do it the proper way?
     
    T0PS3O, Nov 28, 2006 IP
  3. travelhead

    travelhead Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I would like to use the API to save the first 1000 urls in the SERP to a file.
     
    travelhead, Nov 28, 2006 IP
  4. lv211

    lv211 Peon

    Messages:
    168
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Have you tried using the www::mechanize module for perl?

    It is super easy to extract links. All you would have to do after that is figure out how to get the links you want, put them in an array(or whatever), and then do what ever you want.

    It doesn't have to be as complex as that script you linked.
     
    lv211, Nov 28, 2006 IP
  5. KC TAN

    KC TAN Well-Known Member

    Messages:
    4,792
    Likes Received:
    353
    Best Answers:
    0
    Trophy Points:
    155
    #5
    You can have some ideas from my Google API tutorial at:
    http://www.useseo.com/google-api.php

    You need to add in a loop so that your codes will iterate till the 1000th result.
     
    KC TAN, Nov 29, 2006 IP