Google Search Scraping (HELP)

Discussion in 'Google API' started by Vexez, Apr 8, 2010.

  1. #1
    I did tried something with google's new ajax API but is limited to 8 results per query and it does not produce a similar search. So I did some scraping using cURL stuffs.

    I got a problem here that I'm still in doubt.
    Situation with example
    I'm in Singapore and I use www.google.com to do a search with my browser this is what I get.
    
    www.apple.com/iphone/ - Cached - Similar
    www.apple.com/iphone/iphone-3gs/ - Cached - Similar
    en.wikipedia.org/wiki/IPhone - Cached - Similar
    News results for apple iphone
    Image results for apple iphone
    www.cnet.com/apple-iphone.html - 5 hours ago - Cached - Similar
    developer.apple.com/iphone/ - Cached - Similar
    discussions.apple.com › Support › Discussions - Cached - Similar
    www.appleiphoneschool.com/ - Cached - Similar
    www.phonesreview.co.uk/category/mobile.../apple-iphone/ - Cached - Similar
    www.youtube.com/watch?v=ojzgwzU8T7g - Related videos
    www.gsmarena.com/apple_iphone-1827.php - Cached - Similar
    
    Code (markup):
    This above result would return the same even if I use TrafficTravis SEO Analysis (United States)
    But if I use my hosting, I suppose is located in US, to do the scraping on:
    http://www.google.com/search?......................
    This is what I get:
    
    www.apple.com/iphone/ - Cached - Similar
    www.apple.com/iphone/iphone-3gs/ - Cached - Similar
    en.wikipedia.org/wiki/IPhone - Cached - Similar
    store.apple.com/us/browse/home/shop_iphone/.../iphone - Cached - Similar
    www.cnet.com/apple-iphone.html - 5 hours ago - Cached - Similar
    developer.apple.com/iphone/ - Cached - Similar
    discussions.apple.com › Support › Discussions - Cached - Similar
    www.gsmarena.com/apple_iphone_3g-2424.php - 13 hours ago - Cached - Similar
    www.appleiphoneschool.com/ - Cached - Similar
    www.youtube.com/watch?v=ojzgwzU8T7g - Cached - Similar
    
    Code (markup):
    I know the results are close but some other search terms gives slightly different results. I'm just using apple iphone as example. It seems like no matter which one I use, there is still a difference in results. If you're doing SEO Analysis and research you should understand these results does matter. What I really want is the US results, people who do the search in US itself as this is the majority search volume.

    I need some help from people who are in US to tally with me if you do a search using www.google.com on 'apple iphone', does your top 10 results match my top 10 results listed above? Which one did yours match?
    I'm guessing it's the 2nd one though.

    Can some of you guys from US help me do this test? Would be useful information for people who are interested as well.
     
    Vexez, Apr 8, 2010 IP
  2. Vexez

    Vexez Well-Known Member

    Messages:
    339
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    140
    #2
    Anyone from US can help me with this?
     
    Vexez, Apr 8, 2010 IP
  3. zerophean

    zerophean Peon

    Messages:
    91
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Hi, try this one, its awesome :D :

    google-scraper.squabbel (dot) com
     
    zerophean, Apr 12, 2010 IP
  4. webscraper

    webscraper Peon

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Vexez, use google.com/ncr as your starting point for your web scraper. ncr stands for No Country Redirect and will give you the US results. Best of luck.
     
    webscraper, Apr 18, 2010 IP