Downloading all tweets from 5k twitter users into textfile?

Discussion in 'PHP' started by zach catskilson, Feb 28, 2017.

0
  1. #1
    I need to download ALL tweets from my list of 5k twitter users (I have their ID's + screenname) into textfile/s in order to later search through all the content to find specific data.

    One of ideas was creating a PHP curl script that will grab data off website called allmytweets(dot)net which basically displays all tweets published by user.
    The main problem is that their page is divided into pages, you need to scroll down for it to show the next page which could cause problems for the curl script.

    Is there any other simpler/more effective way of doing this?
     
    zach catskilson, Feb 28, 2017 IP
  2. sarahk

    sarahk iTamer Staff

    Messages:
    28,808
    Likes Received:
    4,534
    Best Answers:
    123
    Trophy Points:
    665
    #2
    If you know who they are you can access their feeds one by one and download that. Sounds painful but when you do it from a script it's not so bad.
     
    sarahk, Feb 28, 2017 IP
  3. eritrea1

    eritrea1 Active Member

    Messages:
    182
    Likes Received:
    9
    Best Answers:
    2
    Trophy Points:
    70
    #3
    As someone who has done similar things in the past (not the same site) this is very doable, but you better use the API, or inspect the element and see what happens when you scroll down the page. It is most likely sending a request with numbers like getTweet/?page=... from that you can fetch more
     
    eritrea1, Mar 3, 2017 IP