What is the best language for this huge requirement? Your expert input required please

Discussion in 'Programming' started by WHoSaidThis, May 21, 2020.

  1. #1
    What languages would you use to do the project below and why?

    Summary:
    • Scrape a lot of data (text, pictures, videos) from multiple websites multiple times in a day (one example: all vehicle data from autotrader.co.uk)
    • Combine all data and check for any changes in the data being scraped (e.g new pictures)
    • Make data available on a website on a real time basis (database?)
    • Data is searchable with multiple filters (think autotrader.co.uk again)
    Thank you
     
    WHoSaidThis, May 21, 2020 IP
  2. seomanualsubmission

    seomanualsubmission Well-Known Member

    Messages:
    921
    Likes Received:
    132
    Best Answers:
    4
    Trophy Points:
    165
    #2
    If you are asking about Programing Language then
    PHP is best to manage all your requirement for good price. ASP can be also good but hosting space or server cost a bit high for ASP.
     
    seomanualsubmission, May 21, 2020 IP
  3. WHoSaidThis

    WHoSaidThis Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #3
    Thanks. PHP because it’s fast ?

    what would you use for storage of the data?
     
    WHoSaidThis, May 22, 2020 IP
  4. seomanualsubmission

    seomanualsubmission Well-Known Member

    Messages:
    921
    Likes Received:
    132
    Best Answers:
    4
    Trophy Points:
    165
    #4
    I use Mysql for database for data of my websites. I dont think there any other better options than mysql.
     
    seomanualsubmission, May 22, 2020 IP
  5. sarahk

    sarahk iTamer Staff

    Messages:
    28,821
    Likes Received:
    4,539
    Best Answers:
    123
    Trophy Points:
    665
    #5
    Any database - MySQL, postgres, MSSql
    Language - PHP, python, Perl, Ruby, ASP

    What you're trying to do isn't a huge ask, the big issue is writing the scraper that can read the dom and extract the right info and cope with changes to the layout in a timely manner.

    Outputting the scraped data would be a doddle.
     
    sarahk, May 23, 2020 IP