Can MySQL import a 1 TB text file with no problem?

Discussion in 'MySQL' started by electroze, Aug 6, 2011.

0
  1. #1
    I'm wondering the fastest, most efficient way to build a website using a 1 TB file as its data. If MySQL can import a 1 TB tab delimited .txt file with no problem, or if performance will drop and I should find a different solution?
     
    electroze, Aug 6, 2011 IP
  2. mwasif

    mwasif Active Member

    Messages:
    816
    Likes Received:
    23
    Best Answers:
    1
    Trophy Points:
    70
    #2
    Give it a try. I hope it can import the file without any problem using command line. But it will take some time. What is your server specs?
     
    mwasif, Aug 6, 2011 IP
  3. fydgp

    fydgp Greenhorn

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #3
    Importing 1TB into MySQL (that is a single MySQL server) is probably not a good idea. You'd need a lot of memory and computational power. You probably need a disk-based index or something like Hadoop MapReduce or a cluster of MySQL nodes. 
     
    Last edited: Aug 6, 2011
    fydgp, Aug 6, 2011 IP
  4. electroze

    electroze Active Member

    Messages:
    179
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    60
    #4
    electroze, Aug 6, 2011 IP
  5. Ruddo

    Ruddo Member

    Messages:
    821
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    43
    #5
    Ruddo, Aug 6, 2011 IP
  6. mwasif

    mwasif Active Member

    Messages:
    816
    Likes Received:
    23
    Best Answers:
    1
    Trophy Points:
    70
    #6
    Bigdump will consume more memory as compared to MySQL command line. I don't recommend this.
     
    mwasif, Aug 6, 2011 IP