1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

how to open a very large csv file

Discussion in 'Databases' started by superuser101, Oct 31, 2013.

  1. #1
    Hello,

    I need some help to open a 16gb csv file, how much RAM or CPU do i need in my pc to open it? Thank you !!
     
    superuser101, Oct 31, 2013 IP
  2. ZyreX

    ZyreX Well-Known Member

    Messages:
    104
    Likes Received:
    13
    Best Answers:
    1
    Trophy Points:
    115
    #2
    Try these 2 programs: UltraEdit and EmEditor

    User Comment:
     
    ZyreX, Oct 31, 2013 IP
  3. OwnCoder

    OwnCoder Greenhorn

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #3
    You can try on NotePad++
     
    OwnCoder, Oct 31, 2013 IP
  4. cameronpalte

    cameronpalte Active Member

    Messages:
    672
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    65
    #4
    Or you could try kiwix. 16gb is too large for Notepad ++ because it wouldn't work for my downloaded wikipedia copy (10gb), but kiwix did. Though I'm not sure if kiwix supports all files.
     
    cameronpalte, Oct 31, 2013 IP
  5. kutchbhi

    kutchbhi Active Member

    Messages:
    130
    Likes Received:
    4
    Best Answers:
    2
    Trophy Points:
    70
    #5
    depending upon your requirements , you could also use command line tools to do the job. like grep
     
    kutchbhi, Nov 12, 2013 IP
  6. Dot_Net

    Dot_Net Greenhorn

    Messages:
    30
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    23
    #6
    I'm assuming that you're on Windows, so I'll recommend gVim. Where Notepad++ will choke on very large files, VIM has chowed through those puppies with little problem.
     
    Dot_Net, Nov 13, 2013 IP
  7. nextsmm

    nextsmm Greenhorn

    Messages:
    47
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    8
    #7
    There is a solution in Excel. You can’t open big files in a standard way, but you can create a connection to a CSV file. This works by loading data into Data Model, keeping a link to the original CSV file. This will allow you to load millions of rows.
     
    nextsmm, Feb 14, 2022 IP
    JEET likes this.
  8. Daniel165

    Daniel165 Member

    Messages:
    132
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #8
    You can try Notepad++ and Subline text.
     
    Daniel165, Jul 6, 2022 IP
  9. AquaSwitch

    AquaSwitch Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #9
    I would recommend Looker as the solution for datasets larger than old excel can cope with.
     
    AquaSwitch, Jul 6, 2022 IP
  10. James Cavalcanti

    James Cavalcanti Peon

    Messages:
    14
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    3
    #10
    The amount of RAM or CPU you need to open a 16 GB CSV file will depend on the size of the data contained in the file and the capabilities of your computer. Here are a few factors to consider:

    • The amount of data contained in the file: A CSV file is a simple text file, so the size of the file itself doesn't necessarily reflect the amount of data it contains. If the file has a lot of data, it may require more memory and processing power to open and manipulate it.

    • The capabilities of your computer: The amount of RAM and CPU available on your computer will also affect its ability to open and work with large files. In general, more RAM and a faster CPU will allow your computer to handle larger files more efficiently.
    As a general rule of thumb, it's a good idea to have at least 8 GB of RAM and a relatively fast CPU (such as an Intel Core i5 or higher) to work with large files like a 16 GB CSV. However, the specific requirements may vary depending on the complexity of the data and the specific software you're using to open and manipulate the file.

    If you're having trouble opening a 16 GB CSV file on your computer, you may want to try using a tool specifically designed for working with large datasets, such as Apache Spark or Google BigQuery. These tools are designed to handle very large datasets efficiently and can make it easier to work with large files like a 16 GB CSV.
     
    James Cavalcanti, Dec 16, 2022 IP
  11. knowledgeherald

    knowledgeherald Well-Known Member

    Messages:
    638
    Likes Received:
    39
    Best Answers:
    0
    Trophy Points:
    120
    #11
    1. Spreadsheet software: You can open a very large CSV file using spreadsheet software such as Microsoft Excel, Google Sheets, or LibreOffice Calc. These programs can handle large files and allow you to sort, filter, and analyze the data.

    2. Text editor: You can open a very large CSV file using a text editor such as Notepad++, Sublime Text, or Atom. These programs can handle large files, but they may not be as user-friendly as spreadsheet software and may not have built-in data analysis tools.

    3. Command-line tools: You can use command-line tools such as sed, awk, or grep to open and manipulate very large CSV files. These tools can be powerful, but they may require some knowledge of programming or scripting.

    4. Open source libraries: You can use open source libraries such as pandas or Dask to open and manipulate large CSV files. These libraries are designed to handle large data sets and can be used with programming languages such as Python.
     
    knowledgeherald, Jan 14, 2023 IP
  12. laci272

    laci272 Well-Known Member

    Messages:
    163
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    118
    #12
    This is the best reply.

    But you can also just import in into mysql or sqlite and then use phpmyadmin, adminer or sqlite explorer to browse the data (that's what i would do)
     
    laci272, Feb 15, 2023 IP