I have just started looking into starting a company that is going to require a Database. A rather large database. I would like to say over 10 million Names, and customer information will be held. The customer information will NOT be credit cards, maybe home address, phone number, email...if given. Still thinking if its really needed, along with some other customer information. I would like to know the level of difficulty to maintain a database of such size...people needed. How many servers will I be looking at, security the hole 9 yards so..If we can start there I will be happy to reply to any questions or comments you have Best Regards
What's your experience, and what type of setup are you most comfortable with (Linux, Microsoft, Open Source, erc.)? Also as far as access, will it be all local, or web based? Speaking generally, you could probably handle that efficiently on one decent server. I would personally use a second identical server for failover and a 3rd (low end server with a lot of disk space) for dedicated backups. I have a huge DB with about 5M rows each row with over 100 fields running on a single server. It's currently growing by about 250K rows per week and I have yet to see any noticeable performance hits. One thing you will definitely need to do is carefully plan the DB's structure and how your application accesses it. On a DB of that size, a single bad index or data type can make a .01 ms query jump to 30 seconds or more. If I were going to do this, I would go for a Dual or Single Quad Core Xeon Server, with 8 - 16Gb Ram, and SCSI or SAS drives if you can afford the price and the smaller size. If you're using MySQL, go with a solid x64 *nix distro like Debian x64 or Solaris 10 x64. Otherwise I would use Server 2003 x64 with MSSQL or DB2 or Oracle. The non-MySQL path will be much signifigantly more expensive. I recently setup a Tyan GT20 B5372 with Dual Quad Xeon 2.4's, 16Gb RAM and a RAID 10 SAS drive setup. The total cost came out under $5k, and the server is incredibly fast and stable.
Thanks for the information jstep..I wont necessarily be putting the hardware and all together but I will be funding it and overseeing it so this should help a good bit. We will be accessing the database from hopefully a download able software and a web based app. We would also like for people with mac and who are running out software to be able to access it to...would I run into any problems with that...?
Generally the interface will be the easiest part. However, depending on what type of database you go with, you should definitely make sure that it is easy to connect to it on a variety of platforms. (IE: Connecting to a MSSQL server from a Linux server is not fun!) I would definitely take some time to see what MAC's can easily connect to as you will most likely need to use a 3rd party ODBC or other driver to connect your application to the DB. Otherwise, as long as the software is properly designed to connect to the database, and there is a supported driver for the computer's OS, it shouldn't matter what type of computer is trying to access it. Since this is going to be accessed from different locations, it would be a good idea to have a hardware firewall on the internet side of the database to control all inbound and outbound traffic to the server.
Would there be any problems with starting to build the database now and then transfering it after everything is set up...?
Not a problem at all. Make sure you know how to migrate your database to a new server. The actual process varies with different types of DB's. Also, if you are using MySQL, I highly recommend using Navicat (http://www.navicat.com/). It makes managing indexes, and foreign keys, backups, transfers, and other general database tasks really easy.
That really depends on the application and how the data is being stored. As far as access, it's just as secure as the hardware and computer it's sitting on. This is the same with any database. For encryption, MySQL has built in encryption functions: http://dev.mysql.com/doc/refman/5.0/en/encryption-functions.html. It doesn't support some of the more advanced encryption that MSSQL and others support. MSSQL, DB2, and Oracle support far more advanced encryption. On windows, you can also encrypt the disk itself, so the data can be safe even if someone steals the entire computer. The concern you run into with any encryption situation is key management, because at some point a key must be used to retrieve the data. Depending on how/ how much you need to protect and how you need to do it, MySQL could be perfectly sufficient.
Too start the database...I know there are many ways. I was thinking that we could set up a form on one of the sites back pages so I could have multiple users start entering information that we already have into the database. Do you see any problem in that?