Hi Guys, I am building a scraper and its going to add roughly about 20K of rows everyday on a table called 'tbl_posts'. I am planning to write a bulk insert statements with PHP for a set of about 100 data or so. Is this a good idea? Also if I do that, i will have to remove unique field from the table because bulk insert will fail other wise if it finds duplicate entry and whole 100 data wont be saved. How do I deal with this? Any insight would be very much appreciated. Thanks, Amit
I do not think you would need bulk insert for just 20K entries. Simply connect once before loop, execute queries, and disconnect. That is if you have all 20K ready for inserts. But in case of scrapper, there will always be few milliseconds to seconds gap between each insert and hence its better to keep doing single inserts. Table structure depends on the type of data you are storing within. 20K rows per day is not much of data to create any problem. I would suggest to proceed with single inserts and optimal table design and then see if you really need to empower it.