Just experimenting with setting up the regular delivery of a product catalog in CSV, which I plan to dump into my database on a daily basis, say (for the techie-minded, using the LOAD DATA INFILE in MySQL, which should do it quickly) and then use it in my store. The advertiser I am looking at has a very big catalog, (which is partly why I would use the LOAD DATA INFILE method), but I notice that lots of SKUs are identical, along with the rest of the data on those rows. Any idea what gives? Is that common, for an advertiser to have a lot of dupes in their catalog? Also, for the tech-minded... is there a way to use LOAD DATA INFILE to skip duplicates when importing, because they are really going to increase the size of my database unnecessarily? As I understand it, the IGNORE and REPLACE options refer only to existing records in the database, not the new ones you are importing, am I right?
Well, I can't really - there are maybe 150,000 records, so the whole reason I am importing the CSV file they provide using LOAD DATA INFILE is to put the minimum load on my (shared) hosting. If I start sorting through 150,000 records eliminating duplicates, then that defeats the point...
Well, you know, I really need to keep my post count up so I can get all that shared AdSense revenue, I'm sure you know how it is. Keeps me off the streets too.