I'm trying to upload information to my database from a variety of different sources and I'm finding that I'm inserting a lot of duplicate info. Is there a way to prevent this from happening?
Check if it exist first I guess SQL: select count(column) from table where whatever = "term"; if count > 0, don't insert, otherwise do
Do the items have a unique identifier? If not, consider taking a hash of the values. Then mark one of the fields in the database table "unique" and perhaps use a "ON DUPLICATE KEY" clause in your query to handle dupes. deception's suggestion also works but you may want to make it more efficient by returning the entire table and storing the data and then comparing to the stored list instead of querying every time (resource intensive).