I have a website that lists hundreds of events (football matches, car boot sales, tribute nights etc). When the events expire is there a way using the robots.txt that they can automatically be hidden from search engines?
I'm assuming you are using a database to track the event status I also assume that when the event is expired you change a value in the database(for example active = false) Pseudo-code for this: if event is not active write new line to robots.txt with text "Disallow: /event/id/" Or something like that, if you ask a php developer he could probably whip you up a script like that in an hour or less.