I have a website that lists hundreds of events (football matches, car boot sales, tribute nights etc). When the events expire is there a way using the robots.txt that they can automatically be hidden from search engines?
Yes, you can add specific URLs, However if Google/S.E can find it through a different path it will index it. For example, you are hiding xyz.com/post1.html However in the robots.txt you do not hide the category "Fashion" under which the post is made. In this case the page xyz.com/fashion/ will be indexed or found. This means that the content in the URL that you wanted to be hidden will be found despite being explicitly mentioned in Robots.txt. There are a few more things that can get your URL found by google/SE. 1.Categories as i said 2.Tags 3.Archives 4.Page Number 5.Author posts / Profile link So, it is good to think before you try to hide URLs. Finally not all Bots obey Robots.txt