Can I add specific urls to robots.txt to stop google indexing them?

Discussion in 'robots.txt' started by David Lowe, Jun 25, 2013.

  1. #1
    I have a website that lists hundreds of events (football matches, car boot sales, tribute nights etc). When the events expire is there a way using the robots.txt that they can automatically be hidden from search engines?
     
    David Lowe, Jun 25, 2013 IP
  2. Helge Sverre

    Helge Sverre Prominent Member Affiliate Manager

    Messages:
    840
    Likes Received:
    99
    Best Answers:
    2
    Trophy Points:
    305
    Digital Goods:
    2
    #2
    I'm assuming you are using a database to track the event status
    I also assume that when the event is expired you change a value in the database(for example active = false)
    Pseudo-code for this:

    if event is not active
    write new line to robots.txt with text "Disallow: /event/id/"

    Or something like that, if you ask a php developer he could probably whip you up a script like that in an hour or less.
     
    Helge Sverre, Jun 28, 2013 IP
  3. BabyDoll123

    BabyDoll123 Banned

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #3
    yes sure you can add specific URLs to stop Google indexing them
    Use this string >
    Disallow: / URL
     
    BabyDoll123, Jun 19, 2014 IP