1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Can I add specific urls to robots.txt to stop google indexing them?

Discussion in 'Google' started by David Lowe, Jun 25, 2013.

0
  1. #1
    I have a website that lists hundreds of events (football matches, car boot sales, tribute nights etc). When the events expire is there a way using the robots.txt that they can automatically be hidden from search engines?
     
    David Lowe, Jun 25, 2013 IP
  2. purpleorange

    purpleorange Active Member

    Messages:
    141
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    60
    #2
    Yes, you can add specific URLs, However if Google/S.E can find it through a different path it will index it.

    For example, you are hiding xyz.com/post1.html

    However in the robots.txt you do not hide the category "Fashion" under which the post is made. In this case the page xyz.com/fashion/ will be indexed or found. This means that the content in the URL that you wanted to be hidden will be found despite being explicitly mentioned in Robots.txt.

    There are a few more things that can get your URL found by google/SE.
    1.Categories as i said
    2.Tags
    3.Archives
    4.Page Number
    5.Author posts / Profile link

    So, it is good to think before you try to hide URLs. Finally not all Bots obey Robots.txt
     
    purpleorange, Jun 25, 2013 IP