1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Pages getting indexed despite diallowing them with robots.txt

Discussion in 'robots.txt' started by Alfred_ans, Sep 13, 2005.

  1. #1
    Hi!

    I have a website www.mydomain.com that deals in gift items.We had used rd in the above site. The urls with rd in it has been disallowed from search engines with the help of robots.txt.

    So after using rd the URL : www.mydomain.com/kids/funtoys looks like www.mydomain.com/rd/?/tt2 . Despite disallowing this specific page from being indexed by google, we were surprised to find that google has actually indexed this page in the rd format only.So a keyword like "funtoys" will show the url:www.mydomain.com/rd/?/tt2 without any meta description or title.

    In order to avoid this we have now changed the same url to www.mydomain.com/rd2/?/Kids/funtoys. In this case rd2 has also been disallowed by using robots.txt.

    Now could anybody pls tell me why in the first instance did google index my page despite using rd which had been actually diallowed by using robots.txt. and secondly do i have a risk of getting indexed again after using rd2 in the above format.

    Pls help
     
    Alfred_ans, Sep 13, 2005 IP
  2. Kim Steinhaug

    Kim Steinhaug Peon

    Messages:
    22
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I'm sure you would make it alot more easy by includeing your robots.txt file.
     
    Kim Steinhaug, Sep 13, 2005 IP