1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Meta noindex OR robots.txt OR rel=nofollow

Discussion in 'robots.txt' started by cellphone_store, May 5, 2005.

  1. #1
    I have a website and I noticed that some pages that I "disallow"ed in the robots.txt are showing up in Google, but without the title and cache.

    What code would make Google or any SE not even list those pages in their SERP?

    Robots.txt? (i found out this didn't work)
    Meta noindex ?
    rel="nofollow" ?
     
    cellphone_store, May 5, 2005 IP
  2. City2

    City2 Peon

    Messages:
    542
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #2
    meta no index works fine. Not sure about nofollow
     
    City2, May 5, 2005 IP
  3. dcristo

    dcristo Illustrious Member

    Messages:
    19,776
    Likes Received:
    1,199
    Best Answers:
    7
    Trophy Points:
    470
    Articles:
    7
    #3
    Did you implement the robots.txt file previous to the pages getting indexed?
     
    dcristo, May 6, 2005 IP
  4. J.D.

    J.D. Peon

    Messages:
    1,198
    Likes Received:
    65
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Post your robots.txt file - it may be misconfigured (e.g. you have to use absolute paths, etc). If you believe that is not, you should contact Google and let them know about this problem.

    J.D.
     
    J.D., May 6, 2005 IP
  5. cellphone_store

    cellphone_store Peon

    Messages:
    22
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    User-agent: *

    Disallow: /account.php
    Disallow: /conditions.php
    Disallow: /contact_us.php
    Disallow: /create_account.php
    Disallow: /login.php
    Disallow: /logoff.php
    Disallow: /password_forgotten.php
    Disallow: /privacy.php
    Disallow: /shopping_cart.php

    ------------------
    the robots.txt was implemented before google even found my website.
     
    cellphone_store, May 6, 2005 IP
  6. ServerUnion

    ServerUnion Peon

    Messages:
    3,611
    Likes Received:
    296
    Best Answers:
    0
    Trophy Points:
    0
    #6
    I do not believe you have to use the "/" before filenames, only directories...
     
    ServerUnion, May 6, 2005 IP
  7. J.D.

    J.D. Peon

    Messages:
    1,198
    Likes Received:
    65
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I see nothing wrong with this file. I would contact Google and let them know about this. It might be useful for them if you actually include a few log lines that show how Google got robots.txt first and then ignored its instructions and requested the disallowed pages. Also, I believe that Googlebot retrieves robots.txt only once in a day (may be once in a few hours), so the request for this file may be quite far in the log file from the page request.

    J.D.
     
    J.D., May 6, 2005 IP
  8. J.D.

    J.D. Peon

    Messages:
    1,198
    Likes Received:
    65
    Best Answers:
    0
    Trophy Points:
    0
    #8
    You have to use absolute paths in robots.txt. Relative paths (i.e. those that don't have a leading slash) are not allowed. There's no distinction between files and directories - bots simply match each line in robots.txt to the beginning of the actual path.

    J.D.
     
    J.D., May 6, 2005 IP
  9. ServerUnion

    ServerUnion Peon

    Messages:
    3,611
    Likes Received:
    296
    Best Answers:
    0
    Trophy Points:
    0
    #9
    even after you ad the needed text to the file or robots.txt, it will take a while before the pages are removed from the listing....
     
    ServerUnion, May 6, 2005 IP