robots.txt help?

Discussion in 'Programming' started by mattmdesign, Sep 23, 2006.

  1. #1
    Hi,

    On one of my hosting accounts I have a main domain, then several addon domains. Some of the URL's inside of the sites on the addon domains were accidentally pointing to a page using the subdomain (for example, they were using addonsite.mainsite.com/directory instead of http://www.addonsite.com/directory), so now in Google, a :site search for my main domain shows several addonsite.mainsite.com/directory listings, and I would like them to drop those pages out of the index. I can't use noindex nofollow because I still want those pages to be indexed using the addon domain's URL. Is there a way to get rid of those listings using robots.txt? If so, what would I put in the robots.txt file?

    Thanks, Matt
     
    mattmdesign, Sep 23, 2006 IP
  2. Evoleto

    Evoleto Well-Known Member

    Messages:
    253
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    108
    #2
    If you use PHP you should consider the following approach for the those pages in the <head> section:

    
    if( $_SERVER["HTTP_HOST"] === "addonsite.mainsite.com" ) {
        echo "<meta name='robots' content='noindex, nofollow'>";
    }
    
    PHP:
     
    Evoleto, Sep 24, 2006 IP
  3. wmtips

    wmtips Well-Known Member

    Messages:
    601
    Likes Received:
    70
    Best Answers:
    1
    Trophy Points:
    150
    #3
    I would recommend to set up the 301 permanent redirect with .htaccess like this:
    RewriteCond %{HTTP_HOST} addonsite\.mainsite\.com
    RewriteRule (.*)  http://www.addonsite.com/$1 [R=301,L] 
    Code (markup):
    This might help to save Google PR if you have inbound links to addonsite.mainsite.com/directory.
     
    wmtips, Sep 25, 2006 IP
  4. Evoleto

    Evoleto Well-Known Member

    Messages:
    253
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    108
    #4
    Correct, I subscribe to Wmtips' solution - better than mine :D
     
    Evoleto, Sep 25, 2006 IP
  5. mattmdesign

    mattmdesign Well-Known Member

    Messages:
    1,201
    Likes Received:
    77
    Best Answers:
    0
    Trophy Points:
    118
    #5
    Thanks! Would this go in mainsite.com/.htaccess, or addonsite.com/.htaccess?

    Thanks, Matt
     
    mattmdesign, Sep 25, 2006 IP
  6. wmtips

    wmtips Well-Known Member

    Messages:
    601
    Likes Received:
    70
    Best Answers:
    1
    Trophy Points:
    150
    #6
    mainsite.com/.htaccess
     
    wmtips, Sep 25, 2006 IP
  7. mattmdesign

    mattmdesign Well-Known Member

    Messages:
    1,201
    Likes Received:
    77
    Best Answers:
    0
    Trophy Points:
    118
    #7
    Thanks, but I'm not able to get it working. Here is what I have:

    RewriteCond %{HTTP_HOST} addonsite\.mainsite\.com/directory
    RewriteRule (.*) http://www.addonsite.com/directory/$1 [R=301,L]

    but it is not redirecting. I also tried it without the /directory parts and the subdomain was not directing to the new url either.
     
    mattmdesign, Sep 25, 2006 IP
  8. wmtips

    wmtips Well-Known Member

    Messages:
    601
    Likes Received:
    70
    Best Answers:
    1
    Trophy Points:
    150
    #8
    There is no /directory string in my code example. HTTP_HOST contains only host name without path information, so you need to omit the /directory.
     
    wmtips, Sep 25, 2006 IP
  9. mattmdesign

    mattmdesign Well-Known Member

    Messages:
    1,201
    Likes Received:
    77
    Best Answers:
    0
    Trophy Points:
    118
    #9
    I tried that too and it didn't change anything.
     
    mattmdesign, Sep 26, 2006 IP
  10. wmtips

    wmtips Well-Known Member

    Messages:
    601
    Likes Received:
    70
    Best Answers:
    1
    Trophy Points:
    150
    #10
    Forgot to mention: you need the line
    at the .htaccess beginning to enable rewriting.
     
    wmtips, Sep 26, 2006 IP
  11. mattmdesign

    mattmdesign Well-Known Member

    Messages:
    1,201
    Likes Received:
    77
    Best Answers:
    0
    Trophy Points:
    118
    #11
    Now it is giving a 404 error instead of redirecting when I go to addonsite.mainsite.com.
     
    mattmdesign, Sep 27, 2006 IP