How to prevent search engine bots crawl my site as duplicated contents?

Discussion in 'Site & Server Administration' started by Sxperm, Feb 10, 2006.

0
  1. #1
    Hi, my friends, I've used htaccess + mod_rewrite to rewrite some dynamic URLs from refer.php?id=XX ---> Tell-a-friend-ID-XX.html

    It's work very well but one thing happened. Many bots crawl both of them and that's a duplicated contents. How to prevent bots crawl my dynamic page but only URL I've rewritten for. I tested by many sitemap generator software and they're crawl my site like I mentioned. :D

    Cheers,
    Sxperm
     
    Sxperm, Feb 10, 2006 IP
  2. ServerUnion

    ServerUnion Peon

    Messages:
    3,611
    Likes Received:
    296
    Best Answers:
    0
    Trophy Points:
    0
    #2
    You have one step done. Next you need to disallow the robots from indexing your old URL's. Updating your robots.txt file will help with this. Here is a help file on robots.txt to help with this process. Good luck
     
    ServerUnion, Feb 10, 2006 IP
  3. AWD1

    AWD1 Peon

    Messages:
    191
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I'm not sure how to do this in PHP or mod_rewrite (I know in ASP, that's it) but did you make sure to include the 301 response status in the header of your old referring URL? The 301 status code tells the SEs "hey, the page moved and it's over here now. We're not restoring this page ever again, so please delete."
     
    AWD1, Feb 11, 2006 IP