1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Can I Hide Duplicate Content By Robots.txt

Discussion in 'robots.txt' started by Sumitthewriter, Oct 28, 2010.

0
  1. #1
    I have a few pages on my website with duplicate content.
    These pages are not yet indexed and i have blocked the access to those particular pages for crawlers in robots.txt.

    Is there a chance that Google will still look into the file and flag my website??

    I am confused. Please help!
     
    Sumitthewriter, Oct 28, 2010 IP
  2. AtomicPages

    AtomicPages Peon

    Messages:
    38
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Googlebots will follow the rules you have set in your robots.txt file. Make sure the duplicate content is not included in your sitemap.xml file and all should be well! Be sure your robots.txt file is set correctly!
    http://tool.motoricerca.info/robots-checker.phtml
     
    AtomicPages, Oct 31, 2010 IP
  3. Sudhaa

    Sudhaa Peon

    Messages:
    25
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I suggest you to follow Google guidelines and avoid having duplicate content in your website. It is not sure your pages will not be indexed if its blocked using robots. Use "noindex, nofollow" meta tags in the pages.
     
    Sudhaa, Nov 1, 2010 IP
  4. kwikmed

    kwikmed Guest

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Can you explain why you have duplicate content in general? Why not just delete the duplicate content before it gets indexed?

    If you can't delete the duplicate pages.. then I would suggest adding the canonical tag to them in the header, telling Google which one is the one you want indexed.
     
    kwikmed, Nov 9, 2010 IP
  5. pibc

    pibc Guest

    Messages:
    20
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    If duplicate content is there for indexing of pages. There are multiple ways to "spin" your content and make old content "as new". Make sure to not take entire texts, but only sections form each and spin them entirely.
     
    pibc, Nov 10, 2010 IP
  6. techvivek

    techvivek Peon

    Messages:
    24
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    use only noindex and nofollow it will do your job pretty easy.
     
    techvivek, Dec 31, 2010 IP
  7. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #7
    use noindex and nofollow as if the crawlers are coming to your web page from a third party source they may skip the robots.txt of your website and might index the duplicate pages.
     
    manish.chauhan, Jan 4, 2011 IP
  8. fsdnetwork

    fsdnetwork Peon

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    To be 100% sure I recommend add the meta tag robots on these pages.

    <META name="Robots" content="NOINDEX,NOFOLLOW">

    I recommend delete these pages ( problematic to future )
     
    fsdnetwork, Feb 9, 2011 IP
  9. cuteboy

    cuteboy Active Member

    Messages:
    229
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    53
    #9
    Yes you can hide the duplicate content by using the robots.txt, but if any search engine crawl your content and check it is duplicate then your site never be high page rank, it will goes down in search engine. So dont use this type of risky work.
     
    cuteboy, Feb 13, 2011 IP
  10. ragnamaniac1

    ragnamaniac1 Peon

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Hi,
    having duplicate content on site is not good for site it will affect the site very badly in search engine....
     
    ragnamaniac1, Feb 14, 2011 IP
  11. designer23

    designer23 Peon

    Messages:
    17
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #11
    duplicate content in a site reduces the search engine rankings....

    you may avoid it by doing:
    <META name="Robots" content="NOINDEX,NOFOLLOW">

    this will go between the
    <head>
    </head>
    portion of ur html page....

    and u can also put a screenshot of your text (if it does not contain any links )....this is because <nofollow> tag is not liked much by google.....

    again if u want to hide ur links ie you dont want search engine crawlers to go to another site following your link then just add fel="nofollow" in the link < a href="siteurl" rel="nofollow" >link</a>
     
    designer23, Mar 3, 2011 IP
  12. shakac

    shakac Peon

    Messages:
    41
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Thank you atomicpage for give this useful robots check link.i checked my site and got some error.
    thank you again.
     
    shakac, Mar 7, 2011 IP
  13. shakac

    shakac Peon

    Messages:
    41
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    thank you for your this checking link.thank you very much.
     
    shakac, Mar 7, 2011 IP
  14. Sujeet Yadav

    Sujeet Yadav Peon

    Messages:
    31
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    <META name="Robots" content="NOINDEX,NOFOLLOW">
     
    Sujeet Yadav, Mar 7, 2011 IP
  15. ACME Squares

    ACME Squares Peon

    Messages:
    98
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #15
    Google isn't going to "flag your website", all that will happen is one version will be picked over the other.
    Use rel="canonical" to tell which version you want google to pick.
     
    ACME Squares, Mar 15, 2011 IP