robots.txt to prevent search engines from crawling/ indexing a page or folder

Discussion in 'robots.txt' started by tradester, Oct 13, 2011.

  1. #1
    Would appreciate if someone can show us example of robots.txt that will prevent search engines like Google from crawling and indexing a page or folder. So that the page will not show up on search engine results.
    Thanks :)
     
    tradester, Oct 13, 2011 IP
  2. sem-profiteer

    sem-profiteer Active Member

    Messages:
    257
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    75
    #2
    User-agent: *
    Disallow: /


    Google Webmaster Tools also has a great resource for this.
     
    sem-profiteer, Oct 20, 2011 IP
  3. ozrocks22

    ozrocks22 Member

    Messages:
    67
    Likes Received:
    4
    Best Answers:
    1
    Trophy Points:
    33
    #3
    And it's as easy as that!
     
    ozrocks22, Nov 1, 2011 IP
  4. rmvalues

    rmvalues Well-Known Member

    Messages:
    326
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    125
    #4
    or you can do it in your .htaccess file
     
    rmvalues, Nov 1, 2011 IP
  5. pr0t0n

    pr0t0n Well-Known Member

    Messages:
    243
    Likes Received:
    10
    Best Answers:
    10
    Trophy Points:
    128
    #5
    Something like this:
    
    User-agent: *
    Disallow: /folder_name/
    Disallow: /filename.php
    
    Code (markup):
    be very careful with previously mentioned example as that one may block EVERYTHING on your site, for ALL search engines:
    
    User-agent: *
    Disallow: /
    
    Code (markup):
    Cheers.
     
    pr0t0n, Nov 1, 2011 IP
  6. bobwarner01

    bobwarner01 Peon

    Messages:
    176
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Thanks for the post.Really quite interesting to know about the info.
     
    bobwarner01, Nov 4, 2011 IP