1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt to prevent search engines from crawling/ indexing a page or folder

Discussion in 'robots.txt' started by tradester, Oct 13, 2011.

0
  1. #1
    Would appreciate if someone can show us example of robots.txt that will prevent search engines like Google from crawling and indexing a page or folder. So that the page will not show up on search engine results.
    Thanks :)
     
    tradester, Oct 13, 2011 IP
  2. sem-profiteer

    sem-profiteer Active Member

    Messages:
    257
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    75
    #2
    User-agent: *
    Disallow: /


    Google Webmaster Tools also has a great resource for this.
     
    sem-profiteer, Oct 20, 2011 IP
  3. ozrocks22

    ozrocks22 Member

    Messages:
    67
    Likes Received:
    4
    Best Answers:
    1
    Trophy Points:
    33
    #3
    And it's as easy as that!
     
    ozrocks22, Nov 1, 2011 IP
  4. rmvalues

    rmvalues Well-Known Member

    Messages:
    326
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    125
    #4
    or you can do it in your .htaccess file
     
    rmvalues, Nov 1, 2011 IP
  5. pr0t0n

    pr0t0n Well-Known Member

    Messages:
    243
    Likes Received:
    10
    Best Answers:
    10
    Trophy Points:
    128
    #5
    Something like this:
    
User-agent: *
Disallow: /folder_name/
Disallow: /filename.php
    Code (markup):
    be very careful with previously mentioned example as that one may block EVERYTHING on your site, for ALL search engines:
    
User-agent: *
Disallow: /
    Code (markup):
    Cheers.
     
    pr0t0n, Nov 1, 2011 IP
  6. bobwarner01

    bobwarner01 Peon

    Messages:
    176
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Thanks for the post.Really quite interesting to know about the info.
     
    bobwarner01, Nov 4, 2011 IP