1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to block a particular folder from website from crawling

Discussion in 'Search Engine Optimization' started by elle19570, Oct 17, 2010.

  1. #1
    Hi,

    I would like to know how to block a particular folder from search engine crawler. I just want to block one folder only and not the entire website.

    for eg. www.website.com/folder-to-block/index.htm

    now how to disallow this "/folder-to-block/" folder by using using robot.txt?

    Thanks
     
    Last edited: Oct 17, 2010
    elle19570, Oct 17, 2010 IP
  2. internetproz

    internetproz Peon

    Messages:
    136
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    you can use this code to block search engine to crawl the folder

    User-agent: *
    Disallow: /folder-to-block/
     
    internetproz, Oct 17, 2010 IP
  3. merockysingh

    merockysingh Peon

    Messages:
    308
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Use this line: Disallow: /folder-to-block/ Thats it.. It will block all content of /folder-to-block/
     
    merockysingh, Oct 17, 2010 IP
  4. bestinktoner

    bestinktoner Peon

    Messages:
    24
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Create a robots.txt file in your server and put the code
    User-agent: *
    Disallow: /folder-name

    This will remove the folder from the indexing..
     
    bestinktoner, Oct 18, 2010 IP
  5. Baby Prakash

    Baby Prakash Peon

    Messages:
    232
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    If we can define Robot.txt file on webmaster tool it will be easy to index your WebPages .In robot text file we need to define which file of folder need not to be indexed.
     
    Baby Prakash, Oct 18, 2010 IP
  6. Puneet Singh

    Puneet Singh Peon

    Messages:
    76
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    yes webmaster tools is perfect place for do this ...
     
    Puneet Singh, Oct 18, 2010 IP
  7. BB-OM

    BB-OM Member

    Messages:
    371
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    35
    #7
    Also, you can make sure that the folder/page is not accessible/linked from any of the pages from the site other than configuring robots.txt.
     
    BB-OM, Oct 18, 2010 IP
  8. haveseo

    haveseo Well-Known Member

    Messages:
    995
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    103
    #8
    But this way doesn't work if the folder is indexed yet.
    If Google indexed this folder, the only way is defining it on robots.txt using Google webmaster tools or any other way, then when Google reads the robots.txt in next times, it will not indexed the pages on that folder more.
     
    haveseo, Oct 18, 2010 IP
  9. ROOFIS

    ROOFIS Well-Known Member

    Messages:
    1,234
    Likes Received:
    30
    Best Answers:
    5
    Trophy Points:
    120
    #9
    As the others are saying you could place a robots statement in the meta (example shown)
    in the header if the document is html.

    <html>
    <head>
    
    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW, NOARCHIVE">
    </head>
    
    Code (markup):

    If the content in question is a pdf, doc, txt or some other exotic file/doc type you can
    create a separate robots txt file on your site name.com/robots.txt or add a no index to
    your .htaccess (if you have one.)







    ROOFIS:cool:
     
    ROOFIS, Oct 18, 2010 IP
  10. denismark

    denismark Peon

    Messages:
    37
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Hello just Do this in robots.txt File

    User-agent: *
    Disallow: /folder-to-block(name Of Folder)/
     
    denismark, Oct 18, 2010 IP
  11. snyxius

    snyxius Peon

    Messages:
    22
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    I want block the pages instantly which is public...let me know how to do these...
     
    snyxius, Sep 5, 2012 IP