how to disallow Robots

Discussion in 'robots.txt' started by MrPJH, Jan 7, 2010.

0
  1. #1
    i want to disallow Robots
    what i need to write
    quoted bellow is fine?
    also it ll be take effect to subdomains or not?
    because i can see them as directory in main domain
     
    MrPJH, Jan 7, 2010 IP
  2. sevnrock

    sevnrock Peon

    Messages:
    241
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    That's fine and you need a robots.txt in each subdomain folder.
     
    sevnrock, Jan 7, 2010 IP
  3. webizpro

    webizpro Guest

    Messages:
    42
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    An additional way to stop your content from being indexed is to use meta tags. You need to place the following tag in the <head> section of your pages.

    <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

    That's just a little more security against indexing. I like to use both.
     
    webizpro, Jan 7, 2010 IP
  4. Hatemaker

    Hatemaker Peon

    Messages:
    38
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    As long as you have that in all your top-level directories most reputable crawlers will ignore your site.
     
    Hatemaker, Jan 18, 2010 IP
  5. harrysmith

    harrysmith Well-Known Member

    Messages:
    466
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    110
    #5
    There are few robots who don't follow rules set in robots.txt file ;)
     
    harrysmith, Jan 20, 2010 IP
  6. Hatemaker

    Hatemaker Peon

    Messages:
    38
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Correct but short of IP banning them individually there really is no way to get rid of them.
     
    Hatemaker, Jan 20, 2010 IP