Robots.txt goes in which directory?

Discussion in 'robots.txt' started by fadetoblack22, Jul 6, 2009.

  1. #1
    I have a test site that has been indexed, but I don't want it to be, so I have used robots.txt to stop it being indexed. After more than a week it is still indexed.

    I have used this in the root:

    User-Agent: *
    Disallow: /

    Say if a bot enters my site higher than the root, how does it know not to index the files?
     
    fadetoblack22, Jul 6, 2009 IP
  2. fadetoblack22

    fadetoblack22 Well-Known Member

    Messages:
    2,399
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    160
    #2
    any help here?
     
    fadetoblack22, Jul 8, 2009 IP
  3. DemCage

    DemCage Peon

    Messages:
    77
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    The robots.txt does not prevent robots from indexing your site. Most of the bots will follow your restrictions in your robots.txt, but some might not care.

    You probably have to wait more than one week if your page is already indexed. ;)
     
    DemCage, Jul 8, 2009 IP
  4. TheMagpie

    TheMagpie Active Member

    Messages:
    225
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #4
    Pretty sure you need to request it be removed. Once it has a backlink it gets crawled.
     
    TheMagpie, Jul 8, 2009 IP
  5. fadetoblack22

    fadetoblack22 Well-Known Member

    Messages:
    2,399
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    160
    #5
    I am referring to google who does follow the robots.txt
     
    fadetoblack22, Jul 8, 2009 IP
  6. fadetoblack22

    fadetoblack22 Well-Known Member

    Messages:
    2,399
    Likes Received:
    62
    Best Answers:
    0
    Trophy Points:
    160
    #6
    But when the bot comes around again it will see the robots file, yeh?
     
    fadetoblack22, Jul 8, 2009 IP
  7. TheMagpie

    TheMagpie Active Member

    Messages:
    225
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    53
    #7
    I run bans and wordpress sites and have tried to stop the bot from crawling the store and it keeps coming.
    That's my experience.
     
    TheMagpie, Jul 8, 2009 IP