robots.txt help

Discussion in 'Google' started by lovetospooge, Sep 6, 2011.

0
  1. #1
    Hi,

    what would I add in the robots.txt file if I don't want a bot to crawl a
    directory for example: /data/ and all the folders and files in that directory example /data/2010/ - /data/2011/. I just need something that would not crawl anything in that directory because there is thousands of files in.

    lovetospooge out :cool:
     
    lovetospooge, Sep 6, 2011 IP
  2. basickno

    basickno Member

    Messages:
    422
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    48
    #2
    use below code

    User-agent: *
    Disallow: /data/
     
    basickno, Sep 6, 2011 IP