Urgent: Robots.txt flle

Discussion in 'Search Engine Optimization' started by Loy Maben, Nov 3, 2008.

0
  1. #1
    Hello
    pls tell me the meaning of following content in robots.txt file::

    User-agent: *
    Disallow: /cgi-bin/ # This is an infinite virtual URL space
    Disallow: /stats/ # these will soon disappear

    Thanks in advance
     
    Loy Maben, Nov 3, 2008 IP
  2. kenny_pure

    kenny_pure Active Member

    Messages:
    604
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    58
    #2
    Disallow: /cgi-bin/

    thats mean you are not giving the permission to crawlers to crawl pages in : /cgi-bin/
     
    kenny_pure, Nov 3, 2008 IP
  3. rena

    rena Peon

    Messages:
    1,987
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Its disallowing Google bot to crawl the specific directories, whatever u add on it
     
    rena, Nov 4, 2008 IP
  4. stevegrant

    stevegrant Peon

    Messages:
    46
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    That is just to stop crawler to crawl particular pages
     
    stevegrant, Nov 4, 2008 IP
  5. Bloomtools

    Bloomtools Peon

    Messages:
    1,361
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #5
    This is stop the crawler to crawl, what ever you mention there.
     
    Bloomtools, Nov 4, 2008 IP
  6. poseidon

    poseidon Banned

    Messages:
    4,356
    Likes Received:
    246
    Best Answers:
    0
    Trophy Points:
    0
    #6
    You are stopping search engine bots to crawl those directories. Robots.txt files are supposed to direct search engines. DP has a full section devoted to it which you can read.
     
    poseidon, Nov 4, 2008 IP