what all Robots.txt directives FAST supports ??

Discussion in 'robots.txt' started by mahavir, Dec 30, 2008.

0
  1. #1
    We want to create a robots.txt for our site for FAST Enterprise Search to Crawl accordingly?

    To specify the locations to crawl and the other to not, we would like to know what all Robots.txt directives FAST supports.

    Does it support 'Allow', 'Request-rate', 'Crawl-Delay', 'Visit-Time ?
     
    mahavir, Dec 30, 2008 IP
  2. FightRice

    FightRice Peon

    Messages:
    1,082
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Why not email them and ask directly to get a for sure answer
     
    FightRice, Jan 4, 2009 IP
  3. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #3
    Robots.txt supports all the parameters..:)
     
    manish.chauhan, Jan 20, 2009 IP