1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt explanation need?

Discussion in 'robots.txt' started by clive2, Apr 25, 2011.

0
  1. #1
    I have one sub domain which is having robots.txt file as follows

    User-agent: *
    Crawl-delay: 30 # Current Standard way to delay bot hits
    Request-rate: 1/30 # Extended Standard to set bot hit rate
    Visit-time: 1500-1700 # Extended Standard to set bot hit time range in UTC
    Disallow: /javascripts/
    Disallow: /stylesheets/
    Disallow: /images/

    and at last sitemap url
    Can any one explain me about crawl delay, request rate and visit time.
     
    clive2, Apr 25, 2011 IP
  2. Balaji Naidu

    Balaji Naidu Active Member

    Messages:
    245
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #2
    Robots.txt is one of the best way to guide the crawler how to look to the website. Crawl-delay, Request-rate, Visit-time are one such guide to rule the crawlers.

    Crawl-delay: It defines how many seconds to wait after each succesful crawling.

    Request-rate: defines pages/seconds to be crawled ratio. 1/30 would be 1 page in every 30 second.

    Visit-time: you can define between which hours you want your pages to be crawled. Example usage is: 1500-1700 which means that pages will be indexed between 03:00 PM – 05:00 PM GMT.
     
    Balaji Naidu, Apr 25, 2011 IP
    manish.chauhan likes this.
  3. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #3
    Nicely explained :)
     
    manish.chauhan, May 4, 2011 IP