Below is a sample robots.txt file for multiple user agents with multiple crawl delays, one for each user agent. The Crawl-delay values are for illustration purposes and will be different in a real robots.txt file. I have searched all over the web for proper answers but could not find one. There are too many mixed suggestions and I do not know which is the correct / proper method. Questions: (1) Can each user agent have it's own crawl-delay? (I assume yes) (2) Where do you put the crawl-delay line for each user agent, before or after the Allow / Dissallow line? (3) Does there have to be a blank like between each user agent group? (4) If I want to set all of the user agents to have crawl delay of 10 seconds, would the code at the bottom be correct? References: http://www.seopt.com/2013/01/robots-text-file/ http://help.yandex.com/webmaster/?id=1113851#1113858 Essentially, I am looking to find out how the final robots.txt file should look using the values in the sample below. Thanks in advance. # Allow only major search spiders User-agent: Mediapartners-Google Disallow: Crawl-delay: 11 User-agent: Googlebot Disallow: Crawl-delay: 12 User-agent: Adsbot-Google Disallow: Crawl-delay: 13 User-agent: Googlebot-Image Disallow: Crawl-delay: 14 User-agent: Googlebot-Mobile Disallow: Crawl-delay: 15 User-agent: MSNBot Disallow: Crawl-delay: 16 User-agent: bingbot Disallow: Crawl-delay: 17 User-agent: Slurp Disallow: Crawl-delay: 18 User-agent: Yahoo! Slurp Disallow: Crawl-delay: 19 # Block all other spiders User-agent: * Disallow: / # Block Directories for all spiders User-agent: * Disallow: /ads/ Disallow: /cgi-bin/ Disallow: /scripts/ (4) If I want to set all of the user agents to have crawl delay of 10 seconds, would the following be correct? # Allow only major search spiders User-agent: * Crawl-delay: 10 User-agent: Mediapartners-Google Disallow: User-agent: Googlebot Disallow: User-agent: Adsbot-Google Disallow: User-agent: Googlebot-Image Disallow: User-agent: Googlebot-Mobile Disallow: User-agent: MSNBot Disallow: User-agent: bingbot Disallow: User-agent: Slurp Disallow: User-agent: Yahoo! Slurp Disallow: # Block all other spiders User-agent: * Disallow: / # Block Directories for all spiders User-agent: * Disallow: /ads/ Disallow: /cgi-bin/ Disallow: /scripts/