robots.txt - How to ban everyone except Google

Discussion in 'robots.txt' started by lim (x² - 5x³) = -∞, Mar 27, 2008.

0
Page 2 of 2
  1. jasp

    jasp Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #21
    This is not true if the robot is standards compliant. I'd be surprised if the big 5 didn't understand the official robots.txt standard.

    To allow only specific robots, like those from google you can use
    
User-agent: *
Disallow: /

User-agent: Googlebot
User-agent: Googlebot-Mobile
User-agent: Googlebot-Image
User-agent: Mediapartners-Google
User-agent: Adsbot-Google
Disallow:
    Code (markup):
    Remembering that the 'allow' directive is non-standard.
     
    jasp, Jun 2, 2008 IP
  2. dermax

    dermax Peon

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #22
    I can confirm jasp's solution.
     
    dermax, Jun 2, 2008 IP
Page 2 of 2