can give an example of a good robots.txt to wordpress

Discussion in 'robots.txt' started by Olvia, Nov 13, 2014.

  1. #1
    hello :)

    I have problems with two wordpres websites that have the same robots.txt
    appear and disappear
    I dont know
    if I can maybe be by robots.txt

    can give an example of a good robots.txt to wordpress?

    I have this for several years,
    is as follows:

    User-agent: Mediapartners-Google *
    Disallow:
    
    
    User-agent: *
    
    
    Disallow: / cgi-bin /
    Disallow: / page /
    Disallow: / search /
    Disallow: / tag /
    Disallow: / wp-admin /
    Disallow: / wp-includes /
    Disallow: / wp-content / plugins /
    Disallow: / wp-content / themes /
    Disallow: / wp
    Disallow: /? S =
    Disallow: / trackback /
    Disallow: / *?
    Disallow: * / trackback /
    
    Allow: / feed / $
    Disallow: / feed
    Disallow: / comments / feed
    Disallow: / * / feed / $
    Disallow: / * / feed / rss / $
    Disallow: / * / trackback / $
    Disallow: / * / * / feed / $
    Disallow: / * / * / feed / rss / $
    Disallow: / * / * / trackback / $
    Disallow: / * / * / * / feed / $
    Disallow: / * / * / * / feed / rss / $
    Disallow: / * / * / * / trackback / $
    
    User-agent: MSIECrawler
    Disallow: /
    
    User-agent: WebCopier
    Disallow: /
    
    User-agent: HTTrack
    Disallow: /
    
    User-agent: Microsoft.URL.Control
    Disallow: /
    
    User-agent: libwww
    Disallow: /
    Code (markup):
     
    Last edited by a moderator: Nov 13, 2014
    Olvia, Nov 13, 2014 IP