1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

can give an example of a good robots.txt to wordpress

Discussion in 'robots.txt' started by Olvia, Nov 13, 2014.

  1. #1
    hello :)

    I have problems with two wordpres websites that have the same robots.txt
    appear and disappear
    I dont know
    if I can maybe be by robots.txt
    SEMrush
    can give an example of a good robots.txt to wordpress?

    I have this for several years,
    is as follows:

    User-agent: Mediapartners-Google *
    Disallow:
    
    
    User-agent: *
    
    
    Disallow: / cgi-bin /
    Disallow: / page /
    Disallow: / search /
    Disallow: / tag /
    Disallow: / wp-admin /
    Disallow: / wp-includes /
    Disallow: / wp-content / plugins /
    Disallow: / wp-content / themes /
    Disallow: / wp
    Disallow: /? S =
    Disallow: / trackback /
    Disallow: / *?
    Disallow: * / trackback /
    
    Allow: / feed / $
    Disallow: / feed
    Disallow: / comments / feed
    Disallow: / * / feed / $
    Disallow: / * / feed / rss / $
    Disallow: / * / trackback / $
    Disallow: / * / * / feed / $
    Disallow: / * / * / feed / rss / $
    Disallow: / * / * / trackback / $
    Disallow: / * / * / * / feed / $
    Disallow: / * / * / * / feed / rss / $
    Disallow: / * / * / * / trackback / $
    
    User-agent: MSIECrawler
    Disallow: /
    
    User-agent: WebCopier
    Disallow: /
    
    User-agent: HTTrack
    Disallow: /
    
    User-agent: Microsoft.URL.Control
    Disallow: /
    
    User-agent: libwww
    Disallow: /
    Code (markup):
     
    Last edited by a moderator: Nov 13, 2014
    Olvia, Nov 13, 2014 IP
    SEMrush