Are there any issues with robots.txt and Wordpress?

Discussion in 'WordPress' started by samie, Jan 27, 2014.

0
  1. #1
    I keep getting a message in my Google Webmaster Tools that my robots.txt is blocking all the links for my website.

    Here robots.txt looks like this:
    User-agent: Googlebot-Image
    Disallow: /

    User-agent: *
    Allow: /


    And I don't have it checked to discourage search engines in the dashboard. I even checked the pages for no-index/no-follow. I even tried deactivating my one plugin which is a Google Sitemap plugin.

    Any ideas?
     
    samie, Jan 27, 2014 IP
  2. homer7

    homer7 Well-Known Member

    Messages:
    268
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    125
    #2
    homer7, Jan 27, 2014 IP
  3. Nigel Lew

    Nigel Lew Notable Member

    Messages:
    4,642
    Likes Received:
    406
    Best Answers:
    21
    Trophy Points:
    295
    #3
    User-agent: Googlebot-Image stops google from including stuff in image search. Unless its NSFW that is not a real sound idea.

    Nigel
     
    Nigel Lew, Jan 27, 2014 IP
    samie likes this.
  4. samie

    samie Member

    Messages:
    269
    Likes Received:
    8
    Best Answers:
    1
    Trophy Points:
    45
    #4
    Thank you for the help. I think I got it figured it now. Seems like it was all more of a propagation issue with Google or something. I tried resubmitting today and there were no errors. It might have been because I was originally blocking search engines while in development.
     
    samie, Jan 27, 2014 IP
  5. RobinInTexas

    RobinInTexas Active Member

    Messages:
    217
    Likes Received:
    14
    Best Answers:
    2
    Trophy Points:
    65
    #5
    There are plugins for WordPress that will dynamically generate a robots.txt when requested, I would strongly advise against using them but create an actual robots.txt file, due to the additional load created on the WordPress engine.
     
    RobinInTexas, Feb 2, 2014 IP