1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Webmaster tools robots.txt problems

Discussion in 'Google' started by Lucas Cardoso, May 4, 2014.

  1. #1
    Hi everyone, I just moved my wordpress site to a new host this week, I got a error message at webmaster tools during that time that google bot couldn't access my robots.txt, problaby because we suffered a couple of hours of downtime. After the migration I fetched robots.txt it as google and it says success but the error message "Google couldn't crawl your site because we weren't able to access robots.txt" keeps staying there.

    This week it the warning changed to a yellow exclamation point. Anyone have any experience in getting this error message to go away? After the date of the errors there are no red dots or blue dots that says failed attempts on my webmaster tool graph.

    I should mention that the only thing at my robots.txt currently is

    #Begin Attracta SEO Tools Sitemap. Do not remove
    sitemap: cdn.attracta.com/sitemap/url.xml.gz
    #End Attracta SEO Tools Sitemap. Do not remove

    Anyone with any experience with this kind of issues? Is there something wrong with my robots.txt? How long does it usually take for webmaster tools to show that the issue have been solved?

    I have attached the graph from my webmaster tool. Any insight would be much appreciated.
     

    Attached Files:

    Lucas Cardoso, May 4, 2014 IP
  2. Sarkari Naukri

    Sarkari Naukri Member

    Messages:
    54
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    43
    #2
    I have the same issued faced a year before, My hosting provider told, that they don't have any hosting issue, but I have faced lot of downtime, thus this issue happenes, Ask your hosting provider and keep watch on site downtime. Also check your robot.txt file, it should be clean...
     
    Sarkari Naukri, May 4, 2014 IP
  3. Kartic

    Kartic Active Member

    Messages:
    54
    Likes Received:
    8
    Best Answers:
    1
    Trophy Points:
    65
    #3
    sometime in after changing host, google still determine same thing and won't update error or graph for some days. in few days check this again and you'll be fine with your site permissions :)
     
    Kartic, May 5, 2014 IP
  4. JoeBikerul

    JoeBikerul Greenhorn

    Messages:
    15
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    13
    #4
    Try to remove robots.txt and rewrite it.
     
    JoeBikerul, May 5, 2014 IP
  5. Joshie

    Joshie Notable Member

    Messages:
    482
    Likes Received:
    50
    Best Answers:
    1
    Trophy Points:
    205
    #5
    Add this on the top of your robot.txt:
    User-agent: *
    Code (markup):
    So that it allows all of search engines to crawl your website and if you have specific folders on your host that you don't want any search engines to crawl then you would have to change and add like in this example:
    User-agent: *
    Disallow: /media
    Code (markup):
    /media is a example and often people would use that folder to upload images, videos and audios in that directory but if you use that code it would prevent all search engines from crawling and index your multimedia files on their search engines for anyone to steal.

    You can add Disallow: then put any folders or pages that you do NOT want any search engines to crawl and possibly index but if you want crawlers to crawl the entire of your website freely then go with the first code I provided above.

    But if you have migrated websites to a new web hosting, give it a time and that warning/alert message will go away as it may take few days or weeks. Took about a week for me.
     
    Joshie, May 5, 2014 IP
  6. Reviva Clinic

    Reviva Clinic Greenhorn

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #6
    you go to your word press web site setting and found search setting update their your robots file .For more information http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress
     
    Reviva Clinic, Aug 6, 2014 IP