1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Google can't Reach any of my site's links + my robots file

Discussion in 'Site & Server Administration' started by YoYaYo, Nov 3, 2013.

  1. #1
    Hello everyone ;

    Google can't Reach any of my site's links + my robots file

    At the begging my site was fine suddenly Google webmasters sent me message at my mail said there is a problem at my robots file i checked it and found no problem it was as it was before nothing change ...

    then i contacted my server provider an said there is no problem from server side

    Now a lot of my links is unreachable and i got second message from google with the same unreachable robots file and three warnings with some of my links that is unreachable too

    Edit

    My site is : http://planeteups.com/

    The message from my Google webmaster

    "Over the last 24 hours, Googlebot encountered 9 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

    You can see more details about these errors in Webmaster Tools.
    Recommended action

    If the site error rate is 100%:
    • Using a web browser, attempt to access http://planeteups.com/robots.txt. If you are able to access it from your browser, then your site may be configured to deny access to googlebot. Check the configuration of your firewall and site to ensure that you are not denying access to googlebot.
    • If your robots.txt is a static page, verify that your web service has proper permissions to access the file.
    • If your robots.txt is dynamically generated, verify that the scripts that generate the robots.txt are properly configured and have permission to run. Check the logs for your website to see if your scripts are failing, and if so attempt to diagnose the cause of the failure.
    If the site error rate is less than 100%:
    • Using Webmaster Tools, find a day with a high error rate and examine the logs for your web server for that day. Look for errors accessing robots.txt in the logs for that day and fix the causes of those errors.
    • The most likely explanation is that your site is overloaded. Contact your hosting provider and discuss reconfiguring your web server or adding more resources to your website.
    • If your site redirects to another hostname, another possible explanation is that a URL on your site is redirecting to a hostname whose serving of its robots.txt file is exhibiting one or more of these issues.
    After you think you've fixed the problem, use Fetch as Google to fetch http://planeteups.com/robots.txt to verify that Googlebot can properly access your site."

    Please help me with this issue

    Thanks
    Last edited: Nov 3, 2013
    YoYaYo, Nov 3, 2013 IP
  2. Corestratagems

    Corestratagems Active Member

    Messages:
    122
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    53
    #2
    People will need a lot more information to be able to help. For starters the URL would be helpful. There are online places that can check the health of a server and DNS configuration to start. http://dnscheck.pingdom.com/ is one example to start with.
    Corestratagems, Nov 3, 2013 IP
  3. YoYaYo

    YoYaYo Greenhorn

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #3
    I have edited my thread please check now
    YoYaYo, Nov 3, 2013 IP
  4. Corestratagems

    Corestratagems Active Member

    Messages:
    122
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    53
    #4
    Were there any changes to your .htaccess file?
    Corestratagems, Nov 3, 2013 IP
  5. YoYaYo

    YoYaYo Greenhorn

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #5
    i did some changes :

    Code (Text):
    1. DirectoryIndex index.php
    2.  
    3. RewriteEngine On
    4. ErrorDocument 404 /404.php
    5. RewriteBase /
    6. RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
    7. RewriteRule ^(.*)$ http://%1/$1 [R=301,L]
    8.  
    9. RewriteRule ^page/(.*)$ index.php?cstart=$1 [L]
    10.  
    11. RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/page,([0-9]+),([0-9]+),(.*).html(/?)+$ index.php?subaction=showfull&year=$1&month=$2&day=$3&news_page=$4&cstart=$5&news_name=$6&seourl=$6 [L]
    12. RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/page,([0-9]+),(.*).html(/?)+$ index.php?subaction=showfull&year=$1&month=$2&day=$3&news_page=$4&news_name=$5&seourl=$5 [L]
    13. RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/print:page,([0-9]+),(.*).html(/?)+$ engine/print.php?subaction=showfull&year=$1&month=$2&day=$3&news_page=$4&news_name=$5&seourl=$5 [L]
    14. RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/(.*).html(/?)+$ index.php?subaction=showfull&year=$1&month=$2&day=$3&news_name=$4&seourl=$4 [L]
    15.  
    16. RewriteRule ^([^.]+)/page,([0-9]+),([0-9]+),([0-9]+)-(.*).html(/?)+$ index.php?newsid=$4&news_page=$2&cstart=$3&seourl=$5&seocat=$1 [L]
    17. RewriteRule ^([^.]+)/page,([0-9]+),([0-9]+)-(.*).html(/?)+$ index.php?newsid=$3&news_page=$2&seourl=$4&seocat=$1 [L]
    18. RewriteRule ^([^.]+)/print:page,([0-9]+),([0-9]+)-(.*).html(/?)+$ engine/print.php?news_page=$2&newsid=$3&seourl=$4&seocat=$1 [L]
    19. RewriteRule ^([^.]+)/([0-9]+)-(.*).html(/?)+$ index.php?newsid=$2&seourl=$3&seocat=$1 [L]
    20.  
    21. RewriteRule ^page,([0-9]+),([0-9]+),([0-9]+)-(.*).html(/?)+$ index.php?newsid=$3&news_page=$1&cstart=$2&seourl=$4 [L]
    22. RewriteRule ^page,([0-9]+),([0-9]+)-(.*).html(/?)+$ index.php?newsid=$2&news_page=$1&seourl=$3 [L]
    23. RewriteRule ^print:page,([0-9]+),([0-9]+)-(.*).html(/?)+$ engine/print.php?news_page=$1&newsid=$2&seourl=$3 [L]
    24. RewriteRule ^([0-9]+)-(.*).html(/?)+$ index.php?newsid=$1&seourl=$2 [L]
    25.  
    26. RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})(/?)+$ index.php?year=$1&month=$2&day=$3 [L]
    27. RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/page/([0-9]+)(/?)+$ index.php?year=$1&month=$2&day=$3&cstart=$4 [L]
    28.  
    29. RewriteRule ^([0-9]{4})/([0-9]{2})(/?)+$ index.php?year=$1&month=$2 [L]
    30. RewriteRule ^([0-9]{4})/([0-9]{2})/page/([0-9]+)(/?)+$ index.php?year=$1&month=$2&cstart=$3 [L]
    31.  
    32. RewriteRule ^([0-9]{4})(/?)+$ index.php?year=$1 [L]
    33. RewriteRule ^([0-9]{4})/page/([0-9]+)(/?)+$ index.php?year=$1&cstart=$2 [L]
    34.  
    35. RewriteRule ^tags/([^/]*)(/?)+$ index.php?do=tags&tag=$1 [L]
    36. RewriteRule ^tags/([^/]*)/page/([0-9]+)(/?)+$ index.php?do=tags&tag=$1&cstart=$2 [L]
    37.  
    38. RewriteRule ^xfsearch/([^/]*)(/?)+$ index.php?do=xfsearch&xf=$1 [L]
    39. RewriteRule ^xfsearch/([^/]*)/page/([0-9]+)(/?)+$ index.php?do=xfsearch&xf=$1&cstart=$2 [L]
    40.  
    41. RewriteRule ^user/([^/]*)/rss.xml$ engine/rss.php?subaction=allnews&user=$1 [L]
    42. RewriteRule ^user/([^/]*)(/?)+$ index.php?subaction=userinfo&user=$1 [L]
    43. RewriteRule ^user/([^/]*)/page/([0-9]+)(/?)+$ index.php?subaction=userinfo&user=$1&cstart=$2 [L]
    44. RewriteRule ^user/([^/]*)/news(/?)+$ index.php?subaction=allnews&user=$1 [L]
    45. RewriteRule ^user/([^/]*)/news/page/([0-9]+)(/?)+$ index.php?subaction=allnews&user=$1&cstart=$2 [L]
    46. RewriteRule ^user/([^/]*)/news/rss.xml(/?)+$ engine/rss.php?subaction=allnews&user=$1 [L]
    47.  
    48. RewriteRule ^lastnews(/?)+$ index.php?do=lastnews [L]
    49. RewriteRule ^lastnews/page/([0-9]+)(/?)+$ index.php?do=lastnews&cstart=$1 [L]
    50.  
    51. RewriteRule ^catalog/([^/]*)/rss.xml$ engine/rss.php?catalog=$1 [L]
    52. RewriteRule ^catalog/([^/]*)(/?)+$ index.php?catalog=$1 [L]
    53. RewriteRule ^catalog/([^/]*)/page/([0-9]+)(/?)+$ index.php?catalog=$1&cstart=$2 [L]
    54.  
    55. RewriteRule ^newposts(/?)+$ index.php?subaction=newposts [L]
    56. RewriteRule ^newposts/page/([0-9]+)(/?)+$ index.php?subaction=newposts&cstart=$1 [L]
    57.  
    58. RewriteRule ^favorites(/?)+$ index.php?do=favorites [L]
    59. RewriteRule ^favorites/page/([0-9]+)(/?)+$ index.php?do=favorites&cstart=$1 [L]
    60.  
    61. RewriteRule ^rules.html$ index.php?do=rules [L]
    62. RewriteRule ^statistics.html$ index.php?do=stats [L]
    63. RewriteRule ^addnews.html$ index.php?do=addnews [L]
    64. RewriteRule ^rss.xml$ engine/rss.php [L]
    65. RewriteRule ^sitemap.xml$ uploads/sitemap.xml [L]
    66.  
    67. RewriteCond %{REQUEST_FILENAME} !-d
    68. RewriteRule ^([^.]+)/page/([0-9]+)(/?)+$ index.php?do=cat&category=$1&cstart=$2 [L]
    69. RewriteRule ^([^.]+)/?$ index.php?do=cat&category=$1 [L]
    70.  
    71. RewriteCond %{REQUEST_FILENAME} !-f
    72. RewriteRule ^([^.]+)/rss.xml$ engine/rss.php?do=cat&category=$1 [L]
    73. RewriteRule ^page,([0-9]+),([^/]+).html$ index.php?do=static&page=$2&news_page=$1 [L]
    74. RewriteRule ^print:([^/]+).html$ engine/print.php?do=static&page=$1 [L]
    75.  
    76. RewriteCond %{REQUEST_FILENAME} !-f
    77. RewriteRule ^([^/]+).html$ index.php?do=static&page=$1 [L]
    78.  
    79. <FilesMatch "(?i)^.*\.(ico|flv|jpg|jpeg|png|gif|js|css|pdf)$">
    80. ExpiresActive On
    81. ExpiresDefault A2592000
    82. </FilesMatch>
    83. <ifmodule mod_deflate.c>
    84. AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css application/x-javascript application/javascript
    85. </ifmodule>
    Last edited: Nov 3, 2013
    YoYaYo, Nov 3, 2013 IP
  6. Corestratagems

    Corestratagems Active Member

    Messages:
    122
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    53
    #6
    When you're logged in to Webmaster tools under Crawl --> Blocked URLs, have you done a user agent test at the bottom? If that passes than it may have been a host down issue. (use something like pingdom.com to monitor uptime) If that fails then it might give you more info on what the issue is, like cannot find the file or error reading the file. If it cannot find the file, then change your .htaccess to only include the top portion and leave out all the rewrite rules except the www one and save it, and re-test. If that fixes the issue, you will have to add your rewrite rules one section at a time and re-test to see what section is causing the issue.

    If the test gives an error like cannot read file format or something then recreate the robots.txt and try with the new file.
    Corestratagems, Nov 3, 2013 IP
  7. YoYaYo

    YoYaYo Greenhorn

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #7
    I removed my robots file comp;etly and leave the upper part only of my .htaccess file Then retest my sitemap and this is came :

    #TypeIssueDescriptionIssues countExampleLineDetected
    1
    [​IMG]
    Errors
    Error
    We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guideline and can be accessed at the location you provided and then resubmit.
    1
    Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.
    -
    Nov 4, 2013
    YoYaYo, Nov 3, 2013 IP
  8. Corestratagems

    Corestratagems Active Member

    Messages:
    122
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    53
    #8
    Its true, I don't see anything at
    http://planeteups.com/sitemap.xml
    Did you have a sitemap at one point? Does it still exist? If so what are the permissions on the sitemap.xml?
    Corestratagems, Nov 3, 2013 IP
  9. YoYaYo

    YoYaYo Greenhorn

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #9
    My Site is DLE (data life engine) and have a lot of .htaccess that i never touch them so i think nothing wrong with them and also i think they are unrelative to my sitemap file and also this problem of "unreachable page" is for all my pages not only my sitemap file ... i called my server provider again and said there is no problem here ...
    YoYaYo, Nov 3, 2013 IP
  10. Corestratagems

    Corestratagems Active Member

    Messages:
    122
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    53
    #10
    The hosting provider most likely isn't going to care enough to see the problem or there is no problem when they connect to it from the same network (not unlikely) I would stop bothering to check with the hosting provider unless you know of a specific problem that is re-creatable to them that shows them the problem. Regardless of the bigger picture, you never answered the question about getting a 404 error when trying to get to the sitemap. Maybe it has nothing to do with the problem or maybe it has the same problem as other pages in your setup. Why can't I get to the sitemap.xml at http://planeteups.com/sitemap.xml ? Is the sitemap missing? Are the permissions incorrect? With that question answered you may get further along.
    Corestratagems, Nov 3, 2013 IP
  11. YoYaYo

    YoYaYo Greenhorn

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #11
    http://planeteups.com/uploads/sitemap.xml this is my sitemap which should be redirects to http://planeteups.com/sitemap.xml by htaccess file

    i tested http://planeteups.com/uploads/sitemap.xml on google but the same problem appears :(
    YoYaYo, Nov 3, 2013 IP
  12. Corestratagems

    Corestratagems Active Member

    Messages:
    122
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    53
    #12
    If you bring everything down to the simplest level you can as you have started to do. Remove robots.txt, use a simple .htaccess, check file permissions etc, in webmaster tools under Crawl --> Fetch as Google, can Google fetch any of your pages, including the main homepage? If not then you should look in to the hosting provider blocking Google IP's at the firewall. By removing the robots.txt and .htaccess and permissions from the mix, you are proving out that the issue isn't from anything under your control.
    Corestratagems, Nov 3, 2013 IP
  13. Corestratagems

    Corestratagems Active Member

    Messages:
    122
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    53
    #13
    Is this on a shared hosting account where you have other domains hosted on the same server? If so are any of them set up in webmaster tools as well? And can Google get to them without a problem? That would help narrow the issue further as well
    Corestratagems, Nov 3, 2013 IP
  14. Abiolas Gabriel

    Abiolas Gabriel Member

    Messages:
    73
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #14
    bro i clicked on your website links,nothing is working i got this message "PAGE NOT FOUND"
    Abiolas Gabriel, Nov 4, 2013 IP
  15. YoYaYo

    YoYaYo Greenhorn

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #15
    I just checked now ... The "Not Found Page" was due to .htaccess modifications now everything is going alright I Can't Believe My Self :)

    Thanks So much for helping me and your effort ..
    YoYaYo, Nov 4, 2013 IP
  16. Abiolas Gabriel

    Abiolas Gabriel Member

    Messages:
    73
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #16
    Ok,well if google can't still crawl your website,you can solve this in the webmaster tools.
    Login into webmaster at the left side go to CRAWL->fetch as google. Add a link from your
    site,if google can fetch the link,then the problem is not from you. And make sure you not
    blocking google crawler in your htaccess.
    Abiolas Gabriel, Nov 4, 2013 IP
  17. YoYaYo

    YoYaYo Greenhorn

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #17
    yes i did fetch as google and it successfully fetched i hope the problem not coming back

    Thanks
    YoYaYo, Nov 4, 2013 IP
  18. omkaar

    omkaar Member

    Messages:
    70
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #18
    dont see any problem with your robots.txt or sitemap now
    google should be able to crawl
    omkaar, Nov 5, 2013 IP
  19. Abiolas Gabriel

    Abiolas Gabriel Member

    Messages:
    73
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #19
    @YoYaYo that is good news,since you can fetch it on webmaster tools,then everything will is fine.
    Abiolas Gabriel, Nov 5, 2013 IP
  20. YoYaYo

    YoYaYo Greenhorn

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #20
    hehehehehehe Now Google can't reach my robots
    YoYaYo, Nov 5, 2013 IP