Google can't Reach any of my site's links + my robots file

Hello everyone ;

Google can't Reach any of my site's links + my robots file

At the begging my site was fine suddenly Google webmasters sent me message at my mail said there is a problem at my robots file i checked it and found no problem it was as it was before nothing change ...

then i contacted my server provider an said there is no problem from server side

Now a lot of my links is unreachable and i got second message from google with the same unreachable robots file and three warnings with some of my links that is unreachable too

Edit

My site is : http://planeteups.com/

The message from my Google webmaster

"Over the last 24 hours, Googlebot encountered 9 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

You can see more details about these errors in Webmaster Tools.
Recommended action

If the site error rate is 100%:

• Using a web browser, attempt to access http://planeteups.com/robots.txt. If you are able to access it from your browser, then your site may be configured to deny access to googlebot. Check the configuration of your firewall and site to ensure that you are not denying access to googlebot.
• If your robots.txt is a static page, verify that your web service has proper permissions to access the file.
• If your robots.txt is dynamically generated, verify that the scripts that generate the robots.txt are properly configured and have permission to run. Check the logs for your website to see if your scripts are failing, and if so attempt to diagnose the cause of the failure.

If the site error rate is less than 100%:

• Using Webmaster Tools, find a day with a high error rate and examine the logs for your web server for that day. Look for errors accessing robots.txt in the logs for that day and fix the causes of those errors.
• The most likely explanation is that your site is overloaded. Contact your hosting provider and discuss reconfiguring your web server or adding more resources to your website.
• If your site redirects to another hostname, another possible explanation is that a URL on your site is redirecting to a hostname whose serving of its robots.txt file is exhibiting one or more of these issues.

After you think you've fixed the problem, use Fetch as Google to fetch http://planeteups.com/robots.txt to verify that Googlebot can properly access your site."

Please help me with this issue

Thanks

 

People will need a lot more information to be able to help. For starters the URL would be helpful. There are online places that can check the health of a server and DNS configuration to start. http://dnscheck.pingdom.com/ is one example to start with.

 

I have edited my thread please check now

 

Were there any changes to your .htaccess file?

 

i did some changes :

DirectoryIndex index.php

RewriteEngine On
ErrorDocument 404 /404.php
RewriteBase /
RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]

RewriteRule ^page/(.*)$ index.php?cstart=$1 [L]

RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/page,([0-9]+),([0-9]+),(.*).html(/?)+$ index.php?subaction=showfull&year=$1&month=$2&day=$3&news_page=$4&cstart=$5&news_name=$6&seourl=$6 [L]
RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/page,([0-9]+),(.*).html(/?)+$ index.php?subaction=showfull&year=$1&month=$2&day=$3&news_page=$4&news_name=$5&seourl=$5 [L]
RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/print:page,([0-9]+),(.*).html(/?)+$ engine/print.php?subaction=showfull&year=$1&month=$2&day=$3&news_page=$4&news_name=$5&seourl=$5 [L]
RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/(.*).html(/?)+$ index.php?subaction=showfull&year=$1&month=$2&day=$3&news_name=$4&seourl=$4 [L]

RewriteRule ^([^.]+)/page,([0-9]+),([0-9]+),([0-9]+)-(.*).html(/?)+$ index.php?newsid=$4&news_page=$2&cstart=$3&seourl=$5&seocat=$1 [L]
RewriteRule ^([^.]+)/page,([0-9]+),([0-9]+)-(.*).html(/?)+$ index.php?newsid=$3&news_page=$2&seourl=$4&seocat=$1 [L]
RewriteRule ^([^.]+)/print:page,([0-9]+),([0-9]+)-(.*).html(/?)+$ engine/print.php?news_page=$2&newsid=$3&seourl=$4&seocat=$1 [L]
RewriteRule ^([^.]+)/([0-9]+)-(.*).html(/?)+$ index.php?newsid=$2&seourl=$3&seocat=$1 [L]

RewriteRule ^page,([0-9]+),([0-9]+),([0-9]+)-(.*).html(/?)+$ index.php?newsid=$3&news_page=$1&cstart=$2&seourl=$4 [L]
RewriteRule ^page,([0-9]+),([0-9]+)-(.*).html(/?)+$ index.php?newsid=$2&news_page=$1&seourl=$3 [L]
RewriteRule ^print:page,([0-9]+),([0-9]+)-(.*).html(/?)+$ engine/print.php?news_page=$1&newsid=$2&seourl=$3 [L]
RewriteRule ^([0-9]+)-(.*).html(/?)+$ index.php?newsid=$1&seourl=$2 [L]

RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})(/?)+$ index.php?year=$1&month=$2&day=$3 [L]
RewriteRule ^([0-9]{4})/([0-9]{2})/([0-9]{2})/page/([0-9]+)(/?)+$ index.php?year=$1&month=$2&day=$3&cstart=$4 [L]

RewriteRule ^([0-9]{4})/([0-9]{2})(/?)+$ index.php?year=$1&month=$2 [L]
RewriteRule ^([0-9]{4})/([0-9]{2})/page/([0-9]+)(/?)+$ index.php?year=$1&month=$2&cstart=$3 [L]

RewriteRule ^([0-9]{4})(/?)+$ index.php?year=$1 [L]
RewriteRule ^([0-9]{4})/page/([0-9]+)(/?)+$ index.php?year=$1&cstart=$2 [L]

RewriteRule ^tags/([^/]*)(/?)+$ index.php?do=tags&tag=$1 [L]
RewriteRule ^tags/([^/]*)/page/([0-9]+)(/?)+$ index.php?do=tags&tag=$1&cstart=$2 [L]

RewriteRule ^xfsearch/([^/]*)(/?)+$ index.php?do=xfsearch&xf=$1 [L]
RewriteRule ^xfsearch/([^/]*)/page/([0-9]+)(/?)+$ index.php?do=xfsearch&xf=$1&cstart=$2 [L]

RewriteRule ^user/([^/]*)/rss.xml$ engine/rss.php?subaction=allnews&user=$1 [L]
RewriteRule ^user/([^/]*)(/?)+$ index.php?subaction=userinfo&user=$1 [L]
RewriteRule ^user/([^/]*)/page/([0-9]+)(/?)+$ index.php?subaction=userinfo&user=$1&cstart=$2 [L]
RewriteRule ^user/([^/]*)/news(/?)+$ index.php?subaction=allnews&user=$1 [L]
RewriteRule ^user/([^/]*)/news/page/([0-9]+)(/?)+$ index.php?subaction=allnews&user=$1&cstart=$2 [L]
RewriteRule ^user/([^/]*)/news/rss.xml(/?)+$ engine/rss.php?subaction=allnews&user=$1 [L]

RewriteRule ^lastnews(/?)+$ index.php?do=lastnews [L]
RewriteRule ^lastnews/page/([0-9]+)(/?)+$ index.php?do=lastnews&cstart=$1 [L]

RewriteRule ^catalog/([^/]*)/rss.xml$ engine/rss.php?catalog=$1 [L]
RewriteRule ^catalog/([^/]*)(/?)+$ index.php?catalog=$1 [L]
RewriteRule ^catalog/([^/]*)/page/([0-9]+)(/?)+$ index.php?catalog=$1&cstart=$2 [L]

RewriteRule ^newposts(/?)+$ index.php?subaction=newposts [L]
RewriteRule ^newposts/page/([0-9]+)(/?)+$ index.php?subaction=newposts&cstart=$1 [L]

RewriteRule ^favorites(/?)+$ index.php?do=favorites [L]
RewriteRule ^favorites/page/([0-9]+)(/?)+$ index.php?do=favorites&cstart=$1 [L]

RewriteRule ^rules.html$ index.php?do=rules [L]
RewriteRule ^statistics.html$ index.php?do=stats [L]
RewriteRule ^addnews.html$ index.php?do=addnews [L]
RewriteRule ^rss.xml$ engine/rss.php [L]
RewriteRule ^sitemap.xml$ uploads/sitemap.xml [L]

RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^.]+)/page/([0-9]+)(/?)+$ index.php?do=cat&category=$1&cstart=$2 [L]
RewriteRule ^([^.]+)/?$ index.php?do=cat&category=$1 [L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^.]+)/rss.xml$ engine/rss.php?do=cat&category=$1 [L]
RewriteRule ^page,([0-9]+),([^/]+).html$ index.php?do=static&page=$2&news_page=$1 [L]
RewriteRule ^print:([^/]+).html$ engine/print.php?do=static&page=$1 [L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^/]+).html$ index.php?do=static&page=$1 [L]

<FilesMatch "(?i)^.*\.(ico|flv|jpg|jpeg|png|gif|js|css|pdf)$">
ExpiresActive On
ExpiresDefault A2592000
</FilesMatch>
<ifmodule mod_deflate.c>
AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css application/x-javascript application/javascript
</ifmodule>

Code (markup):

 

When you're logged in to Webmaster tools under Crawl --> Blocked URLs, have you done a user agent test at the bottom? If that passes than it may have been a host down issue. (use something like pingdom.com to monitor uptime) If that fails then it might give you more info on what the issue is, like cannot find the file or error reading the file. If it cannot find the file, then change your .htaccess to only include the top portion and leave out all the rewrite rules except the www one and save it, and re-test. If that fixes the issue, you will have to add your rewrite rules one section at a time and re-test to see what section is causing the issue.

If the test gives an error like cannot read file format or something then recreate the robots.txt and try with the new file.

 

I removed my robots file comp;etly and leave the upper part only of my .htaccess file Then retest my sitemap and this is came :

#TypeIssueDescriptionIssues countExampleLineDetected
1

Errors
Error
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guideline and can be accessed at the location you provided and then resubmit.
1
Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at the root of your site but were unable to download it. Please ensure that it is accessible or remove it completely.
-
Nov 4, 2013

 

Its true, I don't see anything at
http://planeteups.com/sitemap.xml
Did you have a sitemap at one point? Does it still exist? If so what are the permissions on the sitemap.xml?

 

My Site is DLE (data life engine) and have a lot of .htaccess that i never touch them so i think nothing wrong with them and also i think they are unrelative to my sitemap file and also this problem of "unreachable page" is for all my pages not only my sitemap file ... i called my server provider again and said there is no problem here ...

 

The hosting provider most likely isn't going to care enough to see the problem or there is no problem when they connect to it from the same network (not unlikely) I would stop bothering to check with the hosting provider unless you know of a specific problem that is re-creatable to them that shows them the problem. Regardless of the bigger picture, you never answered the question about getting a 404 error when trying to get to the sitemap. Maybe it has nothing to do with the problem or maybe it has the same problem as other pages in your setup. Why can't I get to the sitemap.xml at http://planeteups.com/sitemap.xml ? Is the sitemap missing? Are the permissions incorrect? With that question answered you may get further along.

 

http://planeteups.com/uploads/sitemap.xml this is my sitemap which should be redirects to http://planeteups.com/sitemap.xml by htaccess file

i tested http://planeteups.com/uploads/sitemap.xml on google but the same problem appears

 

If you bring everything down to the simplest level you can as you have started to do. Remove robots.txt, use a simple .htaccess, check file permissions etc, in webmaster tools under Crawl --> Fetch as Google, can Google fetch any of your pages, including the main homepage? If not then you should look in to the hosting provider blocking Google IP's at the firewall. By removing the robots.txt and .htaccess and permissions from the mix, you are proving out that the issue isn't from anything under your control.

 

Is this on a shared hosting account where you have other domains hosted on the same server? If so are any of them set up in webmaster tools as well? And can Google get to them without a problem? That would help narrow the issue further as well

 

bro i clicked on your website links,nothing is working i got this message "PAGE NOT FOUND"

 

I just checked now ... The "Not Found Page" was due to .htaccess modifications now everything is going alright I Can't Believe My Self

Thanks So much for helping me and your effort ..

 

Ok,well if google can't still crawl your website,you can solve this in the webmaster tools.
Login into webmaster at the left side go to CRAWL->fetch as google. Add a link from your
site,if google can fetch the link,then the problem is not from you. And make sure you not
blocking google crawler in your htaccess.

 

yes i did fetch as google and it successfully fetched i hope the problem not coming back

Thanks

 

dont see any problem with your robots.txt or sitemap now
google should be able to crawl

 

@YoYaYo that is good news,since you can fetch it on webmaster tools,then everything will is fine.

 

hehehehehehe Now Google can't reach my robots