Hi, My website Homepage return a 404 to the spiders. I cna't find the problem. Is there someone that could help me to sort this out? I lost my listing on google because of that. Some people looked at my htaccess. They says it ok from there. But here is anyways: ExpiresActive On ExpiresByType image/gif A2592000 Options +FollowSymLinks RewriteEngine On RewriteBase / #RewriteCond %{HTTP_HOST} ^x10media.com #RewriteRule (.*) http://www.x10media.com/$1 [R=301,L] RewriteCond %{http_host} ^x10media.com$ [nc] RewriteRule ^(.*)$ http://www.x10media.com/$1 [r=301,nc,L] ErrorDocument 404 /404.php Redirect 301 /scripts.php http://www.x10media.com/scripts Redirect 301 /image-hosting-script.php http://www.x10media.com/image-hosting-script Redirect 301 /mp3-script.php http://www.x10media.com/mp3-script Redirect 301 /media-script.php http://www.x10media.com/media-script Redirect 301 /micro-blog-script.php http://www.x10media.com/micro-blog-script Redirect 301 /micro-blogging-script.php http://www.x10media.com/micro-blog-script Redirect 301 /filehost-script.php http://www.x10media.com/filehost-script Redirect 301 /add-ons.php http://www.x10media.com/add-ons Redirect 301 /general-add-ons.php http://www.x10media.com/general-add-ons Redirect 301 /contact.php http://www.x10media.com/contact Redirect 301 /webmaster-resources.php http://www.x10media.com/webmaster-resources Redirect 301 /new http://www.x10media.com # Uncomment following line if your webserver's URL # is not directly related to physical file paths. # Update Your Joomla!/MamboDirectory (just / for root) RewriteRule ^mp3-script$ index.php?option=com_content&task=view&id=5&Itemid=9 [L] RewriteRule ^image-hosting-script$ index.php?option=com_content&task=view&id=14&Itemid=9 [L] RewriteRule ^media-script$ index.php?option=com_content&task=view&id=15&Itemid=9 [L] RewriteRule ^micro-blog-script$ index.php?option=com_content&task=view&id=16&Itemid=9 [L] RewriteRule ^filehost-script$ index.php?option=com_content&task=view&id=17&Itemid=9 [L] RewriteRule ^add-ons$ index.php?option=com_content&task=view&id=18&Itemid=9 [L] RewriteRule ^general-add-ons$ index.php?option=com_content&task=view&id=27&Itemid=9 [L] RewriteRule ^ms-add-ons$ index.php?option=com_content&task=view&id=19&Itemid=9 [L] RewriteRule ^mp3-add-ons$ index.php?option=com_content&task=view&id=20&Itemid=9 [L] RewriteRule ^general-add-ons$ index.php?option=com_content&task=view&id=20&Itemid=9 [L] RewriteRule ^webmaster-resources$ index.php?option=com_content&task=view&id=29&Itemid=9 [L] RewriteRule ^contact$ index.php?option=com_content&task=view&id=21&Itemid=9 [L] RewriteRule ^scripts$ index.php?option=com_content&task=view&id=12&Itemid=9 [L] RewriteRule ^AboutUs$ index.php?option=com_content&task=view&id=23&Itemid=9 [L] RewriteRule ^news$ index.php?option=com_content&task=blogsection&id=0&Itemid=9 [L] RewriteRule ^sitemap$ /index.php?option=com_xmap&sitemap=1 [L] RewriteRule ^power-scripts-information$ index.php?option=com_content&task=view&id=30&Itemid=9 [L] ########## Begin - Joomla! core SEF Section ############# Use this section if using ONLY Joomla! core SEF ## ALL (RewriteCond) lines in this section are only required if you actually ## have directories named 'content' or 'component' on your server ## If you do not have directories with these names, comment them out. # RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d #RewriteCond %{REQUEST_URI} ^(/component/option,com) [NC,OR] ##optional - see notes## RewriteCond %{REQUEST_URI} (/|\.htm|\.php|\.html|/[^.]*)$ [NC] RewriteRule ^(content/|component/) index.php # ########## End - Joomla! core SEF Section ########## Begin - 3rd Party SEF Section ############# Use this section if you are using a 3rd party (Non Joomla! core) SEF extension - e.g. OpenSEF, 404_SEF, 404SEFx, SEF Advance, etc # #RewriteCond %{REQUEST_URI} ^(/component/option,com) [NC,OR] ##optional - see notes## #RewriteCond %{REQUEST_URI} (/|\.htm|\.php|\.html|/[^.]*)$ [NC] #RewriteCond %{REQUEST_FILENAME} !-f #RewriteCond %{REQUEST_FILENAME} !-d #RewriteRule (.*) index.php # ########## End - 3rd Party SEF Section ########## Begin - Rewrite rules to block out some common exploits ## If you experience problems on your site block out the operations listed below ## This attempts to block the most common type of exploit `attempts` to Joomla! # # Block out any script trying to set a mosConfig value through the URL RewriteCond %{QUERY_STRING} mosConfig_[a-zA-Z_]{1,21}(=|\%3D) [OR] # Block out any script trying to base64_encode crap to send via URL RewriteCond %{QUERY_STRING} base64_encode.*\(.*\) [OR] # Block out any script that includes a <script> tag in URL RewriteCond %{QUERY_STRING} (\<|%3C).*script.*(\>|%3E) [NC,OR] # Block out any script trying to set a PHP GLOBALS variable via URL RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR] # Block out any script trying to modify a _REQUEST variable via URL RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2}) # Send all blocked request to homepage with 403 Forbidden error! RewriteRule ^(.*)$ index.php [F,L] # ########## End - Rewrite rules to block out some common exploi Code (markup): Test result for http header: [B]HTTP Headers Test[/B] [B]Domain tested:[/B]www.x10media.com[B]Test performed from:[/B]Seattle, WA[B]Test performed at:[/B]2009-03-25 13:57:27 (GMT -07:00)[B]Status:[/B][COLOR=red]404 - Not Found[/COLOR] [B]Response Time:[/B]0.713 sec [B]DNS:[/B]0.278 sec [B]Connect:[/B]0.091 sec [B]Redirect:[/B]0.000 sec [B]First byte:[/B]0.344 sec [B]Last byte:[/B]0.000 sec [B]HTTP Header:[/B] HTTP/1.0 404 Not Found Date: Wed, 25 Mar 2009 20:57:09 GMT Server: Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8b mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.8 X-Powered-By: PHP/5.2.8 Set-Cookie: bde85fcb1bac46ec83e9c239910b6c1a=- Set-Cookie: mosvisitor=1 Connection: close content-Type: text/html Code (markup): My robots.txt: User-agent: * Allow: / Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /editor/ Disallow: /help/ Disallow: /includes/ Disallow: /language/ Disallow: /mambots/ Disallow: /media/ Disallow: /adult/ Disallow: /modules/ Disallow: /templates/ Disallow: /installation/ Disallow: /new/ Code (markup): Help!
Nothing worng with robot txt more like your .htaccess.. rename your htaccess and try run HTTP headers test again. Try remove unecessary line from your htaccess..
I tested without an .htaccess and it keeps getting a 404 in the http header. My website is on Joomla. After thinking on the result of what you asked me. I've check in the index.php of Joomla. I found that: ob_start(); if ($path = $mainframe->getPath( 'front' )) { $task = strval( mosGetParam( $_REQUEST, 'task', '' ) ); $ret = mosMenuCheck( $Itemid, $option, $task, $gid ); if ($ret) { require_once( $path ); } else { mosNotAuth(); } } else { // header( 'HTTP/1.0 404 Not Found' ); echo _NOT_EXIST; } PHP: As you can see, I commented the else for the 404 and it fixed the problem. I am no good at php, so I am not even sure what I did but it works and the real 404 also works... Thanks!