I have Forum with a few thousand Posts. I guess there should be at least 5000-10000 different Pages. But only 800 Pages are indexed so far. The Site is 3 1/2 years old and I switched from phpbb to SMF about 6 months ago. So of course SE traffic decreased after I switched to the new Software. But my SE Traffic is quite bad, compared to the old Site. Even after months by far not all pages are indexed. I think there were more than 10000 Pages indexed when it was phpbb. Is anyone having an idea, what this could be? I tried the SEF Url Mod for SMF which seems to work quite well, but it is causing some errors, so I can not use it. Anyone willing to check my Site? http://bmx-lounge.com
When you changed over from phpbb to SMF what did you do to redirect the old URLs? Did you set up any mod_rewrite rules or an htaccess file or conf file with 301 redirects? Have you looked at your log files for any errors? Have you got a 404 handler?
If you really want to have a few of those pages specifically indexed then get as many links pointing to them as you can
This is probably the best explanation for why your pages are not being indexed. Also, if you can, try to have a dynamically created html sitemap linked to from the front page. This will ensure all the pages on your site have at least one link to them.
Thanks everyone, I did a mod_rewrite So a non existing Link (http://mywebsite.com/badlink.html) showed the content of the Main Page. After a while I realised that this created thousands of Mirror Pages. So I changed that and now it redirects to a 404 error page. But to be honest I'm not too familiar with .htaccess and all this. So whatever I do, I google it first to check how it's done. This is how my .htaccess look s like: RewriteEngine On RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d ErrorDocument 401 http://mysite.com/404.shtml ErrorDocument 402 http://mysite.com/404.shtml ErrorDocument 403 http://mysite.com/404.shtml ErrorDocument 404 http://mysite.com/404.shtml ErrorDocument 500 http://mysite.com/404.shtml RewriteRule (.*) index.php
I did a Site reconsideration in Google Webmaster Tools a few days ago. I will see if this helps. I always used my Site without the www, but I just switched to www-mysite, so maybe this will help. @createdevelop, I will link to my Sitemap from my Mainpage. Maybe this will work, but my Sitemap is in Google Webmaster Tools as well. Should I rather link to /index.php?action=sitemap (html version) or /index.php?action=sitemap;xml ?
One more thing. Is my robots.txt correct? User-agent: * Disallow: /*printpage Disallow: /index.php?action=printpage Disallow: /index.php?action=printpage* Sitemap: /index.php?action=sitemap;xml