On our homepage, we have several articles from different categories within our site (joomla). When we add a new article, we also post an intro on the homepage. The title links it to the full page. Our recent entries (last 2 weeks) are not being indexed by google. Only what is on the homepage. The full article is not being indexed at all. Why is this happening and how can I fix it. Any help is much appreciated.
indexed? or returned? they're two different things. Returned page - Google will see the snippet of the article on your homepage (which is 50% of your article content by the way, which is too much). It will also the the same content on another page. Given that google will always return your homepage IF it sees the relevant content in there, theres no reason for it to return your article page. Its already done its job and found the info in your homepage. Indexed page: same goes.. the content is near enough full on there in your homepage.. it may not see a reason to index your article, or maybe its sandboxed (more likely) due to content duplication. If it was me i'd shorten my snippets to at least half of what they currently are, if not less. All they need is a couple of sentences. This should stop the problem.
One thing that you could try is to create a more search engine friendly sitemap, preferably an xml sitemap. Then upload that sitemap to Google Webmaster Tools. You might also want to specify that sitemap in your robots.txt file. If this doesn't help getting those new pages indexed, you could try uploading that new sitemap to Google immediately after adding the page to your site. Anothing thing that will help is to build up a stronger link profile for your site. If you have lots of good links, it becomes easier to get more of your site indexed. This is because you would get less pages getting pushed down into Google's supplemental index. That is where many low PR pages end up. I do agree with freeflyer that you might be including too much of your articles on your homepage. You should try either including less on the homepage or write longer articles.
Unique content sitemap.xml strong back link in high page ranked site ( bookmarking sites ) bookmark each pages
Try digging your own content, or even right squidoos and put your link on the page. Sites like Digg, StumbleUpon are highly indexed by google. I myself after getting content on my site and getting it ready I posted it on digg and stumbleupon it was indexed 3 days later by almost every search engine I know. I now have 2/10 PR with a domain that is 1month old!
yes its funny but you should do that for your site page rank bookmarking sites can give strong back links, thats why.
theres only so many times that you can read 'digg' and 'sitemaps' are the solution to everything before you shoot yourself. I'm loading the gun now. Anyway, that aside.. this was intriguing me.. so i ran a little test on your site with xenu. Heres the results.. ok 321 URLs 55.73% forbidden request 134 URLs 23.26% no info to return 1 URLs 0.17% not found 76 URLs 13.19% SSL certificate common name incorrect 1 URLs 0.17% skip type 43 URLs 7.47% Total 576 URLs 100.00% The forbidden requests were random pages. This isnt good for spiders. It seems your server is set to block multiple requests from the same IP too.. this may mean the indexing will be hit and miss as to what pages get indexed and cached. A spider is essentially an IP the same as everything else. For good indexing a link check such as xenu would show all pages returning an ok status. you may be blocking effective indexing at the server. Your server people will dispute this of course, but one thing is for certain and that's that you need a clean report when running xenu.
[shoots himself] to all those saying sitemaps.. http://www.dsflyfishing.com/sitemap.xml . I think its already taken care of.
So, we are still having the issue which is maddening since it is just two of us. Freeflyer - thanks for you input. We have been using a third party program to add keywords in each page url. Now you have me thinking that this program may be doing some of the blocking. Maybe I should get a site map
the prog itself wont block.. this will be a server setting. most servers are set to trigger alerts (and block) if multiple requests are made from the same IP.. yours seems set very low. Theres no way a xenu check would trigger blocks normally though, so perhaps its down to something else. Either way, its server related.