Confused

I recently changed hosts and am in the middle of trying to work my way back up the rankings.

Only thing I'm confused about is the fact that my domain name doesn't appear in the rankings but my domain/index.html does.

If I do a site:domain search my pages appear but my domain doesn't unless I click on the ommited results included thing.

site:www.dsi-for-free.co.uk - Google Search

Anyone have any ideas why this might be?

 

One possible reason: the spiders tried to index your website and it was not available for some reason.

 

The reason this is occuring is that you have canonical issues. Every page should have one and only one URL for refering to the page. This is called the canonical or prefered URL. All other URLs used to reference that same page should be 301 redirected to the canonical URL. This prevents issues w/ duplicate content and split page rank.

For example, I can reference your home page as:

http://dsi-for-free.co.uk
http://dsi-for-free.co.uk/
http://dsi-for-free.co.uk/index.html
http://www.dsi-for-free.co.uk
http://www.dsi-for-free.co.uk/
http://www.dsi-for-free.co.uk/index.html

Google has your home page indexed several ways... Perform a search for site:dsi-for-free.co.uk (without the www). This gives you a better picture of what URLs are indexed for your site... Including the www causes the site: operator to ONLY show URLs indexed w/ www.

If 10 sites link to each of the above URLs, Google sees that as 6 different web pages w/ 10 inbound links from external sites each. This is bad... Google also knows that you are rendering the exact same content under multiple URLs so you have duplicate content issues. Google considers one of these URLs to be the originator of the content and the other 5 to be duplicate content. So the content on 5 of the URLs are being devalued.

You need to decide on rules for creating canonical URLs on your site and enforce them site wide... This is typically done using Mod Rewrite or some equivalent method where you can implement 301s. You need to decide on things like:

1) www vs non-www (http://example.com vs http://www.example.com)
2) show trailing '/' on folders containing default documents or hide the trailing slash (http://www.example.com/ or http://www.example.com)
3) show default document file names in folders containing default documents or hide the default document file name (http://www.example.com/index.html vs http://www.example.com/)
4) if you support HTTPS on your site, which pages should be HTTPS and which should be HTTP (you don't have this issue)

It doesn't matter which rules you pick for determining canonicals... You just need to decide and be consistent. Put 301 redirects in place to enforce the rules site-wide. I usually choose www, show trailing '/', and hide default document names so in the case of your site, MY preference would be
http://www.dsi-for-free.co.uk/. So I would implement the following 301 redirects to ensure that http://www.dsi-for-free.co.uk/ is always shown in the browser and shown for my home page URL to the spiders:

http://dsi-for-free.co.uk ---> 301 redirect ---> http://www.dsi-for-free.co.uk/
http://dsi-for-free.co.uk/ ---> 301 redirect ---> http://www.dsi-for-free.co.uk/
http://dsi-for-free.co.uk/index.html ---> 301 redirect ---> http://www.dsi-for-free.co.uk/
http://www.dsi-for-free.co.uk ---> 301 redirect ---> http://www.dsi-for-free.co.uk/
http://www.dsi-for-free.co.uk/ CANONICAL URL - NO REDIRECT REQUIRED
http://www.dsi-for-free.co.uk/index.html ---> 301 redirect ---> http://www.dsi-for-free.co.uk/

Now anytime someone links to you with any of these URLs, your web server will 301 redirect to your canonical URL... So http://www.dsi-for-free.co.uk/ is the only URL that is ever seen in the browser and the only one ever seen by the search engines... Because of this you'll ONLY have one URL indexed as your home page, you will have eliminated any duplicate content issues... AND http://www.dsi-for-free.co.uk/ will get credit for all inbound links for all 6 URLs. Google will now see this as a single web page with 60 inbound links instead of 6 pages with 10 inbounds each.

PS: Google, Yahoo! and MSN recently announce a less desirable way to handle canonical issues - the old <link> element with a new syntax and new rel="canonical" attribute. It is really made for sites that have difficulty implementing proper 301 redirects or those that cannot implement 301 redirects at all like sites that are pure HTML (no server side scripting) running on IIS instead of Apache (IIS doesn't come w/ Mod Rewrite functionality built in... well until IIS 7.0). 301 redirects are still the BEST choice... per Matt Cutts for obvious reasons once you understand You can add a link to each page of the form:

For example, if you place a <link rel="canonical" href="http://www.dsi-for-free.co.uk/ " in the <head> of your home page HTML... This tells Google regardless of what URL was used to render the home page or which content is rendered on the page, give http://www.dsi-for-free.co.uk/ credit for all inbound links and link text.

But 301s can do much more...

 

This is a brilliantly detailed reply.

Thankyou for that.
I'll get down to creating a 301 redirect for all of these.

Well after I've read up on google about how to do this.

 

that was an EXCELLENT explanation..hats off..thank you..

 

Is there any way of giving people rep on this forum?
That post definitely deserves it


Edit: Never mind, found it

 

Great post Canonical. The name suits.

I am one of the unfortunates that cannot use mod rewrite (IIS, HTML), so I have added the link rel="canonical" to most of the pages on one of my sites, (slowly getting there)

Here's Matt Cutts on the subject. http://www.youtube.com/watch?v=Cm9onOGTgeM

Cheers
James