Homepage Only Index Conundrum

[JonahViaKeyboard]

I'm checking out this site: www.paulayoung.com and have absolutely NO IDEA why the site isn't completely indexed by Yahoo. Hundreds of pages are indexed in Google but only the homepage in Yahoo - which means it's not banned. The site is really old and has hundreds of inbound links, so it's not like it needs more time.

Has anyone seen anything like this before? Is there something obvious I'm missing? Any help would be very appreciated.

[jlawrence]

I have one site that has 4500+ pages indexed in Google, but only the homepage in Yahoo.
Sometimes, with newer sites, it seems to take an age before Yahoo gets of it's arse and shows the pages in its index.
At the moment, it's almost tempting to ban slurp from the site - all it's doing is wasting bandwidth. But then again Yahoo might produce something useful oneday - like an adsense alternative - I won't hold my breath but it might happen.

[JonahViaKeyboard]

Thanks jlawrence, can you give me any details about your site so we can see if our sites are similar? How old is it? Backlinks? Or could you just tell me the URL? Feel free to PM me if you'd rather not post. Thanks!

[Ian]

See these posts:

http://forums.digitalpoint.com/showthread.php?t=19086

http://forums.digitalpoint.com/showpost.php?p=242908&postcount=7

Ian

[wrmineo]

Try to make your code more bot friendly if not W3C compliant/valid.

For starters, your page code does not have a DOCTYPE or ENCODING. If the bot reaches your site and encounters problems, it may fail to index it or move from there.

Also, I'm not sure Yahoo! gets around java very well - having it as low in your code as possible might help too.

[jlawrence]

You could have a point there.
For some reason my pages are no longer validating - they should be and I'm sure they used to.
/me needs to remember to get that sorted. I consider valid markup to be the first thing you need to get right on a site.

[yfs1]

I think Yahoo is definately just slow in this respect.

For example:
www.articledepot.co.uk

A fairly new site (3+ months)

Google:
http://www.google.ie/search?hl=en&q=site%3Awww.articledepot.co.uk&meta=

Yahoo
http://search.yahoo.com/search?p=site%3Awww.articledepot.co.uk&fr=FP-tab-web-t&toggle=1&cop=&ei=UTF-8

All of my other sites started this way too. MSN is somewhere in the middle.

[jlawrence]

I have the adsense search available on that site and it was the code for that that was causing validation errors. For some reason that were quite a few of </input> tags. Anyway, the site validates again now, so I'll see if Yahoo gets of it's arse and adds the site to it's index.
At the moment, MSN is actually providing more traffic than G with Yahoo refering once this month. My site is about 2 months old.
Backlinks:
G 256
Y 1810
M 3113
Pages indexed:
G 4500
Y 1
M 2081

The traffic from Yahoo would be quite useful as the site is still sandboxed in G.

[JonahViaKeyboard]

The thing is this site isn't new: Yahoo is reporting 7,000+ links to this domain!

http://search.yahoo.com/search?p=linkdomain%3Awww.paulayoung.com&prssweb=Search&ei=UTF-8&fr=fp-pull-web-t&fl=0&x=wrt

The entire site is in the /catalog/ directory, but I'm pretty sure the robots.txt file looks good:

User-agent: *
Disallow: /catalog/assets/
Disallow: /catalog/track/
Disallow: /catalog/scripts/
Disallow: /catalog/site_graphics/
Disallow: /affiliates/manager/
Disallow: /secure_affiliates/manager/

Any other ideas? This is unrelated to the current Yahoo update, as this has been going on for months. Thanks again.

[jlawrence]

As wrmnieo said above, try and get the site to validate - see http://validator.w3.org/check?uri=http%3A%2F%2Fwww.paulayoung.com%2F&charset=%28detect+automatically%29&doctype=Inline&verbose=1
for details as to why it doesn't.
Also, make the links at the bottom more visible. It's possible that yahoo may be seeing them as hidden text - they're certainly only just visible in firefox.

[JonahViaKeyboard]

Thanks jlawrence. I know Y!'s most recent cache of the homepage includes those links at the bottom which point to static, simple pages outside of /catalog/ that should get indexed no problem - but those are relatively new. The real question is what's up with the rest of the site's 100s of pages.

I'll see what I can get going on validation. Still, this is about the most inexplicable thing I've ever seen.