Google's Ability To Spider 'Difficult' URLS

Hi all,

Not sure whether it's down to the latest new Bot they have running around but it seems to me like there's no mountain to big to clime when it comes to URL spidering. Look at this one with multiple variables, most of them named with the 'definite no-no' letters "ID".

[nourl]http://www.argos.co.uk/webapp/wcs/stores/servlet/ArgosBrowseCounts?storeId=10001&catalogId=2501&langId=-1&categoryId=16419[/nourl]

Indexed perfectly fine and ranks really well for a particular, fairly competitive, KW.

Any reason left for us database programmers to be careful with variables in URLs?

 

Just better safe than sorry I think. I have several variables in my urls but I make sure than none are "id". Since yours are not just "ID", you're probably fine.

 

YES
there are many reasons why to be careful in presenting URLs and pages
facts - real facts
look at your own access_log stats and look at the % of
G - quiet sophisticated bot
Y - bot failry OK
MSN - VERY much still a "baby" bot and far from ideal and mature yet

and then look at how many % these SE bring traffic into your site
and look at all other bots / SEs combined

most of all NON-Google bots still are extremely fragile in their crawling "strength"

sufficient reasons why to keep on serving nice URLs and nicely presented ( validated ) pages in clean style

 

You're right mopacfan, no ID and no 2 kilometres urls.. Might not help but won't hurt

 

I'd wish Argos was my company but it isn't and that site isn't mine.

Hans, you are right in saying that Google isn't the only one out there and of course "better safe than sorry" still counts as well.

 

Hey tops, I've never really found that the letters 'id' are a no-no in a variable. What I have found is that if the variable is called ID (so not forumid or some variant) it has a very hard time getting indexed. We've been using a lot of variables on our site like 'forumid' or 'threadid' for the last couple of years already without a problem.

I have noticed that we have started to get pages indexed with three url variables though - something new in Google.

Mind you, at the same time, we've just started flattening our important URLs; mainly to ensure we are prepared for the other SEs.

 

found here
http://www.google.com/webmasters/guidelines.html

 

Maybe not every Search Engine crawler, but the Googlebot definitely does.

 

I know of a site that just launched about 3 weeks ago that has session ids in the URL, among other variables. The site is completely indexed, session IDs and all, in Google, Yahoo and MSN. Ofcourse I would never recommend launching a site like that, but I found it very interesting.

-Tyson

 

Well there it is, by trying to have twice the size of pages indexed, there will be some serious garbage in Google's index now, but as long as the proportion garbage / valid url is steady... (http://www.google.com/googleblog/)

 

I just found out that Google is now able to index pHpBB forums with session IDs. There are no mods installed on my forum. Used to be that only the index page is indexed by Google.

http://www.google.com.ph/search?hl=en&q=site:www.textforum.com&btnG=Google+Search&meta=

Maybe they took a cue from MSN tech preview

 

So is there something special you have to do with phpBB to get this to work? I still only show my index page: http://www.google.com.ph/search?hl=en&q=site:forums.mccoypottery.com&btnG=Search&meta=

 

Sounds good.. I think I will go that route.. I use phpBB cause it's free

 

yep I hacked up my phpbb site good (is this even a proper sentence?). Anyway, I found when I removed Session ID's, users had difficulty logging in.

I was a die-hard Phpbb addict, now I had to face reality and recommend VBullitin or Invasion instead...

 

Honeslty, I was about to apply the indexing mod to the forum. I checked it first and found out the indexed pages. So I never really did any mods to it. If you want to remove the SIDs, you can use this great mod

http://www.phpbb.com/phpBB/viewtopic.php?t=200361

Thanks.

 

that's the mod I used. It effects log in's by users and cookies. Not recommended, sorry!

 

Newbie here everyone, how is everyone ('s rank?)

If this is so then I hate Google even more! I just rebuilt one of our entire sites (5000) pages with html catalog to combat this problem.

Once again Googlefied.

 

The PHPBB hack to remove session ids is piss easy to use, its great.

as for variables in the url, for every variable the PR of the targeted phrase generally drops 1 automatically,

(yes I know that there are exeptions if you deep link externally to the page but generally if you get all the PR off your frontpage or site pages its true.)