Mortgage - Home Loan - Cheap Flights - Unblock Myspace - Car Insurance

PDA

View Full Version : Problems with Inktomi Crawler and Metatraffic


TampaDave
Mar 30th 2004, 8:38 am
Metasun.com makes a good stats package you can run off Access or SQL Server. I'd been using it for some time when, at the beginning of March, I noticed the Inktomi bot was getting 500 errors when it tried to crawl my site.

Much back and forth between myself and MarketLeap (I paid for inclusion). No resolution.

By use of a free page optimizer, I found out some bots weren't seeing my <body> tag. If I took out the Metatraffic include, the page optimizer saw it.

So I left it out and two days later, Inktomi indexed the pages no problem.

More experimentation revealed that the lost <body> tag only occured when using the #include or Server.Execute command. Using the javascript logging method didn't seem to cause a problem.

Just a caution to those of you who might be using Metatraffic as your stats package.

TampaDave

hans
Mar 30th 2004, 12:17 pm
to my knowledge the Inkotmi bot is slurp
21 2772 0.56% Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/
that is the slurp visit this month - for free :-) inclusion

i have seen no slurp problems exept that slurp seems NOT to process a 404 and dump that page - but slurp comes back for YEARS to missing files ..

googlebot had about 9000 hits ( theoretically each file 3 x a months ) in march

TampaDave
Mar 30th 2004, 1:18 pm
Yah-

Slurp is free, but when you launch a new site, you can pay your $39 and get listed in 48 hours instead of 4 weeks to 6 months. I only paid for home page inclusion.

Google didn't seem to have anyproblems indexing even when using the #include or Server.Execute

TampaDave