loopy62
Jun 23rd 2006, 10:25 am
Look at the following crawl from googlebot:
Is this normal? Googlebot is scrambling my site and thus it's giving me supplemental results. grrrr!!!
/Society/Celebrity/Internet/Forum/
Http Code: 200 Date: Jun 23 05:07:29 Http Version: HTTP/1.1 Size in Bytes: 26396
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Sports/Travel/Insurance/Mortgage/
Http Code: 200 Date: Jun 23 05:07:30 Http Version: HTTP/1.1 Size in Bytes: 8509
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Sports/Travel/Education/Colleges/
Http Code: 200 Date: Jun 23 05:07:30 Http Version: HTTP/1.1 Size in Bytes: 9738
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Society/Celebrity/Leisure/Travel/
Http Code: 200 Date: Jun 23 05:07:31 Http Version: HTTP/1.1 Size in Bytes: 36872
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Sports/Travel/Regional/Countries/
Http Code: 200 Date: Jun 23 05:07:31 Http Version: HTTP/1.1 Size in Bytes: 8510
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Society/Celebrity/Insurance/Life/
Http Code: 200 Date: Jun 23 05:07:32 Http Version: HTTP/1.1 Size in Bytes: 8493
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Is this normal? Googlebot is scrambling my site and thus it's giving me supplemental results. grrrr!!!
/Society/Celebrity/Internet/Forum/
Http Code: 200 Date: Jun 23 05:07:29 Http Version: HTTP/1.1 Size in Bytes: 26396
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Sports/Travel/Insurance/Mortgage/
Http Code: 200 Date: Jun 23 05:07:30 Http Version: HTTP/1.1 Size in Bytes: 8509
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Sports/Travel/Education/Colleges/
Http Code: 200 Date: Jun 23 05:07:30 Http Version: HTTP/1.1 Size in Bytes: 9738
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Society/Celebrity/Leisure/Travel/
Http Code: 200 Date: Jun 23 05:07:31 Http Version: HTTP/1.1 Size in Bytes: 36872
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Sports/Travel/Regional/Countries/
Http Code: 200 Date: Jun 23 05:07:31 Http Version: HTTP/1.1 Size in Bytes: 8510
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
|
|
|
/Society/Celebrity/Insurance/Life/
Http Code: 200 Date: Jun 23 05:07:32 Http Version: HTTP/1.1 Size in Bytes: 8493
Referer: -
Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)