|
|
|
|
#1
Mar 8th 2004, 9:53 am
|
||||
|
||||
|
Spiders Supporting HTTP 1.1?
I'm curious if anyone knows of any spiders that support HTTP 1.1? I know Googlebot doesn't, and neither does Yahoo's Slurp bot. Both are HTTP 1.0 only.
It seems like search engines spiders would want to be HTTP 1.1 compliant, as it can cut down on bandwidth costs for them tremendously. One of the nice things about HTTP 1.1 is it can support gzip compression for web pages. Text files (HTML files) typically compress down about 5 to 1, so it would save quite a bit of bandwidth. Of course the real reason I want it is because *I* want to save on bandwidth when the spiders visit.  - Shawn
__________________
~ Shawn @Twitter Keyword Tracker Please do not PM, IM or email me for product or tool support (they will go unread/ignored), and don't "friend" me unless we are really friends. 
|
|
#2
Mar 18th 2004, 7:52 pm
|
||||
|
||||
|
Something I overlooked this morning when posting about the new Googlebot, is that the new test Googlebot described here, is now running HTTP 1.1, so it looks like it's more than just a JavaScript reading upgrade.
- Shawn
__________________
~ Shawn @Twitter Keyword Tracker Please do not PM, IM or email me for product or tool support (they will go unread/ignored), and don't "friend" me unless we are really friends.
|
|
#3
May 29th 2004, 5:09 pm
|
||||
|
||||
|
Shawn, you can also check this website out.... It will give you tons of information on the different bots out there as well as track them when they crawl your website.
http://bots.pcpropertymanager.com/index.php Hope it helps Wayne Last edited by digitalpoint; May 29th 2004 at 5:44 pm. Reason: signatures should be in signatures, not body
|
|
#4
May 29th 2004, 5:44 pm
|
||||
|
||||
|
Yep... that was written by one of our members.
__________________
~ Shawn @Twitter Keyword Tracker Please do not PM, IM or email me for product or tool support (they will go unread/ignored), and don't "friend" me unless we are really friends.
|
|
#5
Jun 2nd 2004, 11:38 pm
|
||||
|
||||
|
do you think they aren't adding the support for it because of the CPU overhead on their end to decompress the gzip'd pages?
__________________
tutorials and guides on DP: Digg's Gaming You (new!) Unsandboxed vs Never Sandboxed Domains
|
|
#6
Jun 3rd 2004, 8:15 am
|
||||
|
||||
|
Doubtful... I imagine they will support it at some point. The tiny CPU overhead would save them 80% of their bandwidth.
__________________
~ Shawn @Twitter Keyword Tracker Please do not PM, IM or email me for product or tool support (they will go unread/ignored), and don't "friend" me unless we are really friends.
|
 |
| Bookmarks |
| Thread Tools | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| PHP XML HTTP Post | 5speed | PHP | 3 | Sep 30th 2008 12:54 pm |
| ??? www -vs- http and PR ??? | SEOGulper | Search Engine Optimization | 13 | May 5th 2008 4:18 pm |
