Problems with Spider Simulator. Urgent help needed

Discussion in 'Site & Server Administration' started by link_dev, Feb 6, 2007.

0
  1. #1
    Spider simulator cannot read my website.

    I am using following site to check my website
    http://www.webconfs.com/search-engin...-simulator.php

    and it shows all garbage characters.

    Did anybody see this behavior?
    If so, what is the fix?
    The character set is something like this
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
    <link href="style.css" rel="stylesheet" type="text/css">

    This is how the text looks like
    Spidered Text :
    ‹í}]oI¶Øûûj´¹³w±n5¿EÙ–}ì±eëšÚqvÆ’ Qì.²Ëìîꩪ&ÍA°˜$@Þò”\ È£7òpŸ’¿bì/É9UÝ)Q’Ù²Mz¶5c‰ìsêœ:u¾êTÕãPÇÑþ¯õ8d4À¿šëˆí‰$a¾& ®Cò, 8MH?Ã!“ŠÐ$ :d\Å¡Kê±k߁—c¦)IhÌö¶¦|ÉSÍE²E|‘h–è½**censored**cÊ£) ÉÒ€j¦È@Å xºÚ"î L¨uê°3>ÞÛ:²ï;Ó”ÍAÓìv‘€GÄ©TLïq%Å“n·½ëÔ·RÄ“ %ìm)=Ø¶¯‰dQþ]…Œé-¢nÀ7Ý‚!}L‰iLå'{[µ-±žûj?C†¡ž»0áñ»a+íGŒä—êµÚßm‘¾“æñ!Àò*&@E ) ž Ímü®Rêçß â„¢V<`~»ÿ˜Zã·è²‡ˆÝ'?î%"`ns§Ûúm.„¿-Lâ„¢GȼknÅ“:²PȽßtøßþ3I$G"‹@¼¹2!)•úT‰á¡ ð/â_ˆ¾cÚÆßýL&Lÿ,]ú–N%²Ÿ¨ûç¦Ûí´ZV»ë®Ln†& Ä$! E&Õ#ÒÓü@ÁáÐÓ,†à倱GµÿÈùÝf‘©@ëÐ4–…“ÓØ©wn»Õ]â„¢\Ppä\Å ×ä$Ȭˆ“‡äLHF^¾R éñÚ|Tï0ÃŒ|95ôƒ3"§î0}9*±q«wò©yŸ¼±ï¯É…EêT6À© b•€m'L»Hüû†Ï ¦À‹YÒ x›àÛà**censored**\#3-"Q‰ ºúüJlòìü)ð‘¸çßåI**censored**mÔ¥öüËpQÈ4.ⶍ ` ïmŽç½ÓNA¤ôôpx¢—,r1Éó£“Wäœ^7õ$G&ŽKÅ“>؃™ ©Kñ $ÈiëðÃ’Sx €c&5êRdS 4¬Y`nròÏÁÃW@BŸÿ”ûøÝn«ÛrwwÁOyŸ‘KTEÞDÄ,ñx⁠”cÖî.Á)8r\À$o&&º0ÍñFÈ(ØdR±rÃ’|yê³LCÅ“]ª¦\3A¿[ÍtO¹êõÈù"*#_ye‘‘ùÚ¬Ù©Õwv\Ã…$gÊ.Ã}/§T=±zÍ‹JrÍÀ&¨ä©…JÃŽ@Èò›øõ‚1¸ƒ°- 6ÿöòi_D4®çÔ˜¶6ꆔZ»Ý(—RÛzRMð\1“z$d¦3I^S Zú~?ÙÚ¼!Òƒ—Â$‡—c¤^k6]É°VÁÒtDÇXxÔ³U¹1òÚÀÄÔòÂüa=$Ä| ½SCb¦¥0i Å“Z¯ZS!¥ÛB.f,BæÀˆp²ÄIý) °9ÿÞ'ðþùÑæ‰Å5ÍÙnµvë w×`üdI‚ 4i =æiá%Bê°œTÅ“"Lrha‚‚40ÁƐ—“<’Q#sótÇtºŒG áSâ‡ÐБg:£‘y¨¤drR %)ùkTbn²“|6•”‡brf›ï“xÃŒžçÙDG•z¬RkI=VEjó Ej½ P•uüg¿Ìü—#Ì’¡i'÷˜Ñø[GS¹:¡Ìž¹H±–'“ÊTùàºÀFcˆ)KTöÌa!r’cy@ 4vùAŽfSJ¾uc娂iuˆÓšÍ²Õ^èzOµe>$„P•Å’9Lz~ K(Äsbb¸´Æ)T!M¿RûÐ(2L`9~–:XÅ“P§î´!¶ƒ~'ð¥D ™€õ$LnÅ’e)ÉA“ßÔ’¼Mràëa &}

    Please help, how to make this spider simulator read my website.
     
    link_dev, Feb 6, 2007 IP
  2. andre75

    andre75 Peon

    Messages:
    1,203
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Yeah I have seen this b4. Its gzip compressed.
    Modify the script so it sends the correct headers and indicates that this spider does not accept gzip compression.
    I saw this on one of my own scripts and banged my head for a long time on this :)
    Don't worry about it. Google will be able to read your site. The "spider simulator" has a flaw :)
    If you just want to "see" how it "looks like" to a spider, you can use a text only browser like lynx.
     
    andre75, Feb 7, 2007 IP
  3. link_dev

    link_dev Peon

    Messages:
    292
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #3
    thank you. Even msn has problem reading gzip compressed sites. However, i lost two advertisers trying to fix this issue.
     
    link_dev, Feb 8, 2007 IP
  4. andre75

    andre75 Peon

    Messages:
    1,203
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Well, in my scripts i simply changed the part of the header that proclaimed to accept gzip and it worked. I wonder why msn would send the wrong headers in the first place. If they don't, the website won't be compressed :confused:
     
    andre75, Feb 8, 2007 IP