Looksmart and Inktomi are slowing my site to a crawl today. I just added a crawl-delay of 30 seconds to my robots.txt but I'm not sure they're obeying it... do they? My visitors can't even get to my site today, should I just block them?? On a related note, does Googlebot 2.1 obey crawl-delay, I can't find a solid answer.
Most bots only read the robots.txt once every 24 hours, therefore any changes made today will unlikely alter their indexing.
Hello Rose*** Please Can You Write To Me SO I can Knw or Think Of A Wai To Assist You Asap** My Email is maccollarme@yahoo.com Thnx For Ur Understandn. (+2348062409580)
I am not familiar with this thing right now. But want to know about this in future may be it would prove beneficial for me.
It sounds new to me. Though, I better stick on the main search engines for me to market more my site.