My affiliate's spider visits my site every night to check if their ads are placed on my site. Just recently, they are saying that they were not able to crawl through even though my server log says http code 200 next to the spider log. How could this be possible? Are they pulling one on me? BTW: I have "disalow" in my robot text but they are for my javascript, stylesheet and image folders which has nothing to do with the ad on the page. Clickalot
200 means its accessible by server language, but if you buddy is looking for ads on your web site and you didnt place any on there, then whats the point of doing an archive on your site?
If the ads are there, then it's kind of strange issue. I would suggest to double check your robots.txt, if that is causing all this. If not, then I would suggest to inform your affiliate about this, if you can.
webgk, that's why I wrote in my title "mystery". As far as I am concerned if my weblog shows 200 and I see download kilobytes recorded in the log, it means the robot crawled successfully. My robots txt is validated and OK. I have contacted the affiliate and they said they'd be back but they haven't.