How do I check if my site has been spidered?

[rbay]

How do I check if my site has been spidered?

[digitalpoint]

Check your HTTP logs for the user agent of the spider you are looking for.

[ResaleBroker]

If you insert this code [modify the email] into your PHP page [for instance your index page] it will send you an email every time Google spiders that page.

If Google spiders your site a lot be prepared for a lot of emails ;)

<?
if(eregi("googlebot",$HTTP_USER_AGENT))
{
if ($QUERY_STRING != "")
{$url = "http://".$SERVER_NAME.$PHP_SELF.'?'.$QUERY_STRING;}
else
{$url = "http://".$SERVER_NAME.$PHP_SELF;}
$today = date("F j, Y, g:i a");
mail("you@yourdomain.com", "Googlebot detected on http://$SERVER_NAME", "$today - Google crawled $url");
}
?>

[rbay]

Thank you both for you answers. I will go try those suggestions.

[expat]

..assuming you have a log analyser.

A good one to start off with is the light version @ http://weblogexpert.com/
you can load it to your windows pc and than get the log files either directly from your webhost or ftp them to your win pc.
I use awstats (standard) it's free, if it's not installed on your host you can get a copy and install it in the cgi directory.
Both give you info about which spider when keywords keyphrases and of course visitors/pages
M

[sarahk]

Or I could hook you up with Botspotter (http://botspotter.net), 'bout time I got it going again.

Sarah

[PR Weaver]

You may try RobotStats (http://www.robotstats.com/en/)...
It's an open source free application I wrote in PHP/MySQL, which analyzes the visits of search engine robots.

A new version will be released within 15 days...

Best,
PR Weaver

[T0PS3O]

Or check Google's cache dates and cache in general if G is what you are interested in.