How do I check if my site has been spidered?

How do I check if my site has been spidered?

 

If you insert this code [modify the email] into your PHP page [for instance your index page] it will send you an email every time Google spiders that page.

If Google spiders your site a lot be prepared for a lot of emails

<?
if(eregi("googlebot",$HTTP_USER_AGENT))
{
if ($QUERY_STRING != "")
{$url = "http://".$SERVER_NAME.$PHP_SELF.'?'.$QUERY_STRING;}
else
{$url = "http://".$SERVER_NAME.$PHP_SELF;}
$today = date("F j, Y, g:i a");
mail("you@yourdomain.com", "Googlebot detected on http://$SERVER_NAME", "$today - Google crawled $url");
}
?>

 

Thank you both for you answers. I will go try those suggestions.

 

..assuming you have a log analyser.

A good one to start off with is the light version @ http://weblogexpert.com/
you can load it to your windows pc and than get the log files either directly from your webhost or ftp them to your win pc.
I use awstats (standard) it's free, if it's not installed on your host you can get a copy and install it in the cgi directory.
Both give you info about which spider when keywords keyphrases and of course visitors/pages
M

 

You may try RobotStats...
It's an open source free application I wrote in PHP/MySQL, which analyzes the visits of search engine robots.

A new version will be released within 15 days...

Best,
PR Weaver

 

Or check Google's cache dates and cache in general if G is what you are interested in.