1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How do I check if my site has been spidered?

Discussion in 'Traffic Analysis' started by rbay, Aug 25, 2004.

0
  1. #1
    How do I check if my site has been spidered?
     
    rbay, Aug 25, 2004 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    Check your HTTP logs for the user agent of the spider you are looking for.
     
    digitalpoint, Aug 25, 2004 IP
  3. ResaleBroker

    ResaleBroker Active Member

    Messages:
    1,665
    Likes Received:
    50
    Best Answers:
    0
    Trophy Points:
    90
    #3
    If you insert this code [modify the email] into your PHP page [for instance your index page] it will send you an email every time Google spiders that page.

    If Google spiders your site a lot be prepared for a lot of emails ;)

    <?
    if(eregi("googlebot",$HTTP_USER_AGENT))
    {
    if ($QUERY_STRING != "")
    {$url = "http://".$SERVER_NAME.$PHP_SELF.'?'.$QUERY_STRING;}
    else
    {$url = "http://".$SERVER_NAME.$PHP_SELF;}
    $today = date("F j, Y, g:i a");
    mail("you@yourdomain.com", "Googlebot detected on http://$SERVER_NAME", "$today - Google crawled $url");
    }
    ?>
     
    ResaleBroker, Aug 25, 2004 IP
  4. rbay

    rbay Well-Known Member

    Messages:
    102
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    118
    #4
    Thank you both for you answers. I will go try those suggestions.
     
    rbay, Aug 25, 2004 IP
  5. expat

    expat Stranger from a far land

    Messages:
    873
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #5
    ..assuming you have a log analyser.

    A good one to start off with is the light version @ http://weblogexpert.com/
    you can load it to your windows pc and than get the log files either directly from your webhost or ftp them to your win pc.
    I use awstats (standard) it's free, if it's not installed on your host you can get a copy and install it in the cgi directory.
    Both give you info about which spider when keywords keyphrases and of course visitors/pages
    M
     
    expat, Aug 26, 2004 IP
  6. sarahk

    sarahk iTamer Staff

    Messages:
    28,500
    Likes Received:
    4,460
    Best Answers:
    123
    Trophy Points:
    665
    #6
    Or I could hook you up with Botspotter, 'bout time I got it going again.

    Sarah
     
    sarahk, Aug 26, 2004 IP
  7. PR Weaver

    PR Weaver Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    You may try RobotStats...
    It's an open source free application I wrote in PHP/MySQL, which analyzes the visits of search engine robots.

    A new version will be released within 15 days...

    Best,
    PR Weaver
     
    PR Weaver, Aug 26, 2004 IP
  8. T0PS3O

    T0PS3O Feel Good PLC

    Messages:
    13,219
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Or check Google's cache dates and cache in general if G is what you are interested in.
     
    T0PS3O, Aug 26, 2004 IP