Need help, Codes to block spiders except google.

Discussion in 'Programming' started by silverunion, May 9, 2008.

  1. #1
    <%
    dim spider
    spider=Request.ServerVariables("HTTP_USER_AGENT")
    if instr(spider,"ALL Spider's characteristic STRING")>0 and not instr(spider,"googlebot")>0 then
    response.Write("pass it.")
    response.end
    end if
    %>

    I want to block all the spiders except Googlebot, what's the characteristic string of all bot and spider?
    Any guys can help me ?
     
    silverunion, May 9, 2008 IP
  2. silverunion

    silverunion Peon

    Messages:
    10
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I had use robots.txt,but it seam not work right?

    User-agent: Googlebot
    Disallow: /help/
    User-agent: *
    Disallow: /
     
    silverunion, May 10, 2008 IP
  3. BlogSalesman

    BlogSalesman Well-Known Member

    Messages:
    1,687
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    100
    #3
    I seem to recall reading that the Robots.txt was your best bet, but even that did not work 100% of the time.

    I don't think it is possible without knowing either the IP address(es) of the bots or some kind of signature they each have.
     
    BlogSalesman, May 12, 2008 IP
  4. Dreamerr

    Dreamerr Peon

    Messages:
    914
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I think it can also be done with .htaccess
     
    Dreamerr, May 12, 2008 IP
  5. seo8k

    seo8k Guest

    Messages:
    70
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #5
    oh it's easy
    use this : " if instr(spider,"googlebot")<=0 then "
    it means it will pass unless it's google
     
    seo8k, May 12, 2008 IP
  6. olcaymy

    olcaymy Active Member

    Messages:
    561
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    85
    #6
    User-agent: Googlebot
    Disallow: /help/
    User-agent: *
    Disallow: /
     
    olcaymy, May 13, 2008 IP