crawler from tamu.edu

Discussion in 'All Other Search Engines' started by hextraordinary, Aug 13, 2006.

0
  1. #1
    I'm getting crawled pretty intensively by a crawler coming from tamu.edu .

    Anyone know anything about it?
     
    hextraordinary, Aug 13, 2006 IP
  2. surchin

    surchin Banned

    Messages:
    372
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #2
    what are you using to determine this? the site tamu.edu is not working for me
     
    surchin, Aug 13, 2006 IP
  3. hextraordinary

    hextraordinary Well-Known Member

    Messages:
    2,171
    Likes Received:
    115
    Best Answers:
    0
    Trophy Points:
    105
    #3
    hextraordinary, Aug 13, 2006 IP
  4. surchin

    surchin Banned

    Messages:
    372
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #4
    ok its working now...maybe I typed it in wrong... but what are you using to determine that they are crawling your site a lot?
     
    surchin, Aug 13, 2006 IP
  5. hextraordinary

    hextraordinary Well-Known Member

    Messages:
    2,171
    Likes Received:
    115
    Best Answers:
    0
    Trophy Points:
    105
    #5
    I see their crawler all over going over my pages....
     
    hextraordinary, Aug 13, 2006 IP
  6. Monty

    Monty Peon

    Messages:
    1,363
    Likes Received:
    132
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Monty, Aug 13, 2006 IP
    hextraordinary likes this.
  7. 50plus

    50plus Guest

    Messages:
    234
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #7
    If you don't want the bot to crawl your site just put its UA in your robots.txt file, it does obey the rules.
     
    50plus, Aug 13, 2006 IP
  8. hextraordinary

    hextraordinary Well-Known Member

    Messages:
    2,171
    Likes Received:
    115
    Best Answers:
    0
    Trophy Points:
    105
    #8
    Thanks, thats what I wanted to know.
     
    hextraordinary, Aug 14, 2006 IP
  9. infonote

    infonote Well-Known Member

    Messages:
    4,032
    Likes Received:
    68
    Best Answers:
    0
    Trophy Points:
    160
    #9
    Did you get any traffic from that site? besides the bot?
     
    infonote, Aug 14, 2006 IP
  10. hextraordinary

    hextraordinary Well-Known Member

    Messages:
    2,171
    Likes Received:
    115
    Best Answers:
    0
    Trophy Points:
    105
    #10
    No. Nothing significant at least.
     
    hextraordinary, Aug 14, 2006 IP