Google Crawler

Discussion in 'robots.txt' started by jkapadia1983, Oct 1, 2009.

0
  1. #1
    Can any one give code for developing a web crawler ??
     
    jkapadia1983, Oct 1, 2009 IP
  2. kanzaki

    kanzaki Peon

    Messages:
    61
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    wow, thx ...!!!i have to try it
     
    kanzaki, Oct 5, 2009 IP
  3. jkapadia1983

    jkapadia1983 Peon

    Messages:
    40
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks...I got STS Web Crawler 1.3.1
     
    jkapadia1983, Oct 7, 2009 IP
  4. AKSIN

    AKSIN Peon

    Messages:
    159
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    User-Agent: *
    Allow: /
     
    AKSIN, Oct 9, 2009 IP
  5. david-mynehost

    david-mynehost Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Downloading it now..will have a try..

    how others who been downloading the crawler ?

    is it any good ?
     
    david-mynehost, Oct 23, 2009 IP
  6. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #6
    manish.chauhan, Oct 28, 2009 IP
  7. php.freak

    php.freak Well-Known Member

    Messages:
    185
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    101
    #7
    what is the benifit of it?
     
    php.freak, Nov 11, 2009 IP
  8. RGInfotech1

    RGInfotech1 Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Thanks for sharing it above Url
    it is so useful for all

    Thanks
     
    RGInfotech1, Nov 13, 2009 IP
  9. diya

    diya Guest

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Thank for nice suggestion.
     
    diya, Nov 19, 2009 IP
  10. johnnylee

    johnnylee Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    thanks for the great sharing
     
    johnnylee, Nov 27, 2009 IP
  11. Angelyss

    Angelyss Member

    Messages:
    42
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    48
    #11
    Angelyss, Nov 30, 2009 IP
  12. redesignunit

    redesignunit Peon

    Messages:
    467
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Good sharing bro
     
    redesignunit, Dec 4, 2009 IP