1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Google Crawler

Discussion in 'robots.txt' started by jkapadia1983, Oct 1, 2009.

0
  1. #1
    Can any one give code for developing a web crawler ??
     
    jkapadia1983, Oct 1, 2009 IP
  2. kanzaki

    kanzaki Peon

    Messages:
    61
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    wow, thx ...!!!i have to try it
     
    kanzaki, Oct 5, 2009 IP
  3. jkapadia1983

    jkapadia1983 Peon

    Messages:
    40
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks...I got STS Web Crawler 1.3.1
     
    jkapadia1983, Oct 7, 2009 IP
  4. AKSIN

    AKSIN Peon

    Messages:
    159
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    User-Agent: *
    Allow: /
     
    AKSIN, Oct 9, 2009 IP
  5. david-mynehost

    david-mynehost Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Downloading it now..will have a try..

    how others who been downloading the crawler ?

    is it any good ?
     
    david-mynehost, Oct 23, 2009 IP
  6. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #6
    manish.chauhan, Oct 28, 2009 IP
  7. php.freak

    php.freak Well-Known Member

    Messages:
    185
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    101
    #7
    what is the benifit of it?
     
    php.freak, Nov 11, 2009 IP
  8. RGInfotech1

    RGInfotech1 Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Thanks for sharing it above Url
    it is so useful for all

    Thanks
     
    RGInfotech1, Nov 13, 2009 IP
  9. diya

    diya Guest

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Thank for nice suggestion.
     
    diya, Nov 19, 2009 IP
  10. johnnylee

    johnnylee Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    thanks for the great sharing
     
    johnnylee, Nov 27, 2009 IP
  11. Angelyss

    Angelyss Member

    Messages:
    42
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    48
    #11
    Angelyss, Nov 30, 2009 IP
  12. redesignunit

    redesignunit Peon

    Messages:
    467
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Good sharing bro
     
    redesignunit, Dec 4, 2009 IP