What programming language are robots (spiders) written in?

Discussion in 'Programming' started by whiteblue1942, Jul 15, 2008.

0
  1. #1
    What programming language are robots (spiders) written in? Does anyone know? Lets say I am trying to make a search engine and I need to make something that can crawl websites on its own, does anyone have any idea how to make these?

    Someone suggested javascript and c++ but I cant see how this is possible, or maybe it is a different programming language all on its own???
     
    whiteblue1942, Jul 15, 2008 IP
  2. enslon

    enslon Peon

    Messages:
    12
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    kapowtech.com can help you if you are parsing not the entire web.

    IF you need such development - pm me or skype me: thinkactcreate
     
    enslon, Jul 15, 2008 IP
  3. abluegrape

    abluegrape Peon

    Messages:
    1,029
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #3
    At a guess you could use any language you are comfortable using - I have made a very basic crawle rin the past using Perl.

    With available Perl modules it was very easy!

    RY
     
    abluegrape, Jul 15, 2008 IP
  4. Arkserver

    Arkserver Banned

    Messages:
    2,533
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I;ve written one in java so i can confirm it can be done in Java.
     
    Arkserver, Jul 16, 2008 IP
  5. firmaterra

    firmaterra Peon

    Messages:
    756
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I wrote one in java too. You can write a crawler in almost any language. Its been done in PHP, JAVA, C++ and Python.

    If you want a ready made one you can check out hadoop written in Java. Yahoo is using an advanced version of Hadoop and even hired the programmer to work for it.
     
    firmaterra, Jul 18, 2008 IP
  6. shallowink

    shallowink Well-Known Member

    Messages:
    1,218
    Likes Received:
    64
    Best Answers:
    2
    Trophy Points:
    150
    #6
    Almost any language could work. Which is best is another matter. Perl is good for small to medium scale searches but will run into serious issues if it was asked to fetch the entire WWW. Java, if there wasn't anything else. Google uses python, C++ and java.
     
    shallowink, Jul 18, 2008 IP