I need a program that will scan websites for a keyword - help?

Discussion in 'Programming' started by chris123, Jul 29, 2009.

0
  1. #1
    I need a program that will scan a list of URLS looking for the word affiliate - this is my crude way of detecting whether they have an affiliate scheme.

    I could use a bot to do this but can't seem to find any that will do the job, most bots seem to look for email addresses.

    The list of URLs in large too - 100'000+

    Does anyone know of a program I could use? Im open to paying for the software.
     
    chris123, Jul 29, 2009 IP
  2. tguillea

    tguillea Active Member

    Messages:
    229
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    90
    #2
    I think you could do this with PHP. Do you have a server you could execute it on? Maybe one that doesn't timeout / you can edit the php.ini file?
     
    tguillea, Jul 29, 2009 IP
  3. L4N

    L4N Guest

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Coding this in python would also be fairly easy using urllib and regular expressions to scan the data. I sent you a PM with details. Looking forward to hearing your response.
     
    L4N, Jul 30, 2009 IP
  4. vagothcpp

    vagothcpp Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    It would also be simple to do it in C++, since windows sockets are at basic level the same as Linux/Unix Sockets (BSD Sockets). PM Me if your looking for it in an app
     
    vagothcpp, Aug 1, 2009 IP