Hi, I'm trying to write some scraping software to download data from betting websites into spreadsheets. It appears that most of the data is stored in text files that are downloaded by various javascripts/applets. But I don't know the URL's of thse files. How can I find these URL's? In an old job I had I used to be able to browse the proxy logs of our servers. However, I'm just using a windows XP machine at the moment and don't have access to those logs (don't even know if the requests are checked). Does anyone know how I can find out all the http requests made from my browser? Regards, fcmisc.
Ethereal (though I think it's just gone through a rename) from http://www.ethereal.com can do that for you. It sniffs everything so you will need to filter a lot of stuff out (all doable through the GUI) but it will definitely be able to find out what URLs you're getting.
My understanding is - you want to capture "the link behind the applet" right ? Try http://www.link-rank.com/sniffer/ you can capture the link, if the data is in XML format, then you can import it in Excel (XP or 2003) Hope helpful