Getting threads from forums

I am in a way of developing a new script.For that I need to get each and every thread of forums.

I mean if I am entering :

$url = "http://forums.digitalpoint.com/";

I should get back all the threads of DP forum.

Is it possible ?

 

Threads: 1,070,926, Posts: 9,452,724, Members: 225,426, Active Members: 61,009

I would suggets not doing it but....its your choice, even if the script loads 10 threads/second the script will take 300 hours+ to fully load.

And by the way IT IS POSSIBLE, but recommended NOT TO BE DONE. And the code shall be pretty large too.

 

I don't want to take the threads from big forums like DP.But just want to know the method or idea.

Can you please suggest ?

 

CURL can do it.

 

Yup cURL can do it easily, I have posted a small tut about cURl on my blog you can find in my singnature although that shall not be enough for retrieving the threads. You will need to use preg_match and preg_match_all to retreive sub-categories/forum addresses, then again to retrieve the thread page addressed then again to retrieve its code. good luck!

 

With cURL you'd be able to simulate a web browser almost 100% perfectly. There are of course some javascript problems which may be difficult to come around, but generally you should be able to do about anything else.

 

I know curl.But not getting the idea.If you can help it will be great.

Your site is not loading

 

sorry shifting it to my server, will be fixed soon

 

I can see nothing abt threading getting i your blog.Please help if you can.

 

I havent posted the exact method, see the cURl basics on how to get pages from the internet. Then use peg_match to retreive forum and thread urls. Google both "PHP: cURL" and "preg_match" or "preg_match_all" for more info.

 

can you please tell the idea of it ?

 

Google cURL ;-)
You can even log in to the DP forum with cURL, and crawl the only-users categories

 

That all I can do ......!
But want to know how to crawl each and every thread.

 

Alright here it is step by step

1) Open desired forum using cURL.
2) Use preg_match_all for getting the exact links to forums.
3) Open each forum and take exact link of the posts again using preg_match_all.
4) You can either just us ethe URL or again retreive the specific words written by the author using preg_match.

I know this is a long and tedious job, but good luck