We have a site with 80K + pages tried the fee ones that run from your Pc but they take 3 days to do the job and then when your go back into them they lock up. Looked at few that run on the server but reading the support forums large sites that run a few database ours has nine, the program take up most of the CPU and cause problems that way. So were are looking for a paid remote web based one that will do it from there server any about or would it be better to use a PC based one and do five smaller ones instead of one big one Thanks Barrie,
yes i can make u a sitemap in $50 for links upto 99999 50% advance Only need your sitename Approx time 3-4 days Regards Alex
I don't know if you have tried A1SG, but 80K should not be any problem, at least with a recent version of A1SG. To increase capacity and speed, you can check this article. If you encounter any problems, please PM or email me. If your website it splitted in sections, that may be a quite good idea. Please note that if your website has heavy SQL/database usage, all crawler based approaches (PC, remote, etc.) will either: a) hit your webserver and increase load b) need to be configured to crawl slowly (thus increasing time before done)
Yes the site has a small forum, 6 directory's, 2 items for sale sections, one trade and one General public and about 2000 static pages we get just short of 3 million visitors a year we have put Gsitecrawler on its own computer with nothing running that has speeded it up I will read the article on as A1 was my next port of call if Gsitecrawler did not improve Thanks for you help Barrie,