GSiteCrawler, who uses it?

I'm creating a sitemap with that free program, but it's running for like 5 hours now and it's still not finished.
Is this normal?
I can see that it's still active, but could there be anything else that causes the slowness?

 

I used it for a while until I realised that the datafile got to 1gig and it stopped functioning, then rather than mess about I just removed it from my computer. It is free for now because we are basically beta testing it for the developers...and I must say that when the glitches are sorted out it will be a very useful little program.

 

I found it very useful on smaller sites. Really good for finding dupe content etc. I have also seen it run very slow on bigger sites which stopped me from using it full time.

Openg I see you were in my neck of the woods a couple of weeks back.

 

What are you using instead now?

 

I use Xenu to check for broken links etc but also to grab a list of valid urls on the site.

 

if you have an huge web site , it's normal . it must crawl all of your pages

 

I use it to index my local site. Then I replace localhost with my sites domain name and then uploads the sitesmap to the server. This saves my bandwidth and my sites bandwidth limit

 

That is a quite good idea as long your website isn't CMS / forum / blogging based

(Albeit, for some strange reason, my Apache runs very slowly on my own local computer.
I am not quite sure why, but it happened some months ago. Probably configuration related.)

 

First I downloads the database and updates my local database. Then I runs the program. Hence it doesn't matter whether its a Forum or Blog. Provided you have access to the database.

 

Hey thnx...i have been searching for this software since a pretty long time!!

 

I just remembered something about GSiteCrawler. As far as I know it checks valid urls on your site by using Google results, this can slow down the whole process. You can disable the option when going through the new sitemap wizard.

 

On my site, GSiteCrawler takes nearly 2.1/2 hrs for about 2500 links.

Thank you turfsniffer . Xenu's Link Sleuth will go into the FOSS directory I am building.

 

Try this webmaster tool for finding broken links, plus building sitemaps.

2500 links should only take minutes!

 

interesting!

 

yep, it should take minutes for 5000 links.

But i suppose it depends how many links you have on each page and the length of content, and also how many duplicate links you have.

 

Ok, after trying out a few other services I am back with gsitecrawler.

It doesn't seem to be creating the huge databases on this update, but I am a bit confused as to how the site I am crawling (the one in my sig linked to ecommerce templates that has about 240 products) returns almost 3000 pages, and has crawled for aroiund 6 hours so far and says it has 10 hours to go.

I am using oscommerce 'ultimate seo url' plugins and hope it is not replicating loads of pages that are both seo urls and don't seem to want to be removed from the cache as unfriendly url's.

Give me strength

Dom

 

I use Gsitecrawler and it takes about 14 hours to spider my site.

I've used it for months and it has been extremely useful

 

it eats up a lot of bandwidth, lol i tried to use it on a forum with 10k + posts lol not even half way done after 6 hrs, there is like 20k links in there

 

I had heard of gsitecrawler but never used it before. that good, huh?

 

cheers for the xenu link been looking for something like that for ages!