Our members have made a total of 156905 posts, but sitemap only 4000 links?

Our members have made a total of 156905 posts
We have 3753 registered members

But my sitemap only came out to 4,000 urls, is that right? Should there be atleast one url for each topic? J/W

 

actually no, because u can have 10 posts on the one page, or more depending on what you have it set to, but then again it should still be more than 4000 urls, is the software you are using only limited to 4000 urls???

 

No, it's limited to 5,000 urls.
I am using freesitemapgenerator.com

When I try and use a software app to generate my sitemap, it gets really crazy with like 200k+ urls. So I always think something is going wrong and stop the software. Is that right? Should it be hundreds of thousands of URL's?
I figured maybe google wouldnt really like such a large sitemap file..

 

yeah im not really sure, i am yet to find a good sitemap program. i think i might have to make one, ill keep you posted if it eventuates.

 

phpSitemap is a good one.

 

http://www.automapit.com/ is what I use.

-RonMo

 

I am using "softplus Gsite Crawler"
I am just gonna let it ride, as of now after almost 24 hours it's at 81,000 crawled and 108,000 waiting to be crawled. Note, the "wait to be crawled" keeps going up lol

I dunno how many url's this thing is gonna have when done =x

 

The problem might be memory usage...Even if you have loads of memory, 32bit apps are still limited to 2 gigabyte memory address space. (3GB if using a switch)... With ~200k pages... Troubles can quickly occur if the program also stores lots of other data (e.g. redirections, links to and from, titles, response codes etc. you name it).

 

sessionID's may be causing problems in the URLs. Unless a spider is built to handle cookies, your forums see it as a new visitor and add session ID's to the end of the URL (PHP will also do this unless you set it not to). This could lead to hundreds of the same page with a different ID each time.

ini_set("use_only_cookies","1");

Another issue may be a poorly coded website/forum. For spiders that rely on HTML parsing, open tags and poor code can really confuse them. The validator at http://validator.w3c.org ought to shed some light on this. While most sites unfortunately have errors, serious errors can hinder a spider or even mess up in different browsers.

Of course, this could be the generators fault by not handling relative links correctly or being unable to find links in A tags that are 'out of sequence' from the norm. Some may look at a tag that puts a class="" before the href="" and not recognize it. Generally any sitemap tool 'works' to a point, but may be unable to handle quirks and not suitable for a particular website.

One solution is to use a web-based sitemap service that offers you a 'web bug' which is a small image on your page. The image is actually a script on the services website that captures your URL and inserts it into the database of your websites pages, then returns an image. This way, when it's time to create your sitemaps, all of the obscure URLs are added to your sitemaps.

AutoMapIt was built to handle cookies after problems I had with this early on. I do use an HTML parser so that I can report on various SEO factors and for other reasons, but I also offer the web bug images if you need to go that route. If you can't get Gsite to work, I'll be happy to help you out. I used to use Gsite myself until I wanted something more

 

Most default forums are a killer for crawlers - they generate upwards from 10 URLs for the same content. This is not a problem with Sitemap generators / crawlers -- it's a problem with the way forums are set up. Every new URL is a new page.

I have seen forums with 1000 postings with over 100k generated URLs!

So what do you do? One possibility would be to trim the URLs with the sitemap tool and only submit the ones you like (my GSiteCrawler will let you do that). However, doing that is only a small band-aid on the real problem: the search engines will find all those URLs anyway and play poker to guess at which ones you prefer. The only clean solution is to go through your forum and make sure that it only generates one crawlable + indexable URL per content block.

It'll take some time to get cleaned out, once you have it online like that, but it's well worth it in the end. Would you rather have 100 postings with the same content ranking at positions 100+ or 1 posting at position 10? ;-)