Hi there, I own a large forum and have never produced a sitemap before. I tried GSiteCrawler and an online link I saw on here but they took hours upon hours to produce and I didn't let them finish. I wondered if anybody could either recommend to me: a) a quicker method of using these programs... settings I might have to change etc b) alternative ways of producing sitemaps which might be more suitable for a site with a large number of pages. Many thanks, Rob
Read my Google Sitemap FAQ: http://forums.digitalpoint.com/showthread.php?t=531765 GSitemap is great imo for crawling any website, its fast, free and reliable.
Then I would check out your site. I have been using it for ~ 1 year without a problem. Are you using the current version?
Yes, I downloaded it from the official site yesterday. It's very strange that it does this - perhaps I will try again tomorrow.
g-sitecrawler is the best program outthere I know. But if you site is very large, your computer slow, and your upload bandwith low, it can take a while. And you also need enough RAM, to make it run smooth. If you have enough of the above, you can go to the settings and get more bots to crawl your website which makes the hole process faster. My g-sitecrawler runs for more than a day due to my small upload bandwith :/
why does google bother with this? With a large forum, all the links are there, and they are dynamic and growing every day. Google should just spider. The robots.txt method of telling google what NOT to crawl is fine. If my forum has 1,000,000 threads to crawl, it is a ridiculous job to have a site map, isn't it?
How many pages are we roughly speaking? and how many hours did you give it? e.g. 100.000 pages is going to take at least a few hours no matter what. Forum sites in particular can be rather DB extensive and thus sometimes slow if not located on a very capable server. Anyways, before you give up, I suggest you also try the newest version of A1 Sitemap Generator. One thing you may want to consider for a forum website: Filter out "profile", "login" etc. pages.
I've found that creating a sitemap for my forum using a plug-in for vbulletin, and then creating a separate one for the rest of my site is much easier. Is there any disadvantage to having two sitemaps as opposed to just one? Rob
Since you are using vBulletin i Strongly recommend you to use vBSEO sitemap generator. It is free, and it is quite easy to install&configure. You can get it from vbulletin.org You will also see an increase in traffic from search engines as it has an automatic pinging capabilities on new sitemap creation to Google and Yahoo crawlers..
That's the plug-in I've installed, and I couldn't be happier with it - it's doing everything automatically for me. My outstanding issue is the rest of my site which is not forum-based. I'd ideally like my vbSEO Sitemap to include it, but I don't think it's possible
There is aotehr text file where you can also submit URL's outside of your forum. I think that file is called something like extra-urls.txt. You need to put one URL per line like this http://www.mysite.com/ http://www.mysite.com/media/ http://blog.mysite.com/ I hope you get the idea. Once you get this settled, you owe me a beer.
Do I have to place every single URL of my site in the extra-urls.txt In effect I'm typing up my own sitemap then?! Or will it crawl those URLs for other non-forum URLs? If I get it set up I'll buy you TWO beers - I'm in a Christmas mood!
You will have to place any subdomain URL's and website directories, and then the sitemap generator will take care of the rest of links inside subdomain/directories. Just like my three examples above, i.e. if you place http://www.mysite.com/media/ then sitemap generator will find links such as: http://www.mysite.com/media/paris-hilton-uncovered.html etc... Regards.