1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Sitemap Help for Large Forum

Discussion in 'Google Sitemaps' started by robsharpe, Oct 28, 2007.

  1. #1
    Hi there,

    I own a large forum and have never produced a sitemap before. I tried GSiteCrawler and an online link I saw on here but they took hours upon hours to produce and I didn't let them finish.

    I wondered if anybody could either recommend to me:
    a) a quicker method of using these programs... settings I might have to change etc
    b) alternative ways of producing sitemaps which might be more suitable for a site with a large number of pages.

    Many thanks,
    Rob
    SEMrush
     
    robsharpe, Oct 28, 2007 IP
    SEMrush
  2. astup1didiot

    astup1didiot Notable Member

    Messages:
    5,927
    Likes Received:
    269
    Best Answers:
    0
    Trophy Points:
    280
    #2
    astup1didiot, Oct 28, 2007 IP
  3. robsharpe

    robsharpe Peon

    Messages:
    258
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I just tried GSitemap a few times but it keeps stalling after around 300 entries.
     
    robsharpe, Oct 28, 2007 IP
  4. catanich

    catanich Peon

    Messages:
    1,921
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Then I would check out your site. I have been using it for ~ 1 year without a problem. Are you using the current version?
     
    catanich, Oct 29, 2007 IP
  5. robsharpe

    robsharpe Peon

    Messages:
    258
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Yes, I downloaded it from the official site yesterday. It's very strange that it does this - perhaps I will try again tomorrow.
     
    robsharpe, Oct 29, 2007 IP
  6. evera

    evera Peon

    Messages:
    283
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #6
    g-sitecrawler is the best program outthere I know. But if you site is very large, your computer slow, and your upload bandwith low, it can take a while.
    And you also need enough RAM, to make it run smooth.
    If you have enough of the above, you can go to the settings and get more bots to crawl your website which makes the hole process faster.

    My g-sitecrawler runs for more than a day due to my small upload bandwith :/
     
    evera, Nov 1, 2007 IP
  7. ruffnat

    ruffnat Peon

    Messages:
    394
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #7
    why does google bother with this? With a large forum, all the links are there, and they are dynamic and growing every day. Google should just spider. The robots.txt method of telling google what NOT to crawl is fine.

    If my forum has 1,000,000 threads to crawl, it is a ridiculous job to have a site map, isn't it?
     
    ruffnat, Nov 3, 2007 IP
  8. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #8
    How many pages are we roughly speaking? and how many hours did you give it? e.g. 100.000 pages is going to take at least a few hours no matter what. Forum sites in particular can be rather DB extensive and thus sometimes slow if not located on a very capable server.

    Anyways, before you give up, I suggest you also try the newest version of A1 Sitemap Generator.

    One thing you may want to consider for a forum website: Filter out "profile", "login" etc. pages.
     
    websitetools, Nov 7, 2007 IP
  9. ruffnat

    ruffnat Peon

    Messages:
    394
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #9
    so how often should you replace your sitemap for a large forum?
     
    ruffnat, Nov 9, 2007 IP
  10. kmap

    kmap Well-Known Member

    Messages:
    2,215
    Likes Received:
    29
    Best Answers:
    2
    Trophy Points:
    135
    #10
    kmap, Nov 9, 2007 IP
  11. robsharpe

    robsharpe Peon

    Messages:
    258
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #11
    I've found that creating a sitemap for my forum using a plug-in for vbulletin, and then creating a separate one for the rest of my site is much easier.

    Is there any disadvantage to having two sitemaps as opposed to just one?

    Rob
     
    robsharpe, Dec 5, 2007 IP
  12. Ahmad9999

    Ahmad9999 Peon

    Messages:
    229
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #12
    me also i need help with the site map my url www.Dove4.com please help
     
    Ahmad9999, Dec 6, 2007 IP
  13. ablaye

    ablaye Well-Known Member

    Messages:
    4,022
    Likes Received:
    96
    Best Answers:
    0
    Trophy Points:
    100
    #13
    I agree with that. Sitemaps are not practical for large forums.
     
    ablaye, Dec 6, 2007 IP
  14. ruffnat

    ruffnat Peon

    Messages:
    394
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #14
    I doubt this forum has a sitemap
     
    ruffnat, Dec 6, 2007 IP
  15. profire

    profire Well-Known Member

    Messages:
    199
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    103
    #15
    Since you are using vBulletin i Strongly recommend you to use vBSEO sitemap generator. It is free, and it is quite easy to install&configure.
    You can get it from vbulletin.org

    You will also see an increase in traffic from search engines as it has an automatic pinging capabilities on new sitemap creation to Google and Yahoo crawlers..
     
    profire, Dec 8, 2007 IP
  16. robsharpe

    robsharpe Peon

    Messages:
    258
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #16
    That's the plug-in I've installed, and I couldn't be happier with it - it's doing everything automatically for me.

    My outstanding issue is the rest of my site which is not forum-based. I'd ideally like my vbSEO Sitemap to include it, but I don't think it's possible :(
     
    robsharpe, Dec 9, 2007 IP
  17. profire

    profire Well-Known Member

    Messages:
    199
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    103
    #17
    There is aotehr text file where you can also submit URL's outside of your forum. I think that file is called something like extra-urls.txt. You need to put one URL per line like this

    http://www.mysite.com/
    http://www.mysite.com/media/
    http://blog.mysite.com/

    I hope you get the idea.

    Once you get this settled, you owe me a beer. ;)
     
    profire, Dec 9, 2007 IP
  18. robsharpe

    robsharpe Peon

    Messages:
    258
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Do I have to place every single URL of my site in the extra-urls.txt In effect I'm typing up my own sitemap then?! Or will it crawl those URLs for other non-forum URLs?

    If I get it set up I'll buy you TWO beers - I'm in a Christmas mood! :p
     
    robsharpe, Dec 9, 2007 IP
  19. profire

    profire Well-Known Member

    Messages:
    199
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    103
    #19
    You will have to place any subdomain URL's and website directories, and then the sitemap generator will take care of the rest of links inside subdomain/directories.

    Just like my three examples above, i.e. if you place http://www.mysite.com/media/ then sitemap generator will find links such as: http://www.mysite.com/media/paris-hilton-uncovered.html :D etc...

    Regards.
     
    profire, Dec 9, 2007 IP