1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Sitemap exclusions

Discussion in 'Google Sitemaps' started by ultimatehandyman, Aug 3, 2006.

  1. #1
    Hi,

    I have a diy website that has over 400 static pages and a phpbb forum.

    For the sitemap I use the online generator at auditmypc.com

    If I do a full crawl of my site it generates a sitemap that is >12mb and this makes submitting the sitemap more complicated as it has to be <10mb. I started to exclude files and folders from my forum and now the sitemap is roughly 260kb, but I am concerned that most of the forum is not being crawled.

    Does anyone know which files and folders I should be including from the forum and which ones I should be excluding?

    Thanks in advance.
    SEMrush
     
    ultimatehandyman, Aug 3, 2006 IP
    SEMrush
  2. MaxPowers

    MaxPowers Peon

    Messages:
    261
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #2
    For phpBB, there are a number of exclusions that I filter on my site. First of all, anything with 'highlight' as a query key... no need for it. I also filter out any forums pages where 'mode=quote', 'mode=reply', 'mode=newtopic', 'folder=inbox'

    I also block out any URL with 'search_author' as a query key.

    I remove certain keys from the string, but still allow URLs that use 'postdate' and 'postorder' as keys because these just sort the same page up and down, but it still shows the same posts without these in the URL and it cuts way down on the number of URLs.

    All of these URLs are different versions of the same page really and while they help users get around and work in phpBB, they don't really offer any new content so I avoid them on my sitemap. I'm not sure if these filters are available through your sitemap service or not, but there are loads of unique filters where I get my site maps ;)
     
    MaxPowers, Aug 4, 2006 IP
  3. livingingermany

    livingingermany Active Member

    Messages:
    311
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    58
    #3
    Great advice MaxPowers, I also have the same problem on one of my forum sites, will try your tips now.
    Thanx
     
    livingingermany, Aug 4, 2006 IP