How much is TOO much?

Discussion in 'Google Sitemaps' started by tkroll, Aug 14, 2006.

0
  1. #1
    Hello peoples!

    I'm wondering if there is a point with G sitemaps that triggers the spam sensors? If I include all my forum posts as individual pages, topic pages, etc., am I just asking for it? :eek:

    Related question: If a post can be on its own page and on a topic summary page, should I be worried about duplicate content?

    Thanks,
    Ty
     
    tkroll, Aug 14, 2006 IP
  2. bizhobby

    bizhobby Peon

    Messages:
    692
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Don't do it! I've submited 1.2 million pages and it just started removing pages from the index. Then sometime today I dropped it and now it's going back up... slow but steady... The thing is... Sitemaps is NOT going to get your pages in faster. So don't bother. It will eventually find it.

    I only did it because I wanted to tell it not to recrawl them over and over again and to have in mind I have a LOT more that it thought...
    I've had 30,000 pages indexed when I dropped down to 13,000 I freaked out deleted the sitemaps and now I'm going steady up to like 14,500 (last few hours)
     
    bizhobby, Aug 14, 2006 IP
  3. Design1

    Design1 Active Member

    Messages:
    388
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    78
    #3
    I've been wondering this myself.. and if indeed we should not index forums, what about all the sites out there that are basically nothing but forums? I started running a site map generator on one of my sites and the forums are about the smallest part of the site but yet accounted for a lot of pages. Most of which have nothing to do with the theme of my site..
     
    Design1, Aug 21, 2006 IP
  4. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #4
    If your sitemap generator (of type: website crawler) supports various filters, you may for a large part be able to avoid duplicate content appearing in your generated xml sitemaps.
     
    websitetools, Aug 22, 2006 IP
  5. bizhobby

    bizhobby Peon

    Messages:
    692
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #5
    That is a great idea!!!!!!! Avoiding duplicate content in your sitemap. WOW! GENIUS!!! THANKS!

    Because my site is using Associate-O-Matic (an Amazon store engine) and therefore there IS a lot of duplicate content and that's probably why I get penalized even though there are tons of good and legit pages.
     
    bizhobby, Aug 23, 2006 IP
  6. GeorgR.

    GeorgR. Peon

    Messages:
    2,831
    Likes Received:
    78
    Best Answers:
    0
    Trophy Points:
    0
    #6
    yes....but WHAT exactly to filter ? I am using gsitecrawler...and yes, if i let it rip it would probably return something in the zillion-pages.....so what do i filter ?

    I need a setup/filter so it submits/crawls maybe a max. 5000 relevant page. The crawler would go into everything...i still havent figured out how exactly and what to filter..i do NOT want to filter the whole store either.

    (talking about AoM/Associate-o-matic now)
     
    GeorgR., Dec 30, 2006 IP