Big Sites Dont Need All Pages on Sitemap

Discussion in 'Search Engine Optimization' started by seofighter, Jul 16, 2010.

  1. #1
    Hi,,
    every one


    As we know sitemaps are intended to help the bots crawl the site better. For bigger sites listing all the pages could be a nightmare; sites like forums, classified ads etc.

    I think if we have a nice internal linking structure, we dont need to list all the pages on sitemap-we can just list the major pages.

    waiting your reply:)
     
    seofighter, Jul 16, 2010 IP
  2. freeseoguide

    freeseoguide Guest

    Messages:
    95
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    there are different opinions on sitemapas and if they have any influence on positions but yes, you are right... nice internal linking structure will for sure improve crawling your website.

    also, important to mention, if you have large number of pages, and want all of them to be indexed, you should have enough link jouce... website with 5000 pages with 3 incoming links, is it normal ? :)
     
    freeseoguide, Jul 16, 2010 IP
  3. webdevelopmentcompany

    webdevelopmentcompany Peon

    Messages:
    139
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    agree with u but as per my opinion u should create sitemap for whole website indexing.
     
    webdevelopmentcompany, Jul 16, 2010 IP
  4. jitendraag

    jitendraag Notable Member

    Messages:
    3,982
    Likes Received:
    324
    Best Answers:
    1
    Trophy Points:
    270
    #4
    Search engines recognize that issue and allow multiple sitemaps. If your site has GREAT navigation and no dangling pages, ideally you don't even need a sitemap to get all pages indexed.
     
    jitendraag, Jul 16, 2010 IP
  5. mohithanche

    mohithanche Peon

    Messages:
    70
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    yeah i agree with you, but why to take chance, as Major SE Guideline is to submit a XML sitemap.
     
    mohithanche, Jul 19, 2010 IP
  6. blings

    blings Peon

    Messages:
    152
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    I agree on this. If your site's interlinking structure allows even the deepest web pages to being discover by bots, then having it declared through sitemap isn't necessary.
     
    blings, Jul 19, 2010 IP
  7. Mikeamit

    Mikeamit Guest

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    yes, I agree with you but as per my opinion you should create sitemap for whole website indexing. very important is xml sitemap.
     
    Mikeamit, Jul 19, 2010 IP
  8. subburajacmic

    subburajacmic Peon

    Messages:
    162
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Sitemap is very important for search engine crawl. If you have sitemap, all the pages will index in search engine. We can't leave any option in SEO.
     
    subburajacmic, Jul 19, 2010 IP
  9. social-media

    social-media Member

    Messages:
    311
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    35
    #9
    Sitemap.xml files shouldn't be needed for any sites EXCEPT very large sites with hundreds of thousands or millions of URLs. They do NOT get more pages indexed unless your site has crawlability issues preventing the crawlers from finding all of your URLs. They do NOT get your pages indexed faster. Google uses it to "assist" with the crawl, meaning to find URLs they might not be able to find otherwise and to prioritize which pages are most important (using the <priority> element in the sitemap.xml).

    Sitemaps are good for 3 situations IMO:

    1) to get a new site with no inbound links indexed
    2) to help with sites that have crawlability issues
    3) to prioritize URLs for sites with hundreds of thousands or millions of URLs.

    You really should never need a sitemap.xml for #1 above. Simply build inbound links from other sites and Google will discover your site naturally. Getting indexed won't get you ranked. Inbound links will.

    If you have a good internal linking structure and navigation, you shouldn't have crawlability issues with your site. Google and the other engines should be able to find all of your pages by following links on your if you have a good linking structure. So using a sitemap for #2 above should be VERY rare.

    Sites with even 10s of thousands of URLs should not require a sitemap assuming, again, that they have a good internal linking structure from which the search engines can derive which pages on the site are most important. But if you have LOTS of pages (like in the hundreds of thousands to millions range), you may want to use the <priority> element in the sitemap.xml to specify which pages are most important to be indexed.

    Other than those situations, sitemap.xmls really have no value from an SEO perspective.
     
    social-media, Jul 19, 2010 IP
  10. johnk681

    johnk681 Active Member

    Messages:
    240
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    63
    #10
    I generally create site map separately for all major site, like main site, forum site, social site etc.
     
    johnk681, Jul 20, 2010 IP
  11. Matthew0126

    Matthew0126 Guest

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    divide your sitemap into serveral files.
    PS.follow the structure of your website
     
    Matthew0126, Jul 20, 2010 IP
  12. asghar.paracha

    asghar.paracha Well-Known Member

    Messages:
    720
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    118
    #12
    I am little confuse about my website sitemap because it has 500+ pages including blog links and most of the sitemap generators dont allow to create a sitemap for no more then 500 links
     
    asghar.paracha, Jul 20, 2010 IP
  13. blings

    blings Peon

    Messages:
    152
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    @social-media that's a comprehensive response you got there. It should help others understand more about the story behind why sitemap existed.
     
    blings, Jul 20, 2010 IP
  14. edfranklin

    edfranklin Peon

    Messages:
    82
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #14
    Yes, I agree with you. If you have good internal linking structure then there is no need sitemap. Using the sitemap we are telling search engines to crawl these pages or giving intimation about our site. It is better to create sitemap but that is not mandatory
     
    edfranklin, Jul 20, 2010 IP
  15. catanich

    catanich Peon

    Messages:
    1,921
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #15
    The reason the SEs created sitemaps was to speed up indexing of the site. For those of you that have large sites and don't use sitemaps, then you may have found out why those deep link pages are not indexed yet.

    A sitemap tells the SEs what pages you have and they should index. Not using a complete list of urls is just dumb guys.
     
    catanich, Jul 20, 2010 IP
  16. catanich

    catanich Peon

    Messages:
    1,921
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #16
    And for those of you that have large sites (>500 pages), use gSiteCrawler to do the very large sites.
     
    catanich, Jul 20, 2010 IP
  17. webuyhousecash1

    webuyhousecash1 Guest

    Messages:
    57
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #17
    Big sites need sitemap. Because we have to concentrate all pages for search engine crawl. If we have sitemap, that is easy for indexing all the pages.
     
    webuyhousecash1, Jul 20, 2010 IP
  18. ASPMachine

    ASPMachine Peon

    Messages:
    723
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Using sitemap is good. But SE never claim to be indexed all the links that have included into the sitemap. it's never mind what kind of sitemap you're using. I think we may try another way to get index all of our internal pages instead of using sitemap. Fist of all, when I'd used sitemap at Google, only 10-15% of my link were crawled by Google. So, I used a another tics to get indexed my pages. Now I've enough indexed pages at Google for my movie website.

    We all know it very well that Google index homepage of a website instead of internal pages more often. So, I've designed my site such a way that when I add any new movie, the link displays at homepage. When Google reindex my homepage, he gets a dynamic update of my homepage and crawl the internal link. It's not a theory. I got full prove of it. And now I don't need any sitemap to get index my internal page link. This was my story. :)

    Keep reflect your update/added page at homepage by an anchor link with little more contents

    This will definitely help you.
     
    ASPMachine, Jul 20, 2010 IP
  19. rahulamaze

    rahulamaze Active Member

    Messages:
    287
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    53
    #19
    Hi,

    Site map is to help to see the all page in the simply way, and it's also very helpful for visitors, for my opinion, you should create all pages in site map/......
     
    rahulamaze, Jul 20, 2010 IP