1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Does your robots.txt file include sitemap references?

Discussion in 'Google Sitemaps' started by The SEO Man, Mar 7, 2019.

  1. #1
    I've noticed that a lot of sites I've recently evaluated do not have the XML and HTML sitemaps referenced with the robots.txt file.

    Including sitemaps in the robots.txt is known to be good for SEO and helps GoogleBot with crawlability.
    SEMrush
    How important is it to you that your robots.txt file includes sitemap references?
     
    The SEO Man, Mar 7, 2019 IP
    SEMrush
  2. qwikad.com

    qwikad.com Illustrious Member Affiliate Manager

    Messages:
    5,759
    Likes Received:
    1,069
    Best Answers:
    21
    Trophy Points:
    400
    #2
    I always thought that Google will crawl your site (and sitemap) whether you have a Google webmaster account or not, or whether you include sitemaps in your robots.txt file or not. As long as you don't block Googlebot it will crawl your site one way or another. Maybe including sitemaps in your robots.txt expedites the process, but I don't see how it's necessarily good or beneficial for SEO in the long run.
     
    qwikad.com, Mar 7, 2019 IP
  3. mmerlinn

    mmerlinn Notable Member

    Messages:
    2,055
    Likes Received:
    248
    Best Answers:
    6
    Trophy Points:
    240
    #3

    The only time sitemaps are really needed is for very complex sites where the indexing bots might not find important pages in a timely manner.

    I never even considered a sitemap for my site until I had over 9,000 pages and NEVER had a problem being indexed. If maintaining sitemaps wasn't so easy, I would probably not have a sitemap on my site today, even though I currently have over 13,000 pages.

    Comparing before and after, I see NO DIFFERENCE in how my site is indexed nor where it lands in the SERPS.
     
    mmerlinn, Mar 8, 2019 IP
  4. sarahk

    sarahk iTamer Staff

    Messages:
    24,831
    Likes Received:
    3,365
    Best Answers:
    98
    Trophy Points:
    615
    #4
    Sitemaps are pretty easy to create with most CMS so I'd make one and register it but I suspect for an actively maintained, well structured site that it's overkill. Adding it to Robots seems like trying to double dip - the file is really to tell Robots what NOT to do, rather than point them towards content.
     
    sarahk, Mar 8, 2019 IP
    dcristo likes this.