1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Does your robots.txt file include sitemap references?

Discussion in 'Google Sitemaps' started by The SEO Man, Mar 7, 2019.

0
  1. #1
    I've noticed that a lot of sites I've recently evaluated do not have the XML and HTML sitemaps referenced with the robots.txt file.

    Including sitemaps in the robots.txt is known to be good for SEO and helps GoogleBot with crawlability.

    How important is it to you that your robots.txt file includes sitemap references?
     
    The SEO Man, Mar 7, 2019 IP
  2. qwikad.com

    qwikad.com Illustrious Member Affiliate Manager

    Messages:
    7,151
    Likes Received:
    1,656
    Best Answers:
    29
    Trophy Points:
    475
    #2
    I always thought that Google will crawl your site (and sitemap) whether you have a Google webmaster account or not, or whether you include sitemaps in your robots.txt file or not. As long as you don't block Googlebot it will crawl your site one way or another. Maybe including sitemaps in your robots.txt expedites the process, but I don't see how it's necessarily good or beneficial for SEO in the long run.
     
    qwikad.com, Mar 7, 2019 IP
  3. mmerlinn

    mmerlinn Prominent Member

    Messages:
    3,197
    Likes Received:
    818
    Best Answers:
    7
    Trophy Points:
    320
    #3

    The only time sitemaps are really needed is for very complex sites where the indexing bots might not find important pages in a timely manner.

    I never even considered a sitemap for my site until I had over 9,000 pages and NEVER had a problem being indexed. If maintaining sitemaps wasn't so easy, I would probably not have a sitemap on my site today, even though I currently have over 13,000 pages.

    Comparing before and after, I see NO DIFFERENCE in how my site is indexed nor where it lands in the SERPS.
     
    mmerlinn, Mar 8, 2019 IP
  4. sarahk

    sarahk iTamer Staff

    Messages:
    28,500
    Likes Received:
    4,460
    Best Answers:
    123
    Trophy Points:
    665
    #4
    Sitemaps are pretty easy to create with most CMS so I'd make one and register it but I suspect for an actively maintained, well structured site that it's overkill. Adding it to Robots seems like trying to double dip - the file is really to tell Robots what NOT to do, rather than point them towards content.
     
    sarahk, Mar 8, 2019 IP
    dcristo likes this.
  5. Gadi and Deborah Slade

    Gadi and Deborah Slade Greenhorn

    Messages:
    14
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    13
    #5
    It is good practice for SEO.
     
    Gadi and Deborah Slade, Apr 30, 2019 IP
  6. Tech9logyCreators

    Tech9logyCreators Member

    Messages:
    25
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #6
    It is a beneficial factor to include your sitemap in robots.txt file. You don't need to make much efforts in that, Just copy and paste the .xml sitemap link in your robots.txt file. The benefit of it is that when bots go through your robots file it also go through your sitemap which has a good effect on your crawling
     
    Tech9logyCreators, May 15, 2019 IP
  7. Paul Nieman

    Paul Nieman Greenhorn

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #7
    Plenty of people out there do seem to think it's worth the trouble learning good syntax and because I'm weird I enjoy getting it right. Make friends with the bots and they will reward you. I have a robots.txt syntax question. If I write the following
    Disallow: /*.pdf$
    Will it apply throughout the site including directories or should I repeat the code for every directory level. Thanks.
     
    Paul Nieman, Aug 23, 2019 IP
  8. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #8
    I would not reference HTML sitemaps in robots.txt - only your XML sitemaps
     
    websitetools, May 1, 2020 IP
  9. jeep98

    jeep98 Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #9
    It is not necessary to have a map site in robots.txt, but it's better than not having it
    It is important that the site map is optimized
    Use this plugin to create an optimal sitemap
    wordpress.org/plugins/google-sitemap-generator
     
    jeep98, Jan 15, 2022 IP