Which robots.txt should I use?

Discussion in 'robots.txt' started by knight1, Oct 16, 2009.

0
  1. #1
    Hello,

    I know a xml sitemap looks like this: "http://www.yoursite.com/sitemap.xml"

    and adding .gz at the end is for your zipped compressed version.

    Okay, now I've heard the purpose for the compressed version is for search
    engines pretty much. But I don't see any million dollar sites or blogs using
    a compressed version for their robots.txt. They all use just their regular
    xml sitemap.

    Is there really any difference which one you use for your robots.txt? Is it okay to use "http://www.yoursite.com/sitemap.xml.gz" or should I just use the regular one?

    Thanks for any advice, I did search and search before I asked such a newb question.

    cheers!
     
    knight1, Oct 16, 2009 IP
  2. shanmugasundaram

    shanmugasundaram Active Member

    Messages:
    230
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    70
    #2
    if you are using a wordpress blog then need not worry about sitemaps or anything

    just promote your site and never mind about crawling

    if you add contents daily then after few months your contents will be indexed within hours i.e. spider will assume that your blog gets updated daily so it will visit often
     
    shanmugasundaram, Oct 17, 2009 IP
    knight1 likes this.
  3. Dan Schulz

    Dan Schulz Peon

    Messages:
    6,032
    Likes Received:
    436
    Best Answers:
    0
    Trophy Points:
    0
    #3
    If you have a "small" XML sitemap then don't worry about compression. When you get into the tens of thousands of links though, then start looking at it.
     
    Dan Schulz, Oct 18, 2009 IP
    knight1 likes this.
  4. knight1

    knight1 Peon

    Messages:
    281
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Appreciate it guys, thanks for the responses to my robots.txt question. I understand the difference between them now.
     
    knight1, Oct 18, 2009 IP
  5. CODForum

    CODForum Guest

    Messages:
    141
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    compressed sitemaps are for HUUUUUUUUUUUGE forums that have thousands of links
    webmasters compress their sitemaps so search engines dont eat up so much bandwidth
    robots.txt is never compressed, because it is never over 1KB in size :)
     
    CODForum, Oct 22, 2009 IP
  6. harry009

    harry009 Active Member

    Messages:
    93
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    73
    #6
    oh thanks for sharing nice information. from today i understand the difference between compressed sitemaps and robots.txt.
     
    harry009, Oct 22, 2009 IP
  7. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #7
    Hi Knight1, just to make you correct, I want to apprise you that your question was not related to robots.txt but it was about sitemap.xml.
     
    manish.chauhan, Oct 28, 2009 IP
  8. mjreine

    mjreine Well-Known Member

    Messages:
    110
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    101
    #8
    Bottom line...dont worry about compressed sitemaps unless you run an HUGE forum or ecommerce site with 10,000+ products. Do you?
     
    mjreine, Nov 15, 2009 IP