Are you using a robots.txt file?

Discussion in 'Search Engine Optimization' started by Mr.Dog, Nov 12, 2013.

  1. #1
    I honestly don't unless I have some special thing to tell Google. I don't want to block any directory or other content, so I don't use it.

    Although, Google recommends it, it sounds like it's compulsory:
    "Make use of the robots.txt file on your web server"

    What is your opinion? Are there any downsides if you don't use one?
     
    Mr.Dog, Nov 12, 2013 IP
  2. Kianu42

    Kianu42 Member

    Messages:
    510
    Likes Received:
    15
    Best Answers:
    5
    Trophy Points:
    43
    #2
    The upside of using a robots.txt is that it's yet another document Google can consult when deciding what to index and what not to index on your site.
    Sometimes, there are duplicate URL issues (in fact, it's not the URLs that are duplicate, but the content on them but you get the idea).
    To an extent this problem is solved with the XML sitemap and/or canonical tags.
    But a robots.txt file could be a good confirmation.
    Actually to me it's never daunting, since a robots.txt generator+editor comes with the on page optimisation audit tool I use.
     
    Kianu42, Nov 12, 2013 IP
  3. SoftDive Technologies

    SoftDive Technologies Greenhorn

    Messages:
    22
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    21
    #3
    Ya, You can use robots.txt for a formalty because google prefer robots.txt file and in robots.txt file you can simply write
    User-agent: *
    Allow: /

    You have no need to block any one url or any thing.
     
    SoftDive Technologies, Nov 12, 2013 IP
  4. Omnitec1018

    Omnitec1018 Member

    Messages:
    92
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    38
    #4
    Yes! I also use it just like a formality nothing else, because I don't have any URL to block in my website.
     
    Omnitec1018, Nov 12, 2013 IP
  5. Mr.Dog

    Mr.Dog Active Member

    Messages:
    912
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    60
    #5
    So you are saying I should just use a "blank" one?

    On another Google page, Google says "No", it's not compulsory. But in the mentioned quote (in my first post here), it seems like they do recommend one.

    Ah well, I guess it's no major fuss about it...
     
    Mr.Dog, Nov 12, 2013 IP
  6. webdev007

    webdev007 Active Member

    Messages:
    1,037
    Likes Received:
    13
    Best Answers:
    3
    Trophy Points:
    88
    #6
    Google just reminds you (if your are missing something.. a Robots.txt file etc.). Also its not compulsory and you can omit it easily. Some of my sites also don't have it all. If you are in doubt, then create one (as others suggested) and do not block anything.
     
    webdev007, Nov 12, 2013 IP