How important is robot.txt in SEO?

Discussion in 'robots.txt' started by circuscircus, Jan 21, 2007.

Thread Status:
Not open for further replies.
0
  1. #1
    If I only care about SEO and it reaching higher ranks on search engines, should a robot.txt be on my priority list?
     
    circuscircus, Jan 21, 2007 IP
  2. dhruv37

    dhruv37 Well-Known Member

    Messages:
    3,222
    Likes Received:
    195
    Best Answers:
    0
    Trophy Points:
    165
    #2
    robot.txt is file which really helpfull when you want certaing pages / folder of the site not to crawl by spiders.
     
    dhruv37, Jan 22, 2007 IP
  3. adverlicious

    adverlicious Peon

    Messages:
    37
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    As you get into more advanced optimization, you'll definitely want to be using your robots.txt file as part of your larger toolkit.

    It can help you, for instance, keep unwanted pages from crowding out your targeted ones. Likewise, it can reduce your risk of duplicate content penalties when you need to have the same content in 2 places.

    It's your friend ;)
     
    adverlicious, Jan 25, 2007 IP
  4. meganerd

    meganerd Peon

    Messages:
    105
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #4
    It's not needed in your case. You can lean more towards a sitemap.
     
    meganerd, Jan 25, 2007 IP
  5. eTIME

    eTIME Banned

    Messages:
    128
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    actully robot.txt help to don't index the pages that you don't want crawal by spiders.
     
    eTIME, Jan 29, 2007 IP
  6. jonbt

    jonbt Peon

    Messages:
    11
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    For a wordpress blog or other dynamic website robots.txt should be considered essential to eliminate duplicate posts and non-seo title pages and filenames.

    Here is my wordpress robots.txt

    
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /wp-admin/
Disallow: /page/
Disallow: /comments/
Disallow: /wp-
Disallow: /xmlrpc.php
Disallow: /index.php
Disallow: /category/
Disallow: /?p
Disallow: /about/trackback/
Disallow: /wp-register.php
Disallow: /wp-login.php
Disallow: /stats/
Disallow: /dh_
Disallow: /*?*
Disallow: /contact/
Disallow: /tag
    Code (markup):
     
    jonbt, Jan 30, 2007 IP
Thread Status:
Not open for further replies.