Question for Robots.txt file

Discussion in 'Search Engine Optimization' started by taham, Jan 21, 2011.

0
  1. #1
    Hi to all,

    I don't want to restrict bots for crawled to all pages for my website. So, in my robots.txt file I write only the below code. Is this affect anythings in my website? Please let me know about that.

    User-agent:*
    Disallow:
     
    taham, Jan 21, 2011 IP
  2. pavanseo

    pavanseo Active Member

    Messages:
    144
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    53
    #2
    This code means that crawler will read everything in your website. Few pages like term & condition etc need to disallow because these pages may have some duplicate content
     
    pavanseo, Jan 21, 2011 IP
  3. Surajdeshmukh

    Surajdeshmukh Peon

    Messages:
    106
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Your robots.txt file code needs to be changed as
    User-agent:*
    Allow:
    Please do these changes. It will help you.
     
    Surajdeshmukh, Jan 22, 2011 IP
  4. iproficientseo

    iproficientseo Peon

    Messages:
    50
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I completely agree with Pavan, TOC and login pages are some of the few pages that should be disallowed by crawling.... it can affect your site at some extent... :)
     
    iproficientseo, Jan 22, 2011 IP