How robot.txt useful to SEO?

Discussion in 'Search Engine Optimization' started by ndarsana, Apr 5, 2011.

  1. #1
    My site is protected by a robots.txt file, which does not allow directory submissions agent to access my site, will it affect link building? What can be done?
     
    ndarsana, Apr 5, 2011 IP
  2. Aquarezz

    Aquarezz Notable Member

    Messages:
    3,009
    Likes Received:
    69
    Best Answers:
    0
    Trophy Points:
    225
    #2
    What exactly do you mean? IMO your post doesn't make any sense at all at the moment. What has blocking search spiders to access one page to do with link building?
     
    Aquarezz, Apr 5, 2011 IP
  3. ndarsana

    ndarsana Peon

    Messages:
    40
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Will robot.txt affect indexing of web page?
     
    ndarsana, Apr 5, 2011 IP
  4. Aquarezz

    Aquarezz Notable Member

    Messages:
    3,009
    Likes Received:
    69
    Best Answers:
    0
    Trophy Points:
    225
    #4
    Kind of. They will crawl your website as they would normally do, but won't crawl the pages you have disallowed ;)
     
    Aquarezz, Apr 5, 2011 IP
  5. onlineprime08

    onlineprime08 Peon

    Messages:
    255
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    If you have disallowed a page of yours into. for example, for directory agents that you are talking about.
    Then your link building will not include directory submission and may give you a limit on your link building for a site.

    Robots.txt helps crawl your site and index them.
     
    onlineprime08, Apr 5, 2011 IP
  6. rosesmark

    rosesmark Active Member

    Messages:
    542
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    75
    #6
    its useful for your website indexing.when bot enter in your website if robot.txt file include so that bot read your robot.txt file and then bot index your page which available on that file.

    This allows all robots to crawl all files.

    User-agent: *
    Disallow:
     
    rosesmark, Apr 5, 2011 IP
  7. Brunzig

    Brunzig Peon

    Messages:
    55
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Thanks to share that information with us.
     
    Brunzig, Apr 5, 2011 IP
  8. John_Marshal

    John_Marshal Peon

    Messages:
    221
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #8
    robot.txt is very useful to enable the crawler to access the webpages and its data. if you disable the crawler by robot.txt it will not cache the webpage for seo purpose you should always enable the web crawlers.
     
    John_Marshal, Apr 5, 2011 IP
  9. Kiranraj

    Kiranraj Well-Known Member

    Messages:
    1,583
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    128
    #9
    It is helpful to get correct page indexed by the Search engine crawler
     
    Kiranraj, Apr 7, 2011 IP