Help with Robots.txt

Discussion in 'robots.txt' started by virtualmisc, Oct 1, 2009.

0
  1. #1
    What is a robots.txt?
    How important is it?
     
    virtualmisc, Oct 1, 2009 IP
  2. shanmugasundaram

    shanmugasundaram Active Member

    Messages:
    230
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    70
    #2
    if you have a wordpress blog then you dont need it but if you want to put some restrictions on scrolling then u need it
     
    shanmugasundaram, Oct 1, 2009 IP
  3. AKSIN

    AKSIN Peon

    Messages:
    159
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    robots.txt helps search engine to determine which pages of your website to crawl or not to crawl. It can be created as follows: make a file as yoursite(.)com/robots.txt and upload
    User-Agent: *
    Allow: /
     
    AKSIN, Oct 9, 2009 IP