help with my robots.txt file

Discussion in 'robots.txt' started by YSNY, Sep 25, 2009.

0
  1. #1
    YSNY, Sep 25, 2009 IP
  2. glo

    glo Peon

    Messages:
    36
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Change Disallow with Allow.
     
    glo, Sep 26, 2009 IP
  3. YSNY

    YSNY Peon

    Messages:
    52
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    allow all????
     
    YSNY, Sep 26, 2009 IP
  4. Baster

    Baster Peon

    Messages:
    44
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Disallow: - this unnecessary
     
    Baster, Sep 27, 2009 IP
  5. contra10

    contra10 Member

    Messages:
    30
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    43
    #5
    only disallow the ones tha you don't need or on the other hand allow the ones u want
     
    contra10, Oct 1, 2009 IP
  6. shanmugasundaram

    shanmugasundaram Active Member

    Messages:
    230
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    70
    #6
    your file will work

    both Disallow: and Allow:/ means the same
     
    shanmugasundaram, Oct 1, 2009 IP
  7. YSNY

    YSNY Peon

    Messages:
    52
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    thanks guys
     
    YSNY, Oct 2, 2009 IP
  8. AKSIN

    AKSIN Peon

    Messages:
    159
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #8
    You can use like this:
    User-agent: *
    Allow: /
    Disallow: /cgi-bin/

    And disallow any files whichever you want to by using word Disallow:
     
    AKSIN, Oct 9, 2009 IP
  9. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #9
    manish.chauhan, Oct 28, 2009 IP
  10. manish0109

    manish0109 Active Member

    Messages:
    164
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    68
    #10
    Thanks all for info
     
    manish0109, Nov 26, 2009 IP
  11. madison123

    madison123 Guest

    Messages:
    184
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    disallow only use when u want to disallow it
     
    madison123, Mar 12, 2010 IP
  12. tata_capl

    tata_capl Peon

    Messages:
    461
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Remove This
    Disallow:
     
    tata_capl, Mar 14, 2010 IP
  13. more_sem

    more_sem Peon

    Messages:
    299
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    The text in red means that you blocked crawlers from accessing any of your sub-directory or even your root directory, change it to Allow. The correct format should be


    # robots.txt
    User-agent: *
    Allow: /
    Disallow: /cgi-bin/
    Sitemap: http://realpracticesolutions.com/google_sitemap.xml.gz

    You might also want to change google_sitemap.xml.gz to just sitemap.xml.gz just for simplicity reasons
     
    more_sem, Mar 14, 2010 IP