Problem with file

Discussion in 'robots.txt' started by shoprat, Sep 23, 2016.

0
  1. #1
    What are the problems with this robots.txt file?

    Thanks
     

    Attached Files:

    shoprat, Sep 23, 2016 IP
  2. Furquan Ahmed

    Furquan Ahmed Well-Known Member

    Messages:
    819
    Likes Received:
    76
    Best Answers:
    1
    Trophy Points:
    135
    #2
    I don't know why you are using this huge robots.txt file on your website. You can just simply make this very simple as below.

    User-agent: *
    Disallow: /links/
    Disallow: /cgi-bin/
    Disallow: /data/
    Disallow: /database/
    Disallow: /generator/
    Disallow: /include/
    Disallow: /forms/
    Disallow: /temp/
    Sitemap: http://www.warlinks.com/sitemap.xml
     
    Furquan Ahmed, Sep 23, 2016 IP
  3. JeffMichaels

    JeffMichaels Well-Known Member

    Messages:
    207
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    125
    #3
    JeffMichaels, Sep 23, 2016 IP