1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Problem with file

Discussion in 'robots.txt' started by shoprat, Sep 23, 2016.

  1. #1
    What are the problems with this robots.txt file?

    Thanks
    SEMrush
     

    Attached Files:

    shoprat, Sep 23, 2016 IP
    SEMrush
  2. Furquan Ahmed

    Furquan Ahmed Well-Known Member

    Messages:
    791
    Likes Received:
    72
    Best Answers:
    1
    Trophy Points:
    110
    #2
    I don't know why you are using this huge robots.txt file on your website. You can just simply make this very simple as below.

    User-agent: *
    Disallow: /links/
    Disallow: /cgi-bin/
    Disallow: /data/
    Disallow: /database/
    Disallow: /generator/
    Disallow: /include/
    Disallow: /forms/
    Disallow: /temp/
    Sitemap: http://www.warlinks.com/sitemap.xml
     
    Furquan Ahmed, Sep 23, 2016 IP
  3. JeffMichaels

    JeffMichaels Well-Known Member

    Messages:
    209
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    125
    #3
    JeffMichaels, Sep 23, 2016 IP