1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt Issues

Discussion in 'robots.txt' started by ketanseo23, Apr 14, 2016.

  1. #1
    Hello Everyone,
    I have auto-generated robots.txt. But i am facing much issues like
    1. in google search results shows: "A description for this result is not available because of this site's robots.txt – learn more."
    2. Many other important Urls not indexed by google.


    Can anyone help me to find the issues? Is this robots file ok or not? My Robots.txt shown as below:
    SEMrush
    # Google Image Crawler Setup
    User-agent: Googlebot-Image
    Disallow:

    # Crawlers Setup
    User-agent: *

    # Directories
    Disallow: /404/
    Disallow: /app/
    Disallow: /cgi-bin/
    Disallow: /downloader/
    Disallow: /errors/
    Disallow: /includes/
    #Disallow: /js/
    #Disallow: /lib/
    Disallow: /magento/
    #Disallow: /media/
    Disallow: /pkginfo/
    Disallow: /report/
    Disallow: /scripts/
    Disallow: /shell/
    Disallow: /skin/
    Disallow: /stats/
    Disallow: /var/

    # Paths (clean URLs)
    Disallow: /catalog/product_compare/
    Disallow: /catalog/category/view/
    Disallow: /catalog/product/view/
    Disallow: /catalogsearch/
    #Disallow: /checkout/
    Disallow: /control/
    Disallow: /customize/
    Disallow: /newsletter/
    Disallow: /poll/
    Disallow: /review/
    Disallow: /sendfriend/
    Disallow: /tag/
    Disallow: /catalog/product/gallery/
    Disallow: /faqs/
    Disallow: /checkout/
    Disallow: /disclaimer-policy/
    Disallow: /return-policy/

    # Files
    Disallow: /cron.php
    Disallow: /cron.sh
    Disallow: /error_log
    Disallow: /install.php
    Disallow: /LICENSE.html
    Disallow: /LICENSE.txt
    Disallow: /LICENSE_AFL.txt
    Disallow: /STATUS.txt

    # Paths (no clean URLs)
    #Disallow: /*.js$
    #Disallow: /*.css$
    Disallow: /*.php$
    Disallow: /*?SID=
     
    ketanseo23, Apr 14, 2016 IP
    SEMrush
  2. MayurKawale

    MayurKawale Well-Known Member

    Messages:
    145
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    108
    #2
    MayurKawale, Aug 4, 2016 IP